Along with the Metaverse, Gartner last year listed Causal AI as one of the most promising emerging technologies. But what is it ? Is the future of AI causal ? And why aren’t more established machine and deep learning methods sufficient ?
Gartner Hype Cycle 2022 - Emerging Technologies
Correlated feature sets like those in the python charts below are the basis for machine learning.
Feature Correlation in Machine Learning
The problem is, standard predictive analytics (“Association-based or correlation-based AI”) often leads to spurious correlations, where features can be related by a high correlation value (e.g. divorce rate and margarine consumption) but have no underlining causal relationship
Causal AI on the other hand is rooted in decision making, and understanding root-causes.
Rather than trying to identify e.g. customer segments or patients based on minimising (or maximising) a given variable (like churn or infection), Causal AI rather recommends more sophisticated tailored strategies (“interventions”) that minimise the same variable. So targeted advertising, price offers or specific treatments applied to different segments or patient groups.
And at a still higher level of sophistication, counterfactuals – or imaginary scenarios – are tested to refute estimated outcomes, such as what-if the patient hadn’t smoked for two years or by testing a placebo. More generally, if a specific event has occurred, could we have prevented or changed our outcome in some way by doing things differently ?
A causal inference analysis often entails drawing a graph of what may be causing what, identifying confounders, and stratifying those to find the effect of a treatment on an outcome. An example of a confounder might be soil moisture on crop yield or as suggested in the picture below, the presence of sand where grass might be expected – the kind of thing that can cause a trained deep learning image classification model to fail (in identifying a cow).
Source: qualcomm
One of the means of performing Causal Analysis is via the PyWhy library, formerly DoWhy, and now co-developed by Microsoft and AWS. Using this directly in Python takes some setup (contact ce.tech for support), but as our Python example* shows, once we have set up our causal graph scope we can relatively quickly analyse p-values (statistical strength of specific treatments on outcomes) and confounders for these treatments (suggested by variations in p-value).
* NB uses Binder - this will either take a few seconds or up to 5 minutes depending on whether Docker image requires rebuilding.
Feel free to clone the GitHub repo here and run locally instead.
Example Causal Graph
PyWhy is more abstract and statistical in syntax – no surprise given the relative recency of Causal AI as a trend - it is therefore somewhat behind more familiar / mainstream python libraries on the maturity curve.
But given the break-neck speed of development we are experiencing across AI and the wider enterprise need to talk to a non-technical audience, NoLo code and Full Stack Data Science has crept into to Causal AI products AI, much as it has in Generative AI / Transformers.
Our partner, Geminos, is one of the leaders in low code Causal AI and will be showcasing their Causeway platform next week (March 29th). Register for the webinar “The Future of AI is Causal: 3 Things You Should Know Now” here and check back soon as we will be looking further into how Causal AI can be leveraged to generate value/ROI for specific industry challenges in the weeks ahead.