Machines and causes

Last time: an AI that diagnosed the hospital instead of the patient, and cattle that seemed to die from their antibiotics. The confounder there was the risk the vet saw on arrival. Both models were stuck on the bottom rung. Today: how AI and causal thinking help each other climb.

What does causal thinking give AI? Three things, at three different stages. First, advice that holds up when you act on it. In Greece, researchers judged sowing advice not by how accurate its forecasts were, but by what following it did. Across 171 cotton fields, that meant an estimated 12 to 17 per cent more yield. That kind of answer works today, when its assumptions hold.

Second, explanations you can check. A causal model is a map of arrows. An agronomist or a vet can look at each arrow and ask: does that make sense? That's partly there. Third, models that keep working in new places. The idea is that causes travel and coincidences don't. The biology that turns nitrogen into grain protein is the same in Lincolnshire as in Kent. A hospital's pneumonia rate isn't.

Researchers have tried to build that into AI. But when the methods are tested carefully, ordinary training often does just as well. It's a promise, not yet a result. Now turn it round. For most of its history, causal inference didn't need AI. It needs AI now because of the data we have.

One farm's records can hold soil tests, rainfall, variety, sowing date, previous crop and every spray. Hundreds of things, tangled together in ways no straight line describes. AI helps in three ways. It can model those tangled background relationships, as you'll see in a moment.

It can find where an effect is bigger or smaller: in Ghana, randomised fertiliser trials showed a typical maize response of 2.8 tonnes a hectare in one zone and 0.9 in another, and machine-learning models were trained to predict those differences. And it can help draft the causal map itself, as a starting point for experts. From observational data alone, those methods can narrow the possibilities. They rarely settle them.

Here's the partnership on a real question. Does an extra pass of nitrogen raise the protein in milling wheat? Across farms, more nitrogen goes with more protein. But the farms applying more aren't a random sample. They may have better soils, different varieties, a wetter spring, or a contract that pays for protein. Each of those moves protein on its own.

Double machine learning works in three steps. One: a model predicts how much nitrogen each field got, from everything we know about it. Subtract that prediction from what actually happened. What's left, the residual, is the part of the decision the background doesn't explain. Two: a second model predicts protein from the same background, ignoring nitrogen. Subtract again. What's left is the protein the background doesn't explain.

Three: compare the two residuals. If fields that got unexpectedly more nitrogen also came in with unexpectedly more protein, that relationship is the estimate. Two details make it work. Each model is trained on one part of the data and used on another, so no model marks its own homework. That's cross-fitting. And the method is built so that small errors in the two models barely move the answer.

The residuals trick was proved in 1933. What's new is machine learning doing the predicting. And here's what it can't do. It only removes the influence of things you measured. If the farms applying more nitrogen differ in something nobody recorded, the grower's skill, say, that's still in the answer.

So the last step is a sensitivity analysis: how strong would a hidden factor need to be to overturn the result? If it would have to be far stronger than soil type, you can be fairly confident. If something weaker than the weather would do it, you can't.

So AI supplies the flexibility. The causal design supplies the right to call the answer an effect. And when you can run an experiment, choosing at random who gets the treatment, it still beats everything in this video. That idea wasn't invented for computers, or for medicine. Much of it was worked out in a wheat field in Hertfordshire. But the story starts earlier: on a ship in 1747, with twelve sick sailors and some oranges and lemons.