Right for the wrong reason

In 2018, researchers trained an AI to spot pneumonia on chest X-rays. On the images it learned from, it did well. Then they looked at what it was using. The model could tell which hospital an X-ray came from, from clues like the type of scanner and the markers stamped on the image. And the hospitals had very different rates of pneumonia.

So part of what looked like diagnosis was the model working out where the patient had been scanned. In hospitals it hadn't seen before, it usually did worse. Nothing in the model was broken. It did exactly what it was built to do: find the pattern that best predicts the answer.

The trouble is, a pattern that predicts is not the same as a cause. This series is about that difference. Where the idea came from, how people learned to measure it, and why it matters for the food we grow and eat. It comes from Vitagri, whose causal engine, Vitagri Vitals, builds causal links between agriculture and food nutrition.

Here's the same trap on a farm. When young cattle arrive at a feedlot after a long journey, some groups get antibiotics straight away, to head off pneumonia before it starts. Vets call it metaphylaxis. Now look back through the records. It's quite possible the treated groups lost more animals than the untreated ones. A model trained on those records would learn that treatment goes with death.

Do the antibiotics kill cattle? No. The vet chose to treat the groups that looked most at risk: lighter calves, mixed from many sources, a long journey, cold rain on arrival. Those same things make animals ill. Something that drives both the decision and the outcome is called a confounder. When the confounder is the very reason for treating, medicine calls it confounding by indication. And here's the hardest part: much of that judgement is never written down. You can't correct for something you never recorded. That's why, when the answer matters, the decision is made at random, so no hidden judgement can line up with the result.

The computer scientist Judea Pearl gave us a simple way to picture the gap: a ladder with three rungs. The bottom rung is seeing. Fields that got more nitrogen grew grain with more protein. Given what I've seen, what should I expect? Most of the AI in use today learns here, from patterns in data it observed.

The second rung is doing. If I put another pass of nitrogen on this field, what will happen to its protein? That's an intervention: a question about changing the world, not just watching it. The top rung is imagining. This field had the extra pass and made thirteen per cent protein, enough for the milling contract. Would it have made thirteen per cent without it? That's a counterfactual: what would have happened instead.

Here's the key point. No amount of data from the bottom rung gets you to the second on its own. You need one of two things. An experiment, where you do the intervening yourself. Or a model of how the world works, drawn up before you look at the data, that tells you what to correct for.

Even a near-perfect pattern isn't enough. On Barn Door Farm, grain zinc tracked fibre across five wheat samples with a correlation of 0.98. Five samples from one season can't tell you what to change. So a model can predict well and still give bad advice, because it learned on the bottom rung. Next time: what happens when AI and causal thinking work together, and where that partnership still falls short.