Home / Academy / Causal inference / 4.3
Causal inference · Module 4: Machines and causes, in depth

Try to break your own answer

How do you stress-test a causal estimate?

4.3 · 3 min 13 s · published

Try to break it before anyone relies on it. Run a placebo test with a fake, shuffled treatment; check a negative control, an outcome the treatment can’t possibly affect; ask how strong an unmeasured confounder would need to be, with a sensitivity analysis or the robustness value; and estimate the effect more than one way.

Written and reviewed by Rob Ward, CEO & Co-Founder, Vitagri.

Key points

  • If a method finds a big effect of a treatment that never happened, don’t trust its real result either.
  • This year’s nitrogen can’t change last year’s yield; if an analysis says it did, something is confounding the comparison.
  • The robustness value, from Cinelli and Hazlett in 2020, puts sensitivity into a single number.
  • Agreement across methods isn’t proof, but disagreement is a warning.

Chapters

  1. 0:00 Recap: honesty
  2. 0:17 The habit
  3. 0:31 A fake treatment
  4. 0:56 An impossible outcome
  5. 1:21 How strong would it need to be?
  6. 2:11 Ask it more than one way

Key terms

Placebo test
Replacing the real treatment with a fake one, shuffled at random, and running the whole analysis again. The effect should vanish. If the method finds a big effect of a treatment that never happened, it is finding patterns in the noise, and its real result shouldn’t be trusted either.
Negative control
An outcome the treatment can’t possibly affect, used to check an analysis. This year’s nitrogen can’t change last year’s yield, so if an analysis says it did, something is confounding the comparison. Negative controls can expose hidden confounding that the causal map missed.
Sensitivity analysis
Asking how strong an unmeasured confounder would have to be to overturn a result, and comparing that with confounders you did measure. Jerome Cornfield’s 1959 answer to Fisher about smoking was the first. If the hidden factor would need to be far stronger than soil type, you can be fairly confident.
Robustness value
A single number from Carlos Cinelli and Chad Hazlett, in 2020: how much of the variation in both the treatment and the outcome a hidden confounder would need to explain to bring the effect to zero. Compared with how much measured factors such as soil type or rainfall explain, it shows how sturdy a result is.

Transcript

Last time's quick check: if the same data chose a group and measured it, the forest could report an impressive difference that was only noise. A good causal analysis doesn't stop at an estimate. It tries to break it. If the answer survives, you trust it more. If it breaks, better that you find out than your customer.

First, a placebo test. Replace the real treatment with a fake one, shuffled at random, and run the whole analysis again. The effect should vanish. If your method finds a big "effect" of a treatment that never happened, it's finding patterns in the noise, and you shouldn't trust its real result either.

Second, a negative control: an outcome the treatment can't possibly affect. This year's nitrogen can't change last year's yield. If your analysis says it did, something is confounding the comparison. Perhaps better farmers both apply more and grow more, every year. Negative controls can expose hidden confounding that the map missed.

Third, Cornfield's question, made modern. A sensitivity analysis asks how strong an unmeasured confounder would have to be to wipe out the effect, and compares that with the confounders you did measure. In 2020, Carlos Cinelli and Chad Hazlett gave this a single number, the robustness value: how much of the variation in both the treatment and the outcome a hidden confounder would need to explain to bring the effect to zero. If that's far more than soil type or rainfall explain, and you measured those, the result is sturdy. If a confounder weaker than the weather would do it, it isn't.

Finally, ask the question more than one way. The Greek cotton study estimated its effect with several methods and found a similar range, from 12 to 17 per cent. Agreement isn't proof, because every method shares the same assumptions about what was measured. But disagreement is a warning you shouldn't ignore. Next: the most ambitious question of all. Can AI find causes by itself?

Questions

What is a negative control?

An outcome the treatment can’t possibly affect. This year’s nitrogen can’t change last year’s yield, so if an analysis says it did, something is confounding the comparison: perhaps better farmers both apply more and grow more, every year. Negative controls can expose hidden confounding that the map missed.

What is the robustness value?

A single number set out by Carlos Cinelli and Chad Hazlett in 2020: how much of the variation in both the treatment and the outcome a hidden confounder would need to explain to bring the effect to zero. If that is far more than measured factors such as soil type or rainfall explain, the result is sturdy.

Sources

  1. Cinelli, C. and Hazlett, C. (2020). Making sense of sensitivity: extending omitted variable bias. Journal of the Royal Statistical Society Series B, 82(1), 39–67. doi:10.1111/rssb.12348
  2. Tsoumas, I., Giannarakis, G., Sitokonstantinou, V. et al. (2022). Evaluating digital tools for sustainable agriculture using causal inference. arXiv:2211.03195. doi:10.48550/arXiv.2211.03195
  3. Cornfield, J., Haenszel, W., Hammond, E.C. et al. (1959). Smoking and lung cancer: recent evidence and a discussion of some questions. Journal of the National Cancer Institute, 22(1), 173–203. doi:10.1093/jnci/22.1.173

All 30 films · Glossary · The Vitagri Academy

Films play here on vitagri.org, with English captions. Nothing downloads until you press play.

Keep in touch

Follow the change in farming

Causal evidence is changing how farming is measured and rewarded. Leave your email and we will tell you when new films, courses and findings are published.

Prefer to read online? Vitagri also writes on Substack. See our privacy policy for how we use your email.