Causal inference · Module 3: Getting an answer: study designs

When observation lied

What is target trial emulation?

3.6 · 3 min 50 s · published

Writing down the randomised trial you wish you could run, then making an observational analysis copy it as closely as possible: who is eligible, exactly what the treatment is, and a time zero when everyone’s clock starts. Analysed this way, the Nurses’ Health Study data gave answers on hormone therapy much closer to the trial’s.

Key points

  • Observational studies linked beta-carotene to less lung cancer; trials in Finland and the US found more, about 18 and 28 per cent more.
  • A nutrient inside a whole food isn’t the same as a high-dose pill.
  • Healthy-user bias: women who chose hormone therapy tended to be healthier and more health-conscious to begin with.
  • Most diet and farming questions will never get a long randomised trial, so design the analysis like one.

Chapters

  1. 0:00 Recap: neighbours
  2. 0:19 The carrot and the capsule
  3. 1:21 Healthier to begin with
  4. 2:04 Analyse it like a trial
  5. 2:45 For food and farming

Key terms

Target trial emulation
Writing down the randomised trial you wish you could run, then making an observational analysis copy it: who is eligible, exactly what the treatment is, and a time zero when everyone’s clock starts. Starting everyone at the same moment stops long-term users being compared with newcomers.
Healthy-user bias
Bias that arises because people who choose a treatment differ from those who don’t, often by being healthier to begin with. Women who chose hormone therapy tended to be healthier, wealthier and more health-conscious, which made observational studies suggest a benefit that the trial didn’t find.

Transcript

Last time's quick check: the non-Bt fields benefited from their neighbours' Bt maize, so they weren't a clean untreated comparison. In the 1980s, observational studies kept finding the same thing: people who ate more fruit and vegetables rich in beta-carotene, or had more of it in their blood, got less lung cancer.

It looked like a cheap way to prevent cancer. So two large trials gave beta-carotene supplements to people at high risk: one in Finland, with about 29,000 male smokers, reported in 1994; one in the United States, reported in 1996. Both found more lung cancer in the supplement groups, not less: about 18 per cent more in Finland, and 28 per cent more in the US trial, which was stopped early.

What went wrong? People who eat plenty of vegetables differ in many ways from people who don't. And a nutrient inside a whole food isn't the same as a high-dose pill. Remember Lind's question: what exactly is "it"? Hormone replacement therapy tells a similar story. For years, observational studies, including the large Nurses' Health Study, suggested that women taking it had less heart disease. Then, in 2002, the Women's Health Initiative trial found a higher risk of heart disease, especially in the first years.

Part of the explanation was healthy-user bias: women who chose hormone therapy tended to be healthier, wealthier and more health-conscious in ways the studies hadn't fully captured. Part was timing: the observational studies mostly saw women who had already been taking it for years, and missed the early risk.

Then something remarkable happened. Miguel Hernán and colleagues went back to the Nurses' Health Study data and analysed it as if it were a trial. They defined who would have been eligible, and the moment each woman started therapy or didn't, called time zero. Then they compared new users with non-users from that moment, just as the trial had.

The observational data then gave answers much closer to the trial's. The approach is called target trial emulation: write down the trial you wish you could run, then make the observational analysis copy it as closely as possible. Most questions about diet, farming practice and health will never get a long randomised trial.

Target trial emulation is how careful researchers use the data they have without fooling themselves: define the treatment exactly, start the clock at the same moment for everyone, and compare like with like from there. That completes the toolbox. Next, in Module 4: the machine-learning methods, in depth.

Questions

Why did observational studies get beta-carotene wrong?

People who eat plenty of vegetables differ in many ways from people who don’t, and a nutrient inside a whole food isn’t the same as a high-dose pill. When two large trials gave beta-carotene supplements to people at high risk, both found more lung cancer in the supplement groups, not less.

What is healthy-user bias?

Bias that arises when the people who choose a treatment are healthier to begin with. Women who chose hormone therapy tended to be healthier, wealthier and more health-conscious in ways the studies hadn’t fully captured, so observational data suggested less heart disease. In 2002 the Women’s Health Initiative trial found a higher risk.

Sources

  1. Alpha-Tocopherol, Beta Carotene Cancer Prevention Study Group (1994). The effect of vitamin E and beta carotene on the incidence of lung cancer and other cancers in male smokers. New England Journal of Medicine, 330(15), 1029–1035. doi:10.1056/NEJM199404143301501
  2. Omenn, G.S., Goodman, G.E., Thornquist, M.D. et al. (1996). Effects of a combination of beta carotene and vitamin A on lung cancer and cardiovascular disease. New England Journal of Medicine, 334(18), 1150–1155. doi:10.1056/NEJM199605023341802
  3. Grodstein, F., Stampfer, M.J., Manson, J.E. et al. (1996). Postmenopausal estrogen and progestin use and the risk of cardiovascular disease. New England Journal of Medicine, 335(7), 453–461. doi:10.1056/NEJM199608153350701
  4. Writing Group for the Women's Health Initiative Investigators (2002). Risks and benefits of estrogen plus progestin in healthy postmenopausal women: principal results from the Women's Health Initiative randomized controlled trial. JAMA, 288(3), 321–333. doi:10.1001/jama.288.3.321
  5. Hernán, M.A., Alonso, A., Logan, R. et al. (2008). Observational studies analyzed like randomized experiments: an application to postmenopausal hormone therapy and coronary heart disease. Epidemiology, 19(6), 766–779. doi:10.1097/EDE.0b013e3181875e61
  6. Hernán, M.A. and Robins, J.M. (2016). Using big data to emulate a target trial when a randomized trial is not available. American Journal of Epidemiology, 183(8), 758–764. doi:10.1093/aje/kwv254

All 30 films · Glossary · The Vitagri Academy

Follow the change in farming

Causal evidence is changing how farming is measured and rewarded. Leave your email and we will tell you when new films, courses and findings are published.

Prefer to read online? Vitagri also writes on Substack. See our privacy policy for how we use your email.