3.3 Before and after, with a comparison

Key points

On its own, before and after can’t separate a change from everything else that moved at the same time.
Parallel trends can be checked before the change, but not afterwards, so it stays an assumption.
UK supermarkets that took sweets, chocolate and crisps off their checkouts saw purchases of those foods fall an estimated 17 per cent straight away.
Mexico’s warning labels had no comparison group, so the estimated 25 per cent fall in added sugar rests on a projected trend.

Chapters

Key terms

Difference-in-differences: Comparing the change in a treated group with the change in a group that wasn’t treated, both measured before and after. If they were moving in parallel beforehand, the comparison group’s change shows what would have happened anyway, and the treated group’s extra change is the estimate. It rests on the parallel-trends assumption.
Parallel trends: The key assumption of difference-in-differences: without the change, the treated and comparison groups would have kept moving together. You can check whether they moved in parallel before the change, by tracking both groups over the earlier period, but not afterwards, so it always stays an assumption.
Interrupted time series: A long run of measurements before and after a change, compared with the trend projected from before it. It works best with a comparison group too. Without one, anything else that changed at the same time, such as a pandemic, could be part of the estimate.
Before-and-after comparison: Comparing an outcome before and after a change, such as Takaki’s navy voyages on different diets. It is the simplest evidence and, on its own, the weakest, because prices, weather, fashions and other policies change at the same time. Adding a comparison group, or measuring every link of a chain, makes it much stronger.

Transcript

Last time's quick check: a propensity score is the probability that a unit was treated, given everything you measured. The simplest evidence is before and after: something changes, and you compare. Takaki's beriberi voyage was one. But many things change over time at once: prices, weather, fashions, other policies. On its own, before and after can't tell them apart.

Add a comparison group that didn't get the change, and measure both before and after. If the two were moving in parallel beforehand, the comparison group's change shows what would have happened anyway. The extra change in the treated group is the estimate. That's difference-in-differences.

Its key assumption is called parallel trends: without the change, the two groups would have kept moving together. You can check whether they did before the change. You can't check afterwards, so it stays an assumption. In the UK, researchers compared supermarkets that removed sweets, chocolate and crisps from their checkouts with supermarkets that didn't, tracking household purchases month by month.

A long run of measurements before and after a change is called an interrupted time series; this one had a comparison too. Purchases of the targeted foods fell an estimated 17 per cent straight away. A year on, the estimate was 15.5 per cent, but less robust. And purchases aren't the same as eating.

In a university's cafeterias, some outlets added carbon labels to their meals and others didn't. Across more than 80,000 meal choices, emissions per meal fell 4.3 per cent where labels appeared. It wasn't randomised, so the estimate rests on the labelled and unlabelled cafeterias behaving alike in every other way.

In Mexico, warning labels on packaged foods became law for everyone, so there was no unlabelled comparison. Researchers projected the sales trend from before the policy and compared it with what happened. Added sugar sold per person fell an estimated 25 per cent, while overall sales volume held steady.

But with no comparison group, anything else that changed at the same time could be part of that number, and this was during the pandemic. The more a design leans on a projected trend, the more you have to believe that nothing else changed. Next: experiments that nobody designed.

Questions

What is an interrupted time series?

A long run of measurements before and after a change. In Mexico, warning labels became law for everyone, so researchers projected the sales trend from before the policy and compared it with what happened: added sugar sold per person fell an estimated 25 per cent. With no comparison group, anything else that changed at the time, during the pandemic, could be part of that number.

Did carbon labels reduce emissions from meals?

In a university’s cafeterias, some outlets added carbon labels to their meals and others didn’t. Across more than 80,000 meal choices, emissions per meal fell 4.3 per cent where labels appeared. It wasn’t randomised, so the estimate rests on the labelled and unlabelled cafeterias behaving alike in every other way.

Sources

Ejlerskov, K.T., Sharp, S.J., Stead, M. et al. (2018). Supermarket policies on less-healthy food at checkouts: natural experimental evaluation using interrupted time series analyses of purchases. PLOS Medicine, 15(12), e1002712. doi:10.1371/journal.pmed.1002712
Lohmann, P.M., Gsottbauer, E., Doherty, A. et al. (2022). Do carbon footprint labels promote climatarian diets? Evidence from a large-scale field experiment. Journal of Environmental Economics and Management, 114, 102693. doi:10.1016/j.jeem.2022.102693
Garduño-Alanis, A., Salgado, J.C., Contreras-Manzano, A. et al. (2026). Changes in sales of packaged food and beverages, energy, and nutrients after the implementation of front-of-pack warning labels in Mexico. Social Science & Medicine, 399, 119209. doi:10.1016/j.socscimed.2026.119209

All 30 films · Glossary · The Vitagri Academy