Opening Scene
Fertilizer runoff doesn’t always stay neatly inside the plot it was applied to. It can seep into the soil of an adjacent, supposedly untreated plot, quietly boosting that “control” plot’s yield too — which makes the measured difference between treatment and control smaller than the fertilizer’s true effect actually is. The trial’s design was sound. The physical world simply didn’t respect the boundary the design assumed.
In Plain English
Interference (or spillover, or network effects) occurs when a treatment applied to one subject affects another subject in the control group, violating the core assumption that each subject’s outcome depends only on its own treatment assignment. This is a particularly common and genuinely serious problem in social and marketplace settings — a referral feature tested on some users can still affect their untreated friends, or a pricing change tested on some sellers in a marketplace can affect the demand faced by untreated sellers.
The Old Way
Before this had formal experimental language, the same underlying problem showed up in earlier field research:
- A public health intervention tested in one part of a community that also protected nearby untreated residents, through reduced disease transmission — a genuine, well-documented spillover effect.
- An agricultural fertilizer trial affected by literal runoff between adjacent plots, exactly the scenario opening this article.
- A new store policy tested in one location that also changed customer behavior at a nearby, untreated location, through word of mouth or shared customer base.
In each case, the treatment’s effect wasn’t cleanly contained within the group it was assigned to.
What’s Changing (and Why AI Is the Reason)
- As digital products, particularly social and marketplace platforms, have grown more interconnected, interference has become a genuinely common and important concern, not a rare edge case — the exact kind of connectivity these platforms are built around also breaks the clean separation experiments need.
- Specialized experimental designs, like cluster randomization (randomizing entire connected groups rather than individuals) and switchback designs (alternating treatment over time rather than across subjects), have been developed specifically to address interference in these settings.
- Statistical methods for detecting and correcting for interference, rather than just hoping it doesn’t happen, have matured considerably, giving experimenters real tools to check for and account for this problem rather than being blindsided by it.
The Metaphor, Fully Extended
| The Field Trial | Interference Concept |
|---|---|
| Fertilizer runoff seeping into a neighboring plot | A treatment’s effect spilling over to a control-group subject |
| A measured yield difference smaller than the fertilizer’s true effect | A measured treatment effect diluted by spillover into the control group |
| Grouping plots to prevent runoff between treatment and control | Cluster randomization to prevent spillover between treatment and control |
| Recognizing that plots aren’t always cleanly independent of each other | Recognizing that subjects in a connected platform aren’t always cleanly independent |
For Beginners: What to Actually Do
- Before designing an experiment on a social or marketplace platform, explicitly ask whether treated and untreated subjects can realistically affect each other.
- Learn the basic idea behind cluster randomization as one common solution to interference.
- Recognize interference as a real, common threat to experiment validity in connected systems, not a rare, exotic edge case.
For Practitioners and Leaders: The Deeper Layer
- Explicitly evaluate the risk of interference for any experiment run on a genuinely interconnected platform — social features, marketplaces, multiplayer systems.
- Invest in specialized experimental designs like cluster randomization or switchback testing where interference is a realistic concern.
- Treat an experiment result on an interconnected platform with real caution if interference wasn’t explicitly considered during design.
Quick Recap
- Interference occurs when a treatment applied to one subject affects another subject’s outcome, violating a core experimental assumption.
- It’s a particularly common concern in social and marketplace platforms, where users are genuinely connected to each other.
- Specialized designs like cluster randomization and switchback testing address this problem directly.
- Interference should be explicitly considered during experiment design, not discovered as a surprise afterward.
Where This Fits in the Series
Article 13 covered spillover between supposedly separate groups. Article 14 covers a different kind of hidden variation: results that differ meaningfully across subgroups.
Subscribe to the Newsletter
Get the latest DataParables articles delivered straight to your inbox.