Too Many Trials at Once

Opening Scene

Run a hundred trials, each testing a fertilizer that genuinely has zero effect, at the conventional 5% significance threshold, and pure chance alone will make roughly five of them look statistically significant. Not because anything worked. Because testing enough things, purely by chance, guarantees some will look interesting. An organization running hundreds of simultaneous experiments faces this exact statistical reality, whether or not anyone notices it happening.

In Plain English

The multiple testing problem (or the multiple comparisons problem) describes how running many statistical tests inflates the overall chance that at least one will show a false positive, purely by chance, even if every single test individually uses a standard 5% significance threshold. Corrections like the Bonferroni method, or more modern approaches like the false discovery rate, adjust the significance threshold specifically to account for how many tests are actually being run.

The Old Way

Before this had formal statistical correction methods, the problem often went entirely unrecognized:

• A researcher running many exploratory statistical tests and reporting only the significant ones, without acknowledging how many tests were actually run — a well-documented source of false findings in scientific research.
• A business running many simultaneous experiments and treating each one’s “significant” result at face value, without accounting for the inflated false-positive rate across the full set.
• Data mining large datasets for any pattern that reaches significance, a classic and well-documented source of spurious, non-replicating findings.

In each case, the sheer number of tests being run quietly inflated the risk of a false conclusion, invisible to anyone looking at a single test in isolation.

What’s Changing (and Why AI Is the Reason)

1. As experimentation platforms, covered in Article 17, have made it dramatically easier to run many simultaneous tests, the multiple testing problem has become far more practically relevant than in eras when running even one careful experiment was a significant undertaking.
2. Formal correction methods, and modern approaches like the false discovery rate that are specifically designed for large-scale testing environments, are increasingly built directly into experimentation platforms rather than requiring manual statistical correction.
3. This connects directly to the pre-specified subgroup discipline from Article 14 — exploring many subgroups after the fact is itself a form of multiple testing, and needs the same kind of correction.

The Metaphor, Fully Extended

The Field Trial	Multiple Testing Concept
Running a hundred trials on treatments with genuinely zero effect	Running many simultaneous statistical tests
A handful looking successful purely by chance	A handful showing statistically significant results purely by chance
Adjusting the bar for “success” based on how many trials were run	Adjusting the significance threshold based on how many tests were run
Recognizing that testing enough things guarantees some interesting-looking noise	Recognizing that the multiple testing problem is a real statistical fact, not a rare edge case

Quick Recap

• The multiple testing problem inflates the overall chance of a false positive when running many simultaneous statistical tests.
• This risk is real and invisible when looking at any single test in isolation.
• Formal correction methods, like the false discovery rate, adjust significance thresholds to account for this.
• This applies directly to subgroup analysis and any exploratory, post-hoc pattern-finding, not just to formally separate experiments.

Where This Fits in the Series

Article 18 covered the statistical risk of running many trials at once. Article 19 covers what happens once a single trial’s winner needs to move from the test plot to the whole farm.