How Big Does the Field Need to Be

Opening Scene

A trial run on three plants, even with a perfect control and perfect randomization, is nearly worthless — plant-to-plant variation alone could easily produce a difference that has nothing to do with the fertilizer. A trial run across thousands of plants, spread over enough plots, can detect a real effect with genuine confidence, even a fairly modest one. The size of the trial isn’t a minor logistical detail. It’s the difference between a test that can actually answer the question and one that can’t.

In Plain English

Statistical power is the probability that an experiment will detect a real effect of a given size, if one truly exists. Sample size calculations determine how many subjects are needed to achieve adequate power, based on the expected effect size, the outcome’s natural variability, and the desired confidence level. An underpowered experiment risks a false “no effect” conclusion, not because the treatment didn’t work, but because the trial simply wasn’t big enough to detect it reliably.

The Old Way

Before formal power calculations existed, trial size was often chosen by convenience rather than by any principled method:

A small medical trial run on whatever patients happened to be available, without any formal calculation of whether that sample was large enough to detect a meaningful effect.
A business testing a change on a small, convenient subset of customers, then drawing broad conclusions from a sample far too small to support them.
A farmer testing a new technique on a handful of plants, without any formal sense of how many would actually be needed to trust the result.

In each case, an underpowered test could easily produce a misleading “no difference” result, simply from having too few subjects to detect a real, meaningful effect.

What’s Changing (and Why AI Is the Reason)

Formal power analysis, now standard practice before launching any serious experiment, calculates the minimum sample size needed given an expected effect size and desired confidence — turning trial size into a principled decision rather than a guess.
Digital experimentation platforms can automatically calculate required sample size and estimate how long a test needs to run to reach it, removing much of the manual statistical work this once required.
As organizations run more experiments simultaneously, understanding power has become essential for triaging which tests are even worth running given realistic available traffic — a theme connecting directly to Article 18’s discussion of running many tests at once.

The Metaphor, Fully Extended

The Field Trial	Statistical Power Concept
A trial on three plants, too small to trust	An underpowered experiment, too small to detect a real effect
A trial across thousands of plants, large enough to detect a real difference	A well-powered experiment, large enough to detect the effect size of interest
Calculating how many plots are needed before planting	Calculating the required sample size before launching a test
A trial too small to reliably tell a real effect from natural variation	A test too small to reliably distinguish a real effect from natural noise

For Beginners: What to Actually Do

Learn to run a basic power calculation before designing any experiment, using an estimated effect size and outcome variability.
Practice recognizing an underpowered experiment when you see one — a small sample size relative to a subtle expected effect is a genuine red flag.
Get comfortable with the idea that “we found no significant difference” and “we proved there’s no difference” are not the same claim, especially in an underpowered test.

For Practitioners and Leaders: The Deeper Layer

Require a documented power calculation before approving any significant experiment, as standard practice.
Set realistic expectations about how long a test needs to run to reach adequate power, given actual available traffic or sample size.
Recognize underpowered tests as a genuine, common source of false negative conclusions, not just a theoretical statistics concern.

Quick Recap

Statistical power is the probability of detecting a real effect, given it actually exists.
Sample size calculations determine the minimum trial size needed for adequate power, based on expected effect size and variability.
An underpowered experiment risks a misleading “no effect” conclusion, not because the treatment failed, but because the test was too small.
Modern platforms now automate much of this calculation, making rigor more accessible than ever.

Where This Fits in the Series

Article 7 covered how large a trial needs to be to trust its result. Article 8 covers a related, equally important question: how long it needs to run.