A Difference That Doesn't Matter

Opening Scene

A large enough trial can detect an astonishingly small real difference — a new fertilizer that reliably increases yield by half a gram per plant, measured with total statistical confidence across ten thousand plants. Statistically significant, without question. Worth the cost of the fertilizer, the labor to apply it, and the change to established farming practice? That’s a completely different question, and statistical significance alone can’t answer it.

In Plain English

Practical significance asks whether a statistically significant effect, established in Article 10, is actually large enough to matter for a real decision. A large enough sample size can make even a genuinely tiny effect statistically significant — which means “statistically significant” and “worth acting on” are not the same claim, despite how often they get treated as interchangeable in practice.

The Old Way

Before this distinction had formal experimentation language, the same confusion showed up in other data-driven fields:

A medical treatment shown to reduce a health risk by a statistically confirmed but clinically trivial amount, raising a real question about whether it’s worth prescribing.
A financial strategy shown to statistically beat a benchmark by a tiny margin, raising a real question about whether it’s worth the transaction costs and risk involved.
A manufacturing process change shown to statistically reduce defects by a fraction of a percent, raising a real question about whether it’s worth the disruption to change.

In every field, a statistically confirmed effect and a practically meaningful one turned out to be genuinely separate questions.

What’s Changing (and Why AI Is the Reason)

As experimentation has scaled to very large sample sizes, the risk of finding statistically significant but practically trivial effects has grown correspondingly — large samples make small effects easy to detect, whether or not they’re worth acting on.
Modern experimentation practice increasingly requires an explicit minimum detectable effect (MDE) to be defined before a test is even run, connecting directly to the power calculations from Article 7, forcing a practical-significance threshold to be set in advance rather than debated after the fact.
Reporting confidence intervals around an effect size, rather than a bare significance flag, has become standard practice specifically because it keeps the actual magnitude of an effect visible alongside its statistical confidence.

The Metaphor, Fully Extended

The Field Trial	Practical Significance Concept
A confirmed but tiny half-gram yield increase	A statistically confirmed but practically trivial effect
Weighing the cost of the fertilizer against the actual yield gain	Weighing the cost of a change against its actual measured benefit
Deciding, in advance, what yield increase would actually be worth adopting	Setting a minimum detectable effect threshold before running a test
A trial result that’s real but simply not worth acting on	A statistically significant result that’s real but not practically meaningful

For Beginners: What to Actually Do

Always ask “how big is the effect, in real, practical terms?” alongside “is it statistically significant?”
Practice setting a minimum detectable effect threshold before designing an experiment, rather than deciding after seeing the result whether it “counts.”
Learn to read a confidence interval around an effect size, not just a significance flag, as part of interpreting any result.

For Practitioners and Leaders: The Deeper Layer

Require an explicit minimum detectable effect and a practical significance threshold to be defined before any major experiment launches.
Push back on decisions justified by statistical significance alone, without a clear account of whether the effect size actually matters for the business.
Weigh the real implementation cost of a change against its measured effect size, not just against a significance threshold.

Quick Recap

Practical significance asks whether a statistically significant effect is actually large enough to matter for a real decision.
Large sample sizes can make even genuinely tiny effects statistically significant.
Defining a minimum detectable effect before running a test forces a practical threshold to be set in advance.
Confidence intervals around effect size keep the actual magnitude of an effect visible, not just its statistical confidence.

Where This Fits in the Series

Article 11 covered the gap between statistical and practical significance. Article 12 covers testing several changes at once while still keeping each one’s contribution isolated.