Two Plots, One Season

Opening Scene

A farmer tries a new fertilizer on this year’s crop and gets a bigger harvest than last year’s. Success? Not necessarily. Last year could have had less rain, worse soil, a different seed batch, or a dozen other differences that have nothing to do with the fertilizer at all. Comparing two things that differ in more than one way at a time proves almost nothing — which is exactly the problem the randomized field trial was invented to solve, more than a century ago, and exactly the problem every modern A/B test still solves today.

In Plain English

A/B testing (or controlled experimentation) compares two or more versions of something — a webpage, a pricing plan, a feature — by randomly assigning subjects to each version and measuring the difference in outcome, while holding everything else as close to identical as possible. The entire discipline exists to answer one deceptively hard question: did this specific change actually cause this specific result, or would it have happened anyway?

The Old Way

Before formal experimental design existed, people judged changes the same way that farmer judged the fertilizer:

A shopkeeper trying a new store layout and judging it by whether sales went up that month, with no comparison to what would have happened without the change.
A doctor trying a new treatment and judging it by whether the patient improved, without accounting for what would have happened anyway.
A farmer comparing this year’s yield to last year’s, across two seasons that differed in far more than just the one change being tested.

In every case, a single before-and-after comparison couldn’t separate the effect of the change from everything else that happened to be different at the same time.

What’s Changing (and Why AI Is the Reason)

Digital products can now run randomized experiments at a speed and scale field trials never could — testing a change on thousands of users simultaneously, with a result available in days rather than an entire growing season.
As AI-driven products and recommendation systems make more automated decisions, rigorously testing whether those decisions actually improve outcomes has become a genuine competitive necessity, not an academic nicety.
Modern experimentation platforms, covered in Article 17, have made running a statistically rigorous test nearly as easy as deploying any other product change, lowering the barrier that once made careful experimentation rare.

The Metaphor, Fully Extended

The Field Trial	A/B Testing Concept
Two plots, planted and harvested in the same season	Two variants, tested on the same population at the same time
Everything held constant except the one change being tested	Random assignment holding all else equal except the treatment
A single before-and-after comparison across two different seasons	A flawed comparison that doesn’t isolate the actual cause
A controlled trial designed specifically to isolate one variable’s effect	A/B testing designed specifically to isolate one change’s effect

For Beginners: What to Actually Do

Before running or trusting any comparison, ask explicitly: were these two things tested under genuinely comparable conditions, at the same time?
Get comfortable with the idea that “it went up after we changed it” is not, by itself, evidence that the change caused the improvement.
Practice identifying real-world examples of the flawed before-and-after comparison in news, marketing, and everyday claims.

For Practitioners and Leaders: The Deeper Layer

Insist on genuine randomized comparison before attributing a business outcome to a specific change, rather than relying on before-and-after trend-watching.
Build organizational habits around testing changes rather than just shipping and observing — this series covers exactly how to do that rigorously.
Recognize that this same rigor connects directly to this content library’s dedicated causal inference series, which covers what to do when a true randomized test genuinely isn’t possible.

Quick Recap

A/B testing compares versions by randomly assigning subjects and measuring outcome differences, holding everything else constant.
A single before-and-after comparison can’t reliably separate a change’s real effect from everything else that happened at the same time.
Digital experimentation now runs at a speed and scale far beyond the original agricultural field trials that inspired it.
Rigorous testing has become a genuine competitive necessity as more decisions are automated by AI systems.

Where This Fits in the Series

This opening article set up the fundamental problem every experiment is designed to solve. Article 2 defines A/B testing in plain terms before the season’s trials get underway.