One Variable at a Time

Opening Scene

A farmer eager to improve yield might be tempted to change everything at once — new fertilizer, new seed variety, a new watering schedule — all in the same season. If the yield goes up, which change deserves the credit? Without isolating each variable, there’s no honest way to answer that question, and the farmer risks repeating an expensive combination when only one part of it was actually doing the work.

In Plain English

Isolating the treatment variable means changing exactly one thing between the control and treatment groups, so that any measured difference in outcome can be attributed specifically to that one change. Testing multiple changes simultaneously in a single treatment group — a redesigned page with a new headline, a new image, and a new call-to-action button all at once — makes it impossible to know which specific element actually drove the result.

The Old Way

Before this discipline was formalized, bundling multiple changes together was common practice:

A product team shipping an entire redesign at once and measuring overall engagement, with no way to know which specific element helped or hurt.
A scientist changing several experimental conditions simultaneously, unable to attribute a result to any single factor.
A farmer changing fertilizer, seed, and irrigation together, unable to tell which change, if any, actually mattered.

Bundling changes together often felt efficient — one test instead of several — but it traded away the ability to actually learn anything specific from the result.

What’s Changing (and Why AI Is the Reason)

Multivariate testing methods, covered directly in Article 12, now let experimenters test several variables at once while still statistically disentangling each one’s individual contribution, rather than fully sacrificing isolation for the sake of speed.
As experimentation platforms have matured, running several small, isolated tests sequentially or in parallel has become nearly as fast as bundling changes together used to be, removing much of the old efficiency argument for bundling.
Organizations increasingly recognize that a bundled test that “wins” still leaves them without an answer to which specific change should carry forward into future work — a real, ongoing cost of skipping isolation.

The Metaphor, Fully Extended

The Field Trial	Variable Isolation Concept
Changing fertilizer, seed, and watering all in the same season	Bundling multiple product changes into a single treatment
Not knowing which change actually raised the yield	Not knowing which change actually drove the result
A trial designed to isolate exactly one variable’s effect	An A/B test designed to isolate exactly one change’s effect
A multivariate trial that disentangles several variables statistically	Multivariate testing that disentangles several changes statistically

For Beginners: What to Actually Do

Before launching a test, write down explicitly which single variable is being changed, and resist the temptation to bundle in “one more small thing.”
Practice recognizing bundled changes in real-world A/B tests you encounter, and ask what specific conclusion can actually be drawn from the result.
Learn the basic difference between a simple isolated A/B test and a multivariate test, covered in more depth in Article 12.

For Practitioners and Leaders: The Deeper Layer

Build a team norm around isolating test variables, even when it means running more, smaller tests rather than one large bundled change.
Weigh the real cost of bundling — an ambiguous, less actionable result — against the apparent speed advantage before defaulting to it.
Invest in multivariate testing capability specifically for cases where isolating every variable individually would be genuinely too slow.

Quick Recap

Isolating the treatment variable means changing exactly one thing between control and treatment groups.
Bundling multiple changes together makes it impossible to know which specific change drove a result.
Multivariate testing offers a rigorous middle ground, disentangling several changes statistically.
Modern experimentation platforms have reduced the old speed advantage that once justified bundling changes together.

Where This Fits in the Series

Article 6 covered keeping a test focused on one true cause. Article 7 covers how big the “field” actually needs to be for a test’s result to be trustworthy.