A Rogue Weed in the Data

Opening Scene

A perfectly designed trial can still fail if something goes wrong in the field itself — a rogue weed choking out one plot more than another, an irrigation line quietly leaking onto the control plot, a labeling mistake that swaps which plot got which treatment. None of these problems are about the experimental design. They’re about execution, and a beautifully designed trial can still produce a completely misleading result if execution goes wrong unnoticed.

In Plain English

Experiment data quality covers the practical, easy-to-overlook ways a well-designed A/B test can still go wrong: a bug that assigns users to the wrong group, a tracking mechanism that fails to log some conversions, sample ratio mismatches where the actual split between groups doesn’t match the intended one. These issues can invalidate an experiment’s results even when its statistical design, covered in Articles 4 through 7, was entirely sound.

The Old Way

Before formal data quality checks were standard experimentation practice, these problems often went undetected:

A scientific instrument malfunctioning without producing an obvious error, quietly corrupting an otherwise well-designed study’s data.
A survey with a subtly broken skip-logic pattern, silently excluding certain respondents in a way that biases the results.
A field trial with an unnoticed irrigation leak, quietly favoring one plot over another for reasons that have nothing to do with the treatment being tested.

In each case, a genuinely well-designed study could still produce misleading results because of a practical execution failure nobody caught.

What’s Changing (and Why AI Is the Reason)

Sample ratio mismatch (SRM) detection has become a standard automated check — comparing the actual observed split between groups against the intended split, and flagging a statistically significant deviation as a likely sign of an assignment or tracking bug.
Modern experimentation platforms increasingly run automated data quality checks alongside the statistical analysis itself, catching broken tracking or assignment issues before they reach a final, potentially misleading conclusion.
As experimentation scales to many simultaneous tests, covered in Article 18, automated data quality monitoring has become essential — manually auditing every single test for execution problems simply doesn’t scale.

The Metaphor, Fully Extended

The Field Trial	Data Quality Concept
A rogue weed favoring one plot over another	A bug or tracking issue favoring one experiment group over another
An unnoticed irrigation leak skewing the comparison	An unnoticed assignment bug skewing the comparison
Checking the actual planted count against the intended plot sizes	Checking the actual sample ratio against the intended assignment split
A trial invalidated by execution problems, despite sound design	An experiment invalidated by data quality problems, despite sound statistical design

For Beginners: What to Actually Do

Learn what a sample ratio mismatch is and how to check for one before trusting any experiment’s results.
Practice treating an experiment’s raw data with the same scrutiny you’d apply to any other data source, rather than assuming the platform got it right automatically.
Get in the habit of asking “could something have gone wrong in execution?” before accepting a surprising or a suspiciously clean result.

For Practitioners and Leaders: The Deeper Layer

Build automated sample ratio mismatch and data quality checks into your standard experimentation pipeline, not as an occasional manual audit.
Treat a flagged data quality issue as a genuine blocker to trusting an experiment’s result, even when the statistical analysis itself looks clean.
Connect this discipline to this content library’s dedicated data quality and observability series, which covers the broader practice this article applies specifically to experimentation.

Quick Recap

A well-designed experiment can still fail if execution problems, like assignment bugs or broken tracking, go undetected.
Sample ratio mismatch detection is a standard automated check for these kinds of issues.
Modern platforms increasingly run data quality checks alongside statistical analysis, catching problems before they reach a conclusion.
Automated monitoring becomes essential as organizations run many simultaneous experiments.

Where This Fits in the Series

Article 9 covered the practical execution risks that can invalidate a sound design. Article 10 turns to actually reading a trial’s result once it’s trustworthy.