When You Can't Turn the Valve

Opening Scene

Some pipes simply don’t have an accessible valve. They’re sealed behind finished walls, buried under a foundation, or connected to a system too critical to safely experiment on. A plumber facing a sealed pipe doesn’t just give up on understanding the system — they turn to a different set of tools, ones built specifically for tracing connections without the freedom to intervene directly.

In Plain English

Many genuinely important causal questions can’t be tested with a randomized controlled trial: it may be unethical (you can’t randomly assign people to smoke, to test its effect on health), impractical (you can’t randomly assign entire countries to different economic policies), or simply too late (the event already happened, and there’s no way to rerun history with a different assignment). Observational causal inference covers the rigorous methods — instrumental variables, matching, difference-in-differences, regression discontinuity, all covered individually in the articles ahead — built specifically for exactly these situations.

The Old Way

Before observational causal inference methods matured, questions that couldn’t be tested experimentally were often either avoided or answered with the weaker reasoning covered earlier in this series:

Historical questions — did a specific policy actually cause an economic outcome — were often argued about informally, without any rigorous statistical method for actually testing the claim.
Ethically sensitive questions were sometimes tested unethically anyway, a well-documented and troubling part of the history of medical and social research.
Many genuinely important causal questions were simply treated as unanswerable rigorously, left to informal debate rather than formal analysis.

Observational causal inference methods emerged specifically to make rigorous progress on exactly these previously intractable questions.

What’s Changing (and Why AI Is the Reason)

A mature toolkit of observational methods — covered individually in Articles 8 through 11 — now provides genuinely rigorous causal estimates for many situations where randomization simply isn’t an option, closing much of the gap that once left these questions to informal argument.
Growing computational power and richer observational datasets have made these methods considerably more powerful and more reliable than in earlier eras when data limitations made a convincing observational causal case genuinely harder to construct.
As AI systems increasingly need to reason about causal effects from historical, observational data — since they can’t run a randomized trial on the past — these methods have become directly relevant to how modern AI-driven decision systems are built and evaluated.

The Metaphor, Fully Extended

Behind the Wall	Observational Causal Inference Concept
A pipe sealed behind a finished wall, with no accessible valve	A causal question that can’t be tested with a randomized trial
A different set of tracing tools, built for exactly this situation	A toolkit of observational methods built for exactly this situation
Tracing a connection carefully without ever directly intervening	Estimating a causal effect carefully without ever randomly assigning treatment
A plumber who’s learned to work rigorously without a valve to turn	An analyst who’s learned to work rigorously without a randomized trial

For Beginners: What to Actually Do

Learn to recognize the specific reasons an RCT might genuinely not be feasible for a given question — ethical, practical, or historical — before reaching for an observational alternative.
Get a working overview of the observational methods ahead in this series before diving deeply into any single one.
Practice being explicit that observational causal estimates rely on stronger, more debatable assumptions than a true randomized trial, even when they’re rigorous.

For Practitioners and Leaders: The Deeper Layer

Build organizational capability in observational causal inference specifically for the many important business and policy questions that genuinely can’t be tested with a randomized trial.
Set appropriate expectations that observational causal estimates, however rigorous, carry more assumption-dependent uncertainty than a true RCT.
Invest in richer observational data collection specifically because it strengthens every method covered in the rest of this series.

Quick Recap

Many important causal questions can’t be tested with a randomized controlled trial for ethical, practical, or historical reasons.
Observational causal inference provides a rigorous toolkit of methods built specifically for exactly these situations.
These methods have matured considerably with growing computational power and richer data.
They rely on stronger, more debatable assumptions than a true randomized trial, even when applied rigorously.

Where This Fits in the Series

Article 7 introduced the observational toolkit needed when randomization isn’t possible. Article 8 covers the first specific method: finding a natural, indirect way to turn the valve after all.