What the Detective Is Actually Looking For

Opening Scene

Two models predict which loan applicants will default. Both are equally accurate on a held-out test set. One of them makes decisions a loan officer can trace, in plain terms, to specific factors in an applicant’s history. The other is a dense ensemble that reaches the same conclusions through a process nobody can meaningfully describe. Accuracy alone can’t distinguish these two models. Something else has to.

That something else is what this entire series investigates.

In Plain English

Interpretability is how easily a human can understand the internal mechanics of how a model reaches its decisions. Explainability is closely related but slightly broader: it’s the ability to produce a human-understandable account of a specific decision, even for a model whose internals aren’t inherently interpretable. In practice, this series uses the terms together, the way a detective’s case file covers both the underlying facts and the story built to explain them.

The Old Way

Before this distinction had formal names, people already intuitively separated “does it work” from “can I understand why it works”:

A recipe that produces a great dish but that nobody, including the cook, can actually explain — reliable, but not repeatable through understanding.
A mechanic who can fix a car through trial and error without fully understanding the underlying system — effective, but not diagnosable when something new goes wrong.
A student who reaches the right answer on a math test through a memorized shortcut, without understanding the underlying method well enough to explain it to someone else.

In each case, the outcome could be correct while the reasoning behind it stayed genuinely opaque.

What’s Changing (and Why AI Is the Reason)

As models have grown in complexity — deep networks, gradient-boosted ensembles, large language models — the natural interpretability that simpler models like linear regression once offered by default has become rarer, not more common.
A dedicated field of interpretability research has matured specifically to recover some of that lost transparency, without forcing a return to simpler, less accurate models.
Growing regulatory and organizational pressure, covered later in this series, has turned “can you explain this decision” from a nice-to-have into a genuine requirement in many consequential domains.

The Metaphor, Fully Extended

The Investigation	Interpretability Concept
A case that’s solved correctly	A model that predicts accurately
A case that’s solved and explained clearly	A model that’s both accurate and interpretable
A detective who reaches the right conclusion through unclear methods	A black-box model that’s accurate but not interpretable
The detective’s ability to walk someone through the reasoning	Explainability — producing a human-understandable account of a decision

For Beginners: What to Actually Do

Practice distinguishing “this model works” from “I understand why this model works” as two genuinely separate claims.
When learning about a new model type, ask specifically how interpretable it is by design, not just how accurate it tends to be.
Get comfortable with the idea that interpretability itself comes in degrees, not as a simple yes-or-no property.

For Practitioners and Leaders: The Deeper Layer

Make interpretability an explicit, named requirement during model selection, alongside accuracy, latency, and cost.
Recognize that a highly accurate but uninterpretable model may carry real organizational risk that a slightly less accurate, more interpretable model does not.
Build shared vocabulary across technical and non-technical stakeholders early — much of this series’ value lies in that shared language.

Quick Recap

Interpretability is how understandable a model’s internal mechanics are; explainability is the broader ability to account for a specific decision.
Accuracy and interpretability are separate properties a model can have independently of each other.
More complex, more accurate model types have generally become less naturally interpretable over time.
A dedicated field of techniques has matured to recover transparency without sacrificing accuracy.

Where This Fits in the Series

Article 1 opened the case; this article defined what the detective is actually looking for. Article 3 looks at models that were transparent by design, long before this became a dedicated field.