When the Suspect Lies

Opening Scene

A suspect can give a story that’s entirely plausible, coherent, and convincing — and still be lying about what actually happened. A good detective knows plausibility and truth are not the same thing, and treats a persuasive story with real suspicion until it’s independently verified. The exact same gap exists in model explanations: a SHAP value or LIME approximation can sound entirely reasonable and still not accurately reflect what the model actually did.

That gap has a name: faithfulness.

In Plain English

An explanation is faithful if it accurately reflects the model’s actual internal reasoning, not just a plausible-sounding story that happens to fit the outcome. An explanation can be plausible — convincing to a human reader — without being faithful, and this gap is one of the most genuinely underappreciated risks in applied explainability. A beautiful, intuitive explanation that doesn’t actually describe what the model did is, in an important sense, worse than no explanation at all, because it creates false confidence.

The Old Way

Before this distinction had a name in machine learning, the same gap between plausible and true showed up wherever people accepted convenient explanations too readily:

A “just-so story” in evolutionary biology — a plausible-sounding explanation for why a trait evolved, accepted without rigorous evidence that it’s actually correct.
A confession obtained under pressure that sounds internally consistent but doesn’t match physical evidence, a well-documented failure mode in real investigations.
A business retroactively explaining a successful decision with a tidy narrative that may have little to do with why it actually worked.

In every case, a coherent story substituted for actual verification, and the substitution wasn’t always caught.

What’s Changing (and Why AI Is the Reason)

As explainability techniques like LIME have become widely adopted, researchers have increasingly studied and demonstrated cases where these methods produce plausible but unfaithful explanations, particularly for genuinely complex models.
New evaluation methods specifically test faithfulness — for instance, by removing a feature the explanation claims is important and checking whether the prediction actually changes as expected — turning faithfulness from an abstract concern into something measurable.
This has led to growing caution around treating any single explanation method as ground truth, favoring the kind of cross-checking across multiple techniques this series has built toward since Article 5.**

The Metaphor, Fully Extended

The Investigation	Faithfulness Concept
A story that sounds convincing but isn’t actually true	A plausible but unfaithful explanation
A story that matches the actual, verified physical evidence	A faithful explanation that reflects the model’s real reasoning
Independently verifying a suspect’s story against the evidence	Testing faithfulness by removing a claimed-important feature
A detective who never fully trusts a single uncorroborated account	Cross-checking explanations across multiple methods

For Beginners: What to Actually Do

Get comfortable with the uncomfortable idea that a clear, intuitive explanation isn’t automatically a correct one.
Practice a simple faithfulness check: remove or perturb a feature an explanation claims is important, and see if the prediction actually shifts as expected.
Treat cross-agreement between SHAP and LIME on the same case as a modest positive signal, not proof — both can share the same blind spots.

For Practitioners and Leaders: The Deeper Layer

Build faithfulness testing into any process that relies on explanations for high-stakes decisions, not just plausibility review by a human reader.
Communicate to stakeholders that an explanation’s clarity is not the same as its accuracy — this distinction matters enormously in regulated or adversarial contexts.
Treat unusually clean, tidy explanations with a healthy amount of extra scrutiny, not automatic trust.

Quick Recap

A faithful explanation accurately reflects a model’s actual reasoning; a plausible one merely sounds convincing.
These two properties can diverge, and the gap is a genuine, well-documented risk in applied explainability.
Faithfulness can be tested directly by perturbing claimed-important features and checking the prediction’s actual response.
No single explanation method should be treated as automatically faithful ground truth.

Where This Fits in the Series

Article 11 raised the risk of explanations that sound right but aren’t. Article 12 looks at the human role that has to sit on top of every technique this series has covered so far.