When Explanations Mislead

Opening Scene

A skilled criminal doesn’t just commit a crime — sometimes they deliberately plant a plausible, misleading trail of evidence for investigators to follow. Explanation techniques, it turns out, can be gamed the same way. A model, or the process around it, can be deliberately or accidentally structured to produce reasonable-looking explanations that mask what’s actually driving a decision.

In Plain English

Explanation gaming (or manipulation) refers to cases where a model’s explanations are deliberately or inadvertently misleading — either because an organization has selectively presented favorable explanations, or because a genuinely adversarial actor has crafted inputs specifically to produce innocuous-looking explanations for a harmful decision. This risk connects directly to the faithfulness concerns from Article 11, but goes further: it treats faithfulness itself as something that can be actively attacked, not just accidentally lost.

The Old Way

Before this had a name specific to machine learning, the same manipulation risk showed up wherever explanations could be selectively shaped:

A company selectively disclosing favorable data in a financial report while technically not lying about anything specific.
A propagandist using true but carefully selected facts to build a misleading overall narrative.
A defendant’s account that’s factually accurate in every individual detail but deliberately omits crucial context.

In each case, individual pieces of an explanation could be technically true while the overall picture remained genuinely misleading.

What’s Changing (and Why AI Is the Reason)

As explainability has become a genuine compliance and trust requirement, covered next in Article 19, the incentive to produce favorable-looking explanations — even when they’re not fully faithful — has grown correspondingly.
Research has demonstrated that some explanation methods can be adversarially manipulated, where small, deliberate changes to a model or its inputs produce explanations that look reasonable while masking the model’s true reliance on a problematic feature.
This has strengthened the case for the independent human review from Article 12 and the faithfulness testing from Article 11, treated together as necessary defenses, not optional extras, particularly for the highest-stakes applications.

The Metaphor, Fully Extended

The Investigation	Explanation Gaming Concept
A deliberately planted, misleading trail of evidence	A manipulated or selectively favorable explanation
Individual facts that are true but assembled misleadingly	An explanation that’s technically accurate but practically deceptive
A detective who checks for planted evidence, not just plausible stories	Faithfulness testing that checks for adversarial manipulation
Independent oversight specifically to catch a motivated cover story	Independent human review as a defense against explanation gaming

For Beginners: What to Actually Do

Learn to recognize the difference between an explanation being technically accurate and being genuinely complete and honest.
Practice healthy skepticism toward unusually favorable-looking explanations, especially in contexts with a real incentive to mislead.
Understand this risk as a real, documented research finding, not a hypothetical worry.

For Practitioners and Leaders: The Deeper Layer

Treat explanation gaming as a genuine risk category in any high-stakes or adversarial deployment context, worth explicit threat modeling.
Maintain independent review processes, from Article 12, specifically resistant to internal incentives to present favorable explanations.
Stay current on adversarial explainability research, since this is an actively evolving area with real practical implications.

Quick Recap

Explanation gaming refers to explanations that are deliberately or inadvertently misleading, despite appearing technically reasonable.
This can happen through selective presentation or genuine adversarial manipulation of a model or its inputs.
Research has demonstrated real cases of explanation manipulation, making this a documented, not hypothetical, risk.
Independent human review and faithfulness testing are the strongest available defenses.

Where This Fits in the Series

Article 18 covered explanations as a potential target for manipulation. Article 19 looks at the growing regulatory response to explainability as a genuine requirement, not an optional courtesy.