The Cold Case File

Opening Scene

Not every case gets investigated while the evidence is fresh. Sometimes a detective is handed a cold case — a model already running in production for years, built by a team that’s long since moved on, with incomplete documentation and no clean access to the original training pipeline. The techniques this series has covered still apply, but the investigation itself gets harder.

In Plain English

Retrofitting explainability means applying interpretability techniques to a model that’s already deployed, often without full access to its original training data, code, or the team that built it. Most post-hoc techniques from this series — SHAP, LIME, PDPs, sensitivity analysis — only need the ability to query the model with inputs and observe outputs, which makes them genuinely usable even on a true cold-case model, as long as that basic access exists.

The Old Way

Before this had a name specific to machine learning, organizations faced the same challenge with any aging, undocumented system:

Reverse-engineering a legacy software system with no remaining documentation, working from its observed behavior rather than its original design intent.
An auditor examining a financial process built by people no longer at the company, working from outputs and available records rather than original design documentation.
A historian reconstructing what happened from surviving records, without access to the original participants’ direct testimony.

The core challenge — understanding a system from its outputs and behavior alone — long predates machine learning specifically.

What’s Changing (and Why AI Is the Reason)

As organizations accumulate more legacy production models over time, connecting directly to the model sprawl concern covered in this content library’s MLOps series, retrofitting explainability has become a genuinely common, recurring task rather than a rare exception.
Model-agnostic techniques like LIME, introduced in Article 6 specifically for this reason, have become especially valuable in cold-case scenarios, since they require no knowledge of the model’s internals at all.
Growing regulatory pressure, covered in Article 19, has pushed organizations to retrofit explainability onto models that were built before any such requirement existed, turning what used to be optional archaeology into a genuine compliance necessity.**

The Metaphor, Fully Extended

The Investigation	Retrofitting Concept
A cold case with incomplete original records	A legacy model with incomplete documentation
Working from available evidence rather than original testimony	Working from model queries rather than training-time access
Model-agnostic techniques requiring no internal knowledge	LIME and similar tools usable on any black-box model
A cold case reopened because new pressure demands answers	A legacy model retrofitted because new regulation demands it

For Beginners: What to Actually Do

Practice applying LIME or SHAP to a model where you deliberately don’t use any internal knowledge, simulating a cold-case scenario.
Learn to work from model input-output access alone as a baseline skill, not just as a fallback when better access isn’t available.
Recognize that retrofitting explainability is a legitimate, common task, not a sign that something was done wrong originally.

For Practitioners and Leaders: The Deeper Layer

Inventory legacy production models specifically for their explainability status, treating gaps as a real, trackable risk alongside the cost tracking from this content library’s MLOps series.
Prioritize retrofitting effort based on a model’s real-world consequence, not just its age or how easy retrofitting happens to be.
Build documentation requirements into new model deployments specifically so future teams don’t face the same cold-case problem.

Quick Recap

Retrofitting explainability means applying interpretability techniques to an already-deployed model with incomplete original documentation.
Model-agnostic, post-hoc techniques are especially valuable here since they need only input-output access.
Growing model sprawl and regulatory pressure have made this a common, recurring organizational task.
Documenting explainability requirements at deployment time prevents future cold cases.

Where This Fits in the Series

Article 15 covered explaining a model without the original context. Article 16 shifts to communicating any of these explanations to people who aren’t technical at all.