An Emergency Landing Procedure, Rehearsed in Advance

Opening Scene

Aviation crews don’t improvise emergency procedures in the moment a genuine emergency happens. Every serious emergency scenario has a rehearsed, well-documented procedure, practiced repeatedly under calm conditions long before it’s ever actually needed, precisely so that if the moment comes, the response is fast, practiced, and calm rather than improvised under real pressure. The rehearsal happens specifically because a crisis is the worst possible time to be figuring out the plan from scratch.

That same principle — a well-rehearsed, ready-to-execute plan prepared well before it’s needed — is exactly what a genuine rollback strategy provides for a deployed model.

In Plain English

A rollback strategy is a tested, ready-to-execute plan for reverting a deployed model to a previous, known-good version quickly, if a serious problem is discovered after deployment. Having a rollback plan that exists only on paper, never actually tested, is a common and dangerous gap — the moment a rollback is genuinely needed, under real pressure, is the worst possible time to discover the plan doesn’t actually work as documented.

The Old Way

Before formal rollback strategies were standard machine learning practice, the same gap between “we have a plan” and “we’ve actually tested the plan works” showed up constantly:

A business’s disaster recovery plan that’s never actually been tested, discovered to have gaps only when a genuine disaster strikes.
A building’s fire evacuation plan that’s never been rehearsed, revealing real problems only during an actual emergency.
Software’s rollback procedure that’s never actually been exercised, failing in unexpected ways exactly when it’s genuinely needed.

In each case, the existence of a documented plan was not the same thing as a tested, reliable one — and the difference between the two only becomes obvious at the worst possible moment.

What’s Changing (and Why AI Is the Reason)

Modern deployment infrastructure increasingly supports fast, automated rollback as a built-in capability, rather than requiring a manual, error-prone process improvised during an actual incident.
Regularly rehearsing rollback procedures — deliberately testing them under controlled conditions, not just documenting them — has become recognized as genuine best practice, directly echoing the aviation emergency-drill principle.
The canary deployment approach from Article 4 inherently limits how much damage needs to be rolled back, since only a small percentage of traffic was exposed to the problematic version in the first place, making rollback faster and lower-stakes when it is needed.

The Metaphor, Fully Extended

Airport Operations	Rollback Strategy Concept
A rehearsed emergency landing procedure	A tested, ready-to-execute rollback plan
Practicing the procedure under calm, controlled conditions	Testing the rollback process before it’s genuinely needed
A crisis being the worst time to improvise a response	A production incident being the worst time to discover a rollback plan doesn’t work
A documented but never-rehearsed emergency procedure	A documented but never-tested rollback plan
Fast, practiced execution when an emergency genuinely occurs	Fast, reliable execution when a rollback is genuinely needed
Limiting an emergency’s scope through careful operational design	Limiting rollback scope through canary deployment’s controlled traffic exposure

For Beginners: What to Actually Do

Understand that a documented rollback plan and a tested one are genuinely different things — never assume a plan works just because it’s written down.
If you’re involved in a deployment process, ask directly whether the rollback procedure has actually been tested recently, not just documented.
Recognize rollback readiness as a core, non-negotiable part of deployment readiness, connecting directly to the checklist discipline covered in Article 2.

For Practitioners and Leaders: The Deeper Layer

Schedule regular, deliberate rollback drills for critical deployed models, the same way aviation schedules emergency procedure rehearsals — under controlled conditions, before they’re genuinely needed.
Invest in deployment infrastructure that supports fast, automated rollback as a built-in capability, reducing dependence on manual, error-prone processes during genuine incidents.
Treat “when did we last actually test our rollback procedure” as a standard, recurring question for any consequential deployed model, not a one-time setup task.

Quick Recap

A rollback strategy is a tested, ready-to-execute plan for reverting a model to a known-good version quickly if a serious problem is found.
This mirrors aviation’s emergency procedure rehearsal discipline — practicing a response well before it’s genuinely needed, not improvising under pressure.
Modern infrastructure increasingly supports fast, automated rollback as a built-in capability.
A documented rollback plan and a genuinely tested one are different things, and only the tested version can actually be trusted.

Where This Fits in the Series

Article 6 covered detecting drift in production; this article covered having a genuinely tested plan ready if a serious problem is found. Article 8 looks at a related discipline — keeping a reliable record of exactly which version of a model is actually running.