Common Batch and Event-Driven Failures (and Mail That Never Arrived)

Opening Scene

The truck fails in boring, familiar ways: it breaks down mid-route and the whole night’s mail sits until morning, or a sorting error sends an entire bin to the wrong side of town at once. The courier system fails in stranger, harder-to-notice ways: one specific package quietly falls out of the queue and nobody notices for days, or a burst of drop-offs overwhelms the dispatch desk and a handful of packages get silently dropped in the confusion. Different systems, genuinely different ways of losing the mail.

In Plain English

Batch failures tend to be large, visible, and all-at-once: an entire run fails partway through and needs to be retried or rolled back, a bad input corrupts a whole day’s output, a job simply doesn’t run on schedule. Event-driven failures tend to be small, distributed, and easy to miss: a single message silently dropped, a consumer that’s been lagging for hours without anyone noticing, a duplicate that slipped past an idempotency check. Both categories are genuinely dangerous, but they demand different detection strategies — batch failures are usually caught by checking whether a job succeeded, while event-driven failures require monitoring the health of a continuous stream, as this series’ article on monitoring covered in more depth.

The Old Way

Before these failure modes were well catalogued and specifically designed against:

Batch failure recovery was often manual and improvised, with no standard playbook for how to safely retry or roll back a partially completed run.
Event-driven failures frequently went undetected for long stretches, since a single lost or stalled message rarely triggered an alert the way a failed batch job did.
Postmortems for each kind of failure were often treated as one-off incidents rather than instances of a well-understood, recurring pattern worth building specific defenses against.

Naming these as recurring, well-understood patterns — not one-off surprises — is what lets a team build specific defenses instead of relearning the same lesson repeatedly.

What’s Changing (and Why AI Is the Reason)

Teams increasingly maintain a documented catalogue of known failure modes for both batch and event-driven pipelines, with specific detection and recovery playbooks for each.
This draws directly on the idempotency, backpressure, and monitoring articles earlier in this series, since those mechanisms are precisely what prevent this article’s failure modes from happening in the first place.
AI systems consuming from either kind of pipeline add a new twist to an old problem: a model silently trained or served on partially failed batch output, or silently acting on a gap left by a dropped event, can produce a confidently wrong result with no obvious error message anywhere in the system.

The Metaphor, Fully Extended

How the Mail Actually Gets Lost	Failure Mode Concept
The truck breaking down mid-route, stalling an entire night’s mail	A batch job failing partway through, requiring retry or rollback
A sorting error sending an entire bin to the wrong side of town	A bad input corrupting a whole batch run’s output
One package quietly falling out of the queue, unnoticed for days	A single event silently dropped or lost in an event-driven system
A burst of drop-offs overwhelming the desk, dropping a few packages	A traffic spike overwhelming a consumer without adequate backpressure

For Beginners: What to Actually Do

Practice listing the difference between a failure you’d notice immediately (a failed batch job) and one you might not notice for days (a single dropped event).
Learn to recognize retry-and-rollback as the standard batch recovery pattern, and dead-letter queues as the standard event-driven equivalent.
Get comfortable with the idea that “no error message” doesn’t mean “nothing went wrong,” especially in event-driven systems.

For Practitioners and Leaders: The Deeper Layer

Maintain a documented, living catalogue of known failure modes for each production pipeline, with a specific detection and recovery playbook for each.
Revisit the idempotency, backpressure, and monitoring practices covered earlier in this series as the concrete defenses against this article’s failure catalogue.
Audit AI systems for silent failure propagation, specifically checking whether a model can act confidently on data quietly corrupted by a batch failure or missing due to a dropped event.

Quick Recap

Batch failures tend to be large, visible, and all-at-once; event-driven failures tend to be small, distributed, and easy to miss.
Each category requires a different detection strategy, matched to how that kind of system actually fails.
Documented, recurring failure catalogues with specific playbooks beat treating each incident as a one-off surprise.
AI systems can silently propagate either kind of failure into a confidently wrong result with no obvious error.

Where This Fits in the Series

Article 18 made the honest case for recognizing overkill. Article 19 catalogued how each approach actually tends to fail, and the mail that never arrived. Article 20 closes the series by looking ahead to a postal system that chooses its own delivery method.