When Two Boats Reach the Same Bend at Different Times

Opening Scene

A boat that launched first but got briefly caught on a snag might reach the next checkpoint after a boat that launched later and had a clear run. A guide keeping an accurate log has to record when each boat actually passed the launch point, not just the order the checkpoint happened to observe them arriving. Get that distinction wrong, and the log tells a story that didn’t actually happen — the delayed boat looks like it left second, when it actually left first.

That exact distinction — when something actually happened versus when a system happened to observe it — is one of the most important and most commonly mishandled concepts in streaming.

In Plain English

Event time is when something actually happened in the real world. Processing time is when the streaming system actually observes and processes that event, which can lag behind event time for all kinds of reasons — network delay, a temporary system slowdown, an out-of-order arrival. Watermarks are a mechanism streaming systems use to track how far behind processing time has fallen relative to event time, letting a system make a reasonable, bounded decision about when it’s safe to consider a window “done,” even with some late-arriving events still trickling in.

The Old Way

Early streaming systems often conflated event time and processing time by default — simply treating “when I saw it” as equivalent to “when it happened,” because handling the distinction correctly is genuinely harder to implement. For many use cases with minimal delay between the two, this simplification barely mattered in practice.

But for use cases with real, variable delay — mobile devices with spotty connectivity, systems that batch small delays before forwarding — conflating the two produced genuinely wrong results: a windowed count of “events in the last minute” could be quietly missing events that happened during that minute but arrived a few seconds late, with no indication anything was wrong.

What’s Changing (and Why AI Is the Reason)

Modern streaming frameworks provide mature, built-in support for event-time processing and watermarks. This is no longer something engineers need to build from scratch — established patterns and framework-level support make correctly handling event time versus processing time a well-supported, achievable practice rather than an advanced, rarely-attempted one.
AI is helping tune watermark configuration to actual observed delay patterns. Setting how long a system should wait for late-arriving events before finalizing a window involves a real trade-off between completeness and latency; AI-assisted analysis of actual historical delay patterns can inform a more precise, evidence-based configuration than a fixed, arbitrary guess.
AI systems making time-sensitive decisions are particularly exposed to this distinction going wrong. An AI agent reasoning about “what happened in the last five minutes” based on processing time rather than event time can draw meaningfully incorrect conclusions during periods of variable delay — raising the practical stakes on getting this right as more AI systems consume streaming aggregates directly.

The Metaphor, Fully Extended

River Element	Event Time Concept
When a boat actually launched	Event time: when something actually happened
When the checkpoint guide actually observed the boat arrive	Processing time: when the system actually processes the event
A boat delayed by a snag, arriving later than its actual launch order suggests	An event delayed in transit, arriving later than when it actually occurred
A guide’s rule for how long to wait for a delayed boat before finalizing the checkpoint log	A watermark defining how long to wait for late events before finalizing a window
A guide’s log that reflects true launch order, not just arrival order	A streaming system correctly using event time rather than processing time

For Beginners: What to Actually Do

Practice the exercise of distinguishing event time from processing time for a real scenario you’re familiar with — a mobile app event, a sensor reading — and identify realistic sources of delay between the two.
Get comfortable with the basic idea of a watermark as a deliberate, bounded trade-off: waiting longer catches more late events but delays finalizing results; waiting less finalizes faster but risks missing genuinely late-but-valid events.
When evaluating a streaming system’s windowed results, ask explicitly whether it’s using event time or processing time — this single question surfaces a lot about the system’s actual correctness under real-world delay conditions.
Notice this distinction as one of the more genuinely subtle concepts in this series, worth revisiting until it feels intuitive rather than just memorized.

For Practitioners and Leaders: The Deeper Layer

Audit existing streaming systems for whether they correctly distinguish event time from processing time, particularly for any use case with known variable delay sources (mobile clients, third-party data, cross-region latency).
Treat watermark configuration as a deliberate, evidence-informed trade-off between completeness and latency, not a default setting left untouched — the right balance depends genuinely on the specific use case’s actual delay patterns and tolerance for incompleteness.
For AI systems consuming streaming aggregates, verify explicitly that they’re built on event-time-correct data, especially for time-sensitive decisions — this is a subtle correctness issue that won’t necessarily show up as an obvious error, just as quietly wrong results.
Revisit watermark and delay-tolerance configuration periodically as actual traffic patterns evolve — a configuration tuned for one traffic pattern may become miscalibrated as client behavior, network conditions, or data sources change over time.

Quick Recap

Event time is when something actually happened; processing time is when a streaming system observes it — the two can diverge due to delay, and conflating them produces subtly wrong results.
Watermarks let a streaming system make a bounded, deliberate decision about when to finalize a window despite some late-arriving events still being possible.
Modern frameworks provide mature support for event-time processing, and AI-assisted analysis can help tune watermark configuration to real observed delay patterns.
AI systems making time-sensitive decisions are particularly exposed to this distinction, raising the practical importance of getting it right as more AI consumes streaming data directly.

Where This Fits in the Series

Article 8 covered choosing a deliberate window. This article covered making sure that window reflects when things actually happened, not just when they were observed. Article 10 looks at what happens when a consumer genuinely falls behind the stream altogether.