🏢

Streaming & Real-Time Data

Architectures for data that never stops moving, and the AI systems watching it live.

Part 1

The River Never Stops, the Reservoir Waits

A reservoir fills at its own pace and gets drawn from on schedule. A river never stops moving at all — and building for one when you actually need the other is a genuinely different discipline.

Part 2

Reading the Current, Not the Water Level

Checking a reservoir's water level every hour and reading a river's current continuously aren't the same skill just performed at different speeds — they're fundamentally different ways of paying attention.

Part 3

A Message in a Bottle, One at a Time

A river doesn't move as one undifferentiated mass — it's countless individual drops, each one traceable. Streaming data works the same way: one discrete event at a time, not a shapeless flow.

Part 4

The Rapids Have a Queue, Too

Even a river that never stops moving has a natural holding point where water backs up briefly before the next stretch — that holding point is exactly what a message broker provides for a stream of events.

Part 5

Paddling in Order, Even in Rapids

In a single narrow channel, water passes in the order it arrived, no question. Split that channel into several, and suddenly keeping track of what came before what gets genuinely harder.

Part 6

What Happens When You Miss an Eddy

A guide who misses an eddy can paddle back and try again, catch the next one instead, or just let it go — three genuinely different responses, and a streaming system has to pick one on purpose.

Part 7

Reading the River Before It Reaches You

A guide doesn't just watch the water go by — they actively read it in motion, spotting hazards and opportunities before the raft gets there. Stream processing does the same thing to data in flight.

Part 8

The Guide's Window of Attention

A guide can't watch the entire river's history at once, and doesn't need to — they focus on a deliberate window of what's happening right now. Streaming systems need the exact same discipline.

Part 9

When Two Boats Reach the Same Bend at Different Times

A boat that launched first but got delayed by an obstacle might reach a checkpoint after one that launched later — and a river guide's log has to account for when something actually happened, not just when it was recorded.

Part 10

Catching Up After You Fall Behind

A raft that falls behind the group has real options — paddle hard to catch up, signal the group to slow down, or accept the gap and rejoin at the next calm stretch. A stream consumer needs the same options.

Part 11

Building a Dam Without Stopping the River

You can't drain a river to renovate a dam gate — the water keeps coming whether you're ready or not, and a stream's event structure has to evolve under exactly that same constraint.

Part 12

A Guide Who Remembers Every Rapid

A guide who evaluates every rapid in total isolation, with no memory of what just happened upstream, misses almost everything that actually matters. Some streaming logic needs that same memory.

Part 13

Two Rivers Merging at the Delta

Two rivers merging at a delta don't just combine their water — the merge itself has to happen at the right moment, or you get a flood in one channel and a trickle in the other.

Part 14

The River That Feeds the Lake

A river doesn't just flow for its own sake — much of it eventually feeds into a lake, where it settles and becomes available for entirely different uses. Streaming data has the same eventual destination.

Part 15

Watching the River from the Bank in Real Time

A ranger station's dashboard showing yesterday's water levels is interesting history. One showing this exact moment's conditions is what actually lets someone act while it still matters.

Part 16

When the River Itself Sounds the Alarm

A truly modern river monitoring system doesn't wait for a ranger to notice rising water on a gauge — it sounds the alarm itself, the instant conditions cross into genuinely dangerous territory.

Part 17

Rafts That Steer Themselves Mid-Rapid

A raft that has to radio back to shore for instructions every time the water shifts will always be a beat behind. One that can adjust its own course mid-rapid, on live conditions, is playing an entirely different game.

Part 18

Not Every Trip Needs Whitewater

A calm, scenic float down a gentle stretch of river doesn't need a whitewater guide's specialized skills or gear — and plenty of genuinely valuable data use cases don't need real-time streaming either.

Part 19

Choosing Between the Reservoir and the River

After the whole trip downriver, the practical question every trip planner actually faces is simple to ask and genuinely hard to answer well: reservoir, river, or some of both, and for which specific need?

Part 20

The Whole Watershed, Flowing Together

How every concept from this series fits together as one connected watershed, and where streaming and real-time data are actually headed as AI reshapes what watching a river live even means.