Watching the Walk-In Door

Opening Scene

One way to know what’s changed in the pantry is to count everything, every hour, and compare the count to last time. It works, but it’s slow, and it tells you something changed without telling you exactly what, or when. A faster way: put someone at the walk-in door, watching every single item that goes in or out, the instant it happens. Now you don’t need to guess what changed between counts — you already have a complete, exact record of every single change, as it happened.

That second approach is change data capture, and it’s a fundamentally different way of feeding a pipeline than periodically checking for what’s different.

In Plain English

Change data capture (CDC) watches a source system’s own internal record of changes — the log it already keeps of every insert, update, and delete — and streams those changes to a pipeline as they happen, instead of periodically querying the whole dataset and comparing it to last time. It’s a more precise, more immediate way of doing what Article 7’s incremental loading does, and it’s especially valuable for sources that change constantly and where “as of an hour ago” isn’t good enough.

The Old Way

Traditionally, keeping a downstream system in sync with a source meant periodically querying it — every hour, every night — and figuring out what had changed, usually via a timestamp field as covered in Article 7. This worked, but it had two real limits: it only sees the state at each check, not everything that happened in between, and if something changed and then changed back before the next check, that entire event is invisible. It’s the pantry-counting approach: accurate about the current total, blind to the actual history of comings and goings.

For sources changing rapidly — an order-processing system, a live inventory feed — this gap between checks meant downstream systems were always working with data that was, at best, slightly stale, and at worst, missing transient changes that genuinely mattered.

What’s Changing (and Why AI Is the Reason)

1. CDC infrastructure is easier to stand up than it used to be. Configuring a source’s change log to stream reliably into a pipeline used to require specialized expertise. AI-assisted tooling can now help configure and validate a CDC pipeline for a given source system, lowering the bar to using it where it genuinely helps, rather than defaulting to periodic checks purely because CDC felt too complicated to set up.
2. CDC streams are a natural fit for AI agents that need current state. An agent acting on live business data benefits directly from seeing changes as they happen rather than working from a snapshot that’s already an hour stale — the walk-in door watcher, not the hourly headcount, is what an agent expecting real-time accuracy actually needs underneath it.
3. Anomalies in the change stream itself are easier to catch. AI-assisted monitoring can watch the pattern of changes flowing through CDC — an unusual spike in deletes, an unexpected burst of updates — and flag it as a potential problem at the source, something a periodic snapshot comparison would likely miss entirely between checks.

The Metaphor, Fully Extended

Kitchen Element	Change Data Capture Concept
Counting the whole pantry every hour	Periodic, snapshot-based incremental checking
Watching the walk-in door continuously	Change data capture streaming every insert, update, and delete
An item that goes in and out between counts, invisible to the headcount	A transient change missed by periodic snapshot comparison
The doorwatcher’s running log of every item movement	The source system’s internal change log that CDC reads from
Setting up a reliable doorwatcher shift	Configuring CDC infrastructure for a given source
Noticing an unusual rush of items leaving all at once	Anomaly detection on the pattern of a CDC change stream

Quick Recap

• Change data capture streams every individual insert, update, and delete from a source as it happens, instead of periodically checking for differences.
• Traditional periodic checking is simpler but can miss transient changes and always lags behind real time by at least the check interval.
• AI tools are lowering the barrier to setting up CDC reliably and can monitor the change stream itself for early anomaly signals.
• CDC is a real operational commitment, best reserved for sources where completeness and immediacy genuinely matter.
• AI agents needing current state are a growing reason CDC is worth the added complexity where it used to be optional.

Where This Fits in the Series

Article 12 covered surviving a source that changes shape unexpectedly. This article covered watching a source that changes constantly, on purpose. Article 14 looks at the opposite problem: what a pipeline does when it’s fallen behind and needs to catch up on everything it missed.