A Registry With Ten Million Names

Opening Scene

A small-town registry with a few thousand citizen files can afford a clerk who compares every new filing against every existing file, one by one, and still finish before closing time. Scale that same approach to a national registry with ten million names, and the same clerk, working at the same pace, would need centuries to compare every new file against every existing one. The registry doesn’t respond by hiring centuries’ worth of clerks. It restructures the whole approach — organizing files into indexed, searchable groups so that a new filing only ever needs to be compared against a small, relevant slice of the whole registry, not the entire thing.

Master data matching hits the exact same wall, at a scale most organizations reach faster than they expect.

In Plain English

Performance at scale in MDM is the discipline of keeping matching, reconciliation, and propagation fast and cost-effective as the number of entities grows from thousands into millions or beyond. Naive matching — comparing every record against every other record — has a cost that grows quadratically with the number of records, meaning doubling your entity count doesn’t double your matching cost, it roughly quadruples it. At real-world scale, that growth curve makes naive matching computationally infeasible long before it makes it merely slow, which is why blocking, introduced in Article 3, and the broader architectural techniques covered here aren’t optional refinements — they’re what makes matching possible at all beyond a fairly modest scale.

The Old Way

A few consistent architectural techniques have long made large-scale matching tractable:

Blocking and indexing reduce the comparison space dramatically — grouping records by a shared attribute like postal code, phonetic name encoding, or business category before comparing them means a new record only gets compared against the (much smaller) set of records that share its block, not the entire dataset.
Incremental matching avoids reprocessing the whole dataset on every change — once an initial full match has been run, a well-designed system only needs to match newly added or newly changed records against the existing golden records, rather than recomputing every match in the dataset from scratch each time something changes.
Distributed processing spreads the remaining comparison work across many machines in parallel — even after blocking and incremental processing reduce the workload substantially, matching at real scale typically still requires distributing the remaining comparisons across a cluster rather than a single machine, especially for the periodic full reconciliation passes that catch anything incremental matching might miss.

These three techniques compound — a system using all three handles orders of magnitude more entities than one using none of them, at a small fraction of the raw computational cost naive matching would require.

What’s Changing (and Why AI Is the Reason)

AI-based matching models, particularly those using vector embeddings to represent records, enable approximate nearest-neighbor search techniques that find likely matches dramatically faster than exhaustive comparison, trading a small, well-understood risk of missing an edge-case match for a large, predictable performance gain — the same fundamental tradeoff that vector search makes in other contexts, applied here to entity matching specifically.
AI-assisted blocking can learn better grouping strategies than fixed rules like postal code or phonetic encoding, discovering which attributes actually predict likely matches for a given dataset’s specific patterns, and adjusting the blocking strategy as those patterns shift over time rather than requiring a human to redesign the blocking rules by hand.
The computational cost of AI-based matching itself — running a trained model over potentially every record pair candidate — has to be weighed against its accuracy gains, meaning teams increasingly need to make deliberate tradeoffs between matching sophistication and matching throughput, rather than assuming more sophisticated AI matching is automatically better at every scale.

The Metaphor, Fully Extended

Registry Element	Master Data Management Concept
A clerk comparing every new filing against every one of ten million existing files, one by one	Naive matching, whose cost grows quadratically and becomes infeasible at real scale
Files organized into indexed, searchable regional groups before any comparison begins	Blocking and indexing, reducing the comparison space to a manageable relevant slice
Only processing today’s new filings against the existing registry, not reprocessing every file ever filed	Incremental matching, avoiding a full dataset reprocessing on every change
Multiple clerks working different regional blocks simultaneously rather than one clerk working alone	Distributed processing, spreading remaining comparison work across many machines in parallel
An assistant learning which grouping actually predicts a likely match best for this specific registry	AI-assisted blocking, learning better grouping strategies than fixed, hand-designed rules

For Beginners: What to Actually Do

Understand why naive matching’s cost grows quadratically with entity count — doubling the data roughly quadruples the comparison work, which is the core reason scale techniques exist at all.
Get familiar with blocking as a concept before diving into any specific matching tool, since nearly every real-world matching system depends on some form of it.
Notice the difference between an incremental matching process and a full reprocessing pass, and understand why most systems need both — incremental for day-to-day changes, full passes periodically to catch what incremental processing might miss.
When evaluating a matching approach’s reported accuracy, ask what scale it was tested at — accuracy figures from a small pilot dataset don’t always hold at production scale.

For Practitioners and Leaders: The Deeper Layer

Design your matching architecture around blocking, incremental processing, and distributed computation from the outset, rather than retrofitting them after a naive approach hits a performance wall in production.
Schedule periodic full reconciliation passes even in an otherwise incremental system, since incremental matching alone can slowly accumulate small, uncaught drift over time.
Evaluate AI-based matching techniques, including embedding-based approximate search, specifically for how their performance and accuracy trade off at your actual entity-count scale, not just at pilot scale.
Budget explicitly for the computational cost of AI-based matching at scale, since a more sophisticated model applied to every record pair candidate can become a genuine throughput bottleneck if not architected carefully.

Quick Recap

Naive matching’s quadratic cost growth makes it computationally infeasible well before most organizations reach truly large entity counts.
Blocking, incremental matching, and distributed processing are the three classic techniques that compound to make large-scale matching tractable.
AI-based approximate nearest-neighbor matching and AI-assisted blocking extend these techniques further, at the cost of a genuine, deliberate tradeoff against raw matching sophistication.
Performance planning needs to account for real production scale, not pilot-dataset scale, since matching cost curves bite hardest exactly where naive testing is least likely to reveal them.

Where This Fits in the Series

Articles 10 through 13 covered production concerns: stewardship, propagation, safe merging, and now performance at scale. Article 14 opens the AI-focused arc of this series, starting with AI-assisted entity matching and fuzzy resolution in more technical depth.