🏢

Data Warehousing & Lakehouses

The evolving home for structured and unstructured data, and how AI queries it directly.

Part 1

Two Docks, One Harbor

Every port has two very different ways of handling cargo — a precisely organized container depot and a wide-open wharf where anything can be dropped off — and that's exactly the split between a data warehouse and a data lake.

Part 2

Every Container Has Its Manifest

A container depot only works because nothing gets stacked in the yard without a manifest filled out first — that upfront discipline, applied to data, is what makes a warehouse a warehouse.

Part 3

Cargo Dropped at the Wharf

The open wharf takes cargo first and asks questions later — that deferred, flexible discipline is what makes a data lake genuinely useful, and genuinely risky, in equal measure.

Part 4

One Port, Two Systems, Working Together

A modern port doesn't force every ship to choose between the depot and the wharf — it runs both as one integrated operation, and that's exactly what a lakehouse does with data.

Part 5

Crates, Barrels, and Loose Cargo

A harbor handles three fundamentally different kinds of freight — neatly stacked crates, sealed barrels, and loose cargo with no fixed shape — and every data platform has to handle the same three kinds of data.

Part 6

Label It Now or Label It Later

Every port eventually has to answer the same question for every shipment: fill out the manifest before it's stacked, or leave it for whoever comes looking later? Neither answer is universally right.

Part 7

Sorting by What's Inside, Not When It Arrived

A yard organized by arrival order makes you search the whole lot for one item; a yard organized by contents lets you go straight to the right row — that's the entire case for columnar storage.

Part 8

Zoning the Yard

A container yard the size of a small city only works because it's divided into zones — and a lakehouse handling billions of rows depends on the same idea, called partitioning and clustering.

Part 9

Which Crate Are We Actually Using?

Not every container is built the same way — some are optimized for stacking density, some for quick access, some for specific cargo types. Open file formats are the same choice, one level below the yard.

Part 10

A Ledger for the Open Wharf

A wharf without a reliable ledger can't promise you that the count it just gave you is still accurate a second later — table formats bring that ledger, and real transactional trust, to lake storage.

Part 11

Cranes You Can Rent by the Hour

A port that owns a fixed number of cranes either sits idle during quiet periods or grinds to a halt during a surge — renting cranes by the hour, separate from the yard itself, solves both problems at once.

Part 12

The Port's Master Directory

Even a perfectly zoned, perfectly labeled yard is useless if nobody knows it exists — the port's master directory is what actually lets anyone find anything without walking the whole yard themselves.

Part 13

Who's Allowed on the Dock

A port with a perfect directory still needs a gate — knowing where every container is means nothing if anyone can walk up and open whichever one they want.

Part 14

Paying by the Container or by the Ton

A port can bill by container slot, by weight, by crane-hour, or some blend of all three — and each pricing model quietly rewards a different way of running your operation.

Part 15

Asking the Harbormaster Instead of Reading the Manifest

Most people visiting a port don't want to read a manifest themselves — they want to ask the harbormaster a plain question and get a straight answer. AI is turning that same shift loose on data platforms.

Part 16

A New Kind of Cargo Nobody's Manifested Before

Every cargo type this series has covered fits somewhere on a manifest. Vector embeddings don't — they're a genuinely new kind of freight, and the harbor is still figuring out how to store and find them.

Part 17

Ships That Don't Wait to Dock

Most cargo waits for a ship to fully dock before anything gets unloaded. Some cargo can't wait that long — it needs to move the moment it's ready, straight into the yard, without ever pausing at the gate.

Part 18

Relocating the Whole Port

Moving an entire working port to a new location without ever closing it down is one of the hardest operations in shipping — and migrating a live warehouse or lake to a lakehouse is exactly that hard.

Part 19

Which Dock Does Your Cargo Actually Need?

After walking the whole port, the practical question every shipper actually faces is simple to ask and genuinely hard to answer well: which dock does this specific cargo need, right now?

Part 20

The Whole Harbor, Working as One

How every concept from this series fits together as one connected port, and where warehouses, lakes, and lakehouses are actually headed as AI reshapes what a harbor even needs to do.