Crates, Barrels, and Loose Cargo

Opening Scene

Not everything that comes off a ship looks the same. Crates stack neatly, uniform and predictable, easy to count and slot into a grid. Barrels are sealed and self-contained, each one internally consistent but not identical to the next. And then there’s loose cargo — coiled cable, raw timber, sacks of grain — with no fixed shape at all, handled entirely on its own terms. A harbor that only knows how to move crates is going to have a very bad day the first time a shipment of loose cargo shows up.

Data comes in these same three shapes, and knowing which one you’re looking at changes everything about how you handle it.

In Plain English

Structured data fits neatly into rows and columns with a fixed, predefined format — the crates. Semi-structured data has some internal organization but not a rigid, uniform shape — think JSON or XML, the barrels: self-contained and consistent internally, but varying from one to the next. Unstructured data has no predefined format at all — documents, images, audio, video — the loose cargo, handled on its own terms rather than forced into rows and columns.

The Old Way

Traditional warehouses were built almost exclusively for crates — structured data, and only structured data. Semi-structured data, when it existed at all in older systems, usually got flattened and forced into a rigid table structure, losing some of its natural flexibility in the process, the data equivalent of cutting a barrel apart to stack it like a crate. Unstructured data mostly wasn’t handled by data systems at all — it lived in separate document stores, file shares, or media libraries, disconnected from the analytical systems everyone actually queried.

This wasn’t a limitation anyone chose lightly — early storage and processing technology genuinely struggled with anything that didn’t fit rows and columns. But it meant enormous amounts of an organization’s actual information — contracts, support tickets, product images, call recordings — sat entirely outside the systems built to help people understand the business.

What’s Changing (and Why AI Is the Reason)

Modern lakehouse platforms genuinely handle all three shapes natively. Rather than forcing everything into rows and columns or exiling it to a separate system, contemporary platforms store and query structured, semi-structured, and unstructured data side by side, closing the historical gap this article’s “old way” section described.
AI is what makes unstructured cargo actually useful, not just storable. A lake full of documents and images was always technically possible to store; AI is what makes it possible to search, summarize, and reason over that content at scale — turning previously inert loose cargo into genuinely queryable information.
Semi-structured data is increasingly AI’s native format. API responses, event logs, and the outputs of AI systems themselves are very often semi-structured — this middle category, historically the awkward one to handle well, is becoming one of the most common shapes of data flowing through modern platforms.

The Metaphor, Fully Extended

Harbor Element	Data Concept
Crates, uniform and stackable	Structured data (rows and columns)
Sealed barrels, consistent inside but varying between them	Semi-structured data (e.g., JSON)
Loose cargo with no fixed shape	Unstructured data (documents, images, audio)
Cutting a barrel apart to stack it like a crate	Flattening semi-structured data into a rigid table
Loose cargo stored in a separate facility across town	Unstructured data kept outside core analytical systems
A crew that can now sort, label, and use any cargo shape on the same dock	A lakehouse handling all three data types natively

For Beginners: What to Actually Do

For any dataset you work with, explicitly identify which of the three shapes it is before deciding how to store or query it — the right approach genuinely differs by shape.
Get hands-on with semi-structured data specifically (a JSON API response is a good starting point) — it’s simultaneously more common and less intuitive than structured or fully unstructured data.
Notice where unstructured content already exists at your organization but isn’t connected to any analytical system. That gap is often larger, and more valuable to close, than it first appears.
Don’t assume flattening semi-structured data into a table is always wrong — sometimes it’s the right call. Just make it a deliberate choice, not a default born of tooling limitations.

For Practitioners and Leaders: The Deeper Layer

Audit what fraction of your organization’s total information genuinely sits in unstructured form, disconnected from analytical systems. For most organizations, this is a much larger and more valuable pool than the structured data teams have historically focused on.
Native multi-shape support in a lakehouse platform is a real capability, but it doesn’t remove the need for a deliberate strategy about which shape a given use case actually needs — “just store it as JSON, we’ll figure it out later” reintroduces the swamp risk from Article 3 in a new form.
As semi-structured data becomes more central, invest in your team’s comfort working with it directly, rather than reflexively flattening everything into familiar rows and columns out of habit.
The AI-driven unlock for unstructured data is genuinely new value, not just new complexity — treat previously “dead” unstructured content as an asset worth actively cataloguing and connecting, not merely storing.

Quick Recap

Data comes in three shapes: structured (crates), semi-structured (barrels), and unstructured (loose cargo) — each needs different handling.
Traditional systems handled structured data well and largely excluded the other two from core analytical systems.
Modern lakehouse platforms handle all three natively, and AI is what makes unstructured content genuinely useful rather than merely storable.
Semi-structured data is increasingly the default shape for API and event data, and deserves deliberate handling rather than automatic flattening.

Where This Fits in the Series

Article 4 showed the whole port running as one system. This article named the different cargo shapes that system has to handle. Article 6 goes deeper on the warehouse-versus-lake schema question from Articles 2 and 3, now that all three cargo shapes are on the table.