Which Crate Are We Actually Using?

Opening Scene

Even within a well-zoned yard, the containers themselves aren’t identical. Some are built for maximum stacking density. Some are built for fast access to what’s inside without unloading the whole thing. Some are standardized specifically so any port in the world, run by any company, can handle them the same way. The choice of container isn’t zoning — it’s a layer below that, and it matters just as much.

For a lakehouse, that choice is the file format the actual data gets stored in.

In Plain English

Open file formats — Parquet, ORC, and Avro are the most common — define how data is physically encoded on disk. They’re “open” in the sense that any compatible tool, from any vendor, can read and write them, rather than locking data into one platform’s proprietary format. Parquet and ORC are columnar formats (Article 7’s storage layout, encoded into an actual file), well-suited to analytical queries. Avro is row-based, and better suited to workloads that need whole records at once, such as data actively being ingested.

The Old Way

Before open formats matured and gained broad adoption, data was often stored in proprietary formats tied to a specific database vendor or platform. This worked fine as long as you never needed to move that data anywhere else — but it made switching tools, or even just using a new tool alongside an existing one, expensive and technically painful. Data stored in a vendor’s proprietary format was, in a real sense, held hostage by that vendor’s ongoing cooperation.

This created exactly the kind of standardized-shipping-container problem the real freight industry solved decades ago: before standardized containers existed, cargo had to be repeatedly unloaded and reloaded in different forms at every port that used different equipment. Standardization made cargo genuinely portable across any compatible port. Open file formats did the same thing for data — genuinely portable across any compatible tool or platform.

What’s Changing (and Why AI Is the Reason)

Open formats are now the practical default for serious lakehouse deployments. Choosing an open format over a proprietary one is, for most new lakehouse work, no longer even framed as a debate — it’s close to a starting assumption, specifically because vendor lock-in risk is now well understood across the industry.
AI training pipelines depend heavily on efficient, well-supported formats. Training and fine-tuning workflows often need to read enormous volumes of data efficiently — the performance characteristics of formats like Parquet directly affect how practical large-scale AI training actually is, giving open formats a further, AI-specific reason to matter.
New formats are emerging specifically for AI-era data types. Just as Parquet and ORC were built for structured, columnar analytical data, newer formats and storage approaches are emerging to efficiently handle embeddings and vector data (Article 16) at scale — this is an actively evolving corner of the format landscape, not a settled one.

The Metaphor, Fully Extended

Harbor Element	File Format Concept
A standardized shipping container, usable at any compatible port	An open file format like Parquet
A proprietary container only one company’s equipment can handle	A vendor-specific, proprietary data format
Repeatedly unloading and reloading cargo at incompatible ports	The cost and friction of migrating data locked in a proprietary format
A container built for dense, efficient stacking	A columnar format like Parquet or ORC, optimized for analytical scanning
A container built for fast, whole-load access	A row-based format like Avro, suited to ingestion workloads
A new container design for a genuinely new kind of cargo	Emerging formats for vector/embedding data

For Beginners: What to Actually Do

When you first encounter Parquet, ORC, or Avro, don’t just memorize which is “the standard one” — understand the row-versus-columnar distinction from Article 7 that actually explains why each exists.
Practice identifying which format fits a given workload: analytical querying generally favors Parquet or ORC; active ingestion or whole-record processing often favors Avro.
Get comfortable with the practical reality that “open format” doesn’t mean “no decisions to make” — there are still real trade-offs between the open formats themselves.
If you inherit a dataset in a proprietary format, treat migrating it to an open format as a genuine investment worth prioritizing, not just a nice-to-have cleanup task.

For Practitioners and Leaders: The Deeper Layer

Audit your platform for any data still locked in proprietary formats, and treat the migration cost as a real, quantifiable form of vendor risk, not an abstract concern.
Open formats reduce lock-in at the storage layer, but don’t by themselves guarantee portability — query engines, catalogs, and access controls (Articles 12 and 13) can still create lock-in even when the underlying files are open. Evaluate the whole stack.
As AI training workloads grow, factor format efficiency directly into infrastructure cost planning — the difference between an efficient and inefficient format at genuine training scale is a real, material cost, not a rounding error.
Keep an eye on the emerging vector/embedding storage format landscape (Article 16) — it’s less mature than the structured-data format landscape, and early choices here are more likely to need revisiting as the space matures.

Quick Recap

Open file formats like Parquet, ORC, and Avro define how data is physically encoded, and are usable across any compatible tool rather than locked to one vendor.
Proprietary formats created real vendor lock-in risk, historically; open formats have become the practical default specifically to avoid that risk.
Format choice should match workload: columnar formats like Parquet and ORC suit analytical querying; row-based Avro suits ingestion and whole-record access.
AI training efficiency depends materially on format choice, and new formats are actively emerging for vector and embedding data.

Where This Fits in the Series

Article 8 covered zoning the yard. This article covered the containers themselves. Article 10 covers something neither zoning nor container design provides on its own: a reliable ledger tracking what’s actually in the yard at any given moment.