Choosing Your Kitchen Equipment

Opening Scene

Walk into any serious professional kitchen and you won’t find one machine that does everything. There’s a delivery intake area, a walk-in cooler, prep stations with their own tools, a range, an expo station running the tickets. Each piece of equipment does one job well, and the kitchen’s actual capability comes from how well they work together, not from any single machine being impressive on its own.

The modern data pipeline toolkit looks the same way, and a lot of confusion about “which tool should we use” comes from not first knowing which kitchen station a given tool is actually meant to be.

In Plain English

Pipeline tooling generally falls into a few distinct categories: ingestion tools that handle extraction from source systems, transformation tools that handle cleaning and reshaping, orchestrators that manage scheduling and dependencies (Article 8), and storage/warehouse platforms that hold the result. Most real pipelines combine tools from several of these categories rather than using one tool for everything — the same way a kitchen combines a delivery dock, a walk-in, a prep station, and a range, instead of expecting one machine to do all four jobs.

The Old Way

Traditionally, teams often built custom code for most of this — hand-written extraction scripts, hand-written transformation logic, a custom-built scheduler — because purpose-built tools for each category either didn’t exist yet or didn’t fit a team’s specific needs. This gave total control but meant every team was maintaining its own version of the same basic kitchen equipment, reinventing solutions to problems — retry logic, dependency management, connection handling — that had already been solved elsewhere many times over.

As the ecosystem matured, dedicated tools emerged for each category, and the harder problem shifted from “how do we build this” to “which combination of tools actually fits what we need, and how do we get them working together well.”

What’s Changing (and Why AI Is the Reason)

1. AI is showing up inside every category, not as a separate category of its own. Ingestion tools increasingly use AI to infer schemas (Article 2); transformation tools increasingly use it to suggest cleaning logic (Article 3); orchestrators increasingly use it to draft dependency graphs (Article 8). The tooling landscape isn’t gaining a new “AI station” — every existing station is getting an AI-assisted upgrade.
2. Choosing between tools is becoming less of a one-time, high-stakes decision. Because AI-assisted tools can help translate pipeline logic between platforms, migrating from one tool to another is becoming less catastrophically expensive than it used to be — lowering the cost of an imperfect initial choice, though not eliminating it.
3. Natural-language interfaces are lowering the barrier to entry across categories. Increasingly, someone can describe what they want a pipeline to do in plain language and get a reasonable starting configuration across ingestion, transformation, and orchestration tools alike — a theme this series returns to directly in a later article.

The Metaphor, Fully Extended

Kitchen Element	Pipeline Tooling Concept
The delivery dock	Ingestion / extraction tools
The prep station	Transformation tools
The expo running the ticket rail	Orchestration tools
The walk-in and pantry	Storage and warehouse platforms
A kitchen building all its own equipment from scratch	Custom, hand-written pipeline code for every stage
Buying purpose-built equipment for each station	Adopting dedicated tools for ingestion, transformation, and orchestration
Swapping in a new piece of equipment without rebuilding the whole kitchen	Migrating between tools with AI-assisted translation of pipeline logic

Quick Recap

• Pipeline tools generally fall into a few categories: ingestion, transformation, orchestration, and storage — most real setups combine tools from each rather than using one tool for everything.
• Historically, teams built most of this by hand before dedicated tools matured for each category.
• AI capability is spreading across every category rather than forming a separate “AI tool” category of its own.
• Migrating between tools is becoming less costly, thanks to AI-assisted translation of pipeline logic, which lowers the stakes of an initial choice.
• Evaluate tools on how well they handle the real failure modes covered throughout this series, not just on feature lists.

Where This Fits in the Series

Article 14 covered recovering from a missed window. This article zoomed out to the tools used to build all of this in the first place. Article 16 goes deeper into one specific, fast-moving capability inside that toolkit: describing a pipeline in plain language and having AI draft the first version.