Organizing the Stacks

Opening Scene

A librarian can’t usefully hand a patron an entire multi-volume encyclopedia set in response to a narrow, specific question, and can’t usefully hand over a single, disconnected sentence torn from the middle of a chapter either. Somewhere between “the whole collection” and “one isolated fragment” is the actual unit a librarian retrieves and hands over — a chapter, an article, a well-defined section, sized to be genuinely useful on its own. RAG systems face this exact same sizing problem, and it’s usually the very first decision that shapes everything else.

In Plain English

Chunking breaks source documents into smaller, well-sized pieces before they’re indexed for retrieval — since retrieving and reasoning over an entire large document at once is both impractical and imprecise. Chunk size involves a genuine tradeoff: chunks too small lose important surrounding context; chunks too large dilute relevance and make it harder for retrieval to pinpoint exactly the right information, and can waste context window space, covered in this content library’s LLM fundamentals series, on irrelevant surrounding material.

The Old Way

Before chunking strategy was well understood as its own genuine design decision, document preparation for retrieval was often handled more crudely:

• Early retrieval systems sometimes indexed entire documents as single units, making it impossible to retrieve just the specific, relevant portion of a long document.
• Naive fixed-length chunking — splitting purely by character or word count — ignored a document’s actual logical structure, sometimes splitting a sentence or idea awkwardly in half.
• Chunk size was often chosen arbitrarily, without systematic testing of how different sizes actually affected retrieval quality.

Recognizing chunking as a genuine, consequential design decision — not an incidental preprocessing detail — represents real, accumulated practical RAG experience.

What’s Changing (and Why AI Is the Reason)

1. More sophisticated chunking strategies — respecting document structure like paragraphs and sections, or using semantic chunking that groups genuinely related content together — have replaced naive, fixed-length splitting.
2. Overlapping chunks, where consecutive chunks share some content at their boundaries, have become common practice specifically to avoid losing context at an awkward split point.
3. Growing recognition that chunk size and strategy meaningfully affect downstream retrieval quality has made chunking a genuine, deliberate experimentation target, connecting directly to the RAG evaluation practices covered in Article 18.

The Metaphor, Fully Extended

The Library	Chunking Concept
Handing over an entire encyclopedia set for a narrow question	Retrieving an entire document when only a specific section is relevant
Handing over one isolated, disconnected sentence	Retrieving a chunk too small to carry meaningful context
A well-defined chapter or section, sized to be useful on its own	A well-sized chunk, carrying enough context to be independently useful
A librarian who’s learned exactly how finely to divide the collection	A practitioner who’s learned exactly how to size chunks for their specific content

Quick Recap

• Chunking breaks source documents into smaller, well-sized pieces before indexing, since retrieving whole documents is impractical and imprecise.
• Chunk size involves a genuine tradeoff between losing context (too small) and diluting relevance (too large).
• More sophisticated strategies now respect document structure and use overlapping chunks to preserve context at boundaries.
• Chunking is a genuine, deliberate design decision worth systematic testing, not an incidental preprocessing detail.

Where This Fits in the Series

Article 4 covered dividing documents into retrievable units. Article 5 covers how those units actually get organized so they can be found by meaning, not just by exact wording.