The Librarian's Judgment Call

Opening Scene

A card catalog search might return twenty books that all technically mention a patron’s topic, ranked purely by how closely their catalog entries match the search terms. A good librarian, standing right there, often reorders that raw list based on real judgment — this one’s outdated, this one’s too narrow, this one’s exactly the right depth and focus for what the patron actually seems to need. That judgment layer, applied on top of raw similarity, is exactly what relevance ranking adds to a basic retrieval result.

In Plain English

Similarity search, covered in Articles 5 through 7, ranks retrieved chunks purely by embedding distance — how mathematically close a chunk’s meaning is to the query’s meaning. This is a genuinely useful starting point, but raw similarity doesn’t always perfectly capture true relevance: a chunk can be semantically similar to a query while still not actually containing the specific answer needed, or a highly relevant chunk might rank slightly lower due to how it happened to be phrased. Additional relevance signals — recency, source authority, structural position within a document — can meaningfully improve on raw similarity alone.

The Old Way

Before additional relevance signals were commonly incorporated, early RAG systems often relied purely on raw similarity scores:

• Early retrieval systems typically ranked results purely by embedding similarity, without incorporating any additional relevance signal beyond that single mathematical measure.
• This sometimes surfaced technically similar but practically less useful content ahead of more genuinely helpful material, a gap purely mathematical similarity couldn’t fully close on its own.
• Recency, source quality, and other practical relevance signals weren’t yet systematically incorporated into most early retrieval ranking approaches.

Incorporating additional relevance signals beyond raw similarity emerged as practitioners recognized the real, practical limits of similarity alone.

What’s Changing (and Why AI Is the Reason)

1. Modern RAG systems increasingly combine similarity scores with additional relevance signals — recency, source authority, metadata filters — producing a more genuinely useful ranking than similarity alone.
2. This connects directly to reranking, covered in Article 11, where a more sophisticated second-stage model applies deeper, more nuanced judgment beyond the initial similarity-based retrieval.
3. As RAG systems have matured, the recognition that similarity is a necessary but not sufficient signal for true relevance has become standard, well-understood practice, rather than a subtle, easily overlooked distinction.

The Metaphor, Fully Extended

The Library	Relevance Ranking Concept
A catalog search returning books ranked by raw keyword or subject match	A retrieval system ranking chunks by raw embedding similarity
A librarian reordering that list based on real, practical judgment	A ranking system incorporating additional relevance signals beyond similarity
Recognizing an outdated book despite its strong topical match	Recognizing a stale or low-quality chunk despite strong semantic similarity
A librarian’s judgment adding real value on top of the catalog alone	Additional relevance signals adding real value on top of similarity alone

Quick Recap

• Raw similarity search ranks results by embedding distance alone, a useful but not always sufficient relevance signal.
• Additional signals like recency and source authority can meaningfully improve on similarity alone.
• Modern RAG systems increasingly combine multiple relevance signals rather than relying on similarity in isolation.
• Recognizing this gap is standard, well-understood practice in mature RAG system design.

Where This Fits in the Series

Article 9 covered the judgment layer beyond raw similarity. Article 10 covers what happens when the whole retrieval process points to the wrong shelf entirely.