A Vocabulary for Describing What's Tasted

Opening Scene

Sommeliers share a genuinely specialized vocabulary — words like “tannic,” “oaky,” “mineral” — that lets them communicate precisely about qualities spanning taste, smell, and even texture, in a way anyone outside that shared vocabulary would struggle to fully grasp. A multimodal AI system needs an equivalent shared vocabulary too, except its version is mathematical: a shared embedding space where concepts from genuinely different modalities can be compared and related directly.

In Plain English

A shared multimodal embedding space represents concepts from text, images, audio, and video within the same underlying mathematical space, so that a picture of a dog and the word “dog” end up positioned close together, letting the model recognize their relatedness directly, numerically, rather than through some external translation step. This directly extends the embedding concepts covered in this content library’s vector embeddings and vector databases series, applied here across multiple modalities rather than within just one.

The Old Way

Before shared multimodal embedding spaces were well developed, comparing concepts across modalities required awkward workarounds:

Comparing a concept across modalities often required converting everything to text first, as a common intermediate format, losing modality-specific nuance in the process.
There wasn’t yet a well-established way to train a single embedding space that meaningfully represented text, images, and other modalities together.
Retrieval systems that needed to find, say, an image matching a text query often relied on separately trained, loosely connected systems for each modality.

Genuinely shared multimodal embedding spaces emerged specifically once training techniques allowed a single space to represent related concepts across modalities directly, without a lossy text intermediate step.

What’s Changing (and Why AI Is the Reason)

Multimodal embedding spaces increasingly represent text, images, and other modalities together directly, without needing a lossy text-conversion intermediate step.
This directly extends the embedding concepts covered in this content library’s vector embeddings and vector databases series, applied across modalities rather than within just text.
This connects directly to multimodal retrieval-augmented generation, where a text query can now retrieve relevant images directly, extending the retrieval techniques covered in this content library’s dedicated RAG series.

The Metaphor, Fully Extended

The Sommelier	Shared Embedding Space Concept
A specialized vocabulary spanning taste, smell, and texture	A shared mathematical space spanning text, image, audio, video
Words positioned to precisely capture related sensory qualities	Concepts positioned close together when meaningfully related
Communicating precisely without needing a separate translation	Comparing modalities directly without a lossy text-conversion step
A vocabulary that makes cross-sensory comparison genuinely possible	An embedding space that makes cross-modal comparison genuinely possible

For Beginners: What to Actually Do

Practice exploring how a multimodal system retrieves a relevant image directly from a text query, without an intermediate text-description step.
Learn the basic idea of a shared embedding space, connecting directly to this content library’s vector embeddings and vector databases series.
Get comfortable recognizing why positioning related concepts close together, numerically, is what enables direct cross-modal comparison.

For Practitioners and Leaders: The Deeper Layer

Evaluate whether a project’s cross-modal retrieval needs are well served by a genuinely shared embedding space, versus a more limited text-intermediate workaround.
Connect shared embedding space practice directly to this content library’s vector embeddings and vector databases series and dedicated RAG series.
Recognize multimodal retrieval as an increasingly practical capability for applications needing to search across mixed media directly.

Quick Recap

A shared multimodal embedding space represents concepts from different modalities within the same underlying mathematical space.
This lets a model compare and relate concepts across modalities directly, without a lossy text-conversion intermediate step.
This directly extends the embedding concepts covered in this content library’s vector embeddings and vector databases series.
It enables multimodal retrieval, extending the techniques covered in this content library’s dedicated RAG series.

Where This Fits in the Series

Article 10 covered the shared vocabulary enabling cross-modal comparison. Article 11 turns to how the palate — the model itself — actually gets trained to develop this shared understanding.