Embeddings: Turning Meaning Into Geometry
A practical mental model for understanding embeddings, how they work, and why they matter for AI engineers
This is Part 1 of a 4-part series on embeddings. Part 2: How Embeddings Actually Work in Practice | Part 3: Retrieval Is Not Top-K | Part 4: From Retrieved Context to a Grounded Answer
A practical mental model for beginners and AI engineers
A small story to start
You have 100,000 documents.
A user types a simple question:
How do I cancel my policy?
You run a keyword search.
Nothing useful comes back.
You try again:
- terminate policy
- end coverage
- close insurance
Some results appear. Many are irrelevant. Some are outdated. Some obvious matches are missing.
The problem is not search.
The problem is that computers do not know when two things mean the same thing.
This is the problem embeddings were created to solve.
The core problem embeddings exist for
Computers work well with:
- Numbers
- Exact strings
- Fixed rules
Humans communicate with:
- Meaning
- Context
- Approximation
When a human sees these two sentences:
- How do I cancel my insurance?
- How can I end my policy?
They immediately know they are asking the same thing.
To a computer, these are just different strings.
So instead of asking:
Are these two texts equal?
We ask a different question:
Can we represent meaning in a way that makes similar things comparable?
Embeddings are the answer to that question.
What an embedding actually is
At its simplest:
An embedding is a list of numbers that represents an object.
That object can be:
- A sentence
- A paragraph
- A document
- An image
- A user
- A product
The key property is this:
If two embeddings are close together, the things they represent are likely similar in meaning.
A simple example
| Text | Expected relationship |
|---|---|
| How do I cancel my insurance? | Close |
| How can I end my policy? | Close |
| What is the capital of France? | Far |
That is all an embedding promises.
Not understanding. Not correctness. Just proximity.
Why vectors and not keywords or labels
Keywords break when wording changes.
Labels require humans and do not scale.
Rules become brittle and complex.
Vectors give you something much simpler and more powerful: distance.
With vectors, you can:
- Measure similarity
- Find nearest neighbors
- Form clusters
- Compare many items efficiently
A useful analogy is a map.
Two cities being close on a map does not mean they are the same city. It means traveling between them is easy.
Embeddings are maps of meaning, not definitions.
The famous king–queen analogy (explained properly)
You may have seen this example before:
king − man + woman ≈ queen
This is not magic. It is geometry.
What is actually happening
Embedding models learn relationships, not facts.
They notice patterns like:
- king and queen are related
- man and woman are related
- gender differences form a direction in vector space
So when you subtract one relationship and add another, you often land near a related concept.
Visual intuition
| Concept | Encoded idea |
|---|---|
| king | royalty + male |
| queen | royalty + female |
| man | male |
| woman | female |
When you do:
- king − man → mostly royalty
- royalty + woman → queen
Important clarification:
This works because the model learned consistent patterns across large data. It is not reasoning. It is alignment.
And it does not always work.
How embeddings are learned (without math)
Embedding models are trained on massive datasets where relationships exist.
They see:
- Things that appear together
- Things that appear in similar contexts
- Things that rarely appear together
The training goal is simple:
Put related things closer together. Push unrelated things farther apart.
One critical idea to understand:
Embeddings are not trained to be correct. They are trained to be useful for certain tasks.
This explains:
- Why different models behave differently
- Why embeddings shift across domains
- Why there is no single "true" embedding for a sentence
What embeddings unlock in practice
Once meaning becomes geometry, many problems become easier.
| Problem | What embeddings enable |
|---|---|
| Search | Meaning-based retrieval |
| Recommendations | Users and items in the same space |
| Clustering | Discover structure without labels |
| RAG | Select relevant context for LLMs |
| Deduplication | Detect near-duplicates |
| Personalization | Represent users implicitly |
A good summary:
Embeddings turn unstructured problems into geometric ones.
What embeddings are not
This is where most misunderstandings begin.
Embeddings are not:
- Memory
- A database
- A source of truth
- Interpretable
- Stable forever
A practical failure example
Two documents are almost identical:
- One is an outdated policy
- One is the current policy
Their embeddings will be very close.
The embedding does not know which one is correct.
Closeness means similar, not correct.
This distinction matters later when systems fail in production.
Why embeddings still matter with large language models
Large language models can reason and generate text.
They cannot cheaply scan your entire private corpus and decide what matters.
You still need a way to select information.
Embeddings do that selection.
A useful mental model:
Embeddings decide what the model sees. The language model decides what the model says.
Or:
Embeddings are the librarian. The LLM is the author.
Without the librarian, the author is guessing.
The mental model to keep
If you remember one thing, remember this:
Embeddings are lossy compression for meaning, designed for retrieval and routing.
They trade:
- Precision for scale
- Truth for usefulness
- Accuracy for speed
They are infrastructure, not intelligence.
Once you see embeddings this way, their strengths and limitations become clear.
What comes next
If embeddings are this simple, an obvious question follows:
Why do they fail so often in real systems?
The answer has very little to do with models, and a lot to do with how we use them.
Most failures come from configuration, not the embedding model itself. Part 2 covers the practical decisions that determine whether retrieval works: chunking strategies, dimension selection, and how meaning gets broken before it ever reaches the embedding model.
Next: Part 2: How Embeddings Actually Work in Practice → Part 3: Retrieval Is Not Top-K → Part 4: From Retrieved Context to a Grounded Answer
References and further reading
- Mikolov et al., Efficient Estimation of Word Representations in Vector Space
- Google AI Blog, Understanding Embeddings
- OpenAI documentation on embeddings
- Sentence Transformers documentation
- Karpukhin et al., Dense Passage Retrieval for Open-Domain Question Answering