CN
Aman|
HomeExperienceProjectsBlogsAboutContactTools
Connect
X (Twitter)
LinkedIn
Medium
Github
Instagram
← All Articles

Part 4: From Retrieved Context to a Grounded Answer

Why good retrieval still produces bad answers, and where most systems actually fail

2026-01-04•3 min read
Part 4: From Retrieved Context to a Grounded Answer

This is Part 4 of a 4-part series on embeddings. Part 1: Embeddings: Turning Meaning Into Geometry | Part 2: How Embeddings Actually Work in Practice | Part 3: Retrieval Is Not Top-K

By now, the system should be in good shape.

  • Embeddings represent meaning reasonably well
  • Chunking preserves ideas instead of breaking them
  • Dimensions are chosen intentionally
  • Retrieval is not naïve top-k
  • Redundancy is controlled
  • Relevant context is available

And yet, answers still go wrong.

They sound confident but are incomplete. They ignore relevant chunks. They hallucinate steps that are not present.

At this point, the failure is no longer about embeddings or retrieval.

It is about how generation uses context.

The final boundary that matters

There is a boundary most systems underestimate:

retrieval → generation

Retrieval decides what information is available. Generation decides what information is used.

Those are not the same thing.

You can retrieve the perfect chunks and still get a bad answer.

Why models ignore retrieved context

This is not mysterious once you see the dynamics.

Large language models:

  • are trained to be fluent
  • prefer internal priors when unsure
  • do not inherently "respect" retrieved text

If the prompt allows it, the model will:

  • generalize
  • fill gaps
  • smooth over missing steps

Even when the correct information is right there.

This is not a hallucination bug. It is a control problem.

Common failure patterns at this stage

1. Partial grounding

The answer uses one retrieved chunk and ignores others.

This often happens when:

  • chunks are redundant
  • one chunk is phrased more confidently
  • the model latches onto the first strong signal

Result:

  • technically plausible
  • incomplete or wrong

2. Conflicting evidence collapse

Multiple chunks disagree slightly:

  • old vs new policy
  • exception vs general rule

The model merges them into a single answer.

This sounds reasonable but is often incorrect.

LLMs are very good at smoothing contradictions. They are not good at flagging them unless forced.

3. Prior override

The model answers from its training data instead of the provided context.

This happens when:

  • the prompt does not strongly constrain sourcing
  • retrieved context is long or noisy
  • the model "knows" a generic version of the answer

This is why answers look confident even when unsupported.

The core mistake: treating context as background

Most prompts implicitly say:

Here is some context. Now answer the question.

To the model, this reads as:

  • context is optional
  • fluency is primary

If you want grounding, context must be treated as evidence, not background.

Evidence vs synthesis are different modes

This distinction matters more than most systems acknowledge.

Evidence-driven answers

Examples:

  • policy rules
  • steps in a process
  • eligibility conditions
  • factual definitions

These require:

  • quoting
  • attribution
  • refusal if missing

Synthesis-driven answers

Examples:

  • summaries
  • comparisons
  • explanations
  • tradeoffs

These allow:

  • abstraction
  • combination
  • reasoning

Many failures occur when systems use synthesis behavior for evidence tasks.

If the task is evidence-based, the model must be constrained accordingly.

What actually improves grounding (not prompt magic)

The following patterns work consistently because they change incentives, not wording.

1. Explicit sourcing expectations

When the model is told:

  • answers must be supported by provided context
  • unsupported claims are errors

It behaves differently.

This shifts the objective from "sound right" to "be justified".

2. Structured answer requirements

For example:

  • answer → then cite
  • step → supporting text
  • claim → source chunk

This forces alignment between output and input.

The model now has to reconcile its answer with evidence.

3. Separation of reasoning and quoting

Let the model reason internally, but require:

  • explicit quotes
  • or explicit references
  • or clear acknowledgment when information is missing

This reduces confident invention.

4. Explicit "not found" behavior

If the answer is not in the retrieved context, the model must say so.

This feels restrictive but dramatically increases trust.

Users tolerate uncertainty better than confident wrongness.

Why redundancy hurts generation

Even after retrieval improvements, redundancy can sabotage grounding.

If five chunks say almost the same thing:

  • the model sees signal saturation
  • nuance disappears
  • edge conditions are ignored

This is why retrieval diversity matters all the way through to generation.

Clean context beats more context.

Evaluation must change at this layer

At this stage, retrieval metrics are no longer sufficient.

You need to evaluate:

  • did the answer cite correct evidence?
  • did it invent steps?
  • did it merge conflicting rules?
  • did it acknowledge uncertainty?

This evaluation is task-specific.

There is no universal score.

But without this layer of evaluation, systems regress silently.

The full mental model (the series in one view)

This is the clean way to think about the stack:

  • Embeddings decide what is close
  • Chunking and dimensions decide what survives compression
  • Retrieval dynamics decide what is considered
  • Generation control decides what is said

Failures almost always happen at boundaries between layers, not inside a single component.

Once you see this, debugging becomes tractable.

Final takeaway

Embeddings are not intelligence. Retrieval is not understanding. Generation is not truth.

But when these layers are designed intentionally and constrained correctly, they produce systems that are reliable, explainable, and useful.

That is the real goal.

And that is where this series ends.

Previous: Part 1: Embeddings: Turning Meaning Into Geometry | Part 2: How Embeddings Actually Work in Practice | Part 3: Retrieval Is Not Top-K

References and further reading