An analyst asks a question about a credit agreement. The answer comes back in two seconds, cleanly written, with a citation to page 47. The analyst opens page 47. The passage is there, and it is about the right covenant. It just doesn’t say what the answer says.
That is the failure we worry about most in retrieval-augmented generation (RAG). Not the missing answer, not the slow answer, but the confident answer with a citation that looks like proof and isn’t. In legal, finance, and health work, that single gap carries most of the risk.
This article explains how we test for it in every RAG pipeline we build: why the usual retrieval metrics miss it, the five checks we run, and what you can set up on your own system this week.
Why Retrieval Metrics Miss the Problem
Most RAG evaluation starts with retrieval. Did the system find the right chunks? Recall and precision at k, mean reciprocal rank, hit rate. These are useful numbers, and we track them too.
But they measure pages, not claims. A retriever can return exactly the right passage, and the generator can still overstate it, merge it with another passage, change a number, or fill a gap with something plausible from its training data. The retrieval score stays perfect. The answer is wrong.
Citation faithfulness asks a narrower question: does every claim in the answer follow from the passage it cites? Faithfulness is closer to a fact-check than to a search metric, and it needs its own tests.
Five Faithfulness Checks
We split faithfulness into five checks. Each one catches a different failure, and each one can be automated at least in part.
1. The source exists
Does the cited passage exist, word for word, in the document the system says it came from? This sounds trivial. It isn’t. Models paraphrase quotes, merge two passages into one, or cite the right document with the wrong page. A string match against the stored chunk, with some tolerance for whitespace and OCR noise, catches most of these.
2. The claim is supported
Does the passage support the claim, or does it only sit near the topic? This is the hard one. A separate model acts as a judge and labels each claim as supported, contradicted, or not addressed by its passage, and people review a sample of those verdicts. A passage that is “about the covenant” but says nothing about the threshold is not support.
3. The citation matters
Would the answer change if the citation were removed? Run the question again with the cited passage taken out of the context. If the answer stays the same, the citation was decoration: the model answered from somewhere else and attached a source afterwards.
4. Figures are exact
Are numbers, dates, names, and units copied exactly? Extract every figure from the answer and look for it in the cited passage. In finance and health, a transposed digit or a wrong unit is the most expensive kind of error, and the easiest one to check mechanically.
5. Silence is allowed
Does the system say “not found” when the sources are silent? Keep a set of questions whose answers are deliberately absent from the documents. A faithful system declines. A system that always answers will, sooner or later, fill the silence with something plausible.
How We Run the Checks in a RAG Pipeline
Checks only help if they run every time the system changes. In practice that means three habits.
An evaluation set per use case. A few hundred questions written with the people who will use the system, each tagged with the document and passage that answers it, plus the deliberate “not found” questions. We write it before we build the pipeline, because it is the clearest specification anyone will produce.
Automatic checks on every release. Checks one, three, four, and five are mostly mechanical. Check two uses a judge model, so a person reviews a sample of its verdicts, because a judge can be wrong in the same ways as the system it judges.
A trace for every request. Log the question, the retrieved chunks, the prompt, the answer, and the citations, so any answer can be replayed and explained later. When a user reports a bad answer, the trace shows whether retrieval, generation, or the source document was at fault.
The figures check is a good first automation, because it is short and it catches expensive errors:
import re
FIGURE = re.compile(r"\d+(?:[.,]\d+)*") # 3,25 / 1.000 / 2027
def missing_figures(answer: str, cited_passage: str) -> list[str]:
"""Figures in the answer that do not appear in the cited passage."""
in_passage = set(FIGURE.findall(cited_passage))
return sorted(set(FIGURE.findall(answer)) - in_passage)
print(missing_figures(
"Net leverage must stay below 3,5x, tested quarterly.",
"Net leverage shall not exceed 3,25x on each quarter date.",
)) # ['3,5']: the passage says 3,25
It will not catch everything. It will catch the answer that quietly rounds a covenant threshold, which is the one that ends up in a credit committee.
Outside Eyes on Our Own Work
Testing your own system has a blind spot: you test for the failures you already imagine. So we asked for outside eyes.
Sparky* is a partner of the University of Edinburgh CDT in Designing Responsible NLP. A team of PhD researchers picked our project on citation faithfulness in high-stakes RAG for their Responsible NLP audit, supervised on our side by Nika Jerković, our Data Science Lead. They have four weeks and our evaluation setup, and a clear brief: find what breaks.
We will publish what they find, including the parts we would rather not read.
What to Do on Monday
If you run a RAG system in production, three steps will tell you more about its faithfulness than any leaderboard:
- Write 50 questions with known answers and ten with no answer in your documents. Run them through the system and read every answer against its citation.
- Add the missing-figures check. It takes an afternoon and catches the most expensive errors.
- Log a trace for every request, so the next bad answer can be explained instead of argued about.
Retrieval gets you the right pages. Faithfulness decides whether anyone can trust the answer.
