Something has shifted in how companies write their AI project requirements. A year ago the ask was "build us a chatbot over our documents." Now, increasingly, the ask reads like an acceptance spec: measure factual accuracy and retrieval quality, return a source reference for every claim, add guardrails against invented facts and standards, keep a human in the loop, and give us an audit trail.
That is not only a request to build RAG. It is a request to trust RAG. For these projects, getting a retrieval pipeline to answer fluently is no longer enough. The harder question is whether the fluent answer is supported, and whether the team can prove it.
RAG systems can fail silently in production even when their demos look convincing. These failures are difficult to notice precisely because the output still looks confident and well-written.
Failure modes hidden by fluent output
These are ways a deployed RAG system can give a wrong answer without flagging it:
- Answers from memory, not from the context. The retriever returns the right document, and the model ignores it and answers from training-data memory anyway. Sometimes it's right by luck. Sometimes it isn't.
- Cites the wrong span. There's a citation, so it looks grounded. But the cited passage doesn't actually support the claim. A reviewer who trusts the citation is trusting nothing.
- Flattens and loses cross-source context. The answer needed to connect a decision in one document to its owner in another. The model collapsed them, or dropped the link, and produced a plausible but incomplete answer.
- Invents facts, standards, or numbers. The model fills a gap with a confident fabrication: a regulation that doesn't exist, a dosage that's wrong, a figure that was never in the evidence.
- Confidently wrong on weak retrieval. When the retrieved context only partially supports the answer, the model answers with the same confidence as when it fully supports it. The user has no signal to know which is which.
Each can produce a fluent, plausible answer without throwing an error. A small demo with friendly questions can easily miss them. Without a pinned evaluation and failure log, a team cannot say how often they occur in its own deployment.
Why "it looks good" stops being an answer
The moment a RAG system sits in front of a regulated decision (a clinician, a lawyer, an asset-management engineer, an auditor), someone asks the only question that matters: how do you know it's right? And "we tested it and it looked good" is not an answer a security or quality review accepts. They want a number, a method, and the failing cases.
Many RAG failures are not model failures. They can originate in retrieval, chunking, ranking, or prompting. An honest evaluation tells you which, so you fix the lower-cost application or retrieval problem first and only reach for model adaptation when the model genuinely ignores good evidence. You cannot make that call without measuring it.
What I'm building
I'm building a reproducible way to measure exactly this: whether a RAG or agent system answers from its retrieved evidence. It will score groundedness, citation accuracy, and unsupported-claim rate, return the actual queries where the system failed, and separate the failures that belong in the application layer from the ones that belong in the model. The output will be a reproducible evidence pack, the kind of artifact a review can actually consume.
I did the same thing for tool-output prompt injection on a private GPT-OSS-20B: pinned probes, real numbers, published hashes, honest limits. The next post in this series will report RAG groundedness results on a public benchmark you can reproduce.