When RAG Gets It Wrong
Part 7 of the RAG series, and the one most vendors skip. A system you cannot trust is worse than no system, so let us be specific about the failures.
RAG, the technology behind every example in this series, is honest machinery. Given good documents and a clear question, it finds the right pages and answers from them. But it has real ways of getting things wrong, and you should know every one of them before you let it near documents that matter.
The documents are the problem
The most common failure is not the AI. It is the pile. Scanned pages a machine reads badly. The old version of a procedure sitting next to the new one, with no mark saying which is live. Two files that contradict each other, both written by your own company, both presented to the system as truth.
RAG reads what it is given, faithfully. If the pile contains a lie, a draft or a fossil, the answer can too. This is why we tell clients the first project is never the AI. It is getting one clean, current set of documents. Unfashionable work, and the difference between a tool and a toy.
The question is the problem
Vague questions get vague retrievals. “What does the contract say about ending it” might find the termination clause, or a paragraph about ending a trial period, depending on which wording sits closer to the question. The fix is partly on the system, which should show its sources so a wrong turn is visible, and partly a habit worth learning: ask the way you would ask a careful colleague, with the specifics in the sentence.
The cutting is the problem
Remember the chunks from Part 2. Cut a clause in the wrong place and its condition lands in one chunk while its conclusion lands in another, and the answer loses the “only if” that made it true. Good implementations cut along the document's own structure, headings, clauses, sections, and this is one of the places where careful engineering quietly matters more than a bigger model.
The model fills a gap
And yes, the famous one. If the passages found do not actually contain the answer, a model may still try to be helpful and produce something fluent and wrong. A well built system is instructed, plainly, to say “I could not find that in these documents” and stop. That sentence is the most valuable sentence in the whole product. Any demo you watch, ours included, should be tested with a question the documents cannot answer, just to see whether it admits it.
How you work with a system like this
Ask questions whose answers you already know, first, and check the sources. Treat the system as a very fast, very literal assistant: excellent at finding and summarizing, not a substitute for the person accountable for the decision. Keep the pile clean. Retire old versions like you mean it.
Trust in a tool does not come from it being right in a presentation. It comes from knowing exactly how it fails, and seeing that it was built by people who know it too. Next, the question owners ask us more than any other: is it safe to point this at real company documents at all?
Ten questions about how work actually moves through your operation. No call, no slide deck.
If you want a second opinion on a specific decision, that conversation is free.
More from The Bridge
Don't Take My Word for It: Ask Your Own Documents
The series ends with a door, not a pitch: a working demo I built, with sample contracts, procurement and claims sets, so you can ask real questions and check the sources yourself.
Is RAG Safe for Company Documents?
Do your documents train the AI? Where do they live, and who inside your own company can find what? The safety questions worth asking any vendor, answered plainly.
Example 3 of 3: RAG in Insurance Claims: The Answer Is in the File, Somewhere
A claim file already holds every answer, buried in growing paperwork. RAG joins the documents together and surfaces what is stalled, missing or wrong. Example 3 of 3.