Why Your AI Chatbot Gives Wrong Answers, Even With RAG

"We added RAG, so hallucinations are solved." Not quite. RAG reduces wrong answers, but it also introduces new ways to be wrong.

The Usual Culprits

• Bad chunking: Split a policy in the middle of a sentence and the exception ends up in a different chunk than the rule. The model sees half the truth.
• Retrieval misses: The right document exists, but the search never surfaced it. The user says "cancel," the doc says "terminate."
• Stale data: The vector store still holds last year's pricing because nobody re-indexed.
• Too much context: Stuffing 20 chunks into the prompt buries the relevant one. Models often pay less attention to the middle of long contexts.
• No reranking: Vector similarity gets you candidates, not the best answer. A reranker fixes the ordering.
• No permission to say "I don't know": Unless you tell the model to answer only from the context, it fills gaps from its training data.
• No evaluation: Most teams "test" by trying five questions and feeling good.

How to Debug RAG Failures

1. Log the retrieved chunks for every answer.
2. Check whether the right chunk was retrieved. If not, it's a retrieval problem.
3. If the chunk was retrieved but the answer is still wrong, it's a prompt or model problem.

Most teams jump straight to changing the prompt when the real fix is upstream in chunking, metadata, or hybrid search (keyword plus vector).

Build a small test set of 50 real questions with known answers and run it after every change. That single habit separates demos from products.

What's the most confident wrong answer your chatbot has given?