A retrieval system can score well and still produce an answer that feels useless. The correct passage may appear somewhere in the top results, yet the final response can ignore it, combine it with unsupported claims, or present it without enough context for the user to act. That is why I do not want to evaluate RAG as a single number.
I separate the pipeline into questions. Did retrieval find the evidence? Did ranking place the best evidence where the model could use it? Was the answer grounded in that evidence? Did it answer the user’s actual intent? Each question needs its own examples and failure labels. Otherwise, improving one layer can hide damage in another.
The test set should also resemble the product, not a benchmark assembled only because it is easy to score. For a one-user document assistant, I care about incomplete notes, similar filenames, follow-up questions, and queries whose answer does not exist in the collection. A good system must know when to answer, when to ask for clarification, and when to say that the evidence is missing.
I prefer an evaluation report that keeps examples close to the metrics. A score can show direction, but a failed query shows the mechanism. When I can read the question, retrieved chunks, reranker output, answer, citations, and judge explanation together, the next engineering decision becomes much easier.
The goal is not to make every metric green. It is to build a feedback loop that tells me which part of the system deserves the next hour of work.