TL;DR: Evaluate RAG in two layers: did the retriever find the right evidence, and did the generator answer faithfully from that evidence? Add a labeled dataset, deterministic retrieval checks, human review, model-based grading, and production telemetry.
Why a Demo Is Not an Evaluation
A RAG demo can answer five hand-picked questions and still fail in production. It may retrieve a neighboring section instead of the governing policy, use an outdated document, cite a source that does not support the sentence, or confidently answer when the corpus contains no answer.
Quality therefore needs more than a thumbs-up button. You need to know whether the failure came from ingestion, chunking, indexing, retrieval, prompt assembly, generation, or the user interface.
Build a Representative Evaluation Set
Collect questions from real support tickets, search logs, subject-matter experts, and expected user workflows. Include easy lookups, multi-hop questions, ambiguous questions, permission-sensitive questions, and questions with no answer in the corpus.
Each example should record the question, expected answer or key facts, acceptable source documents, tenant or permission context, and an expected refusal condition when applicable. Keep a private test set that is not used while tuning prompts.
Measure Retrieval Separately
Retrieval recall asks whether the evidence needed to answer appears in the returned context. Precision asks how much of the returned context is useful. A large context can increase recall while reducing the model’s ability to focus.
Log the query, filters, retrieved chunk identifiers, scores, source version, and latency. Without this trace, the team cannot tell whether a bad answer was caused by the model or by missing evidence.
Test chunk size, overlap, metadata filters, query rewriting, hybrid keyword-plus-vector search, and reranking as separate experiments. Change one major variable at a time and preserve the previous configuration for comparison.
Measure the Generated Answer
Answer quality has several dimensions:
- Groundedness: are claims supported by retrieved evidence?
- Completeness: does the answer include the important expected facts?
- Citation correctness: does each citation actually support the nearby claim?
- Relevance: does the response answer the user’s question directly?
- Abstention: does it avoid inventing an answer when evidence is absent?
Use human labels for a seed set, then use automated graders to scale regression checks. Automated judges are useful assistants, not unquestionable truth; calibrate them against human decisions.
Test Permission Boundaries
Multi-tenant retrieval must be evaluated as a security property. A query from tenant A must never retrieve tenant B’s documents, even if the text is semantically similar. Test missing filters, malformed tenant identifiers, administrative roles, revoked access, and deleted documents.
Keep authorization outside the model. Retrieval filters should enforce access before context assembly, and the final response should not expose internal identifiers or restricted metadata.
Track Cost and Latency
A system that is accurate but takes twenty seconds or costs more than the business value will not survive. Track embedding cost, retrieval latency, reranking latency, model input tokens, output tokens, retries, and cache hit rate.
Create budgets for common and worst-case requests. Alert when a release increases p95 latency, context size, refusal rate, or cost per successful answer.
Prevent Regressions
Run a small deterministic suite on every prompt, chunking, embedding, or retrieval change. Run the larger human-reviewed suite before production. Store the configuration, model version, index version, and evaluation result together so a result can be reproduced.
Production Feedback Loop
Capture explicit feedback, reformulated queries, unanswered questions, citation clicks, and escalation events. Sample conversations for review, remove sensitive data, and turn recurring failures into new evaluation examples.
Practical Launch Checklist
- A labeled dataset includes answerable and unanswerable questions.
- Retrieval traces show source, version, score, filter, and latency.
- Tenant and document permissions are tested independently.
- Groundedness and citation quality are reviewed by humans.
- Cost and p95 latency have budgets and alerts.
- Prompt, index, and model changes run regression tests.
- The system can abstain clearly when evidence is missing.
A Reference Evaluation Table
| Layer | Question | Example signal | Typical fix |
|---|---|---|---|
| Ingestion | Did the source arrive completely? | missing pages or broken extraction | parser, OCR, or ingestion retry |
| Chunking | Does each chunk preserve useful meaning? | answers split across boundaries | section-aware splitting |
| Retrieval | Did the right evidence rank highly? | low recall at top-k | filters, hybrid search, reranking |
| Context | Did the prompt contain usable evidence? | irrelevant or duplicated chunks | deduplication and context budget |
| Generation | Did the answer stay within evidence? | unsupported claims | prompt, model, or refusal policy |
| Product | Can the user trust and act on it? | low citation clicks or high escalation | UX, source display, feedback loop |
This table prevents the common mistake of changing the model when the real problem is a stale index or an incorrect tenant filter.
Designing Human Review
Give reviewers a clear rubric and show the question, answer, retrieved chunks, citations, and document versions together. Ask reviewers to label each dimension independently instead of assigning one vague quality score. Two reviewers should label an overlap set so you can identify ambiguous instructions.
Keep review tasks small and rotate difficult cases through a subject-matter expert. Reviewers should be able to mark “not enough information” rather than being forced to choose between correct and incorrect. That label is essential for measuring safe abstention.
Experiment Without Losing the Baseline
Store every evaluation run with a configuration identifier. The identifier should include embedding model, chunking settings, retrieval filters, reranker, prompt version, generation model, and index build. When a score changes, you should be able to explain which variable changed.
Compare systems on the same examples and report confidence intervals or at least the number of cases behind a percentage. A jump from 80% to 82% may be noise when the dataset contains only fifty questions.
Improve Retrieval Before Prompt Tuning
If the correct source is absent from context, prompt changes cannot manufacture reliable evidence. Inspect failed traces in this order: source ingestion, permissions, chunk boundaries, metadata, query formulation, ranking, and only then generation. This order saves time and produces fixes that generalize.
Use synthetic questions carefully. They are useful for coverage, but real users ask incomplete, misspelled, ambiguous, and context-dependent questions that synthetic data often misses.
Launch Checklist
- A labeled dataset includes answerable and unanswerable questions.
- Retrieval traces show source, version, score, filter, and latency.
- Tenant and document permissions are tested independently.
- Groundedness and citation quality are reviewed by humans.
- Cost and p95 latency have budgets and alerts.
- Prompt, index, and model changes run regression tests.
- The system can abstain clearly when evidence is missing.
- Every release can be compared with a reproducible baseline.
If you need help taking an AI prototype into a measurable product, our backend development and API services can support ingestion, retrieval, evaluation, and production observability.