How to design a small RAG evaluation sprint

Define five evaluation dimensions

Retrieval-augmented generation (RAG) supplies a model with retrieved material to help it answer a question. Evaluate retrieval and the resulting answer separately.

• Retrieval recall: did the required source reach the candidate set?
• Groundedness: is every material claim supported by retrieved context?
• Citation accuracy: does the cited passage support the claim beside it?
• Completeness: did the answer preserve required exceptions and conditions?
• Abstention: does the system decline when the corpus cannot support an answer?

Create a small, versioned test set

Start with representative questions, expected source documents, required conditions, and a few intentionally unanswerable cases. Include difficult policy exceptions and stale duplicates; easy questions alone will make a weak system look reliable.

Freeze the set for the sprint. If the questions change while the retrieval configuration changes, the comparison is not interpretable.

Compare configurations, not impressions

Run the same questions against a baseline and one deliberate change when you need to attribute a result. The illustrative scorecard changes chunking, reranking, and retrieval depth together, so it cannot show which change caused a difference. Its percentages are example values, and completeness is left unscored; no evaluation run is reported.

Classify each failure

• For a missed exception, test hierarchy-aware chunking and parent headings.
• For a stale policy ranking above the current version, test effective-date metadata and freshness weighting.
• For unsupported synthesis, inspect claim-level citations and test grounding checks.
• For false confidence on an absent answer, test explicit abstention behavior against unanswerable cases.

Set a release gate by risk

Do not hide one dangerous category inside an average score. Set minimum thresholds by risk class, log the retrieved candidates and citations, and route low-confidence or high-impact questions to a person. Re-run the frozen evaluation set on every retrieval or prompt change.