RAG Evaluation Harness

What you'll learn

• Why manual eval is necessary but insufficient
• Automated retrieval metrics (precision, recall, MRR)
• Answer faithfulness scoring
• Building a regression test suite for prompt changes

Architecture

Eval dataset (queries + expected answers + expected docs)
  → RAG pipeline (configurable: chunk size, search type, reranking)
  → Retrieval metrics (precision@k, recall@k, MRR)
  → Generation metrics (faithfulness, answer relevancy via RAGAS)
  → Report (JSON + human-readable summary)

Steps

1. Create an eval dataset: 30+ queries with ground-truth answers and expected source documents
2. Run each query through your RAG pipeline, log: retrieved chunks, LLM answer, latency, token count
3. Compute retrieval metrics: precision@5, recall@5, MRR
4. Use RAGAS to score faithfulness (does the answer follow from the context?) and relevancy
5. Run the harness after every prompt or retrieval change — treat it like a test suite
6. Track results over time in a JSON file or simple dashboard