Production RAG Observability

What you'll learn

• Containerizing a Python AI app
• Caching LLM responses to save costs and reduce latency
• Implementing distributed tracing for LLM calls
• Running automated, dataset-driven evaluations

Architecture

Dockerized FastAPI Service
  → Redis Cache (Check for exact or semantic matches)
  → LangSmith (Log trace of LLM call latency, token usage, cost)
  → Output to User
  
CI/CD Pipeline
  → Runs RAGAS evaluation on a golden dataset on every push

Steps

1. Take the RAG API you built in Phase 07 and write a Dockerfile for it.
2. Add a caching layer using Redis. If a user asks the exact same question, return the cached answer immediately without hitting the LLM API.
3. Integrate an observability tool like LangSmith, Phoenix (Arize), or Datadog. Ensure every API request logs the full trace of retrieval chunks and generation tokens.
4. Create a "golden dataset" of 20 questions and expected ground-truth answers.
5. Write an evaluation script using RAGAS or DeepEval to automatically score your RAG pipeline's context precision and answer faithfulness against the golden dataset.
6. (Optional) Set up a GitHub Action to run the evaluation script whenever you change your prompt or chunking strategy.