Setup and repo tour

Welcome! I'm Param. In this workshop we are going to take a baseline RAG pipeline and turn it into something you can actually ship. Query rewriting, sub-graph decomposition, Presidio PII scrubbing, RAGAS scoring, and an offline eval harness. By the end, every answer your pipeline produces has a score attached to it.

The app is a FastAPI service that ingests documents, answers questions, and streams metric scores as the answer is produced. Under the hood, a LangGraph parent graph orchestrates anonymize, rewrite, retrieve, synthesize, and evaluate as separate nodes. The retriever itself is a compiled child graph the parent invokes once per rewritten query.

terminal
bash
git clone https://github.com/learnwithparam/advanced-rag-evaluation.git
cd advanced-rag-evaluation

# One command to set up and run
make dev

make dev creates a virtual environment with uv, installs dependencies, and starts the FastAPI server at http://localhost:8000/docs.

.env
bash
# Copy the example and add one API key
cp .env.example .env

# Pick any supported provider. Fireworks or Gemini have free tiers.
LLM_PROVIDER=fireworks
FIREWORKS_API_KEY=your_key_here

# Embeddings run locally via sentence-transformers, no key needed.
EMBEDDING_MODEL=all-MiniLM-L6-v2
CHROMA_PATH=./chroma_db
CHROMA_COLLECTION=advanced_rag

You only need one LLM key. Embeddings run on CPU locally. ChromaDB persists to disk under ./chroma_db so reingestion is not needed between restarts.

Files you will touch in this workshop

The workshop code is split so each concept lives in its own file. You will edit or extend every file on this list.

Validation checklist: Setup checklist

Loading practice…