End to end architecture

Now that you know what grounded video QA looks like, let us walk through the pipeline that produces it. Every stage has a specific job, and every stage has to preserve the timestamp information or the citation breaks.

The video QA pipeline

Five stages, from YouTube URL to cited answer.

The first half is offline indexing. Parse the URL, fetch the transcript, chunk it into time windows, and build two indexes: a FAISS vector index for meaning, and a BM25 keyword index for exact terms.

The second half is online retrieval. A user types a question. A small LLM classifies it as a global question or a specific lookup. Global questions get the full transcript. Specific lookups get the top fused results from hybrid retrieval. The main LLM answers with inline timestamp links.

repo layout
bash
video-rag-faiss/
  app.py                # Streamlit UI with two panel layout
  transcript.py         # URL parsing, youtube-transcript-api, chunking
  metadata.py           # oEmbed title, thumbnail, channel
  embedder.py           # sentence-transformers + FAISS
  keyword_index.py      # BM25 via rank-bm25
  retrieval_fusion.py   # Reciprocal rank fusion
  chat.py               # Router + hybrid retrieval + LLM call
  test_retrieval.py     # Offline tests for retrieval quality
  style.css             # UI polish
  Makefile              # make run, make dev, make test
  .env.example          # LLM_MODEL, OPENAI_API_KEY, OPENAI_API_BASE

Each file has one clear job. We will visit every module in order, so by the end you will have touched the whole system.

terminal
bash
# Clone the workshop repo and set up
git clone https://github.com/learnwithparam/video-rag-faiss.git
cd video-rag-faiss

# Copy env template and add your OpenRouter or OpenAI key
cp .env.example .env

# One command to install deps and run the Streamlit app
make dev

Clone the workshop now so you can follow along. OpenRouter has a free tier that works with the default model.

Matching exercise: Match each stage to its job

Loading practice…

Checkpoint: Foundations checkpoint

Loading practice…