LLM classifier for query routing

"Summarize this video" is not a retrieval query. If you stuff it through vector search, you get five random chunks that are semantically close to the word "summarize". Useless. The fix is a small router that looks at the question and decides: is this a global ask or a specific lookup?

chat.py
python
ROUTER_PROMPT = """You are a query classifier for a YouTube video Q&A assistant.

Classify the user's question as one of two types:

"global"  - The question requires understanding the ENTIRE video.
            Examples: summarize, overview, main topics, key takeaways,
            chapters, structure, what is this video about, full recap.

"rag"     - The question is about a SPECIFIC fact, moment, person, concept,
            or timestamp in the video. Examples: when did X happen,
            what did the speaker say about Y, explain concept Z.

Reply with ONLY a JSON object: {"route": "global"} or {"route": "rag"}
No explanation. No other text."""

The router prompt defines two crisp buckets with examples. Forcing JSON output keeps parsing trivial. "No explanation" reduces token waste and makes the router near-instant.

chat.py
python
from openai import OpenAI
import os

# Model id comes from the environment. The default is a fast,
# cheap OpenRouter model that handles routing and answers well.
MODEL = os.environ.get("LLM_MODEL", "google/gemini-2.5-flash-lite")

# The client is created once in app.py and passed into every
# function in this module:
#
#   client = OpenAI(
#       api_key=os.getenv("OPENAI_API_KEY") or os.getenv("OPENROUTER_API_KEY", ""),
#       base_url=os.getenv("OPENAI_API_BASE", "https://openrouter.ai/api/v1"),
#   )

The OpenAI SDK talks to any OpenAI-compatible endpoint, so pointing base_url at OpenRouter gives us cheap routing calls with the same client. LLM_MODEL, OPENAI_API_KEY, and OPENAI_API_BASE all live in .env, matching the .env.example you copied during setup.

chat.py
python
def classify_query(user_message: str, client: OpenAI) -> str:
    """Classify as global or rag. Falls back to rag on any error."""
    try:
        response = client.chat.completions.create(
            model=MODEL,
            messages=[
                {"role": "system", "content": ROUTER_PROMPT},
                {"role": "user", "content": user_message},
            ],
            temperature=0,
            max_tokens=20,
        )
        raw = response.choices[0].message.content.strip()
        parsed = json.loads(raw)
        route = parsed.get("route", "rag")
        return route if route in ("global", "rag") else "rag"
    except Exception:
        return "rag"

Temperature 0 for determinism, max_tokens 20 because the output is tiny. Defaulting to "rag" is a safe fallback: retrieval is cheap and still produces grounded answers for most queries.

Flashcards: Flashcards

Loading practice…

Quiz: Quiz

Loading practice…

Checkpoint: Retrieval layer checkpoint

Loading practice…