LLM classifier for query routing
"Summarize this video" is not a retrieval query. If you stuff it through vector search, you get five random chunks that are semantically close to the word "summarize". Useless. The fix is a small router that looks at the question and decides: is this a global ask or a specific lookup?
ROUTER_PROMPT = """You are a query classifier for a YouTube video Q&A assistant.
Classify the user's question as one of two types:
"global" - The question requires understanding the ENTIRE video.
Examples: summarize, overview, main topics, key takeaways,
chapters, structure, what is this video about, full recap.
"rag" - The question is about a SPECIFIC fact, moment, person, concept,
or timestamp in the video. Examples: when did X happen,
what did the speaker say about Y, explain concept Z.
Reply with ONLY a JSON object: {"route": "global"} or {"route": "rag"}
No explanation. No other text."""The router prompt defines two crisp buckets with examples. Forcing JSON output keeps parsing trivial. "No explanation" reduces token waste and makes the router near-instant.
from openai import OpenAI
import os
# Model id comes from the environment. The default is a fast,
# cheap OpenRouter model that handles routing and answers well.
MODEL = os.environ.get("LLM_MODEL", "google/gemini-2.5-flash-lite")
# The client is created once in app.py and passed into every
# function in this module:
#
# client = OpenAI(
# api_key=os.getenv("OPENAI_API_KEY") or os.getenv("OPENROUTER_API_KEY", ""),
# base_url=os.getenv("OPENAI_API_BASE", "https://openrouter.ai/api/v1"),
# )The OpenAI SDK talks to any OpenAI-compatible endpoint, so pointing base_url at OpenRouter gives us cheap routing calls with the same client. LLM_MODEL, OPENAI_API_KEY, and OPENAI_API_BASE all live in .env, matching the .env.example you copied during setup.
def classify_query(user_message: str, client: OpenAI) -> str:
"""Classify as global or rag. Falls back to rag on any error."""
try:
response = client.chat.completions.create(
model=MODEL,
messages=[
{"role": "system", "content": ROUTER_PROMPT},
{"role": "user", "content": user_message},
],
temperature=0,
max_tokens=20,
)
raw = response.choices[0].message.content.strip()
parsed = json.loads(raw)
route = parsed.get("route", "rag")
return route if route in ("global", "rag") else "rag"
except Exception:
return "rag"Temperature 0 for determinism, max_tokens 20 because the output is tiny. Defaulting to "rag" is a safe fallback: retrieval is cheap and still produces grounded answers for most queries.
Flashcards: Flashcards
Loading practice…
Quiz: Quiz
Loading practice…
Checkpoint: Retrieval layer checkpoint
Loading practice…