Inline timestamp citations and multi video switching

The final trick is getting the LLM to cite timestamps inline, not dump them at the end. We instruct the model to write markdown links of the form [MM:SS](youtube-url-with-t-seconds). Streamlit renders markdown for free, so clicking the timestamp opens the video at the exact moment.

chat.py
python
SPECIFIC_SYSTEM_PROMPT = """You are an intelligent video assistant. You have been given relevant excerpts from a YouTube video transcript.

Instructions:
- Answer the question based ONLY on the provided transcript excerpts.
- Cite 1 to 3 timestamps INLINE in your answer using this EXACT markdown format:
  [MM:SS](https://www.youtube.com/watch?v={video_id}&t=Xs)
  where X is the timestamp in seconds. Always include the s suffix. Example: &t=315s not &t=315
- Only cite a timestamp when it directly supports the specific sentence you are writing.
- Do NOT list all timestamps at the end, weave them naturally into the answer.
- If the context does not contain the answer, say: \"I could not find information about that in this video.\"
- Never make up information not present in the provided context.
"""

Explicit format rules keep the model honest. The s suffix is the one detail that breaks silently if you skip it, so we spell it out twice with a positive and negative example.

transcript.py
python
def format_timestamp(seconds: float) -> str:
    """Convert seconds to MM:SS or HH:MM:SS."""
    seconds = int(seconds)
    h = seconds // 3600
    m = (seconds % 3600) // 60
    s = seconds % 60
    if h > 0:
        return f"{h}:{m:02d}:{s:02d}"
    return f"{m}:{s:02d}"

A small helper from transcript.py that turns raw seconds into the human-readable labels the context and the citations use. Videos over an hour get the HH:MM:SS form automatically.

chat.py
python
def build_rag_context(chunks: list[dict]) -> str:
    """Format retrieved chunks into a timestamped context block."""
    parts = []
    for chunk in chunks:
        ts = format_timestamp(chunk["start_time"])
        end_ts = format_timestamp(chunk["end_time"])
        parts.append(f"[{ts} - {end_ts}]\n{chunk['text']}")
    return "\n\n---\n\n".join(parts)

Each chunk gets a visible timestamp range in the context. The model sees those ranges and picks the one that best matches the sentence it is writing.

chat.py
python
def extract_sources_from_reply(reply: str, video_id: str) -> list[dict]:
    """Parse [MM:SS](url) links the LLM wrote inline."""
    import re
    pattern = (
        r"\[([\d]{1,2}:\d{2}(?::\d{2})?)\]"
        r"\((https://www\.youtube\.com/watch\?v=[\w-]+&t=(\d+)s?)\)"
    )
    matches = re.findall(pattern, reply)
    seen = set()
    sources = []
    for ts_label, url, seconds_str in matches:
        seconds = int(seconds_str)
        if seconds not in seen:
            seen.add(seconds)
            sources.append({
                "timestamp": ts_label,
                "seconds": float(seconds),
                "link": url,
            })
    return sources

We parse the same markdown the model wrote so the UI can show the citations as chips alongside the message text. Deduplicated by seconds so the same moment never shows twice.

chat.py
python
GLOBAL_SYSTEM_PROMPT = """You are an intelligent video assistant. You have been given the COMPLETE transcript of a YouTube video with timestamps.

Instructions:
- Answer the user's question using the full transcript comprehensively.
- For summaries: cover ALL major sections, not just the beginning.
- For "main sections" style questions: identify distinct topic shifts and list each with its start timestamp.
- Cite timestamps inline using this EXACT markdown format: [MM:SS](https://www.youtube.com/watch?v={video_id}&t=Xs)
  where X is the timestamp in seconds. Always include the s suffix.
- Be well structured, use numbered lists or clear sections.
- Only cite timestamps that are genuinely relevant to that point.
"""

MAX_FULL_TRANSCRIPT_WORDS = 60_000


def build_full_transcript_context(all_chunks: list[dict]) -> str:
    """Concatenate ALL chunks sorted by time, capped at a word budget."""
    sorted_chunks = sorted(all_chunks, key=lambda c: c["start_time"])
    parts = []
    words_so_far = 0
    for chunk in sorted_chunks:
        chunk_words = len(chunk["text"].split())
        if words_so_far + chunk_words > MAX_FULL_TRANSCRIPT_WORDS:
            parts.append("[... transcript truncated for length ...]")
            break
        ts = format_timestamp(chunk["start_time"])
        parts.append(f"[{ts}] {chunk['text']}")
        words_so_far += chunk_words
    return "\n".join(parts)

This is the global route the router promised. No retrieval at all: every chunk goes into the context sorted by start time, each line prefixed with its timestamp, capped at a word budget that fits comfortably in the model context window. The prompt demands the same inline link format as the specific route, so full-video summaries stay clickable too.

chat.py
python
def chat_with_video(user_message, conversation_history, index, chunks,
                    video_id, client, top_k=5, keyword_index=None):
    """Route the query, build context, call the LLM, extract sources."""
    route = classify_query(user_message, client)

    if route == "global":
        context = build_full_transcript_context(chunks)
        system_prompt = GLOBAL_SYSTEM_PROMPT.replace("{video_id}", video_id)
        max_tokens = 1800
    else:
        vector_results = search_index(user_message, index, chunks,
                                      client, top_k=10)
        keyword_results = keyword_index.search(user_message, top_k=10)
        retrieved = fuse_and_get_top_k(keyword_results, vector_results,
                                       top_k=top_k, rrf_k=60)
        context = build_rag_context(retrieved)
        system_prompt = SPECIFIC_SYSTEM_PROMPT.replace("{video_id}", video_id)
        max_tokens = 900

    messages = [
        {"role": "system", "content": system_prompt},
        {"role": "system", "content": f"CONTEXT:\n\n{context}"},
    ]
    messages.extend(conversation_history)
    messages.append({"role": "user", "content": user_message})

    response = client.chat.completions.create(
        model=MODEL,
        messages=messages,
        temperature=0,
        max_tokens=max_tokens,
    )
    reply = response.choices[0].message.content
    sources = extract_sources_from_reply(reply, video_id)
    return reply, sources

The orchestration function app.py calls for every chat message, condensed slightly from the repo version. One route decision, one context build, one LLM call, then the inline links are parsed back out for the citation chips. The repo version also falls back to vector-only retrieval when the keyword index is missing.

app.py
python
# Multi-video switcher using session_state
video_ids = list(st.session_state.videos.keys())
if video_ids:
    labels = [
        st.session_state.videos[vid]["meta"]["title"][:60]
        for vid in video_ids
    ]
    idx = st.selectbox(
        "Active video",
        range(len(video_ids)),
        format_func=lambda i: labels[i],
        index=video_ids.index(st.session_state.active_video_id)
            if st.session_state.active_video_id in video_ids else 0,
    )
    st.session_state.active_video_id = video_ids[idx]

The switcher is a tiny select box backed by session_state.videos. Changing the active video flips both the iframe and the chat history in one interaction.

YouTube still accepts plain seconds in practice, but our regex in extract_sources_from_reply uses a permissive pattern (s? makes the suffix optional) so we capture both shapes. The prompt still asks for the s suffix because it is the canonical format and some YouTube URLs only play correctly with it.

AI prompt: Try it: verify your citations work

Loading practice…

Quiz: Quiz

Loading practice…