Time windowed chunks that preserve citations
The transcript arrives as many short snippets, often two or three seconds each. That is too fine-grained to embed usefully. We need to group snippets into chunks that are big enough to carry meaning but small enough to cite precisely.
From snippets to time windowed chunks
Flatten to word level, then slide a fixed window over the stream.
def chunk_transcript(transcript: list[dict],
chunk_size: int = 300,
overlap: int = 50) -> list[dict]:
"""Chunk transcript into overlapping windows, preserving timestamps."""
chunks = []
words_buffer = []
word_timestamps = []
# Flatten transcript into word-level with interpolated timestamps
for entry in transcript:
words = entry["text"].split()
start = entry["start"]
duration = entry.get("duration", 2.0)
for i, word in enumerate(words):
t = start + (duration * i / max(len(words), 1))
words_buffer.append(word)
word_timestamps.append(t)
# Slide window
step = chunk_size - overlap
chunk_id = 0
i = 0
while i < len(words_buffer):
end_idx = min(i + chunk_size, len(words_buffer))
chunk_words = words_buffer[i:end_idx]
chunk_times = word_timestamps[i:end_idx]
chunks.append({
"chunk_id": chunk_id,
"text": " ".join(chunk_words),
"start_time": chunk_times[0],
"end_time": chunk_times[-1],
})
chunk_id += 1
i += step
if end_idx == len(words_buffer):
break
return chunksTwo phases. First, flatten the transcript into a flat list of words with interpolated timestamps. Then slide a fixed-size window with overlap. Each chunk carries its own start and end times.
A concept can straddle a chunk boundary. If the model mentions "gradient descent" at the end of one chunk and "learning rate" at the start of the next, a query about their relationship would hit neither chunk cleanly. Overlap gives the retriever a second chance to find that joint context.
Quiz: Quiz
Loading practice…