Context compaction

Every model has a context limit. Hit it and the API refuses to answer. A real assistant that runs for weeks will easily blow past any limit if you keep appending turns forever. The fix is simple: when the session gets big, summarize the old stuff and keep the recent stuff verbatim.

Split, summarize, merge

Old turns get compressed into a single summary block. Recent turns are kept as is.

07-context-compaction/bot.py
python
def estimate_tokens(messages: list[dict]) -> int:
    total_chars = 0
    for msg in messages:
        content = msg.get('content', '')
        if isinstance(content, str):
            total_chars += len(content)
        else:
            total_chars += len(str(content))
    return total_chars // 4  # rough heuristic: 4 chars per token

COMPACT_THRESHOLD = 30000  # start compacting when we hit this many tokens
KEEP_RECENT = 20            # keep the last N messages verbatim

A character-based heuristic is more than accurate enough to decide when to compact. Being precise here is over-engineering.

07-context-compaction/bot.py
python
def compact_session(messages: list[dict]) -> list[dict]:
    if estimate_tokens(messages) < COMPACT_THRESHOLD:
        return messages

    to_summarize = messages[:-KEEP_RECENT]
    recent = messages[-KEEP_RECENT:]

    summary_prompt = (
        'Summarize this conversation so the assistant remembers the key facts, '
        'decisions, and open questions. Be concise.'
    )
    summary_response = client.messages.create(
        model=MODEL,
        max_tokens=1024,
        messages=to_summarize + [{'role': 'user', 'content': summary_prompt}],
    )
    summary_text = summary_response.content[0].text

    summary_turn = {
        'role': 'user',
        'content': f'[Prior conversation summary]\n{summary_text}',
    }
    return [summary_turn] + recent

The older half becomes one summary turn. The recent turns stay exactly as they were so the ongoing flow is intact.

It disappears from the live context. That is why compaction alone is not enough for a truly long-lived assistant. Up next, the agent gets a separate long-term memory it can write to and search, so important facts survive compaction.

Quiz: Quiz

Loading practice…