Context compaction: Split, summarise, merge

Before the code: why assistants forget, and how to fix it

Every language model has a limited memory of the conversation, called the context window. After enough back-and-forth, the oldest messages start falling off. Your assistant starts forgetting what it learned about you. The fix here is compaction: at some threshold, you summarise the old turns into one paragraph and keep only the recent turns word-for-word. The model still has the gist, the cost goes back down, and the assistant feels coherent again. Long-term memory then adds little notes the assistant writes to itself that survive across sessions.

Long conversations: compact the old, keep the recent

Old turns get summarised into a single paragraph. Recent turns stay word-for-word. The system message is untouched.

Quiz: Quiz

Loading practice…

Every model has a finite context window. qwen3-coder has a generous one but no model has infinite. A long conversation eventually hits the wall, and once it does, every new call gets more expensive and more brittle. Compaction is the answer: at some threshold, summarise the old turns and keep only the recent ones in detail.

07-context-compaction/bot.py
python
def estimate_tokens(messages: list[dict]) -> int:
    """Estimate token count using chars/4 heuristic."""
    total_chars = 0
    for msg in messages:
        content = msg.get("content") or ""
        total_chars += len(content)
        for tc in msg.get("tool_calls", []) or []:
            total_chars += len(json.dumps(tc))
    return total_chars // 4

A cheap, fast heuristic: characters divided by 4 is roughly tokens. Good enough to decide when to compact. Switch to a real tokeniser only if you need exact accounting.

07-context-compaction/bot.py
python
def compact_session(messages: list[dict]) -> list[dict]:
    """Split old vs recent, summarise old, merge."""
    if len(messages) <= RECENT_KEEP:
        return messages

    old_messages = messages[:-RECENT_KEEP]
    recent_messages = messages[-RECENT_KEEP:]

    old_text_parts = []
    for msg in old_messages:
        role = msg.get("role", "unknown")
        content = msg.get("content") or ""
        if content:
            old_text_parts.append(f"{role}: {content[:500]}")
        for tc in msg.get("tool_calls", []) or []:
            name = tc.get("function", {}).get("name", "tool")
            old_text_parts.append(f"{role}: [called {name}]")

    summary_response = client.chat.completions.create(
        model=MODEL,
        max_tokens=1024,
        messages=[{
            "role": "user",
            "content": f"Summarize this conversation concisely:\n\n{chr(10).join(old_text_parts)}",
        }],
    )
    summary = summary_response.choices[0].message.content or ""

    return [
        {"role": "user", "content": "[Previous conversation summary follows]"},
        {"role": "assistant", "content": f"[Summary]\n{summary}"},
    ] + recent_messages

The compaction strategy. Split the history into 'old to summarise' and 'recent to keep verbatim'. Ask the model to summarise the old part. Rebuild the history as: (summary marker, summary, recent turns).

Before and after compaction

The oldest turns collapse into one assistant message. Recent turns are kept verbatim because the model needs them to understand "what we were just talking about".

Because 'what we just said' carries the most signal for the next turn. The model needs the exact phrasing of the last few user messages to follow pronouns, recent code, and immediate context. A summary loses all of that detail and feels jarring. So we keep the last few turns intact and only sacrifice the older stuff.

Quiz: Quiz

Loading practice…

07-context-compaction/bot.py
python
def run_agent_turn(user_id: str, user_text: str) -> str:
    history = load_session(user_id)
    history.append({"role": "user", "content": user_text})

    # Compact before the API call if needed
    tokens = estimate_tokens(history)
    if tokens > TOKEN_THRESHOLD:
        history = compact_session(history)
        save_session(user_id, history)

    # ... usual loop

The agent loop now compacts BEFORE the model call if it sees the threshold has been crossed.