Summarization memory

What if instead of dropping old messages, we compressed them? Use the LLM to summarize old conversations into a brief paragraph. This summary replaces the old messages in the system prompt, preserving key facts while using far fewer tokens.

Only when memory exceeds the threshold. If you set the threshold to 6 messages, the LLM call happens once every 6 turns, not every turn. The one-time cost of summarization is much cheaper than sending a growing history every single call.

05-memory-basics.ipynb
python
def summarize_history(history):
    """Ask the LLM to compress conversation history."""
    prompt = (
        "Summarize the following conversation "
        "briefly, preserving all key facts "
        "and names mentioned:"
    )
    history_text = "\n".join(
        [f"{m['role']}: {m['content']}" for m in history]
    )
    summary_response = completion(
        model=DEFAULT_MODEL,
        messages=[{
            "role": "user",
            "content": f"{prompt}\n\n{history_text}"
        }]
    )
    return summary_response.choices[0].message.content

The summarize function sends old messages to the LLM and asks for a compressed version.

Now let us build the agent that uses this summarization function. When memory exceeds a threshold, it compresses old messages into a summary and starts fresh.

05-memory-basics.ipynb
python
class SummaryAgent:
    def __init__(self, summary_threshold=6):
        self.memory = []
        self.history_summary = "No history yet."
        self.summary_threshold = summary_threshold

    def respond(self, user_content: str):
        # Compress when memory gets too long
        if len(self.memory) >= self.summary_threshold:
            self.history_summary = summarize_history(
                self.memory
            )
            self.memory = []  # Clear old messages

        self.memory.append(
            {"role": "user", "content": user_content}
        )

        system_msg = {
            "role": "system",
            "content": (
                "You are a helpful assistant. "
                "Here is a summary of our past "
                f"conversation: {self.history_summary}"
            )
        }

        full_messages = [system_msg] + self.memory
        response = completion(
            model=DEFAULT_MODEL, messages=full_messages
        )
        content = response.choices[0].message.content
        self.memory.append(
            {"role": "assistant", "content": content}
        )
        return content

The SummaryAgent compresses old messages into a summary that goes in the system prompt. Recent messages stay as-is.

Summarization flow

05-memory-basics.ipynb
python
# Compare: after 20 turns of conversation

# Sliding Window (max_messages=6):
# Sends: system + last 6 messages = ~7 messages
# Loses: first 14 messages completely

# Summarization (threshold=6):
# Sends: system(with summary) + last 6 messages
# Retains: key facts from all 20 turns in summary
# Trade-off: extra LLM call for summarization

Summarization preserves key facts from the entire conversation, not just recent turns.

It depends on your use case: - Sliding window: best for short-lived tasks where only recent context matters (customer support, quick Q&A). Cheapest option. - Summarization: best when early context matters long-term (personal assistants, ongoing projects). Costs an extra LLM call per compression. - Hybrid: use sliding window for recent messages + periodic summarization of older ones. Best of both worlds.

Matching exercise: Match scenarios to memory strategies

Loading practice…