Persistent and bounded

Agent memory is one of the most misunderstood layers. People either store nothing and the agent forgets who it was talking to, or they store everything and the prompt balloons until requests time out and costs explode. The fix is to make memory bounded by default. Keep recent turns verbatim, drop the oldest when full.

layers/memory.py
python
from collections import defaultdict, deque
from typing import Deque, Dict, List


MAX_TURNS_PER_THREAD = 20


class MemoryLayer:
    def __init__(self) -> None:
        self._threads: Dict[str, Deque[dict]] = defaultdict(
            lambda: deque(maxlen=MAX_TURNS_PER_THREAD)
        )

    def append(self, thread_id: str, role: str, content: str) -> None:
        self._threads[thread_id].append({"role": role, "content": content})

    def history(self, thread_id: str) -> List[dict]:
        return list(self._threads[thread_id])

    def clear(self, thread_id: str) -> None:
        if thread_id in self._threads:
            self._threads[thread_id].clear()

A bounded deque is the right default. Once the thread has 20 turns, the oldest gets evicted automatically. Swap the in-memory dict for Redis or Postgres later; the interface stays the same.

Read before orchestrate, write after guardrails

Memory participates twice in every request, read early and write late.

Because the stored version is the one future turns will see. If a reply contained a phone number, the guardrails layer redacts it in the response, and you want that redacted version in memory too. Writing after guardrails makes sure PII never survives in the store, even if it briefly passed through the model.

Quiz: Quiz

Loading practice…