Memory eviction policy

A bounded buffer plus a rolling summary handles short term memory, but the underlying store still grows across days and weeks. Without an eviction policy, the database silently turns into a costly archive. The fix is two coordinated rules: a per-thread cap by token budget, and a global cap by age.

layers/memory_eviction.py
python
from datetime import datetime, timedelta, timezone

MAX_TOKENS_PER_THREAD = 8_000
MAX_AGE = timedelta(days=30)


def evict(memory_store) -> dict:
    now = datetime.now(timezone.utc)
    evicted = {'aged_out': 0, 'token_trimmed': 0}

    for thread in list(memory_store.threads()):
        if now - thread.last_active > MAX_AGE:
            memory_store.delete_thread(thread.id)
            evicted['aged_out'] += 1
            continue

        if thread.token_count > MAX_TOKENS_PER_THREAD:
            memory_store.compact(thread.id, target_tokens=MAX_TOKENS_PER_THREAD // 2)
            evicted['token_trimmed'] += 1

    return evicted

Two passes per cleanup. Aged-out threads get deleted outright. Live but bloated threads get compacted by the summarisation step you already built. The eviction loop runs on a cron, not on a request.

Two safety rails. Eviction is a background job, never inline on a user request: deleting state during a live turn breaks consistency. And every eviction emits a structured log line with the trace ID of the cron run, so you can audit what got trimmed and why.

Quiz: Quiz

Loading practice…