Persona drift detection

A long call is a long context window, and long contexts are where personas start to drift. The agent emits a list of bullet points that no human would say out loud, or apologises for being an AI. You catch this by sampling replies and grading them against the persona, then nudging the prompt.

Drift detection loop

Sample replies, grade against the persona contract, alert on regressions, and refresh the system prompt before the next turn.

agent/persona_grader.py

python

GRADER_PROMPT = '''Score the assistant reply against the persona below.
Return JSON {\"score\": 0..1, \"violations\": [str]}.

Persona: short spoken sentences, no markdown, no apology for being AI, no bullet lists.

Reply:
\"\"\"{reply}\"\"\"
'''

async def grade(reply: str) -> dict:
    raw = await ask(GRADER_PROMPT.format(reply=reply), max_tokens=120)
    return parse_json_safely(raw)

async def maybe_grade(reply: str, turn: int):
    if turn % 5 != 0:
        return
    result = await grade(reply)
    if result['score'] < 0.7:
        log.warning('persona_drift', extra=result)

Sample one in five turns. Grade against the persona contract. Log when the score drops. Cheap, asynchronous, and surfaces drift before the user notices.

Quiz: Quiz

Loading practice…