Token budget

Iteration count is one axis. Tokens are the other. A single huge tool result can blow the context window even if you only ran a couple of iterations. A cheap byte-per-token approximation is usually good enough to stop the loop before the API does.

agent/agent.go
go
const approxTokenBytes = 4
const maxTokens = 100_000

func approxTokens(msgs []Message) int {
	total := 0
	for _, m := range msgs {
		total += len(m.Content) / approxTokenBytes
	}
	return total
}

// inside the loop, after appending tool results
if approxTokens(messages) > maxTokens {
	return "", fmt.Errorf("agent: token budget exceeded (~%d)", approxTokens(messages))
}

Approximate tokens with bytes divided by four. Not perfect, but it does not need to be. You need a signal loud enough to stop before the API returns a context-length error.

Flashcards: Flashcards

Loading practice…

Quiz: Quiz

Loading practice…