Avoiding dead air

On a text chat, a two-second spinner is normal. On a phone call, two seconds of silence reads as a dropped call. When the agent decides to call a tool, you need to speak something within roughly 300 milliseconds or the caller starts getting nervous.

agent.py
python
async def respond_with_tools(self, user_text: str):
    """Stream a reply that may invoke tools, covering tool latency with holder audio."""
    provider = get_llm_provider()

    async for event in provider.generate_stream_with_tools(
        prompt=f"{self.system_prompt}\n\nCaller said: {user_text}",
        tools=FLIGHT_TOOLS,
    ):
        if event["type"] == "tool_call":
            # Speak the holder phrase immediately so the caller hears something
            yield {"kind": "audio", "text": "one moment, let me check that"}

            name = event["name"]
            args = event["arguments"]
            result = globals()[name](**args)

            # Continue generation with the tool result
            async for token in provider.continue_with_tool_result(event["call_id"], result):
                yield {"kind": "audio", "text": token}
        elif event["type"] == "text":
            yield {"kind": "audio", "text": event["text"]}

The trick is to speak the holder phrase the moment the LLM decides to call a tool, not after the tool returns. The TTS stream covers the tool latency, so the caller hears one continuous, natural reply rather than a thinking-pause and a second reply.

Covering tool latency with a holder phrase

The caller hears one continuous reply while the tool runs in the background.

Fixed text for the first second, then the LLM can take over. Fixed holders land instantly because you do not wait for another LLM round trip. You can keep a small set of holder phrases and pick one at random so it does not feel scripted. After the holder has been speaking for a moment, the continuation can be fully model-generated, which is where the natural variation shows up.

Quiz: Quiz

Loading practice…