Partial transcripts

Even with a fast STT, waiting for a finalized transcript costs real milliseconds. Streaming STT providers emit partial transcripts as confidence rises. You can speculatively start the LLM on the latest partial, then cancel and retry if the final transcript disagrees with what you acted on.

Speculative LLM on partial transcripts

Start the LLM early and cancel if the final transcript disagrees.

agent.py
python
import asyncio

async def respond_speculative(self, partial: str, final_event: asyncio.Event, final_text: list[str]):
    """Start generating on partial, cancel if final disagrees."""
    task = asyncio.create_task(self.respond(partial))

    # Wait until either the task finishes or the final transcript arrives
    done, pending = await asyncio.wait(
        {task, asyncio.create_task(final_event.wait())},
        return_when=asyncio.FIRST_COMPLETED,
    )

    if not final_event.is_set():
        return await task  # generation finished before final arrived

    final = final_text[0]
    if final.startswith(partial):
        return await task  # partial was a correct prefix, keep the work

    task.cancel()
    return await self.respond(final)  # partial was wrong, restart

This is the shape of speculative execution in the voice loop. Start work on the partial. When the final arrives, keep the work if the partial was a prefix, otherwise cancel and restart. The savings only matter when the partial was right, which with good STT is most of the time.

For most phone agents, yes. Partial transcripts are right as a prefix roughly 80 to 90 percent of the time with a modern streaming STT. In the good case you save 200 to 400 ms of latency, which the caller feels. In the bad case you spend a few cents of extra LLM tokens. For a phone agent where responsiveness is the product, that trade is one-sided in favor of shipping it.

Checkpoint: STT seam checkpoint

Loading practice…