Audio frames in and out

FastRTC hands you audio in small chunks whenever the caller is speaking. Your job is to accept the bytes, run the agent turn, and yield audio bytes back. The async generator shape matters here because the handler can start sending audio while the agent is still producing the rest of the reply.

router.py
python
from fastrtc import Stream, ReplyOnPause

async def _audio_handler(audio_chunk):
    """Called by FastRTC on each incoming audio chunk/utterance."""
    try:
        audio_bytes = audio_chunk if isinstance(audio_chunk, (bytes, bytearray)) else bytes(audio_chunk)
    except Exception:
        audio_bytes = b""
    transcript, reply, audio_out = await _agent.handle_turn(audio_bytes)
    logger.info(f"[phone] transcript={transcript!r} reply={reply!r}")
    yield audio_out

stream = Stream(handler=ReplyOnPause(_audio_handler), modality="audio", mode="send-receive")
stream.mount(app, path="/phone/webrtc")

The handler is an async generator. yield audio_out hands bytes to FastRTC, which encodes them into outbound RTP frames. send-receive mode means the same session carries caller audio one way and reply audio the other.

agent.py
python
async def handle_turn(self, audio_bytes: bytes) -> tuple[str, str, bytes]:
    """One full realtime turn: audio -> text -> LLM -> reply -> audio.
    Returns (transcript, reply_text, reply_audio_bytes).
    """
    transcript = self.transcribe(audio_bytes)
    reply = await self.respond(transcript)
    audio_out = self.synthesize(reply)
    return transcript, reply, audio_out

Today handle_turn is a simple pipeline. Transcribe, respond, synthesize. Later we will replace this with streaming versions so audio can leave before respond has finished generating the full reply.

Because the reply is not one buffer. A real TTS stream emits audio in chunks as it synthesizes. An async generator lets your handler yield chunk one, then keep working and yield chunk two. A function that returns bytes would force you to wait for the full sentence before anything plays. On a phone call, that difference is the gap between natural and broken.

Quiz: Quiz

Loading practice…