Whisper streaming

The agent ships with a mock transcriber so the server boots without an OpenAI key. Today we replace it with Whisper. The interesting part is not the API call. It is the fact that transcribe is a named seam on the VoiceAgent class, which lets us swap providers without touching the rest of the loop.

agent.py
python
def transcribe(self, audio_bytes: bytes) -> str:
    """Convert audio bytes to text.

    Uses Whisper if available, else returns a mock transcript so the
    demo runs offline.
    """
    if not audio_bytes:
        return ""

    if self._openai_client is not None:
        try:
            import io
            buf = io.BytesIO(audio_bytes)
            buf.name = "chunk.wav"
            result = self._openai_client.audio.transcriptions.create(
                model="whisper-1",
                file=buf,
            )
            return (result.text or "").strip()
        except Exception as e:
            logger.warning(f"Whisper STT failed, falling back to mock: {e}")

    # Offline fallback, lets smoke tests pass with no keys
    return "[mock transcript]"

The seam pattern: a named method that can be replaced without changing callers. handle_turn only cares that transcribe returns a string. Whether that string comes from Whisper, Deepgram, or a mock is invisible upstream.

For production phone agents, the next upgrade is streaming transcription. Instead of waiting for the full utterance and then calling Whisper, you feed audio frames into a streaming STT session and receive partial transcripts as the caller speaks. That partial text lets the LLM start thinking before the caller finishes the sentence.

AI prompt: Design a streaming transcribe seam

Loading practice…

Quiz: Quiz

Loading practice…