Shipping the agent

Your agent runs on localhost. That is great for learning, not great for customers. Moving this to a real environment is a handful of decisions: where the Python worker lives, how secrets are injected, and how you watch the system when it misbehaves.

A production voice agent topology

Frontend on a CDN, backend on a worker host, LiveKit Cloud for transport, Deepgram for speech.

Secrets never land in the frontend bundle. The frontend only ever gets a short-lived token minted by your backend. Your LiveKit API secret, Deepgram key, and LLM provider key live as environment variables on the backend host.

fly.toml
bash
app = "voice-agent-worker"
primary_region = "sjc"

[build]
  dockerfile = "Dockerfile"

[env]
  PYTHONUNBUFFERED = "1"

[processes]
  worker = "uv run python restaurant_agent.py start"

[[services]]
  internal_port = 8080
  protocol = "tcp"
  [[services.tcp_checks]]
    interval = "15s"
    timeout = "2s"

Fly keeps the worker always on, restarts it on crashes, and pulls secrets from the fly secrets store at boot time. Replace sjc with a region close to your users for lower audio round-trip latency.

Observability for a voice agent is not just logs. Track session duration, tool call counts, STT latency, TTS latency, and end-of-turn delays. When a conversation feels off, those metrics are where you find the culprit. Structured logs plus a dashboard over room events go a long way.

One worker process can handle many concurrent sessions, but each session holds a live audio stream and an LLM call in flight. In practice you scale horizontally: several worker machines, each accepting a handful of sessions. LiveKit dispatches incoming rooms across available workers.

Quiz: Quiz

Loading practice…