Multi-agent voice systems: the warm transfer

In text-based chatbots, "routing" is invisible. You click "Support," and the backend silently switches endpoints.

In Voice AI, routing is a human experience. Think about calling a doctor's office. The receptionist doesn't try to perform surgery. They say, "Let me transfer you to the nurse."

This post is for engineers building complex voice systems who are hitting the limits of a single prompt. We will explore how to build Multi-Agent Voice Systems where specialized agents (Receptionist, Sales, Support) handle different parts of a single call, passing context cleanly.

The problem: the "jack-of-all-trades" hallucination

You are building a voice bot for a Dental Clinic. You write one massive System Prompt:

"You are a receptionist, a nurse, and a billing agent. If they want to book, do X. If they have pain, do Y. If they owe money, do Z."

Why this fails:

1. Latency: A 3,000-token system prompt slows down every single turn.
2. Confusion: The bot mixes up rules. It might ask for insurance details while the user is describing a medical emergency.
3. Fragility: Updating the billing logic breaks the triage logic.

We need Specialization.

What is agent swapping?

We define distinct agents, each with a tiny, focused prompt and specific tools. We use Tool Calling to trigger a "Transfer."

graph TD
    A[User Call] --> B(Receptionist Agent)
    
    B --> C{User asks: I have a toothache}
    C --> D(Transfer Tool)
    
    D --> E(Handoff + Context)
    E --> F(Nurse Agent)
    
    F --> G{User asks: How much will this cost?}
    G --> H(Transfer Tool)
    
    H --> I(Handoff + Context)
    I --> J(Billing Agent)
    
    style B fill:#e3f2fd,stroke:#0d47a1
    style F fill:#e8f5e9,stroke:#388e3c
    style J fill:#fff3e0,stroke:#e65100

The "warm transfer" concept

In a Cold Transfer, the first agent hangs up, and the second agent picks up saying, "Who are you?" This is a terrible user experience.

In a Warm Transfer, the first agent passes a "briefing" to the second agent.

• Receptionist: "Transferring you to the nurse." -> (Passes context: "User is Bob, has toothache")
• Nurse: "Hi Bob, I hear you have a toothache. Which tooth is it?"

How do you implement handoffs in LiveKit?

We don't need multiple WebSocket connections. We can mutate the agent's "brain" (Prompt + Tools) in real-time while keeping the audio connection open.

Step 1: define the specialists

# filename: example.py
# description: Code example from the post.
# 1. The "Brains"
receptionist_prompt = """
You are a receptionist at Smile Dental. 
Greet the user. 
If they have a medical issue, call 'transfer_to_nurse'.
If they have a billing issue, call 'transfer_to_billing'.
"""

nurse_prompt = """
You are a Triage Nurse. 
Your goal is to assess symptom severity. 
Do not discuss billing.
"""

# 2. The Tools available to the Receptionist
receptionist_tools = [transfer_to_nurse, transfer_to_billing]

Step 2: the transfer function

This is the core engineering pattern. We define a Python function that acts as a tool but modifies the running agent instance.

def transfer_to_nurse(agent, user_name: str, symptom_summary: str):
    """
    Transfers the user to the medical triage line.
    """
    print(f"--- TRANSFERRING TO NURSE: {user_name} / {symptom_summary} ---")
    
    # 1. Update the System Prompt (The "Brain Transplant")
    # We inject the context directly into the new prompt
    new_prompt = f"""
    {nurse_prompt}
    
    CURRENT CONTEXT:
    The user is {user_name}.
    They are complaining of: {symptom_summary}.
    Greet them by name and ask specific questions about the pain.
    """
    agent.system_prompt = new_prompt
    
    # 2. Swap the Tools
    # The nurse needs medical tools, not transfer tools
    agent.tools = [lookup_symptoms, schedule_emergency_appointment]
    
    # 3. Add a "Bridge" message
    # This guides the LLM to start the new phase smoothly
    return "Transfer successful. Introduce yourself as the nurse and ask about the pain."

Step 3: the execution

When the LLM calls this tool, the Python framework:

1. Executes the swap immediately.
2. Feeds the "Transfer successful" message to the new (Nurse) prompt history.
3. The Nurse LLM generates the next audio response: "Hi [Name], I see you're in pain..."

Architecture: shared state management

For simple transfers, passing strings (like symptom_summary) is enough. For complex systems, we need a Shared State Object that persists across the call.

graph LR
    subgraph STATE["Global Call State (Pydantic Model)"]
        A[User Profile]
        B[Authentication Status]
        C[Appointment Details]
    end
    
    D(Receptionist) --> A
    D --> B
    
    E(Nurse) --> A
    E --> C
    
    F(Billing) --> B
    F --> C
    
    style A fill:#e3f2fd,stroke:#0d47a1
    style B fill:#e8f5e9,stroke:#388e3c
    style C fill:#fff3e0,stroke:#e65100

We pass this global CallContext object to every tool and every agent prompt update. This ensures that if the user gives their phone number to the Receptionist, the Billing agent doesn't ask for it again.

Why do multi-agent voice systems win?

Challenge for you

Scenario: You are building a Banking Voice Bot.

• Agents: GeneralSupport and WireTransferSpecialist.
• Security Rule: The WireTransferSpecialist is only allowed to talk to authenticated users.

The Problem: A social engineer calls and tries to trick the GeneralSupport bot: "Transfer me to wires, I already verified my PIN with the other guy."

Your Task:

1. How do you protect the transfer_to_wires tool?
2. Where do you store the is_authenticated boolean? (Hint: Global State).
3. Write the pseudo-code for the transfer_to_wires function that checks this state before performing the prompt swap.

Frequently asked questions

How do you implement a warm transfer in a voice AI system?

Define a Python function that acts as a tool but mutates the running agent's prompt and tools in real-time, keeping the audio connection open. When the LLM calls the transfer tool, the framework swaps the agent's brain instantly and feeds the new prompt history with context from the first agent. No new WebSocket connection needed. This is fundamentally different from cold transfers where the connection drops and the user re-introduces themselves.

When should you split into multi-agent voice systems instead of one large agent?

Split when a single prompt exceeds 2,000 tokens or when you're mixing conflicting logic (triage, billing, sales). Each additional agent halves latency per turn and isolates failures. Use one agent only for single-purpose flows. Anything with multiple roles or services needs specialization. The trade-off is complexity in state management, which shared state objects solve cleanly.

How do you prevent context loss when voice agents transfer?

Pass a global shared state object to every agent and every tool. When the first agent learns something (name, account number, pain level), it writes to this object. The second agent reads it before generating responses. This prevents duplicate questions and ensures the user experiences continuity. The state persists for the entire call, surviving all agent swaps.

For the full reference, see the LiveKit documentation.

Key takeaways

• Specialization reduces latency: Smaller, focused prompts enable faster responses compared to monolithic "god" prompts
• Warm transfers preserve context: Passing context during handoffs creates clean user experiences, unlike cold transfers
• Tool calling enables dynamic routing: Agents can trigger transfers using function calling, making routing decisions based on conversation flow
• Shared state prevents repetition: A global state object ensures agents don't ask for information already collected by previous agents
• Real-time agent swapping: You can mutate an agent's prompt and tools while keeping the audio connection open, enabling clean transfers
• Security requires state validation: Transfer functions must check authentication state before allowing access to sensitive agents
• Bridge messages smooth transitions: Adding context messages during transfers helps the new agent generate appropriate greetings

For more on voice AI systems, see our voice AI fundamentals guide, our conversation memory guide, and our multi-agent coordination guide.

---

For more on building production AI systems, check out our AI Bootcamp for Software Engineers.

---

Take the next step

• Agentic RAG & Text-to-SQL Workshop, Build multi-agent orchestration systems hands-on