How to Build an In-House AI Calling Stack (2026)

To build a production in-house AI calling system in 2026, engineering teams deploy a cascaded architecture balancing sub-500ms conversational turn-taking: LiveKit or Pipecat for real-time WebRTC/SIP media transport, Silero VAD v5 for barge-in interruption detection, Deepgram Nova-3 or Sarvam Saaras for streaming STT, Claude 3.5 Haiku or Groq Llama 3.3 70B via LiteLLM for low-latency reasoning, and Cartesia Sonic or ElevenLabs Flash v2.5 for streaming TTS. Closing the quality loop requires Model Context Protocol (MCP) automated post-call telemetry to diagnose production failure modes and feed continuous self-learning evaluation rubrics.

┌────────────────────────────────────────────────────────────────────────────────────────┐
│                      PRODUCTION IN-HOUSE VOICE AI CALLING ARCHITECTURE                 │
├────────────────────────────────────────────────────────────────────────────────────────┤
│ 1. Telephony Ingress  │ SIP Trunk (Twilio / Plivo / Exotel) ──> LiveKit SIP Gateway   │
│ 2. Media Transport    │ WebRTC Audio Pipe (8kHz G.711u / 16kHz Opus) ──> Audio Buffer   │
│ 3. Voice Activity     │ Silero VAD v5 (30ms frame analysis, 0.65 threshold, barge-in)  │
│ 4. Streaming STT      │ Deepgram Nova-3 / Sarvam Saaras v2 (WebSocket chunk streaming) │
│ 5. Agent Brain        │ LiteLLM Gateway ──> Claude 3.5 Haiku / Groq Llama 3.3 70B      │
│ 6. Streaming TTS      │ Cartesia Sonic / ElevenLabs Flash v2.5 (110-135ms TTFA)        │
│ 7. Audio Egress       │ Jitter Buffer (40ms) ──> RTP Media Egress ──> Caller Mobile    │
│ 8. Self-Learning Loop │ MCP Telemetry Audit ──> Tough Tongue AI Simulated Stress Drills│
└────────────────────────────────────────────────────────────────────────────────────────┘

1. The 500ms Conversational Latency Budget

In human telephone conversations, psychological conversational turn-taking occurs in 300ms to 500ms. If an artificial intelligence agent exceeds 650ms to deliver its first synthesized voice syllable, human callers perceive the pause as an awkward robotic hesitation or assume the cellular call was disconnected.

Every millisecond across the cascaded pipeline must be rigorously accounted for:

Conversational Latency Budget (Target: < 480ms Total Round-Trip)
├─ 1. SIP Ingress & RTP Jitter Buffer      :  40 ms  ██
├─ 2. Voice Activity Detection (Silero VAD):  80 ms  ████
├─ 3. Streaming STT End-of-Utterance (EOU) : 140 ms  ███████
├─ 4. LLM Time to First Token (TTFT)       : 120 ms  ██████
├─ 5. TTS Time to First Audio Chunk (TTFA) :  90 ms  █████
└─ 6. RTP Egress & Mobile Telecom Buffer   :  30 ms  ██
──────────────────────────────────────────────────────────
TOTAL END-TO-END ROUND TRIP               : 500 ms (Threshold Met)

2. Framework Selection: LiveKit vs. Pipecat vs. LiteLLM

When building in-house, developers frequently confuse these three tools because modern enterprise architectures use all three in unison across distinct layers of the OSI model.

┌────────────────────────────────────────────────────────────────────────────────────────┐
│                              THE 3-TIER VOICE STACK                                    │
├──────────────────────────┬──────────────────────────┬──────────────────────────────────┤
│ Layer                    │ Recommended Technology   │ Primary Responsibility           │
├──────────────────────────┼──────────────────────────┼──────────────────────────────────┤
│ Media & Room Transport   │ **LiveKit (Agents)**     │ WebRTC SFU, SIP Inbound/Outbound │
│ Audio Pipeline Frames    │ **Pipecat**              │ Modular STT ──> LLM ──> TTS      │
│ Model Gateway & Proxy    │ **LiteLLM**              │ Multi-provider fallback & routing│
└──────────────────────────┴──────────────────────────┴──────────────────────────────────┘

LiveKit Agents: The Media & Telephony Layer

LiveKit provides the real-time communications (RTC) media server. It handles WebRTC streaming, network packet loss concealment, multi-party room routing, and native SIP trunking.

• Best For: Teams that need battle-tested infrastructure capable of terminating thousands of simultaneous PSTN telephone calls, managing WebRTC client connections in web browsers, and orchestrating distributed Python or Go worker nodes.
• Key Strength: Built-in SIP Dispatch Rules that map incoming phone numbers directly to agent instances without custom FreeSWITCH or Asterisk servers.

Pipecat: The Frame Pipeline Layer

Pipecat is an open-source Python framework designed specifically to organize the real-time conversational loop. It treats audio and text as Frames traveling down a pipeline.

• Best For: Developers who want modular flexibility. Swapping Deepgram for AssemblyAI or Cartesia for ElevenLabs takes three lines of configuration code.
• Key Strength: Flawless handling of asynchronous task cancellation when a user interrupts the bot mid-sentence.

LiteLLM: The Model Abstraction & Resilience Layer

LiteLLM is not an audio library; it is a high-performance proxy for LLM APIs.

• Best For: Production voice applications where an OpenAI or Anthropic outage would take down enterprise phone operations.
• Key Strength: Automatic fallback routing (e.g., if Claude 3.5 Haiku hits a 429 rate limit or latency spike over 300ms, LiteLLM instantly re-routes the next token stream to Groq Llama 3.3 70B without dropping the live call).

3. Voice Activity Detection (VAD) & Barge-In Interruption Tuning

Voice Activity Detection (VAD) is the sensory system of a voice agent. If VAD is misconfigured, the bot either speaks over the user (rude) or gets cut off every time the user breathes, coughs, or experiences background noise (conversational deadlock).

VAD Model Comparison

Production Tuning Checklist for Silero VAD v5:

To prevent false barge-in interruptions while maintaining crisp response times, configure these parameters in your audio frame processor:

1. speech_threshold: 0.65 (Do not use the default 0.50). Elevating the threshold to 0.65 prevents background urban office noise or car air conditioners from prematurely halting the bot.
2. min_speech_duration_ms: 250ms: The user must speak for at least 250 milliseconds continuously before the bot cancels its audio output. This prevents sudden single-syllable noises ("uh", throat clears, keyboard clicks) from cutting off the bot.
3. min_silence_duration_ms: 450ms: The amount of continuous silence required before the pipeline concludes the user has finished their sentence. For sales calls where prospects hesitate before speaking, set to 450ms - 550ms. For transactional lookup bots, tighten to 350ms.

4. SIP Telephony Configuration & Media Codec Engineering

Connecting an AI agent to the traditional Public Switched Telephone Network (PSTN) requires integrating with telecom carriers (Twilio, Plivo, Exotel, Telnyx, Tata Tele Business).

┌────────────────────────────────────────────────────────────────────────────────────────┐
│                        SIP SIGNALING & RTP MEDIA NEGOTIATION                           │
├────────────────────────────────────────────────────────────────────────────────────────┤
│ Caller Phone ──> Telecom Carrier PSTN ──> SIP INVITE (UDP/TLS) ──> LiveKit SIP Gateway │
│ LiveKit SIP Gateway ──> 100 Trying ──> 200 OK (SDP Negotiated: PCMU/8000)               │
│ Carrier ACK ──> Bidirectional RTP Audio Stream Active (20ms packet chunks)             │
└────────────────────────────────────────────────────────────────────────────────────────┘

Critical Telephony Configurations:

• Audio Codec Negotiation: Standard PSTN calls negotiate G.711 µ-law (PCMU) or G.711 A-law (PCMA) sampled at 8,000 Hz with 20ms packet frames (160 samples per packet). Ensure your audio ingest resamples 8kHz directly to 16kHz for STT models using high-performance SIMD resampling libraries (e.g., libsamplerate or soxr) to prevent high-frequency aliasing distortion.
• SIP Signaling Latency (Target: < 800ms): When dialing outbound, minimize the time from SIP INVITE to 200 OK. Deploy your media gateways in the same cloud availability region as your carrier's media proxy (e.g., AWS ap-south-1 for Indian carriers like Exotel/Tata; AWS us-east-1 for Twilio).
• Jitter Buffer Sizing: Set an adaptive jitter buffer between 30ms and 50ms. Overly aggressive buffering introduces lag; overly small buffers cause audio packet drops that make your synthesized voice sound robotic or garbled.

5. Production Failure Modes: Where Voice Bots Break

When teams transition from local development to running 10,000+ live phone calls a day, voice bots fail across four distinct operational vectors:

┌──────────────────────────────────────────────────────────────────────────────┐
│                    THE 4 VOICE BOT PRODUCTION TRAPS                          │
├──────────────────────────────────────────────────────────────────────────────┤
│ 1. Conversational Deadlock: Silence detection timeout triggers duplicate speech│
│ 2. Tool-Call Lag: Bot freezes for 2.4 seconds while querying CRM databases    │
│ 3. Acoustic Hallucination: Transcribing silent line static as words ("Thank you")│
│ 4. Sentiment Escalation Failure: Arguing with frustrated human callers       │
└──────────────────────────────────────────────────────────────────────────────┘

1. Acoustic Hallucination: In low-signal cellular areas, background static or breathing can cause models like Whisper to repeatedly transcribe phrases like "Thank you for watching" or "Bye". Mitigate this by enforcing strict VAD probability filters before passing audio buffers to the STT API.
2. Tool-Call Latency Freezes: When the agent needs to check an appointment calendar or CRM record, a synchronous database call taking 1,200ms freezes the audio stream. Engineering Fix: Implement conversational filler tokens ("Let me pull up your account details right now...") streamed immediately via TTS while the async tool execution runs in the background.
3. Interruption Thrashing: If a prospect says "Yeah, yeah, okay", an uncalibrated bot stops speaking instantly, leaving an awkward dead air pause.

6. The Model Context Protocol (MCP) Self-Learning Loop

Leading voice AI engineering teams deploy automated evaluation pipelines using the Model Context Protocol (MCP). Instead of human quality assurance (QA) agents sampling 1% of recorded calls, an MCP server processes 100% of call telemetry automatically.

┌────────────────────────────────────────────────────────────────────────────────────────┐
│                        MCP AUTOMATED SELF-LEARNING AGENT PIPELINE                      │
├────────────────────────────────────────────────────────────────────────────────────────┤
│ 1. Call Session Ends ──> Telemetry (Audio, VAD events, Transcripts, Latencies) Logged │
│ 2. MCP Server Ingests Payload ──> Dispatches Evaluator LLM (100-Point Rubric)          │
│ 3. Evaluates 5 Dimensions: Latency (ms), Script Compliance, Objection Handling, Tone  │
│ 4. Automated Error Tagging: Identifies prompt drifts, interruptions, and hallucinations│
│ 5. Tough Tongue Scenario Studio: Converts failed calls into interactive voice drills   │
│ 6. Continuous Regression Testing: Re-tests agent system prompts prior to production CI │
└────────────────────────────────────────────────────────────────────────────────────────┘

By connecting call recordings to an automated evaluation suite like Tough Tongue AI's Scenario Studio, newly reported edge-case failures are converted into automated voice test suites. The agent is stress-tested against simulated callers before the revised prompt is released into production.

Frequently Asked Questions

Should I choose LiveKit or Pipecat for my voice agent?

If you require native SIP telephony trunking, multi-participant video/audio rooms, and managed cloud infrastructure, choose LiveKit. If you prefer a lightweight, highly composable Python framework to construct custom audio processing pipelines across various transports, choose Pipecat. Many enterprise teams run Pipecat pipelines inside LiveKit rooms for the ideal developer experience.

What is the best Speech-to-Text model for real-time voice bots?

For English-first global pipelines, Deepgram Nova-3 is the gold standard for streaming speed (~140ms endpointing). For Indian languages and mixed Hinglish, Sarvam Saaras v2 delivers the lowest Word Error Rate on mobile 8kHz telephony.

How do you prevent voice bots from interrupting the caller too early?

Configure your Voice Activity Detection (VAD) with a minimum speech duration of 250ms and a silence cutoff padding of 450ms to 500ms. This prevents ambient coughing, throat clearing, or short verbal acknowledgments from canceling the agent's synthesized speech.

Why do voice agents sound robotic over standard phone lines?

PSTN telephone networks downsample audio to 8,000 Hz G.711 µ-law. If your Text-to-Speech (TTS) engine generates 24kHz or 48kHz audio and downsamples using poor linear interpolation algorithms, high-frequency distortion creates a harsh, metallic acoustic quality. Always configure your TTS engine to output native 8kHz or 16kHz PCM audio.

What role does LiteLLM play in a voice architecture?

LiteLLM serves as the model orchestration and resilience gateway. It standardizes OpenAI, Anthropic, Gemini, and local vLLM API calls into a unified format and provides automated failover if your primary LLM experiences rate limits or latency spikes during a live phone call.

Authoritative References & Engineering Documentation

• LiveKit Documentation: Building Real-Time Voice Agents with WebRTC & SIP
• Pipecat Framework: Open Source Pipeline Orchestration for Voice AI
• Silero VAD Architecture: Deep Neural Network Voice Activity Detector
• Internet Engineering Task Force (IETF): RFC 3261: Session Initiation Protocol (SIP)
• Auto Interview AI Research: Voice AI Latency Benchmarks & Sales Simulation Architectures