How do voice AI agents work? A simple explanation

1. 1

   What is a voice AI agent?

   A voice AI agent is an AI agent you talk to instead of type to. Like any agent, it can use tools: look up an order, check a calendar, send a text message. The difference is that everything happens live, in a conversation, often over a phone call.

   You have probably met one already: a support line that understands full sentences, a clinic bot that books slots, or ChatGPT’s and Gemini’s voice modes. AI mock interviews, like the one on this site, use the same building blocks.

   • Customer support: answer common questions, check order status, hand off to a human.
   • Sales calls: qualify leads and book meetings.
   • Appointment booking: clinics, salons, service visits.
   • Practice: mock interviews and cold-call roleplay.
2. 2

   The cascade: speech-to-text, LLM, text-to-speech

   The classic design chains three models. OpenAI’s Agents SDK describes its voice pipeline in exactly these three steps: turn audio into text, run your agent, turn the reply text back into audio. Vapi’s docs describe the same three parts as the transcriber, the model and the voice.

   Speech-to-text (STT, also called ASR, automatic speech recognition) writes down what the caller said. The LLM reads that text, decides what to say and which tools to call. Text-to-speech (TTS) turns the reply into a voice. To save time, each part streams: the LLM starts writing before the transcript is final, and TTS starts speaking the first words before the full reply exists.

   1. Caller speaks↓→
   2. Speech-to-text↓→
   3. LLM thinks and calls tools↓→
   4. Text-to-speech↓→
   5. Caller hears the reply

   Key idea

   A cascade lets you pick the best part for each job and read the text in the middle, which helps with logging and safety checks.

3. 3

   What is a speech-to-speech (realtime) model?

   A speech-to-speech model takes audio in and gives audio out, with no text step in between. OpenAI launched its Realtime API in October 2024 and made it generally available on 28 August 2025 with the gpt-realtime model, adding phone calling over SIP, remote MCP servers and function calls that can run while the model keeps talking. OpenAI has released newer realtime models since; check its docs for current names.

   The upside is natural speech: tone, emphasis and accents survive, and interruptions are handled inside the session. The trade-offs, according to OpenAI’s own voice agent guide and builders who run both, are less control over each step and higher cost. Deepgram’s Voice Agent API and ElevenLabs’ platform sit in between: one connection, with speech-to-text, LLM and text-to-speech run together on their side.

   1. Caller speaks↓→
   2. One realtime model (audio in, audio out)↓→
   3. Caller hears the reply
4. 4

   Why is speed so hard? The latency budget

   Latency is the delay between you stopping talking and the agent starting to reply. In normal conversation people answer fast: the open "Voice AI & Voice Agents" primer (updated June 2026) calls about 500 ms typical, and suggests voice agents aim for 1,500 ms or less voice-to-voice. Vapi’s docs set a tighter target of under 500–700 ms.

   Every step spends part of that budget. Here is the primer’s example for a well-tuned cascade:

   Speech-to-text + deciding you finished:  300 ms
   LLM time to first token:                 650 ms
   Text-to-speech time to first audio:     120 ms
   Microphone, network, buffers:           the rest
   Total:                                about 1,293 ms

   Key idea

   The LLM is usually the biggest slice of the wait, which is why voice agents often use fast, smaller models.

5. 5

   Turn-taking, voice activity detection and barge-in

   The agent must guess when you are done. Voice activity detection (VAD) is a small model that hears whether someone is speaking. The simple rule is "reply after a short silence", which OpenAI calls server_vad. But people pause mid-sentence, so OpenAI also offers semantic_vad, which judges from your words whether you finished your thought. LiveKit, Deepgram and ElevenLabs each describe their own turn-detection models for the same job.

   Barge-in means you interrupt the agent and it stops talking to listen. That sounds easy, but coughs, background voices and the agent’s own echo can trigger false interruptions. The primer also advises saving only the words the caller actually heard, so the agent does not think it said a sentence that was cut off.

   • Too eager: the agent talks over you.
   • Too patient: awkward silences.
   • Good agents tune this per use case, for example waiting longer when someone is reading out a phone number.
6. 6

   Phone calls, tools during calls, and the platforms

   To use a real phone number, the agent connects to the phone network through a telephony provider, usually over SIP (Session Initiation Protocol, the standard for internet phone calls). LiveKit, ElevenLabs, OpenAI’s Realtime API and others support SIP; ElevenLabs also lists a native Twilio integration.

   Function calling (tool calling) works as in text agents, but every call adds a wait. The primer suggests saying "one moment, let me check" for slow tools. Retell’s docs list ready-made call actions such as transferring the call, booking a calendar slot and sending an SMS.

   • Speech-to-speech APIs: OpenAI Realtime API.
   • Hosted voice agent platforms: Vapi, Retell, ElevenLabs Agents (ElevenAgents), Deepgram Voice Agent API.
   • Open-source frameworks: LiveKit Agents (Python and Node.js, Apache 2.0) and Pipecat (Python, BSD-2, maintained by Daily).

   Key idea

   Pick a hosted platform to launch fast, or an open-source framework when you need full control.

From the people who built it

Straight from the source

The original idea, then the same idea taken apart, then how you use it today.

OpenAI

Makers of the Realtime API and ChatGPT’s Advanced Voice Mode

Introducing the Realtime API (October 2024)

1 · Understand

Older voice assistants chained transcription, a text model and text-to-speech, which OpenAI says "often resulted in loss of emotion, emphasis and accents, plus noticeable latency". A single speech-to-speech model streams audio in and out instead.

2 · Break it down

• The cascade loses information: tone and emphasis disappear when speech becomes plain text.
• Each hand-off between models adds delay.
• A realtime session can also handle interruptions and call functions, such as placing an order.

3 · Use it

When you try a voice agent, notice whether it keeps your tone and handles interruptions; that tells you a lot about how it is built.

OpenAI

Maintainers of the OpenAI Agents SDK

Agents SDK voice pipeline and Voice agents guide

1 · Understand

Even OpenAI documents both designs: a three-step pipeline (speech-to-text, your agent, text-to-speech) and a realtime speech-to-speech session, each with its own strengths.

2 · Break it down

• Chained: inspect the text, run policy checks, and swap any part.
• Realtime: one model hears, decides and speaks, and manages turns and interruptions.
• Neither is always better; the choice depends on control, cost and naturalness.

3 · Use it

Start with whichever your platform makes easy, then test both on your real calls before committing.

6 terms

Words to know

Speech-to-text (STT / ASR)

A model that turns spoken audio into written text.

Text-to-speech (TTS)

A model that turns written text into a spoken voice.

Speech-to-speech (realtime) model

One model that takes audio in and gives audio out, with no separate text step.

Voice activity detection (VAD)

A small model that detects when someone is speaking, used to decide when a turn starts and ends.

Barge-in

When the caller interrupts and the agent stops speaking to listen.

SIP

Session Initiation Protocol, the standard used to connect internet voice systems to phone numbers.

1 video · plays here

Watch

Hand-picked from the people who explain this best. The video loads only when you press play.

Hands-on · 10 minutes or less

Try it

Stress-test a voice agent in 10 minutes

1. 1Open any voice AI you can use, such as the voice mode in the ChatGPT or Gemini app, or the free AI mock interview on this site.
2. 2Latency: ask a simple question and count, roughly, how long until it starts speaking.
3. 3Turn-taking: start a sentence, pause for two seconds mid-thought ("I worked at... a startup"), and see whether it jumps in too early.
4. 4Barge-in: while it is answering, interrupt with "wait, shorter please". Does it stop and adjust?
5. 5Write down one thing that felt natural and one that felt robotic, and match each to a step in this lesson.

Free AI mock interview (voice)

3 questions · instant feedback

Check yourself

Pick an answer to see if you are right, and why.

Question 1

Question 1 of 3

In a cascaded voice agent, what is the correct order?

AText-to-speech, LLM, speech-to-textBSpeech-to-text, LLM, text-to-speechCLLM, speech-to-text, text-to-speechDSpeech-to-text, text-to-speech, LLM

Question 2

Question 2 of 3

The agent keeps cutting people off when they pause to think. What should you tune first?

AThe text-to-speech voiceBTurn detection (VAD settings or a semantic turn model)CThe phone numberDThe size of the knowledge base

Question 3

Question 3 of 3

Why might a team choose a cascade over a speech-to-speech model?

ACascades cannot call toolsBCascades let them read and check the text between steps and swap each partCSpeech-to-speech models cannot speakDCascades never have latency

Lesson 25 of 36

Done? Keep going

Mark this lesson complete to keep your place. Progress is saved on this device, no login.

Mark lesson complete

Up next · Lesson 26 · 13 minOpen models: run an LLM on your laptopOpen-weight models like Gemma, gpt-oss, Llama, Qwen, DeepSeek and Mistral can be downloaded and run on your own computer, so your chats stay on your machine and work without internet once the model is downloaded. The two easiest tools are Ollama (one install command, then "ollama run") and LM Studio (a desktop app with a chat window). The main limit is memory: aim for 16 GB of RAM, start with a small model, and use a 4-bit quantized version.← Previous: Evals: does your AI work?

People also ask

Questions people ask

How do AI voice agents work?+

Most chain three models: speech-to-text turns your voice into text, a language model decides the reply and calls any tools, and text-to-speech speaks it. Some newer models do speech-to-speech in one step. Around this sit turn detection, interruption handling and a phone connection.

What is the difference between a voice agent and an IVR?+

An IVR (the old "press 1 for billing" menu) follows fixed paths and keywords. A voice AI agent understands free-form speech, keeps track of the conversation, and can decide which tool to call, such as checking an order or booking a slot.

What is good latency for a voice AI agent?+

People typically reply in about half a second. Builders aim for roughly one second or less from when you stop talking to when the agent starts; the Voice AI primer treats 1,500 ms as an upper target and Vapi aims for under 500–700 ms.

13 sources · checked Nov 2026

Sources and further reading

• OpenAI: Introducing the Realtime API (Oct 2024)
• OpenAI: Introducing gpt-realtime and Realtime API updates (Aug 2025)
• OpenAI API docs: Voice agents
• OpenAI API docs: Voice activity detection
• OpenAI Agents SDK: Voice quickstart
• Voice AI & Voice Agents: an illustrated primer (updated June 2026)
• LiveKit docs: Agents introduction
• LiveKit blog: Voice agent architecture, STT-LLM-TTS pipelines explained (Feb 2026)
• Pipecat docs: Introduction
• Vapi docs: Introduction
• Retell AI docs: ChatGPT app
• ElevenLabs docs: ElevenAgents overview
• Deepgram docs: Voice agent architecture

Related on this site

Previous: Evals, does your AI work?Next: Run an LLM on your laptopTool calling (function calling)How AI sees, hears and speaksVoice AI calling: latency, cost per minute and lawsFree AI mock interviewGTM engineer guide