Voice Agent Latency on Indian Phone Calls, Hop by Hop

On this page

How fast a voice agent needs to reply
Where the time goes between a caller stopping and the agent replying
Why Indian phone calls add time that logs cannot see
Engineering choices that cut reply time
How to measure latency on a real phone call
Sources

How fast a voice agent needs to reply

People expect replies on the phone to come almost at once, because that is how conversation works. A study of ten languages found that the average gap between one speaker stopping and the next starting was about 208 ms. The median gap was 100 ms and the most common was 0 ms (Stivers et al., PNAS, 2009). Language averages ranged from 7 ms in Japanese to 469 ms in Danish, so listeners everywhere notice pauses measured in fractions of a second.

The phone network has its own budget for delay. ITU-T G.114, the international recommendation on one-way transmission time, says most applications feel essentially transparent when mouth-to-ear delay stays below 150 ms. It also says 400 ms one-way should not be exceeded in general network planning (ITU-T G.114). Those figures cover the line alone. An AI agent adds its own listening, thinking and speaking time on top.

No standard sets a target for AI agents, and published targets vary widely. Retell's guide aims for under 800 ms end to end (Retell). Murf, citing a benchmark from Daily, says natural conversation needs voice-to-voice times under 1,500 ms (Murf). BenchLM argues there is no universal cutoff and suggests a median target plus a ceiling for the p95, the time that 95% of replies stay under (BenchLM). The p95 matters because callers remember the slow replies, not the typical ones.

Where the time goes between a caller stopping and the agent replying

A voice agent's reply on a phone call is a relay of seven or eight steps, and most of the delay sits in the steps that wait rather than the ones that compute. The caller's audio crosses the phone network to the carrier's media server, which streams it to the agent. A voice activity detector (VAD) notices silence, the turn logic decides the caller has finished, and speech-to-text (STT) delivers the final transcript. The language model (LLM) starts writing, text-to-speech (TTS) speaks the first phrase, and the audio travels back the way it came.

Vendors split the budget differently. Vapi's playbook gives a voice agent "a total latency budget of roughly 1,000 milliseconds", split into 200 ms each for speech-to-text, text-to-speech and the network, and 400 ms for the LLM (Vapi playbook). Splits like this are useful for planning, but they leave out the wait to decide that the turn has ended, which on real calls is often the largest single piece.

The steps also overlap, which is why per-step figures never add up neatly to a total. A streaming pipeline can send the LLM a transcript before the turn is formally closed, and start speech synthesis on the first clause while the model is still writing the rest. Two latency terms come up throughout. Time to first token (TTFT) is how long the model takes to produce its first word, and time to first byte (TTFB) is how long synthesis takes to produce its first audio.

Deciding the caller has finished

An agent cannot reply until it is reasonably sure the caller has stopped, and that certainty costs time by design. Pipecat's Silero VAD defaults to 0.2 seconds of silence before it reports that speech has stopped (Pipecat Silero VAD docs). LiveKit's agents wait at least 0.5 seconds after the last speech before declaring the turn complete, and at most 3 seconds. With its audio turn detector the defaults become 0.3 and 2.5 seconds (LiveKit turn-handling options).

This is a tuning choice, not a fixed cost. Shorten the pause and the agent talks over callers who stop mid-sentence to think, read out a number or search for a word. Lengthen it and every reply feels slow. Semantic turn models, which judge from the words or the audio whether a sentence is finished, can shorten the pause safely, but they add their own compute to every turn. On our CPU servers that cost outweighed the saving, so we do not run one. The guide to barge-in on "haan" and "hmm" covers the opposite problem, an agent that gives up its turn too easily.

The language model's first token

Once the turn closes, the model's time to first token is usually the largest single computing step. It depends on the model's size, how long the prompt is, how busy the server is and how far away it sits. A large model hosted on another continent pays the distance on every turn, before it has written a word.

Reasoning models add a trap of their own: they can spend their first tokens thinking rather than speaking, which delays the first speakable word. A short prompt helps less than people expect when the server is far away or overloaded, so measure time to first token from the machine that runs your calls, not from your laptop.

Waiting for a phrase that can be spoken

Speech synthesis cannot start on a single token; it needs a phrase it can say naturally, so most pipelines wait for the end of the first sentence or clause. In Indian languages that wait can become the whole reply. Many sentence splitters only know the Latin full stop, question mark and exclamation mark. Pipecat's own fix for non-Latin scripts says that without it TTS text would "accumulate until flush" instead of going out sentence by sentence (pipecat PR 3617).

We tested our splitter across India's 22 scheduled languages on 2 October 2026. Hindi, Marathi, Bengali, Nepali, Odia, Punjabi and seven others end sentences with the danda (।), and Urdu, Kashmiri and Sindhi use the Arabic-script full stop (۔). Only five languages passed with the Latin set, and only because they borrow the Latin full stop. Our engine now splits on all of these, and sends the first clause of a reply once it has at least six words and reaches a comma.

Speech synthesis

Synthesis is usually the smallest piece of the reply, provided it streams. Read vendor figures carefully, though: one analysis notes that a widely quoted 75 ms TTS figure is model inference time only, excluding application and network latency (MajLeads). For comparison, Sonex Labs states that Pāṇini TTS starts streaming audio in under 200 ms (time to first byte); the Pāṇini text-to-speech API documents how that stream is consumed.

Flowchart of voice agent latency on Indian phone calls, from carrier and turn detection to LLM, TTS and back — The relay behind one voice agent reply on a phone call, from the caller's last word to the first audio they hear.

Why Indian phone calls add time that logs cannot see

The time your logs miss is often the time callers notice, because it sits on the phone network's side of the agent. Audio passes from the handset through the mobile or fixed network to the carrier's media server before your agent sees it, and the same path runs in reverse. Each end also holds audio in a jitter buffer, a short queue that smooths out packets arriving unevenly, which adds delay in exchange for clean sound.

A thread on LiveKit's agents repository titled "Latency doubles in telephony contexts" shows how large this can get (livekit/agents issue 3685). An Indian developer measured 2,000 to 3,000 ms on calls through a Twilio trunk left in its default US region, with LiveKit in India. Moving the trunk to Twilio's Singapore region saved about 500 ms, and the total stayed at 2,000 to 2,500 ms. A LiveKit maintainer had earlier put the LiveKit-to-Twilio leg at around 100 to 150 ms, with the rest between Twilio and the caller on the public phone network.

Distance is the cost you can remove. LiveKit's own guide for India recommends deploying agents in its Mumbai region, co-locating speech recognition, the model and synthesis, and using a regional SIP trunk provider (LiveKit, building voice agents for India). It reports about 1.67 seconds end to end for a Mumbai-hosted agent, about a second faster than an equivalent stack hosted in another region. Indian rules point the same way. Plivo says TRAI regulations require voice traffic for India calls to be anchored on servers in India (Plivo, calling in India), and the guide to putting an agent on a +91 number explains the settings.

Remote services inside the turn

Every service the agent calls during a turn adds its own round trip. A LiveKit user on India West saw the warning "turn detection transport latency is too high: 583ms", which a later comment traced to the network hop rather than to inference (livekit/agents issue 6415). The same applies to a database, a CRM lookup or a tool call: if it sits in another region, every turn that touches it pays for the distance.

The way output audio is paced matters too. A Pipecat issue reported gaps of 100 to 350 ms between words over WebSocket transports, traced to how the transport paced its writes (pipecat issue 3315). Choppy speech is not latency in the strict sense, but callers hear it as hesitation. The guide to silent calls on Twilio Media Streams covers the carrier buffer and clear messages behind it.

Carriers and call setup

Which Indian carrier is fastest is harder to answer than it looks. Each agent sets its own end-of-turn pause, speech recognition and model, and those settings usually move reply time more than the carrier does. A fair test runs the same agent on each carrier from ordinary phones. The comparison of Exotel, Plivo, Ozonetel and Vobiz sets out that test.

Call setup is a separate cost from reply time. Between the carrier opening its stream and the pipeline being ready to speak, the agent loads its configuration and connects to its models. On our own pipeline that takes about a second. That is why it pays to render the greeting before the call starts rather than synthesising it after the caller picks up.

Engineering choices that cut reply time

The biggest wins usually come from removing waits around the models rather than from faster models. Each choice below is easy to undo by accident, so measure before and after any change:

Put the carrier's media server, the agent and the models in the same Indian region, and keep tool calls and databases in that region too.
Stream every stage: interim transcripts into the model, tokens into the splitter and audio out of synthesis as it is made.
Split sentences on the danda (।) and the Arabic-script full stop (۔), and flush a first clause early so short replies start speaking at once.
Keep databases and fresh network connections off the per-turn path. Reuse warm connections, connect with a short timeout and retry fast, because one lost packet on a new connection can stall a reply for a second.
Send uncompressed audio between servers on a fast private link. In our tests, Opus encoding and decoding cost more time than the bandwidth it saved; on slow or metered links the answer may differ.
Tune the end-of-turn pause per agent and per language, and use filler audio to cover the model's think time only where it sounds natural.

How to measure latency on a real phone call

A latency figure means little until you know where its clock starts and stops. The span that matters to callers runs from the end of the caller's speech to the first audio they hear, on a real phone line. Server logs start when audio reaches the server and stop when the reply leaves it, so they always read lower than what callers hear.

The simplest fair test is to call your agent from an ordinary phone and record both sides, or to use a test line that records the call. Measure the gap between the end of your speech and the start of the agent's audio on the recording, over enough turns to report a median and a p95. Repeat it from a mobile network in India and with each language you serve, because turn pauses and sentence splitting behave differently in each.

Reading vendor latency claims

Published figures rarely measure the same span. A widely shared Show HN post on a voice agent built from scratch reported about 400 ms from the caller stopping to the agent's first syllable. It argued that voice is "a turn-taking problem, not a transcription problem" (Show HN). A commenter elsewhere on Hacker News put modern LLM voice agents on real calls at 500 ms to 1 second (Hacker News comment). LiveKit's 1.67 seconds for a Mumbai-hosted stack is end to end, while many TTS figures cover inference alone.

Sonex Labs, for its part, states that its voice agents answer in about 300 ms end to end, from when the caller stops speaking to when the agent starts. Whichever figure you are weighing, ask the same three questions: where the clock starts, where it stops, and whether it was taken on a phone line or in a browser. Then run the recorded-call test on the voice agents you are considering, with your own languages and callers.

Sources

Change log: 24 Oct 2026, Rewritten before publication: Sonex's own measured reply times and per-hop figures removed (owner decision, 24 Oct 2026); the budget is now explained with attributed public figures (Stivers 2009, ITU-T G.114, Vapi, LiveKit and Pipecat defaults, LiveKit's Mumbai results) and generic engineering

Why voice agents feel slower on Indian phone calls, and where the time goes