Cascaded STT → LLM → TTS or a speech-to-speech model? The trade-off for Indian languages

Two ways to build a voice agent

A cascaded voice agent chains three models. Speech-to-text (STT) turns the caller's audio into text, a large language model (LLM) writes the reply, and text-to-speech (TTS) speaks it. Each hand-off is text you can see and change.

A speech-to-speech model, sold as a "realtime" or "live" API, takes audio in and sends audio out from one model, with no text hand-off you control in between. Its appeal is that a single model can hear how something was said as well as what was said, and answer without waiting for each stage to finish.

The choice is not strictly either-or. A half-cascade lets a speech-to-speech or audio-understanding model hear the caller and decide the reply, while a separate TTS engine speaks it, so you keep natural listening but choose the voice, the language and how numbers are read. Platforms that sell a bring-your-own-TTS option are selling that design.

Language coverage decides first for Indian callers

For Indian callers, the first question is not speed or price but whether the model speaks your callers' language well. A cascade lets you pick an STT engine that is strong in Tamil and a TTS voice that sounds right in Marathi, independently. A speech-to-speech model gives you whatever its single model learned, in every language at once.

The published language lists are the quickest test. Microsoft, which offers OpenAI's realtime models on Azure, publishes the list for gpt-realtime and gpt-realtime-mini. It explains that the model was trained on 98 languages, but OpenAI lists only those below a 50% word error rate (Microsoft Learn, 24 September 2026). Word error rate is the share of words a recogniser gets wrong, so "listed" is a low bar. Google says its Live API supports 99 languages (Google Live API guide).

Neither list settles quality on a phone line. Phone audio is narrow-band 8 kHz, which most models were not trained on, as our guide to 8 kHz phone audio and 16 kHz models explains. Treat a listed language as a candidate to test on real calls, using a method like the one in our guide to testing a multilingual voice agent.

OpenAI has also released newer realtime models, gpt-realtime-2 and 2.1, at the same audio prices. Microsoft's page publishes a language list only for gpt-realtime and gpt-realtime-mini, so check the newer models' coverage before assuming it matches.

What speech-to-speech does better, and where the gap is smaller than it looks

A single model can respond to tone, hesitation and interruptions, because it hears the audio rather than a transcript of it. There is no wait for a final transcript before the reply starts, and no wait for a full sentence before speech starts, so the design removes hand-offs that a cascade has to engineer away.

The latency gap is smaller than the marketing suggests. A March 2026 tutorial by a team of 14 researchers measured about 702 ms audio-to-audio for a cloud speech-to-speech API. Their own streaming cascade, with full function calling, reached 755 ms to first audio (Qiu et al., arXiv 2603.05413). The two numbers are not measured identically, but they are in the same range. In a cascade, much of that time is the LLM's first token, which our article on LLM time to first token on phone calls looks at in detail.

Speech-to-speech also has documented rough edges with tools. A LiveKit issue opened in April 2025 collected about 28 comments from developers whose realtime agent sometimes skipped function calls and invented answers instead (livekit/agents issue 1953). It was closed in October 2025 without a fix, with the maintainer attributing it to model behaviour. That matters for any agent whose job is to book, look up or confirm something; cascades fail in their own ways here too, as our guide to voice agents that ignore their tool result shows.

Noise and accents are another weak point. τ-Voice, a March 2026 benchmark of full-duplex voice agents on 278 customer-service tasks, found they completed 31 to 51% of tasks on clean audio and 26 to 38% with background noise and diverse accents (Ray and colleagues, arXiv 2603.13686). A text-only reasoning model given the same tasks completed 85%. That baseline is not a cascaded voice agent, so the study does not settle cascade against speech-to-speech, but phone calls in India are rarely clean audio.

What each design costs per minute

Realtime APIs bill audio as tokens. OpenAI counts the caller's audio at 1 token per 100 ms and the model's spoken audio at 1 token per 50 ms, which is 600 and 1,200 tokens a minute. It also sends the entire conversation to the model for each response, so later turns cost more than early ones (OpenAI realtime costs guide).

At gpt-realtime's list price of $32 per million audio input tokens and $64 per million output (OpenAI pricing), take a minute with 30 seconds of caller audio and 30 seconds of agent audio. It costs 300 × $32/1M + 600 × $64/1M = $0.0096 + $0.0384 = $0.048, before any re-sent context. The same minute on gpt-realtime-mini ($10 and $20) is $0.003 + $0.012 = $0.015.

Re-sent context is the part that grows. Cached audio input is $0.40 per million tokens on gpt-realtime, so a well-cached call adds little, but OpenAI says caching is "best-effort and not guaranteed", and uncached re-sends are billed at the full $32 rate. A long call can therefore cost noticeably more per minute than the first minute suggests.

Text in the middle is where control lives

The text hand-off in a cascade is not just overhead. It is the one place you can inspect and change what the agent understood and what it is about to say. You can write ₹1,499 out as words in the caller's language before TTS reads it, or mask an Aadhaar or card number before it reaches a log. You can check a transcript against what a tool returned, and swap the LLM without touching the voice.

With a speech-to-speech model, those controls move into the prompt or disappear. Most realtime APIs can return a transcript of the caller, but it comes from a separate transcription model you configure, so it records the audio rather than what the main model understood. Number reading, pronunciation of names and the choice of voice are whatever the model does in that language, which matters for amounts in lakh and crore, as our guide to Hindi TTS for rupees, lakh and crore shows.

Debugging follows the same split. When a cascaded agent gives a wrong answer you can see whether STT misheard, the LLM misread, or TTS mispronounced. When a speech-to-speech agent does, you have audio in and audio out.

Self-hosting all but rules out speech-to-speech today

If you plan to run models on your own GPUs, for cost or data residency, the choice is mostly made for you. The March 2026 tutorial tried an open end-to-end speech model three ways. The vendor's cloud API worked but cannot be self-hosted, a local vLLM deployment served only the text-generation half, and a full local deployment took about 146 seconds, which the authors called far too slow for realtime.

The authors concluded that the streaming cascade "remains the practical architecture for self-hosted realtime voice agents". A cascade also lets you self-host in stages. Our self-hosting research found the order that pays first is TTS, then STT, with the LLM last or never, because TTS is the largest line in a typical API bill. Our guide to self-hosting voice AI costs works through the numbers. You cannot split a single speech-to-speech model that way.

Hybrids exist and can be the right answer. One pairs a realtime model for understanding with your own TTS; another uses a speech-to-speech model for English calls and a cascade for regional languages, routed by the caller's language at the start of the call. Deepgram's bring-your-own-TTS price shows the market already sells that split. If a cascade is where you land, Sonex AI voice agents are built that way, and the decision chart below is the test we would apply either way.