On this page
What published Indic TTS benchmarks measure
Three kinds of public evidence exist, and each answers a different question well. None was designed for the phone. That is not a flaw in the studies, which set out to answer other questions, but it means a team choosing a voice for calls has to fill the gap itself.
Listener preference at scale
The largest study is "Preferences of a Voice-First Nation", from researchers at AI4Bharat and IIT Madras, first posted in April 2026 and revised in June (arXiv 2604.21481). It evaluated seven TTS systems on more than 5,000 native and code-mixed sentences across 10 Indian languages. Over 1,900 native raters made more than 120,000 pairwise comparisons, which means hearing two versions of the same sentence and picking the better one.
Raters also judged six perceptual dimensions: intelligibility, expressiveness, voice quality, liveliness, noise and hallucinations, where a system says words that are not in the text. Pairwise judgements are more consistent across many raters than asking each person for a score out of five. The study is the best public guide to which voices people prefer when they listen carefully, but it does not time anything and does not test phone audio.
Accent, one sound at a time
PSP, posted in April 2026, measures accent rather than preference (arXiv 2604.25476). It splits accent into six dimensions. Four are specific sounds: retroflex collapse, where consonants made with the tongue curled back, such as ट, come out flat; aspiration, the breath that separates ख from क; vowel length; and the Tamil sound ழ. The other two compare the overall sound and rhythm of a system's output with native speech.
The pilot sets cover Hindi, Telugu and Tamil, and the systems include commercial voices and the open Indic Parler-TTS. PSP is the right tool for asking whether a voice sounds Indian or foreign, and its author leaves a formal comparison with listener scores to a second version. It covers neither latency nor telephone audio.
Vendor rankings and what they disclose
The one ranking we found that times first audio is a post by Caller Digital, a voice AI company, dated 10 July 2026 (Caller Digital benchmark). It timed the first audio chunk through four systems' streaming interfaces from India-routed endpoints and reports Hindi medians between 180 and 280 ms. A panel of 12 native speakers per language rated the audio blind. The test set is available only on request, the post does not say it tested phone-line or 8 kHz audio, and the company routes its own calls across the four systems it ranks.
Other vendor comparisons are explicit that they measure nothing. Smallest AI's October 2026 comparison of six Hindi TTS interfaces describes documented coverage, not quality, and its author sells two of the six (Smallest AI comparison). We do not cite community "arena" leaderboards, because their rankings have not been independently checked.
Sonex Labs, which publishes this page, sells a TTS model, Pāṇini, with native Hindi voices, trained for 8 kHz phone audio. That stake is why this page sets out a method anyone can repeat, and ranks no system, ours included.
| Benchmark | What it measures | What it does not measure |
|---|---|---|
| Preferences of a Voice-First Nation (arXiv 2604.21481) | Listener preference: 120K+ pairwise votes, 10 languages, 7 systems | Latency, phone-band audio |
| PSP accent benchmark (arXiv 2604.25476) | Accent by phonological dimension, Hindi, Telugu, Tamil | Latency, phone-band audio, number reading |
| Caller Digital Indic TTS benchmark (July 2026) | Opinion scores, code-switching, names, median first audio from India-routed endpoints | Phone line, 8 kHz; test set not public |
| Smallest AI Hindi TTS comparison (October 2026) | Documented language coverage and features of six interfaces | Any measured latency or quality |
Why a phone line changes the result
Two things change between a benchmark server and a caller's ear: when the first audio arrives, and what it sounds like when it does. A third difference, how the system reads raw text, matters more on calls than anywhere else, because an agent's text is messier than a test sentence.
When the first audio arrives
Time to first audio is the time from sending text to a TTS service to receiving the first piece of playable sound. For a phone agent it matters more than total synthesis time, because a streaming system starts playback as soon as that first piece arrives. A caller hears silence until then.
Measured from a data centre, the figure covers only the TTS service. On a call, the request starts on your call server, crosses the network to the TTS provider and comes back, and the audio then crosses the phone network to the handset. Your server can time the first part, but the carrier's share in each direction never appears in your logs. Measuring voice agent latency on real Indian calls covers how to see the whole path.
What the 8 kHz phone band removes
Phone calls carry audio at 8,000 samples a second, and sampling at that rate can only represent frequencies up to 4 kHz, half the sample rate. Many TTS systems synthesise at 24 kHz, including the open models IndicF5 and Kokoro, so their output has to be downsampled before a caller hears it. Hissing sounds such as स and श lose some of their crispness, and a voice that wins a careful listening test at full quality can lose part of what made it win.
Carriers that stream call audio to your server keep the phone rate, typically as G.711 μ-law (a compressed 8-bit telephone encoding) or as 16-bit PCM, depending on the carrier. Why 8 kHz phone audio breaks 16 kHz speech models explains the formats and the resampling step. For a benchmark the rule is simple: convert every system's output with the same resampler, so that the differences you hear come from the voices and not from the conversion.
Raw numbers, dates and Hinglish
Numbers are where phone agents most often fail in practice. An agent's text arrives from a language model or a CRM as digits, rupee signs and dates, not as neatly spelt-out words, and the TTS system has to decide how to say them.
Open models show how badly this can go. In an issue opened in May 2026, a user reported that IndicF5 turned "Your OTP is 4582", and other numbers, dates and amounts, into garbled noise. Replies suggested expanding numbers into words before synthesis, and one user said doing that by hand fixed it (IndicF5 issue 31). A benchmark that sends only spelt-out words would never see this failure. How Hindi TTS should read rupees, lakh, crore and dates sets out the readings a caller expects, and Hinglish text to speech covers mixed Hindi and English.
A phone-line benchmark you can run yourself
The method below is deliberately plain, so that two teams testing different systems get numbers they can compare. It has four parts: the sentences, the timing, the phone path and the scoring.
Send the same sentences, unchanged
Every system gets identical input. The set should cover amounts and OTPs, dates, phone numbers, lakh and crore figures, Hinglish in both Devanagari and Roman script, Indian names and one long sentence that exposes pacing problems. Ask a native speaker to write each language's set, because a translated English test misses how people actually write.
Before testing anything, write down the reading you will count as right for each item. A date such as 03/11/2026 is 3 November to most Indian readers and 11 March to an American system, so a scorer who has not agreed the answer in advance cannot mark it fairly.
Time first audio from your call server
Run the timing from the machine, or at least the region, where your call server runs, against each system's streaming interface. Record the arrival of the first byte of audio, not the first byte of the response: a streamed WAV file starts with a header that carries no sound.
Run each sentence about 20 times per system, at three different times of day, and report the median and the 95th percentile, the time that 95 runs in 100 beat. A median alone hides the slow replies callers remember. Report cold starts separately rather than averaging them in, publish the dates of the runs, and rerun after any vendor update, because providers change their infrastructure without notice.
Play the audio over a real call
Convert each system's output to 8 kHz with the same resampler, encode it in the format your carrier expects, and play it over a real call on an Indian carrier to a handset. Record the audio at the far end. That recording, not the file you sent, is what you score, because it includes the codec, the network and the handset speaker.
One carrier and one handset are one phone path, not all of them. If your callers are mostly on mobile networks, test on mobile. If you can, repeat the far-end recording on a second carrier, and keep the recordings so that anyone who doubts a score can listen.
Score each item right or wrong
Give the far-end recordings to two native listeners per language, without telling them which system produced which clip. Each marks every item right or wrong for the number, date or word that was meant, and a third listener settles disagreements.
A plain right-or-wrong mark is harder to argue with than a mean opinion score, the average of 1-to-5 ratings defined for telephone quality testing in ITU-T P.800. Small panels give noisy opinion scores, and a 3 out of 5 does not say whether the voice misread an amount or simply sounded flat. You can also transcribe the far-end recordings with a speech-to-text system as a quick screen, but that measures the transcriber too, so use it to flag items for listeners rather than as a score.
What this method leaves out
This is not a naturalness or preference ranking; the pairwise study above does that far better than a small panel can. It does not test voice cloning, emotion or long-form narration. It also cannot tell you how a system behaves under heavy load, which needs a separate concurrency test on the hardware or plan you would actually use.
import json
import os
import statistics
import time
import requests
API_URL = "https://api.sonexlabs.com/v1/speech/stream"
HEADERS = {
"Authorization": f"Bearer {os.environ['SONEX_API_KEY']}",
"Content-Type": "application/json",
}
WAV_HEADER_BYTES = 44 # a streamed WAV starts with a header that holds no sound
def first_audio_ms(text: str, voice_id: str) -> float:
body = {"text": text, "voice_id": voice_id, "sample_rate": 8000}
t0 = time.perf_counter()
with requests.post(API_URL, headers=HEADERS, data=json.dumps(body), stream=True, timeout=30) as r:
r.raise_for_status()
received = 0
for chunk in r.iter_content(chunk_size=None):
received += len(chunk)
if received > WAV_HEADER_BYTES:
return (time.perf_counter() - t0) * 1000
raise RuntimeError("stream ended before any audio arrived")
def run(text: str, voice_id: str, n: int = 20) -> None:
times = sorted(first_audio_ms(text, voice_id) for _ in range(n))
p95 = times[min(n - 1, round(0.95 * (n - 1)))]
print(f"median {statistics.median(times):.0f} ms, p95 {p95:.0f} ms, n={n}")
if __name__ == "__main__":
run("आपका OTP 4582 है, किसी के साथ शेयर मत करना।", "REPLACE_WITH_VOICE_ID")
How fast are open Indic TTS models?
Developers have asked this in public for more than a year, and the threads are still open. On IndicF5, an issue opened in March 2025 asks why a 10-word sentence takes 2 to 3 seconds and whether it can get under 200 to 300 ms. Replies suggest checking that PyTorch is actually using the GPU and trying torch.compile, but none reports a measured result (IndicF5 issue 6).
On Indic Parler-TTS, one user reported 10 to 15 seconds for a 10 to 15 word sentence in February 2025, and another about 5 to 6 seconds for 10 tokens on a cloud GPU instance. A member of the AI4Bharat team suggested running the model with flash attention in streaming mode. Later commenters said they had tried the suggested optimisations or could not get them working, and nobody in the thread confirmed a speed-up (Indic Parler-TTS discussion 10).
These are single users with unstated settings, so treat them as questions, not results. They still matter for phone use, because a model that returns audio only once the whole sentence is finished has a time to first audio equal to its full generation time. We found no public measurement, with a method you can repeat, of an open Indic model reaching first audio in 300 ms. Choosing an open-source Hindi TTS model compares the main candidates on licence, size and hardware, which is where a self-hosted test starts.
The Hindi and Hinglish sentence set
These are ten Hindi and Hinglish test sentences, written the way an agent's text actually arrives: digits, rupee signs, dates and English words left as they are. Send each one unchanged to the system you want to test. The point is to hear whether the system expands them correctly, so do not pre-process them into words.
Agree the right readings first. The phone number should be read digit by digit or in the two groups shown, ₹2,35,000 is two lakh thirty-five thousand rupees, and 03/11/2026 is 3 November in the Indian day-month order. The three samples below are Pāṇini reading three of the sentences at full quality, before any phone path, so they show what the raw sentences sound like rather than a result. The same matrix extends to other languages in how to test a multilingual voice agent.
- OTP: आपका OTP 4582 है, किसी के साथ शेयर मत करना।
- Amount and date: आपकी EMI ₹1,499 है, 15 तारीख तक भर देना।
- Indian grouping: पूरा अमाउंट ₹2,35,000 है।
- Crore: लोन की लिमिट ₹1.2 करोड़ तक है।
- Phone number: हमारा नंबर 98765 43210 है।
- Numeric date: आपकी डिलीवरी 03/11/2026 को होगी।
- Hinglish, mixed script: आपका order कल तक deliver हो जाएगा।
- Hinglish, Roman script: Aapka order ship ho gaya hai, kal tak deliver ho jayega.
- Name: डॉक्टर Venkataraman से आपकी अपॉइंटमेंट गुरुवार को है।
- Long sentence: अगर आप आज शाम छह बजे तक पेमेंट कर देते हैं तो आपकी सर्विस बंद नहीं होगी और अगले महीने का बिल भी पहले जैसा ही आएगा।
आपका OTP 4582 है, किसी के साथ शेयर मत करना।
पूरा अमाउंट ₹2,35,000 है।
आपका order कल तक deliver हो जाएगा।
Sources
- Anand et al., Preferences of a Voice-First Nation: Large-Scale Pairwise Evaluation and Preference Analysis for TTS in Indian Languages (arXiv 2604.21481)Official
- Menta, PSP: An Interpretable Per-Dimension Accent Benchmark for Indic Text-to-Speech (arXiv 2604.25476)Official
- Caller Digital: Indic TTS benchmark, 10 July 2026Secondary
- Smallest AI: Hindi TTS APIs, 6 vendors compared, 12 October 2026Secondary
- AI4Bharat/IndicF5 issue 6: Taking 2-3 seconds for a sentence with 10 words?Secondary
- AI4Bharat/IndicF5 issue 31: number pronunciation causing gibberish noiseSecondary
- ai4bharat/indic-parler-tts discussion 10: Inference speedSecondary
- ai4bharat/IndicF5 model card (24 kHz output)Official
- hexgrad/Kokoro-82M model cardOfficial
- ITU-T P.800: Methods for subjective determination of transmission qualityOfficial
- ITU-T G.711: Pulse code modulation (PCM) of voice frequenciesOfficial
Change log: 24 Oct 2026, Rewritten as a method guide and prepared for publication: the promised Sonex benchmark results and our own latency figures were removed; added the Smallest AI comparison, the phone-path and scoring rationale, and corrected the description of carrier audio formats
