Hinglish Speech to Text on 8 kHz Calls: Script and WER

On this page

Why Hindi speech-to-text accuracy figures disagree
Devanagari, Latin or mixed: what your STT returns for Hinglish
A normaliser that breaks Hindi words inflates WER
Phone audio, noise and pauses change the result
What a reproducible Hindi call test set needs
Sources

Why Hindi speech-to-text accuracy figures disagree

Word error rate (WER) is the share of words a system got wrong: substitutions, deletions and insertions divided by the number of words in the reference, the human transcript treated as correct. It sounds objective, but it depends on three choices that published figures rarely state: the audio tested, the script of the reference, and how both texts were cleaned before comparison. Change any one and the figure can move by more than the gap between vendors.

Read speech versus phone calls

Audio is the biggest of the three. AI4Bharat's Vistaar benchmark reports Hindi WER for several systems on several test sets. Its own fine-tuned model, IndicWhisper, scored 10.3% on Kathbath, a set of read speech, and 26.8% on Gramvaani. Gramvaani is spontaneous telephone speech in regional varieties of Hindi, and 60.87% of its training and development audio is sampled at 8 kHz (OpenSLR 118).

A commercial cloud service in the same table went from 14.3% to 59.9%. Vistaar is AI4Bharat's benchmark of its own model, and the commercial figures date from when it was published, so do not read it as today's vendor ranking. Read it as evidence that the test audio decides the number, often more than the system does.

Benchmarks built on phone audio

Phone-call benchmarks have started to appear. Voice of India, released in April 2026 by Josh Talks with AI4Bharat and IIT Madras, is built from 536 hours of unscripted telephone conversations in 15 Indian languages, Hindi among them (Voice of India). Its core evaluation set is private, so no provider can tune to it, and that also means you cannot rerun it on a system it does not list.

It also changes the scoring. Instead of one reference per clip, linguists build a lattice of valid variants, covering alternative spellings, scripts, the nukta dot and number formats, and a transcript is scored against the closest valid path. That is the problem the rest of this page is about: deciding which differences are errors and which are just spelling.

Hindi WER (%) by test set in AI4Bharat's Vistaar benchmark; lower is better
System	Kathbath	Gramvaani (phone speech)
IndicWhisper (AI4Bharat)	10.3	26.8
Nvidia-large	12.7	42.6
Azure STT	13.6	42.3
Google STT	14.3	59.9

Devanagari, Latin or mixed: what your STT returns for Hinglish

Hinglish speakers use English words such as "appointment" or "order" that can be written correctly in Devanagari or in Latin letters. Most speech models write Hindi in Devanagari, and the widely used general models have no "Hinglish" output setting, so the script you get depends on the model and on how you call it.

Asking Whisper for Roman script

A developer wanted romanised Hinglish such as "Pata hai sabse darawani" from Whisper large-v3-turbo. Setting the language to Hindi gave Devanagari, and setting it to English gave an English translation, "Do you know that". The only reply said Hinglish is not a supported language and suggested a Hinglish prompt, while doubting the results would be stable (faster-whisper issue 1246).

A second Stack Overflow question, asking how to turn Hindi audio into Roman letters, was answered with a pointer to translation, which changes the words rather than the script. Translation and transliteration are different operations: one says the same thing in English, the other writes the same Hindi words in another alphabet.

English output when you wanted Hindi

The opposite complaint is just as common. A Stack Overflow question with 4,265 views, "Whisper is returning the transcription in English instead of the native language", has no accepted answer (Stack Overflow 73929778). Its only answer suggests a model called base.hi, which Whisper does not ship: its only single-language models are the English .en variants.

Whisper also detects the language from the first 30 seconds of audio only, so a call that opens with an English greeting can be transcribed in the wrong language (faster-whisper issue 265). The guide to Whisper picking the wrong language on short clips covers that failure in detail.

Models trained to write Hinglish

Some open fine-tunes do target Roman-script output. Oriserve's Whisper-Hindi2Hinglish models, for example, are Whisper large-v3 derivatives trained on about 700 hours of noisy Indian-accented Hindi to write it in Roman letters (Whisper-Hindi2Hinglish-Apex model card). Its card reports WER against Hinglish transliterations of the original Hindi references, which is a sensible choice for that model and a good example of why its figures cannot be lined up against ones scored in Devanagari.

Separate recognition from script

The practical fix is to treat the two as separate decisions. Set the spoken language and the task, transcribe rather than translate, explicitly when your engine offers those settings, so recognition is right. Then transliterate the text afterwards if your system needs Latin script.

A language model reading the transcript copes with either script, but a CRM search, an analytics dashboard or a WER test needs one convention. Engines that detect the language for you, such as Valmiki speech to text for Hindi, which follows switches between languages mid-sentence, still leave you to decide which script your downstream systems expect.

Decision chart for a fair Hinglish speech to text WER test on Devanagari and Latin script output — Three checks before trusting a Hindi or Hinglish speech-to-text WER: who reads the output, the reference script, and the normaliser.

A normaliser that breaks Hindi words inflates WER

Before comparing texts, WER tools normalise them: they lowercase, strip punctuation and fold away differences that should not count. This is where Hindi gets hurt. Whisper's BasicTextNormalizer replaces every character in the Unicode "Mark" category with a space, and in Devanagari the vowel signs (matras) and the virama are marks, so "चाहिए" becomes three fragments, "च ह ए". An open pull request from September 2026 says WER computed on that output is effectively character-level and not comparable with other systems (openai/whisper pull request 2849).

The script below avoids that. It removes punctuation only, keeps vowel signs, and folds two spelling variations that are not recognition errors: the nukta (ज़ and ज) and chandrabindu versus anusvara (हाँ and हां). It scores one hypothesis against an all-Devanagari reference and a mixed-script reference. The same transcript scores 17% against the first, for writing "appointment" in Latin, and 0% against the second. Neither is wrong; they answer different questions, so state which one you report.

Numbers and other valid variants

Numbers raise the same problem. A caller who says five hundred can be transcribed correctly as 500, as ५०० in Devanagari digits or as पांच सौ in words, and a single-reference WER counts two of the three as wrong. Decide one form for your references, convert the transcript to it before scoring, and report errors on numbers separately, because an agent that reads back OTPs depends on them more than on any other word.

There are three broad ways to stop valid variants counting as errors, and each has a cost. Two references per clip, as in the script below, is simple and transparent but covers only the variants you wrote down. A lattice of variants, as Voice of India uses, is more complete but needs linguists to build. Sarvam has proposed asking a language model to judge whether mismatched words are equivalent, keeping plain WER as a baseline alongside it (Sarvam on evaluating Indian-language ASR), which scales well but adds a second model whose judgement you have to trust.

import unicodedata


def normalise(text: str) -> list[str]:
    """Hindi-safe normaliser: removes punctuation, keeps vowel signs and viramas."""
    text = unicodedata.normalize("NFC", text).lower()
    text = text.replace("\u093c", "")         # nukta: ज़ -> ज, फ़ -> फ
    text = text.replace("\u0901", "\u0902")   # chandrabindu -> anusvara: हाँ -> हां
    return "".join(" " if unicodedata.category(c).startswith("P") else c for c in text).split()


def strip_marks(text: str) -> list[str]:
    """What a mark-stripping normaliser does: every vowel sign becomes a space."""
    return "".join(" " if unicodedata.category(c)[0] in "MP" else c for c in text.lower()).split()


def wer(ref: list[str], hyp: list[str]) -> float:
    """Word error rate: (substitutions + deletions + insertions) / reference words."""
    d = list(range(len(hyp) + 1))
    for i, r in enumerate(ref, 1):
        prev, d[0] = d[0], i
        for j, h in enumerate(hyp, 1):
            prev, d[j] = d[j], min(d[j] + 1, d[j - 1] + 1, prev + (r != h))
    return d[len(hyp)] / max(1, len(ref))


refs = {
    "Devanagari": "मुझे कल सुबह की अपॉइंटमेंट चाहिए।",
    "Mixed script": "मुझे कल सुबह की appointment चाहिए।",
}
hyp = "मुझे कल सुबह की appointment चाहिए"

for name, ref in refs.items():
    print(f"{name} reference: WER {wer(normalise(ref), normalise(hyp)):.0%}")
print(f"Mark-stripping normaliser splits 'चाहिए' into {strip_marks('चाहिए')}")

# Devanagari reference: WER 17%
# Mixed script reference: WER 0%
# Mark-stripping normaliser splits 'चाहिए' into ['च', 'ह', 'ए']

Standard library only. Swap in your own reference and STT output per clip, and sum errors and reference words across the test set rather than averaging per-clip WER.

Phone audio, noise and pauses change the result

Test on phone audio exactly as it arrives. Calls reach your server at 8,000 samples a second, as G.711 μ-law or 16-bit PCM depending on the carrier, and that rate keeps nothing above 4 kHz. Many speech models expect 16 kHz, so the audio has to be decoded and upsampled before they hear it. Upsampling restores the rate, not the lost detail.

Why 8 kHz phone audio breaks 16 kHz speech models covers the resampling traps. A test set recorded on good microphones and then downsampled is better than nothing, but it misses codec distortion and line noise, so label it as such when you report results.

Errors that WER does not show

Noise and pauses cause problems beyond word errors. On a noisy mobile line, background voices get transcribed, and some engines invent short words such as "okay" from noise or silence, which on a phone agent can open a false turn. Why Whisper writes text during silence on phone calls explains that failure and the voice activity detection that prevents it.

Timing matters too. Short or broken replies, which phone callers give constantly, can take an engine longer to finalise than full sentences, and a WER test will not reveal that. Time how long each final transcript arrives after the speaker stops, on the same clips you score, and report the 95th percentile as well as the median.

How the stream is billed

Cost depends on billing as well as accuracy. Some engines bill every second a stream is open, including long silent pauses, while others bill only the audio you send or the speech they detect. Valmiki, for example, is billed by the second the stream is open. Phone calls contain a lot of silence, so how per-second STT billing treats silence can matter as much as a few points of WER.

What a reproducible Hindi call test set needs

A test set others can trust is built from consented calls and published with its method. The clips should be real 8 kHz call audio from people who agreed to be recorded for testing, such as staff and volunteers calling a test agent. Do not reuse customer calls collected for another purpose. From 13 May 2027, when the core duties of India's data protection rules start, a recording that can be linked to a person is personal data with a stated purpose (what the DPDP Act means for call recordings).

Each clip needs two human references, one in Devanagari and one in mixed script with English words in Latin. Native speakers write them after listening at least twice, and a third settles disagreements. Cover what phone agents actually hear: names, numbers and dates, yes and no in several forms, Hinglish sentences, short one-word replies, noisy backgrounds and long pauses in the middle of a sentence.

Report WER and character error rate (CER), which counts wrong characters instead of whole words and is gentler on spelling variants and long compound words. Score both against both references, and give the number of clips and minutes, the date of each run and the normaliser code. Keep a held-back portion that no vendor has seen. Publish results by category, not just one total, because a system that is excellent on sentences and poor on numbers is a bad choice for an agent that reads OTPs.

A neutral test set is more useful than a leaderboard. Engines are updated without notice, so a ranking goes stale within months, while a method and a set of clips can be rerun by anyone on any engine. How to test a multilingual voice agent extends the same approach to the rest of the agent, from greetings to the disclosure line.

Sources

Change log: 24 Oct 2026, Rewritten for publication without the planned Sonex test set: removed our own timing figures and the promise of results; added the Voice of India telephone benchmark, number variants, three ways of scoring valid variants, a Roman-script Hinglish model and the consent timing under the DPDP Rules; corrected the description of carrier audio formats and of the faster-whisper Hinglish thread

Hindi and Hinglish speech-to-text on 8 kHz calls: Devanagari vs Latin output and a fair WER test