On this page
Whisper guesses the language once, from the start of the audio
When you do not pass a language, Whisper predicts one before it writes anything. The reference code pads or trims the audio to a 30-second window, computes its spectrogram, a picture of the sound's frequencies over time, and asks the model which language token is most likely (openai/whisper README). Everything Whisper then writes is conditioned on that token.
faster-whisper, a popular faster reimplementation, does the same. Its docstring says the language "will be detected in the first 30 seconds of audio", and its language_detection_segments setting defaults to 1, meaning one window (faster-whisper transcribe.py). With more segments, it stops at the first window whose top language beats language_detection_threshold (0.5 by default), and otherwise takes a majority vote of each window's top guess. That helps long files with misleading openings, but does nothing for a short phone turn, which has only one window.
The design works for a podcast and fails in two familiar ways. If the first 30 seconds are unrepresentative, the whole file is misread: in faster-whisper issue 265, an English video was detected as Hindi (faster-whisper issue 265). And if the whole clip is short, the window is mostly padding, so the guess rests on a second or two of sound. The second case is the one that hurts voice agents, because the turns that matter most on a call are short: "haan", "nahi", "sari", "na'am", a name, or a yes to "Shall I book it?".
Short answers carry too little sound to tell languages apart
A one-word answer often sounds alike across languages, so even a perfect detector would struggle. "Okay" is said the same way on Hindi, Tamil, Arabic and English calls. Spoken Hindi and Urdu share most everyday words, so "theek hai" gives a model nothing to separate them; the only difference you will see is the script that comes back, Devanagari or Urdu's Perso-Arabic script.
Forum threads show the pattern from the user's side. On the Hugging Face forum, a developer running detection on 3-second chunks found Whisper sometimes reported a language that was not in the file at all (Hugging Face forum). On the OpenAI community forum, a developer's Arabic "نعم" ("yes") kept coming back as the Latin-script "Naah" from whisper-large-v3-turbo. One reply said all models do much worse when the audio is a single word, and a later user reported fewer errors after adjusting voice activity detection, chunk sizes and a lower sampling temperature (OpenAI forum, Arabic "Naah").
The clips below are the kind of input that trips detection. Play the Hindi and Urdu ones back to back: they are the same words in the same voice, and only the expected script differs. Why Arabic speech-to-text mishears yes and no looks at the Arabic case in more depth.
हाँ जी
ठीक है
ٹھیک ہے
சரி
نعم
Okay.
Why a wrong guess looks like translation
Whisper has two tasks: transcribe, which writes speech in the language spoken, and translate, which writes it in English. When the language token is wrong and the task is transcribe, Whisper still understands the speech and writes its meaning in the language it picked, so the output reads like a translation into that language. A June 2024 thread on the OpenAI forum describes exactly this in a language-practice app, where accented learners were sometimes "translated" into the language Whisper misdetected; the thread has no reply (OpenAI forum, June 2024).
So passing task="transcribe" does not prevent it, because that is already the default; what prevents it is passing the language. Prompts do not either. A developer whose English audio came back in Portuguese, Russian or Bulgarian found that a "Do not translate" prompt changed nothing. A reply explained that Whisper was not trained to follow instructions: a prompt steers vocabulary and style, like the start of a transcript, rather than giving orders (OpenAI forum, October 2023).
In a voice agent, a wrong-language transcript causes more than a bad log line. An agent told to reply in the caller's language may switch languages mid-call, and a "yes" written as an unrelated word in another language may not be read as a yes at all.
Mixed speech such as Hinglish
Callers who mix languages are a different problem. Whisper writes each window under a single language token, so English words may come out in Devanagari, or Hindi may come out in English. In a faster-whisper thread, a contributor said Hinglish is not a supported language and suggested Hindi plus a Hinglish prompt, while doubting the result would be stable (faster-whisper issue 1246). Hinglish speech to text on 8 kHz calls covers what script to expect and how to score it.
Pin the language per caller, then per call
The reliable fix is to know the language before the audio arrives, and you usually do. The campaign was set up for Tamil Nadu, the caller chose Hindi in the IVR menu, the CRM record says Urdu, or the number belongs to a Gulf branch. Pass that language on every request and detection never runs.
Every interface takes it. On the openai-whisper command line it is --language Hindi or --language hi; in Python it is model.transcribe(path, language="hi"), and faster-whisper's transcribe takes the same language argument. OpenAI's hosted transcription interface accepts a language parameter as an ISO-639-1 code such as hi, which its documentation says improves accuracy and latency.
When you genuinely do not know, detect once on a turn long enough to judge, then lock the result for the rest of the call. Restricting the candidates helps too. Azure's speech service builds this in: you list up to four candidate languages for detection at the start of the audio, or up to ten for continuous detection, and it returns one of them. With Whisper you can do the same yourself by reading the detection probabilities and keeping only the languages your line serves, as the sketch below does.
What pinning and restricting cost
Each choice has a cost. Pinning breaks for the caller who answers a Hindi campaign in English, so let the agent switch when a long turn clearly arrives in another language. Restricting candidates means a language outside the list is always mislabelled, because the model must pick one of yours.
Phone audio adds one practical step. The whisper.load_audio function decodes and resamples any file to 16 kHz, but a live call reaches your server at 8 kHz. Resample each turn before detection; why 8 kHz phone audio breaks 16 kHz speech models explains what goes wrong if you do not.
import whisper
model = whisper.load_model("large-v3")
CANDIDATES = ["hi", "en", "ta"] # only the languages this line serves
MIN_SECONDS = 3.0 # shorter turns keep the call's language
MIN_SHARE = 0.8 # how sure we must be to switch
def pick_language(path: str, call_language: str) -> str:
audio = whisper.load_audio(path) # 16 kHz mono float32
if len(audio) / whisper.audio.SAMPLE_RATE < MIN_SECONDS:
return call_language # too short to judge
mel = whisper.log_mel_spectrogram(
whisper.pad_or_trim(audio), n_mels=model.dims.n_mels
).to(model.device)
_, probs = model.detect_language(mel)
scores = {lang: probs[lang] for lang in CANDIDATES}
best = max(scores, key=scores.get)
share = scores[best] / sum(scores.values())
return best if share >= MIN_SHARE else call_language
call_language = "hi" # from the campaign, IVR choice or CRM
lang = pick_language("turn.wav", call_language)
result = model.transcribe("turn.wav", language=lang)
print(lang, result["text"])
Testing one-word answers before launch
No public test we found measures how often Whisper mislabels one-word phone answers in Indian or Arabic languages, and the rate will depend on your callers, your lines and your model size. It is quick to measure yourself. Record yes, no, okay and a few names in every language you serve, spoken by consenting colleagues over a real phone line, and run each clip twice: once with detection, once with the language pinned.
For each run, count the transcripts that come back in the wrong language or the wrong script, and check whether your agent would still read each yes as a yes. Keep the clips, because the same set will tell you whether a model upgrade or a new threshold made things better or worse. How to test a multilingual voice agent puts these short answers into a fuller test matrix, and why Whisper writes text during silence on phone calls covers the related failure of words appearing from no speech at all.
Sonex sells speech to text too. Valmiki speech to text detects the spoken language automatically rather than taking a language setting. The same caution applies to its output on one-word answers: the language you already know about the caller is better evidence than any guess from a second of audio.
Sources
- openai/whisper README: 30-second windows, pad_or_trim and detect_languageOfficial
- SYSTRAN/faster-whisper transcribe.py: language, multilingual, language_detection_threshold and language_detection_segmentsOfficial
- faster-whisper issue 265: Improve language detection (English detected as Hindi)Official
- faster-whisper issue 1246: transcribing Indian English (Hinglish)Official
- OpenAI community: Whisper 3 detects incorrect language, translates instead of transcribes (June 2024)Secondary
- OpenAI community: How can I stop Whisper to translate my audio? (October 2023)Secondary
- OpenAI community: Arabic transcription in whisper-large-v3-turbo returns "Naah" (June 2025)Secondary
- Hugging Face forum: Language detection with Whisper (April 2023)Secondary
- Microsoft Learn: Implement language identification (candidate languages, at-start and continuous)Official
- OpenAI API reference: create transcription (language parameter)Official
Change log: 24 Oct 2026, Prepared for publication: removed the placeholder for our own short-utterance measurements and replaced it with a test readers can run; folded the steps, checklist, takeaways and FAQs into the body; added why wrong-language output looks like translation, why prompts do not stop it, faster-whisper's threshold and majority vote, how each interface takes a language, Azure's limit on mid-sentence switching and the 8 kHz resampling step
