Why Whisper invents sentences on silent phone lines, and how to gate it

Why Whisper writes text when nobody is speaking

Whisper is a sequence-to-sequence model: it reads a window of audio and writes the most likely text for it. It has no reliable way to answer "nothing". When the window holds no speech, a short, common phrase can still be the most probable output, so the model writes one.

The best measurement of this comes from researchers at AGH University of Krakow, who fed Whisper large-v3 301,317 clips of non-speech audio. The clips ranged from street sounds and noise recordings to generated white noise, pink noise and silence. Whisper produced text for 121,378 of them, or 40.3% (Barański et al., ICASSP 2025). The model was set to English with default decoding, so the rates describe English output.

The invented text is not random. The same short phrases came back again and again, and the authors collected them into a "bag of hallucinations" that can be filtered out afterwards. The table shows the most frequent ones. A widely repeated explanation is that Whisper learned from captioned web video, where sign-offs such as "thanks for watching" sit over music and silence, but the paper does not test that cause.

Why phone calls are full of the audio that triggers it

On a call, the caller's channel is mostly not speech. The caller listens while the agent talks, pauses to find an order number, covers the phone to speak to someone else, or puts the agent on hold. Each of those moments sends hiss, room noise, a television or music to the transcriber, and a model that writes text for two in five non-speech clips will find plenty to write about.

Streaming makes it worse. Whisper does not stream natively: it works on 30-second windows, and the reference code pads or trims every input to fit one (openai/whisper). Streaming wrappers therefore cut the call into short chunks and send each one separately. A chunk holding two seconds of line noise becomes a full window of padding and noise, and every chunk is a fresh chance to hallucinate.

Phone audio also arrives at 8 kHz, while Whisper expects 16 kHz input, so the upper half of the frequency range the model learned from is empty. That is mainly an accuracy problem, covered in the guide to 8 kHz phone audio and 16 kHz speech models. It also means the model gets weaker evidence on every chunk, speech or not.

How phantom text derails a voice agent

A wrong word in a meeting transcript is an annoyance. In a voice agent, a transcript is an action: the pipeline treats a final transcript as the end of the caller's turn and asks the language model for a reply. The failure modes below follow from that design rather than from any single measured incident.

• False turns: "Thank you." arrives during a pause, and the agent answers "You're welcome, is there anything else?" to a caller who was still looking for their account number.
• Cut-offs: phantom words that arrive while the agent is speaking look like the caller interrupting, so barge-in logic stops the agent mid-sentence.
• Wrong tool calls: "Thank you, bye" is a plausible goodbye, and an agent with an end-call tool may hang up on a caller who is on hold.
• Language flips: an English stock phrase on a Hindi or Tamil call can push an agent that follows the caller's language into switching to English.
• Polluted records: the call transcript, the summary and any CRM note now contain words the caller never said.

Gate the audio before it reaches the model

Voice activity detection (VAD) is the gate that matters most. A VAD is a small, fast model that marks which parts of the audio contain speech, so only those parts go to the transcriber. In the 2025 study, running Silero VAD first cut detected hallucinations from 21.3% to 0.2% on speech mixed with non-speech audio. Word error rate on that test set fell from 104.8% to 8.0%; it can exceed 100% because inserted words count as errors. The older WebRTC VAD only brought it to 68.3%.

In faster-whisper, a popular reimplementation of Whisper, VAD is a flag, but the defaults are cautious (faster-whisper). The vad_filter option is off by default in WhisperModel.transcribe, though on by default in its batched pipeline. When it is on, it only removes silences longer than 2 seconds (min_silence_duration_ms of 2,000) and pads each speech segment by 400 ms. On a call you want much shorter gaps removed.

from faster_whisper import WhisperModel

# Seed this with the phrases you find in your own logs on silent audio.
# The English ones are from Baranski et al. (2025); with a pinned Indian
# language the stock phrases will be different, so collect your own.
STOCK_PHRASES = {"thank you", "thanks for watching", "thank you for watching",
                 "so", "you", "bye", "subtitles by the amara org community"}

def normalise(text: str) -> str:
    kept = "".join(c for c in text.lower() if c.isalnum() or c.isspace())
    return " ".join(kept.split())

model = WhisperModel("large-v3", device="cuda", compute_type="float16")

def transcribe_turn(path: str, language: str = "hi") -> tuple[str, bool]:
    """Return (text, trusted). Untrusted text must not end, interrupt or act."""
    segments, _info = model.transcribe(
        path,
        language=language,                 # pin it when you know it
        vad_filter=True,                   # Silero VAD drops non-speech first
        vad_parameters={"min_silence_duration_ms": 500, "speech_pad_ms": 200},
        condition_on_previous_text=False,  # one phantom phrase cannot seed the next
        beam_size=1,
        temperature=0.0,
    )
    texts, trusted = [], True
    for seg in segments:
        if seg.no_speech_prob > 0.5 or seg.compression_ratio > 2.4:
            continue                       # likely silence, or a repetition loop
        if normalise(seg.text) in STOCK_PHRASES:
            trusted = False                # keep it, but flag it
        texts.append(seg.text.strip())
    return " ".join(texts), trusted

text, trusted = transcribe_turn("caller_turn_16k.wav")
print(repr(text), "trusted" if trusted else "LOW TRUST: do not act on it alone")

Hold music needs its own gate

VAD handles silence and steady noise, but hold music is harder. Music with vocals contains speech-like sound, so treat any VAD as unproven on your own hold music until you have tested it. The 2025 study removed music from its non-speech set because its tags could not separate instrumental tracks from songs, so its numbers say nothing about music either way.

The cleanest signal is the call itself. When one side of a SIP call puts the other on hold, it re-negotiates the media stream, marking it sendonly, or inactive if it was previously receive-only (RFC 3264, section 8.4). Your telephony layer can pause transcription when it sees that change. If you pay for speech-to-text by the second, closing the stream during hold also stops billing for a silent stream.

Signalling does not catch a caller who puts the handset down next to a radio, or a business line that plays music without signalling hold. For those cases, run a sound-event classifier on the caller's channel. YAMNet, an open model from Google, labels overlapping 0.96-second windows of 16 kHz audio with 521 AudioSet classes, music among them (YAMNet). A run of "music" windows can pause the transcriber. Resume on the first clear speech, and have the agent say something like "I'm still here" rather than answering the music.

How to measure hallucination on your own phone lines

The published figures come from Whisper large-v3 set to English and from the 2023 Whisper API, not from phone audio. Your rates on an 8 kHz line in Hindi, Tamil or Telugu will differ, and the stock phrases a model produces in a pinned Indian language are likely to differ too. So measure before you trust any threshold, using calls you have consent to record and review.

Start with the caller's channel at moments when your recordings show no caller speech: pauses, the stretches while the agent talks, and holds. Transcribe them with your current settings, count how many come back with text, and list the phrases that repeat. That list is your starting blocklist, and the share with text is your baseline phantom rate.

Then add the gates one at a time on the same audio: VAD with a shorter minimum silence, condition_on_previous_text turned off, hold detection and the stock-phrase flag. Measure the other side too. Run the same settings on real caller speech, especially quiet and short replies, and check word error rate, because a gate that removes phantom text by dropping real words has only moved the problem. The guide to testing a multilingual voice agent covers building that kind of test set for each language.

Version-lock the model and its settings, and re-run the test whenever either changes, because thresholds tuned on one model version do not carry over.