Why your Arabic voice bot hears English, and how to fix "نعم"

Why short Arabic answers are hard for speech recognition

A yes/no confirmation is the turn a voice agent's work hangs on: confirm the appointment, approve the payment, agree to the transfer. In Arabic the answer is often a single syllable, such as لا, إيه or إي, said quickly while the caller is already thinking about something else. A speech recogniser then has a fraction of a second of audio and no sentence around it to tell it which language, or which word, it heard.

Language matters more for Arabic than for most languages because Arabic callers switch so freely. A Gulf caller may answer "ok" one minute and "تمام" the next, and many recognisers decide the language of a clip before they decide its words. With so little audio, that first guess is easy to get wrong.

Three different failures follow. The recogniser can decide the audio was English and return an English word. It can recognise the Arabic but write it in Latin letters. Or the word never reaches the recogniser, because the agent's turn-taking rules, the logic that decides when the caller has started or finished speaking, discarded it as too short. Each needs a different fix, and the last is the easiest to miss.

What Azure and Whisper users report

The evidence for these failures is public, scattered across developer forums and issue trackers, and mostly unresolved. Three reports describe the patterns clearly.

Code-switching and dialect spelling make Arabic error rates look worse than they are

Gulf and Egyptian callers mix English into Arabic constantly, and benchmarks have started to measure it. A May 2026 study tested five commercial speech recognition services on code-switched speech, with 300 samples each of Egyptian Arabic with English, Saudi Arabic with English, Persian with English and German with English (Abdoli and others, 2026). The best system overall, ElevenLabs Scribe v2, had a word error rate (WER), the share of words wrongly transcribed, of 13.2%.

The authors' more useful point is about the metric. They say WER overstates the gaps between systems by about three times, because it counts a valid transliteration as an error. An English word written in Arabic script, or the reverse, is marked wrong even when a person would accept it.

Dialect spelling has the same effect. The author of gulf-whisper-srt, a Saudi subtitling tool, found Whisper writes Gulf speech in standard spelling, such as ثاني for the spoken تاني, and calls these "only spelling" errors that a person would accept (gulf-whisper-srt). Their figures come from one episode of one Saudi show, which they say tells you nothing about other dialects. For a yes/no turn the lesson is direct: judge the meaning of the reply, never an exact string match.

Let a one-word answer count as a turn

Start with turn-taking, because a word that is thrown away cannot be recognised. Voice agent frameworks filter short sounds so that a cough or a "hmm" does not interrupt the bot, and an Arabic "لا" can look exactly like one of those sounds.

In Pipecat, the MinWordsUserTurnStartStrategy requires a set number of words before a reply counts while the bot is speaking, and drops to one word once the bot has stopped (Pipecat source). Set it to two words, and a caller who says "لا" over the end of the bot's question is ignored. LiveKit Agents counts speech as an interruption only after 0.5 seconds by default, its min_duration setting (LiveKit turn handling options).

The practical rule is to end the question cleanly, stop speaking, and let one word be enough. If callers often answer before the question ends, shorten the question rather than lowering the threshold for everything else. The same tension, between ignoring backchannel noises and hearing real answers, is worked through for Hindi in why voice bots get interrupted by haan or hmm.

Accept every dialect's yes and no, and treat the rest as unclear

Once the reply reaches the recogniser, accept it in every form your callers use. Yes and no differ by dialect, like most everyday words, as the same sentence in ten Arabic varieties shows. The sharpest example is إيه: it means yes in the Gulf and the Levant but usually means "what?" in Egypt. Callers also answer in English, or in a mix ("ok تمام"), and phrases such as مو مشكلة (no problem) contain a negative word but mean yes.

Normalise the transcript before matching, so that small spelling differences do not decide the outcome. That means stripping tashkeel (the short-vowel marks) and tatweel (the stretching stroke), writing every alef form as plain ا, and lowercasing Latin text. Anything that is empty, contradictory or romanised should come back as unclear, never as a guess.

The table is a starting list, and the Python function below applies it. Both are illustrations written for this article, not lists measured on real calls, and a native speaker of each dialect you serve should correct them before they reach callers.

import re
import unicodedata

YES = {"نعم", "ايوه", "ايوا", "اه", "اي", "ايه", "اكيد", "تمام", "طيب", "ماشي", "صح", "بلى",
       "زين", "واخا", "باهي", "واه", "yes", "yeah", "yep", "ok", "okay", "sure", "aywa", "aiwa", "naam", "tamam"}
NO = {"لا", "لاء", "مو", "مش", "ابدا", "no", "nope", "la", "laa"}
# Phrases that contain a negative word but mean yes ("no problem").
YES_PHRASES = ("مو مشكلة", "مش مشكلة", "ما في مشكلة", "ولا يهمك")
# Tokens that mean different things to different callers: never act on them alone.
AMBIGUOUS = {"nah", "naah", "naahe"}  # Whisper's spelling of نعم, and English "nah" (no)


def normalise(text: str) -> str:
    text = unicodedata.normalize("NFKC", text).lower()
    text = "".join(ch for ch in text if unicodedata.category(ch) != "Mn")  # strip tashkeel
    text = text.replace("ـ", "")  # tatweel
    text = re.sub("[أإآ]", "ا", text).replace("ى", "ي")
    return " ".join(re.findall(r"[\w']+", text))


def yes_or_no(transcript: str, dialect: str = "gulf") -> str:
    """Return "yes", "no" or "unclear" for a caller's reply to a yes/no question."""
    yes_words = set(YES)
    if dialect == "egyptian":
        yes_words.discard("ايه")  # in Egyptian, ايه usually means "what?"
    text = normalise(transcript)
    said_yes = any(phrase in text for phrase in YES_PHRASES)
    for phrase in YES_PHRASES:
        text = text.replace(phrase, " ")
    tokens = text.split()
    if any(t in AMBIGUOUS for t in tokens):
        return "unclear"
    said_yes = said_yes or any(t in yes_words for t in tokens)
    said_no = any(t in NO for t in tokens)
    if said_yes == said_no:  # neither, or both ("لا لا، تمام")
        return "unclear"
    return "yes" if said_yes else "no"


for reply, dialect in [("نَعَم", "gulf"), ("إيه أكيد", "gulf"), ("إيه؟", "egyptian"),
                       ("أيوه", "egyptian"), ("لأ شكراً", "levantine"), ("Naah", "gulf"),
                       ("ok تمام", "najdi"), ("مو مشكلة", "gulf"), ("واخا", "moroccan"), ("لا لا، تمام", "gulf")]:
    print(f"{reply:12} {dialect:10} -> {yes_or_no(reply, dialect)}")

Re-ask instead of guessing

An unclear answer is not a failure if the agent handles it well. Ask again with a different shape: restate the one fact being confirmed ("يوم الخميس الساعة أربع، صح؟"), so the caller can answer with a single word or correct the detail. A second unclear reply is the moment to offer a keypress ("اضغط واحد للتأكيد واثنين للإلغاء"), a good fallback even though callers expect to answer by voice first.

For steps where a wrong answer is costly, such as approving a payment, some teams require both a spoken yes and a keypress. That doubles the effort for the caller, so keep it to the few confirmations that justify it.

Pass the full transcript to the language model as well, so it can read longer answers such as "إيه بس خلها الساعة خمس" (yes, but make it five). Give the model the question being answered and ask it to return yes, no or unclear, rather than a rewritten sentence, so its output can be checked by code in the same way as the word lists.

Test yes/no recognition on your own callers before launch

Published numbers will not tell you how your agent handles your callers, so build a small test of your own. Record consenting speakers from each dialect you serve answering your real confirmation questions, over a real phone line, with short replies, long ones and replies that start while the bot is still talking. Run them through the whole agent, turn-taking included, and score each reply by intent, yes, no or unclear, not by word error rate.

Repeat the test whenever you change the speech-to-text provider or the turn-taking settings, because either can move results more than any prompt change. The checklist for testing a multilingual voice agent covers the rest of the agent.

Sonex Labs has not published Arabic accuracy figures for Valmiki, its speech-to-text model, so this article describes a method, not a result. Valmiki's Arabic speech to text detects the spoken language automatically and takes Arabic as one language, with no dialect setting. That makes a per-dialect test like this one the only way to know how it, or any other recogniser, handles your callers.