How to test a multilingual voice agent before launch

Why each language needs its own tests

Each language fails in its own way. Hindi may read a date wrongly while Spanish handles it fine; Arabic may mishear a one-word "yes" that English never gets wrong. Translating an English test script does not cover this, because the hard cases differ by language: number formats, polite forms, names and the words people mix in from English.

The problem is mostly one of effort. A builder running a support bot in English, Spanish and French asked on Reddit how others automate this, because testing every language by hand after each update was exhausting (r/aiagents thread). The two useful replies described the two approaches that work. One team flagged low-confidence transcript segments, accents and long silences for a person to review. Another ran the same test flows in every language and checked where the Spanish or French behaviour drifted from the English.

The plan below combines both: a fixed matrix for every language, specific tests for switching between languages, a replay harness for the model's decisions, and a final round with native speakers on real phone lines. The chart shows the order; each step has its own section.

A test matrix for every language

The matrix is the same in every language, and only the test lines change. Each row targets a known weak point of voice agents. Numbers, dates and short answers are where most errors are found, because they are short, ambiguous and expensive to get wrong.

Write the lines in the register your callers actually use, and have a native speaker write them rather than translating from English. The same applies to the agent's own spoken lines. Keep one prompt for behaviour, such as which tools to call and when to transfer, but write what the agent says natively per language, because translated prompts tend to produce translated-sounding speech.

For each row, write five to ten lines per language and an expected outcome for each: the slot the agent should book, the digits it should read back, the language it should answer in. An expected outcome turns a vague listening session into a pass or fail you can track from one release to the next.

Testing language switching, and why phone lines make it harder

Language switching needs its own tests, because it fails in the gaps between components rather than inside any one of them. Speech-to-text can hear the right language while the language model replies in another, or the model can reply in a language the voice cannot speak. Three gaps account for most of the failures.

import os
import requests

API = "https://api.sonexlabs.com/v1"
AUTH = {"Authorization": f"Bearer {os.environ['SONEX_API_KEY']}"}

# (label, what the caller says, groups of words; each group needs one match)
CASES = [
    ("es", "Quiero cambiar mi cita para el martes a las tres y media.",
     [["martes"], ["tres", "3"]]),
    ("hi", "कल शाम साढ़े पाँच बजे का slot मिल जाएगा?",
     [["कल"], ["पाँच", "5"]]),
    ("ar", "إيه، تمام، باكر الساعة أربع العصر.",
     [["باكر"], ["أربع", "4"]]),
    ("en", "My number is nine eight four, double one, two zero.",
     [["nine", "9"], ["double", "11"]]),
]

session = requests.Session()
for label, text, groups in CASES:
    speech = session.post(f"{API}/speech", headers=AUTH, json={
        "text": text, "output_format": "wav", "sample_rate": 8000,
    }, timeout=60)
    speech.raise_for_status()

    result = session.post(f"{API}/transcribe",
                          headers={**AUTH, "Content-Type": "audio/wav"},
                          data=speech.content, timeout=60)
    result.raise_for_status()
    heard = result.json()["text"]

    missing = [g for g in groups if not any(w.lower() in heard.lower() for w in g)]
    status = "PASS" if not missing else f"FAIL, missing {missing}"
    print(f"{label}: {status} | heard: {heard}")

Replaying real moments from calls

The language model's decisions are best tested by replaying real moments from calls rather than by making new calls. At Sonex we pick moments from test calls where the agent had to decide something: offer a slot, read back a date, transfer the caller, answer in the caller's language. We freeze the conversation up to that point, send it to the model as it was, and grade the reply against what should have happened, such as the right tool called with the right date.

Two things make this work. First, run each moment many times, because the same prompt can pass on one run and fail on the next, and a handful of runs is too few to tell two prompts apart. Second, keep every moment as a regression test, so a prompt change that fixes Telugu dates is checked against Spanish times and Arabic yes-or-no answers before it ships. We also generate scenarios with their own data, such as a calendar per scenario, and grade each reply against that scenario's calendar.

Native speakers, real phone lines and consent scripts

Before launch, every language needs testing by native speakers calling from ordinary phones. Synthesised test lines are clean, evenly paced and free of background noise; real callers mumble, mix languages, speak over traffic and use words no script anticipated. Recruit at least two native testers per language, ideally of different ages and regions, and give them the matrix as tasks to complete rather than lines to read.

Ask testers to judge the voice as well as the outcome. A line can be correct and still sound stiff, too formal, or wrong for the region. Have a native speaker review every script line, every pre-recorded prompt and every audio sample before customers hear it.

Automated flows do not replace these testers. They catch regressions cheaply and should run on every change, but they use clean, scripted input. Native speakers find what scripts miss: regional words, stiff phrasing and callers who mix languages unpredictably.

After launch: watch each language separately

Launch one language at a time if you can, and keep watching per language after launch. Compare completed tasks, transfers to a person, hang-ups in the first ten seconds, cut-offs where the agent spoke before the caller finished, and repeated questions. A language whose numbers drift away from the others is usually the one with a problem nobody on the team can hear.

One phone number with language detection is usually simpler for callers than one number per language, and it leaves one agent to maintain. Separate numbers make sense when languages need different staff, hours or regulations, or when you advertise a dedicated line to one community. Our guide to a bilingual English and Spanish AI receptionist works through that choice for one pair.

If you are building on Sonex, AI voice agents record every call with its transcript, which is the raw material for the per-language comparison and for the next round of replay tests.