Choosing an LLM for a phone agent: time to first token, prompt size and tool results

Why the first token matters more than generation speed

Most phone agents today are cascaded: speech-to-text (STT) turns the caller's words into text, a large language model (LLM) writes the reply, and text-to-speech (TTS) speaks it. The three run as a relay, and each one waits for the one before it. The LLM is often the longest of these waits, because it has to read the whole conversation before it writes a word.

Time to first token (TTFT) is the gap between sending a request to the model and receiving the first piece of its reply. A token is the unit a model reads and writes, roughly a short word or part of one. On a call, TTFT matters more than how fast the model writes the rest, because voice frameworks stream the reply into TTS as soon as the first phrase is ready. Everything before that first phrase is silence on the line.

Voice frameworks often call the same measurement LLM TTFB (time to first byte), meaning the time to the first streamed chunk of the response. Some timers stop on any chunk, including empty ones that carry only the reply's role, so they can read slightly lower than the first visible word. Before comparing two numbers, check what each one starts and stops on.

How fast a phone agent's LLM needs to be

Published targets disagree mainly because they measure different things. Murf, citing Daily's open-source aiewf-eval benchmark, says natural conversation needs voice-to-voice responses under 1,500 ms, which it puts at roughly 700 ms of TTFT for the LLM (Murf, 10 July 2026). Retell sets a tighter budget for the whole reply, under 800 ms end to end including LLM inference, TTS and network (Retell, 22 October 2026).

Other guides refuse to give one number. BenchLM says "no universal cutoff exists" without knowing the call type and the callers, and recommends setting both a median target and a p95 ceiling for your own calls (BenchLM, 13 October 2026). The p95 is the time that 95 out of 100 replies beat, so it describes the slow replies a caller remembers rather than the typical one.

At the fast end, builders report far lower figures for the model alone. The author of a much-discussed sub-500 ms voice agent credited a hosted provider's roughly 80 ms TTFT as the single biggest win in his build (Show HN thread). That is one developer's account of one setup, but it shows how much of a reply's time the model can take or give back.

A workable rule of thumb, and it is no more than that, is a model TTFT median near 500 ms and a p95 under about one second. Measure both on real calls from the region your call server runs in. A model that misses it is not unusable, but every extra few hundred milliseconds at the model has to be won back somewhere else in the relay.

What drives time to first token

Several things decide how long a model takes to start answering, and most of them are settings or design choices rather than fixed properties of the model. Knowing which one dominates on your agent tells you where to spend effort.

How to measure it on your own stack

Published figures are a starting point, but the number that matters is the one your callers get, with your prompt, from your server. Measure from the machine your call server runs on, so the network to the provider is included. Use your real system prompt and a realistic conversation history, not a one-line test prompt, and time to the first chunk that carries text rather than to the first chunk of any kind.

Run each case many times and read both the median and the p95; a handful of runs will not show the tail. Time a short prompt and a long one side by side, as the sketch below does, to see how much prompt length costs on that provider. Then run a day of real or replayed calls and count errors and timeouts alongside the timings.

Be wary of comparisons across languages. Agents in different languages rarely carry the same prompt, so a gap between two languages can turn out to be a gap between two prompts, which is what we found when we looked at our own agents. Check prompt size per language before believing a language effect.

import statistics
import time

from openai import OpenAI  # pip install openai

client = OpenAI(base_url="https://api.sonexlabs.com/v1", api_key="YOUR_API_KEY")


def first_token_ms(messages: list[dict], model: str = "vyas") -> float | None:
    """Milliseconds from sending the request to the first piece of reply text."""
    start = time.perf_counter()
    stream = client.chat.completions.create(model=model, messages=messages, stream=True)
    for chunk in stream:
        if chunk.choices and chunk.choices[0].delta.content:
            stream.close()
            return (time.perf_counter() - start) * 1000
    return None


faq = "Q: What are your clinic hours? A: 9 am to 7 pm, Monday to Saturday.\n" * 200
for label, system in [("short prompt", "You are a clinic receptionist."), ("long prompt", faq)]:
    messages = [{"role": "system", "content": system}, {"role": "user", "content": "Kal subah slot milega?"}]
    runs = [ms for ms in (first_token_ms(messages) for _ in range(20)) if ms is not None]
    print(label, "median", round(statistics.median(runs)), "ms, max", round(max(runs)), "ms")

Tool results have to reach the model in a form it can read

A fast first token is wasted if the model answers from the wrong information, and on phone agents that information usually comes from a tool: a calendar, an order lookup or a CRM. Speed and tool use have to be judged separately, because a model can be quick and still misread what its tools returned.

One failure mode hits Indian languages hardest. Python's json.dumps escapes non-ASCII text by default, so a tool result in Hindi or Telugu can reach the model as \u codes instead of readable script. A fix merged into Pipecat on 1 October 2026 put the cost plainly: "the model has to decode it, and it costs more" (pipecat PR 5995). In its example, one short Hindi sentence took 104 tokens escaped against 19 readable, and those extra tokens lengthen prefill on every later turn.

Format can go wrong in other ways too. From 5 to 23 October 2026, a setting in our own pipeline made some tool answers arrive late, in a separate escaped message rather than as the tool's reply; we fixed it on 23 October. Any test that put tool results in the normal tool role during those weeks was measuring a different conversation from the live one.

The check is simple: log the exact request your framework sends to the model, not the one you think it sends. Our guide to voice agents that ignore their tool result covers the formats that work and how to test them.

Slow tails and errors count as latency

A model that errors, times out or stalls is the slowest model of all for the caller who meets it, so judge providers on the tail and the failure rate as well as the median. In the same Hacker News thread, one developer said a very fast provider's cheaper tiers were "just not reliable" (Hacker News comment). In our own trials of hosted models, a respectable median sometimes hid a tail several times longer, which a median-only comparison would have missed.

Some teams hide part of the model's wait with a short spoken filler, such as "hmm" or "okay", played while the reply is generated. That helps on slow turns, but it is a mask rather than a fix, and it has costs of its own that our article on filler audio for voice agents sets out.

How to choose an LLM for a phone agent

Choose on quality first, because speed cannot rescue a wrong answer. Shortlist the models that handle your callers' languages and code-mixing well, and that call your tools reliably; our guide to testing a multilingual voice agent describes how to check that before launch.

Then time each shortlisted model with your real prompt, from the region your call server runs in, and read the p95 as well as the median. Finally, run a day of real or replayed calls and count errors and timeouts alongside the timings. If you want to include Vyas in that comparison, the Vyas LLM API is OpenAI-compatible, so the sketch above works against it unchanged.

A cascaded agent is not the only design. Speech-to-speech models skip the text stages entirely and avoid some of these waits, at the cost of other trade-offs that our comparison of cascaded and speech-to-speech voice agents explains.