How many voice calls fit on one 24 GB GPU? What sets the limit, and how to measure it

Why there is no single calls-per-GPU number

A cascaded voice agent runs three models, and each uses a GPU in a different rhythm. Speech-to-text (STT) works for the whole call, because it has to listen to every second of audio. Text-to-speech (TTS) works only while the agent speaks. The large language model (LLM) works in short bursts, for the second or so it takes to read the conversation and write each reply.

So "calls per GPU" has a separate answer for each model, and another when two or three of them share one card. A card that carries many STT streams may carry far fewer LLM conversations, and putting both on it lowers each figure, because they compete for the same memory and memory bandwidth.

The card matters too. The common 24 GB data-centre cards in voice work are the L4 and the A10, and the RTX 4090 is the usual consumer card. Many published TTS and LLM figures, though, come from 48 GB cards such as the L40S or 80 GB cards such as the H100. A figure from a larger card is an upper bound for a 24 GB one, not an estimate of it.

Sonex Labs sells hosted speech and language model APIs and runs its own models on GPUs, so it has a stake in how self-hosting looks. This article does not use Sonex's own capacity figures. It relies on public measurements, says where they come from, and shows how to measure your own.

Memory comes first: weights, the KV cache and what the engine reserves

On a 24 GB card, memory usually runs out before compute does. Everything has to fit at once: the weights of every model on the card, a cache for every conversation in progress, and the working memory the serving software sets aside for itself. When the total does not fit, the model either fails to load or starts pushing calls out of memory under load.

Batching decides whether extra calls share the card or queue

Once the models fit, the next question is whether the software serves several calls in the same pass over the weights. Generating each new token is limited by how fast weights move out of memory, not by arithmetic, so serving several conversations in one pass costs little more than serving one. NVIDIA explains that batching spreads the cost of loading the weights across requests, while each request still needs its own KV cache, which is why memory caps the batch.

Some setups do not batch at all. In livekit/agents issue 7764, opened on 12 October 2026, LiveKit's Hugging Face plugin served sessions on one worker one at a time, because of a lock on each loaded model (issue 7764). With a 0.5-billion-parameter model on an Apple M4, time to first token was 99 ms for one session alone and 1,568 ms for the third of three simultaneous sessions. A queue like that turns every extra call into dead air for the others. The Pipecat vs LiveKit comparison for phone agents covers the framework side of that choice.

Long prompts cause a quieter version of the same problem. Reading a long prompt, called prefill, is heavy work, and while it runs it can hold up token generation for every other call. vLLM's chunked prefill splits long prompts into pieces and mixes them with generation steps. Its guide notes that a smaller token budget per step favours steady generation, while a larger one favours a fast first token (vLLM optimisation guide). If a shared card serves both live calls and long batch jobs, cap the batch prompts or give the live calls priority.

import math

def llm_duty(turns_per_min, ttft_s, out_tokens, itl_s):
    """Share of each minute one call keeps the LLM busy."""
    return turns_per_min * (ttft_s + out_tokens * itl_s) / 60

def gpus_needed(peak_calls, sessions_per_gpu, headroom=0.2, spares=1):
    return math.ceil(peak_calls * (1 + headroom) / sessions_per_gpu) + spares

duty = llm_duty(turns_per_min=4, ttft_s=0.4, out_tokens=60, itl_s=0.01)
print(f"LLM duty per call: {duty:.1%}")                              # 6.7%
print(f"Ceiling at 50 requests in flight: {50 / duty:.0f} calls")    # 750
print("GPUs for 80 peak calls at 44 per GPU:", gpus_needed(80, 44))  # 4

Real-time factor: whether each stream keeps up with live audio

Speech models face a constraint that chat models do not: live audio arrives at its own pace. The real-time factor (RTF) is processing time divided by audio duration, so an RTF below 1 means a model processes audio faster than it is spoken. NVIDIA reports the inverse, RTFX, the seconds of audio handled per second of computation. A card that serves streams at the pace callers speak can only keep up while its total throughput at least matches the number of streams.

NVIDIA's Riva tables for the L4 show what happens at the limit (NVIDIA Riva ASR performance). Take Parakeet-CTC-1.1B multilingual code-switching, a model that follows callers who switch languages mid-sentence. With 160 ms audio chunks, 32 streams average 220 ms of latency, and throughput is 31.5 RTFX, matching the streams. At 48 streams throughput stalls at 39.5 RTFX, below the 48 needed, and average latency jumps to 3.6 seconds. The card has fallen behind, and every further second of audio joins a queue.

That pattern is the clearest signal in any load test: latency stays flat and then climbs steeply within a small change in load. Run up to that point and stop one step short of it. The last load before latency breaks your target is the stream count to plan with.

Why planning figures and vendor claims disagree

Beside the measured tables sit planning figures, and they vary widely. Softcery, a consultancy, models 20 to 50 TTS streams and 100 to 200 STT streams per A10G-class GPU, and labels them "pending a load test" (Softcery, July 2026). io.net's guide to running Whisper at scale advises one GPU per 20 to 40 concurrent streams, using its small or medium models for speed (io.net). Cerebrium, which sells GPU hosting, says one A10 on its own fleet handles roughly 160 to 180 STT connections and about 180 TTS conversations, without describing the test (Cerebrium, July 2026).

For a whole agent on one card the public guidance is much lower. Dograh's sizing table, which it calls "directional starting points", puts a 24 to 48 GB workstation GPU running an 8 to 11-billion-parameter LLM with streaming TTS at single-digit concurrent sessions (Dograh, sizing). Its Whisper guide warns that a clean single-stream test "can still fall apart at 10 concurrent connections" (Dograh, Whisper).

Part of the spread is real: lighter models and looser latency targets carry more streams. The rest is definition. A "stream" may mean a connection held open, audio processed in real time, or a request sent back to back, and the latency may be an average or a 95th percentile. Before comparing two figures, check the model, the card, the audio, the latency target and whether anything else shared the GPU.

How to measure concurrency on your own 24 GB card

Measure in the order the limits bite. Start with memory. Load every model you plan to put on the card, in the serving engine you will run, and note the memory each reports, including how many KV-cache tokens the LLM engine says it has room for. If the models do not fit together with a working margin, no load test will rescue the plan.

Then test each model alone with traffic that looks like your calls. Use recorded calls in your real language mix, not clean English audio, send audio at real-time pace, and make turns arrive in bursts the way callers' turns do. Raise concurrency step by step and record the 95th-percentile latency that matters for each model: final transcript time for STT, first audio chunk for TTS and time to first token for the LLM. For speech models, also check that throughput keeps pace with the number of streams.

The last load before any model breaks its target is that model's number. Then run all the models together on the same card, because shared memory bandwidth slows each of them, and expect the combined figure to be lower than any single one. Finally, size the fleet with headroom for peaks and at least one spare card, using the code earlier in this article. If the result looks expensive, the guide to what self-hosted voice AI really costs compares it with API prices.

A measured stream count is the input every cost comparison needs. The hourly prices for L4, L40S and H100 GPUs turn it into a cost per call-minute, and critical utilisation shows how busy the card must be to beat an API. Many teams keep a small self-hosted base and send traffic peaks to an API. For the LLM stage, a hosted option such as the Vyas LLM API for voice agents leaves capacity planning for that stage to the provider.