Voice AI engineering

Voice agents fail on phone calls in ways a browser demo never shows: phone audio arrives at 8 kHz, the network adds delay your server logs cannot see, and the bot stops talking when a caller says "hmm". These guides come from running phone agents in production. Each starts with the symptom developers search for, explains the cause and shows the fix, with the numbers we measured where we have them.

Connecting the call comes first. Pipecat vs LiveKit for phone agents explains the deciding difference, which is whether calls reach your code as a carrier's WebSocket audio stream or over a SIP trunk. Putting an agent on a +91 number covers the Indian rule that call audio must stay on servers in India, and silence on Twilio Media Streams covers the only format Twilio will play: base64 μ-law at 8000 Hz on a bidirectional stream.

Then check the audio. 8 kHz phone audio in 16 kHz models fails without any error, because the model hears speech at double speed. Whisper hallucinating on silence turns line noise and hold music into words; in a 2025 study, 40.3% of non-speech clips came back as text. For Arabic agents, recognising yes and no is the turn that fails most.

For speed, latency on Indian phone calls breaks the reply down hop by hop: most of it goes on deciding the caller has finished and waiting for the model's first words. Time to first token explains what drives the model's share, chiefly prompt length. Filler audio covers the gap without making the agent faster, and backchannels such as haan and hmm explain why bots stop mid-sentence and how to ignore them without dropping short answers.

Two guides cover model behaviour on calls: empty replies from reasoning models, which usually mean hidden reasoning used up the output cap, and agents that ignore a tool result, usually because the result arrived under the wrong role or as escaped text the model skips.