Your voice agent ignored the answer your tool returned: tool-result formats that work

Where a model expects to find a tool's answer

Tool calling, also called function calling, lets a language model ask the application to run something, such as a calendar check, and then read the result before it replies. Chat models are trained on a fixed conversation shape for this. The assistant message names the tool and gives the call an ID. The next message has the tool role, carries the same ID and holds the result.

A model learns to look in that slot. When the slot holds a placeholder and the result arrives later under another role, the model has to infer that the two belong together, and in the middle of a live turn it often does not. The same thing explains an agent that calls a tool again instead of using its answer: from the model's point of view, a slot that holds a placeholder, an error or nothing readable means the call never finished.

Some roles make this worse. In the chat formats that have it, the developer role carries the app builder's instructions, much like a system prompt, so a result delivered as a developer message looks like guidance rather than data. Some serving stacks also map an unfamiliar role onto another one or render it differently in the model's template; we infer that this can bury a result further, though we have no public source for it.

• Role and pairing: the answer is in a tool-role message whose tool_call_id matches the assistant's call.
• Placement: the answer replaces any placeholder in that slot, rather than arriving later under another role.
• Encoding: serialised once, with non-Latin scripts readable; no \u escapes and no JSON inside a JSON string.
• Content: a short payload with a literal outcome word, such as produced, running or failed, that prompts can branch on.
• Waiting: while a tool runs, the slot says plainly that the result is unknown.
• Failure: a tool that never answers produces a message telling the model to say so and check again.

How a safe setting broke tool answers on our calls

Our own case shows how easily this happens. On 5 October 2026, a caller spoke over our agent's "one minute please" while it was booking an appointment. The framework cancelled the booking tool, the calendar made the booking anyway, and the model, told nothing, said nothing had been booked. The caller rang back and was booked twice.

So we registered our tools so that an interruption would never cancel them. In Pipecat, the open-source voice framework our engine runs on, that is cancel_on_interruption=False. In Pipecat 1.1.0 the setting turns a tool into an "async" tool, and the result changes shape. The tool's own reply becomes a note that the task is running and that results will arrive later as developer messages.

When the answer came, it was appended at the end of the conversation as a separate developer message. Our handlers returned JSON strings, which were serialised again inside another JSON object, so every Telugu letter arrived as an escaped code. The model treated the message as noise, and on 13 October an agent read a Telugu calendar answer for the 15th as the 13th. The excerpt at the end of this section shows one answer as the model received it before and after the fix.

Pipecat's current documentation describes a refinement: an async result that arrives before the conversation has moved on now "settles in place as an ordinary tool result" (Pipecat function calling docs). A result that arrives later still comes in as a developer message. Check the behaviour of the version you run rather than relying on either description.

Before (5 to 23 October 2026)

role: tool   tool_call_id: call_1
content: {"type": "async_tool", "status": "running", "tool_call_id": "call_1", "description": "An asynchronous task associated with this tool_call_id has started running. Expect results to arrive later as developer messages ..."}

... the caller's next turn ...

role: developer
content: {"type": "async_tool", "tool_call_id": "call_1", "status": "finished", "description": "This is the final result for the asynchronous task ...", "result": "\"{\\\"outcome\\\": \\\"produced\\\", \\\"day\\\": \\\"శుక్రవారం\\\", \\\"free\\\": [\\\"15:00\\\", \\\"15:30\\\"]}\""}

After (from 23 October 2026)

role: tool   tool_call_id: call_1
content: {"outcome": "produced", "day": "శుక్రవారం", "free": ["15:00", "15:30"]}

The fix: each answer in its own tool reply, readable

We kept tools safe from interruption and changed only where the answer goes. A custom assistant aggregator, the Pipecat component that writes the model's side of the conversation, now puts each tool's answer into that tool's own reply, replacing the placeholder. Results that are not already strings are serialised once, with ensure_ascii=False, so Telugu stays Telugu. Nothing is abandoned: a booking still finishes whatever the caller says.

The placeholder was rewritten to be useful while it is there. Instead of telling the model to expect developer messages, it says plainly that the check has not answered yet, so the model must not state anything the answer would tell it. A second component holds the caller's turn back while any tool is still running. When the answer lands, Pipecat runs the model again as it always does after a tool, and the model answers the caller's words and the result together.

A hold needs a way out. If a tool runs past 30 seconds, the turn goes through anyway and the incident is logged as an error, so a hung tool cannot leave a call silent. Silence has other causes worth ruling out at the same time, such as a reasoning model that spends its whole output budget thinking.

Results improved, but more than one thing changed at once, so read them with care. On our booking-agent eval, the commit that shipped this fix moved the score from 545 to 691 of 800 scenarios; the same commit also added the hold and a prompt change. Cases where a tool gave no answer at all used to leave the model guessing. They passed 40 of 40 once the failure message told it to say it did not know and check again on the caller's next turn.

import json

from pipecat.processors.aggregators.llm_response_universal import LLMAssistantAggregator

TOOL_STILL_RUNNING = json.dumps({
    "outcome": "running",
    "message": ("This check has not answered yet, so you do not know its result. "
                "Do not say anything its answer would tell you."),
})


class ToolAnswerInPlaceAggregator(LLMAssistantAggregator):
    """Writes each tool's answer into that tool's own reply, readable."""

    async def _handle_function_call_in_progress(self, frame):
        await super()._handle_function_call_in_progress(frame)
        self._update_function_call_result(frame.function_name, frame.tool_call_id, TOOL_STILL_RUNNING)

    async def _handle_function_call_finished(self, frame, in_progress_frame):
        self._function_calls_in_progress.pop(frame.tool_call_id, None)
        result = frame.result
        if result is None or result == "":
            result = "COMPLETED"
        elif not isinstance(result, str):
            result = json.dumps(result, ensure_ascii=False)
        self._update_function_call_result(frame.function_name, frame.tool_call_id, result)

Evals must use the format the engine really sends

The worst part of the bug was that our evals looked fine throughout. An eval here is a set of scripted conversations replayed against the model and scored automatically. Our test cases wrote tool results the textbook way, as tool-role messages with readable content. From 5 October the live engine sent something else, so for 18 days the evals measured a conversation no caller was having.

We rebuilt the eval harness so that the message format is a choice, not an assumption. Each case can now be rendered three ways: the live format, the old async format, and a readable answer placed separately as a developer message. That let us reproduce the misread dates on the old format and confirm the fix on the new one. The reference format is now taken from the real request the engine builds, by running the same aggregator and request builder the calls use, rather than from fixtures typed by hand.

Two checks catch this class of bug early. Capture the exact request body your engine sends to the model on a real call, including roles, tool_call_id values and raw content strings. Diff it against your eval fixtures whenever you upgrade the framework or change tool settings. Then render that request through your serving engine's chat template, so you see the tokens the model sees: a role mapped somewhere unexpected, or a wall of \u escapes, is obvious there and invisible in a pretty-printed log.

The same discipline applies beyond tools. Our guide to testing a multilingual voice agent covers replaying calls in every language you serve. Escaped results also inflate every later prompt, and our guide to LLM time to first token on phone calls explains why a longer prompt costs time on each turn.

Where this leaves agents that act during a call

Tool calls are what turn a voice agent from a talking FAQ into something that books, checks and updates records, and Sonex agents support MCP and tool calling during a call for exactly that reason. The lesson from our incident is that the tool itself is rarely the weak point. The message that carries its answer back is, and it can change underneath you with a framework upgrade or a setting meant to fix something else.

If you are building AI voice agents that use tools on live calls, treat the request the model receives as an artefact to inspect, not an implementation detail. Log it, diff it, and test against it, and a tool result the model ignores becomes a bug you catch in review rather than one a caller finds. The Vyas LLM API uses the standard OpenAI-compatible tool format, so the same checks apply if you test against it.