A latency budget for voice agents, millisecond by millisecond

People are very good at hearing a machine. Not because of the voice, which is now excellent, but because of the timing. A human takes a beat before replying. An agent that takes three beats sounds like it is on hold. An agent that takes none sounds like it is not listening. Getting the pause right is the hardest engineering in a voice agent, and it is almost entirely a latency problem.

This is the budget we build to. The numbers are targets, not promises: every line and every language moves them.

The turn#

A single turn is: the caller stops speaking, the agent starts speaking. Everything below happens in that gap.

Add those up and a good turn lands somewhere between half a second and a second after the caller stops. That is the whole budget. Every design decision below is about protecting it.

Endpointing is the biggest lever#

Most teams tune the model and ignore endpointing, which is backwards. Endpointing is the decision "the caller has finished". Make it too eager and the agent interrupts people mid-thought. Make it too patient and every turn carries a fat, dead pause.

Pure silence detection is the baseline: wait for N milliseconds of quiet. It is also the reason so many agents feel slow, because N has to be long enough to cover a thinking pause, and thinking pauses are long.

The better approach combines signals:

• Acoustic silence, short, as the trigger.
• The partial transcript, to ask whether the sentence looks complete. "I want to move my appointment to" is not finished. "I want to move my appointment to Tuesday" probably is.
• Prosody, where the ASR exposes it. Falling pitch at the end of a phrase is a strong signal in most languages.

With those together, the wait after a complete sentence can be short, and the wait after a dangling one can be long, which is exactly how people do it.

Stream everything, start early#

The second lever is to never wait for a whole anything.

• ASR emits partials continuously, so by the time the caller stops, most of the transcript is already there.
• The model streams tokens, and TTS starts on the first sentence boundary, not on the full reply.
• TTS streams audio chunks, and playback begins on the first chunk.

Done properly, the caller hears the agent begin its reply while the model is still writing the end of it.

Speculate on the partial, cancel on the final#

The trick that buys the most is to start reasoning before endpointing fires, on the best current partial transcript, and to throw the result away if the final transcript changes it. Most of the time the partial was right and the reply is already in flight when the caller stops. When it was wrong, you cancel and start again, and you have lost nothing but some compute.

Cancellation has to be a first-class operation everywhere in the pipeline. This is the shape of the turn loop we use, in TypeScript, with an AbortController per attempt.

type Turn = { controller: AbortController; startedAt: number };
let current: Turn | null = null;
 
function cancelCurrent(reason: string) {
  if (!current) return;
  current.controller.abort(reason);
  tts.flush();          // drop any queued audio, stop playback within ~100 ms
  current = null;
}
 
asr.on("partial", (text, { looksComplete }) => {
  // Speculate: start the reply on a complete-looking partial, before endpointing fires.
  if (looksComplete && !current) {
    const controller = new AbortController();
    current = { controller, startedAt: performance.now() };
    void reply(text, controller.signal);
  }
});
 
asr.on("final", (text) => {
  // The final transcript disagrees with what we speculated on: start over.
  if (current && !sameIntent(text, current)) {
    cancelCurrent("final transcript changed");
    const controller = new AbortController();
    current = { controller, startedAt: performance.now() };
    void reply(text, controller.signal);
  }
});
 
asr.on("speech-start", () => {
  // Barge-in: the caller started talking while we were. Stop immediately and listen.
  if (tts.isPlaying()) cancelCurrent("barge-in");
});
 
async function reply(text: string, signal: AbortSignal) {
  const stream = await llm.stream({ input: text, signal });
  for await (const sentence of sentences(stream, signal)) {
    if (signal.aborted) return;
    await tts.speak(sentence, { signal });   // first audio ~100 ms after first sentence
  }
  current = null;
}

The details that matter: flush() on the TTS side must be fast, because a barge-in that takes half a second to stop the agent's voice feels rude. And sameIntent should be cheap and forgiving, otherwise you cancel replies that were fine.

Barge-in#

Callers interrupt. They do it more with agents than with people, because they have learned that phone systems need to be talked over. The agent must stop within about a hundred milliseconds of the caller starting to speak, and it must not lose the caller's words while it stops.

That means echo cancellation has to be solid, because the agent's own voice coming back down the line looks exactly like a barge-in to a naive detector. We treat the first few hundred milliseconds after the agent starts speaking as a guarded window with a higher threshold, and we always keep the ASR running through playback.

Fillers, used sparingly#

When the budget is blown, a short acknowledgement ("Sure, one moment") buys time and keeps the caller informed. It is a good tool and an easy one to overuse. An agent that says "let me check that" before every reply sounds like it is stalling, because it is. We trigger fillers only when the expected reasoning time is above a threshold, such as a tool call that has to hit a slow system.

Measure from the audio, not from the logs#

Application logs will tell you what your code thinks happened. The line tells you what the caller heard. We compute turn latency from the recording itself: time from the end of the caller's last speech to the first agent audio, per turn. The script is short.

import numpy as np, soundfile as sf
 
def speech_segments(x, sr, thresh_db=-40, min_gap=0.25):
    frame = int(sr * 0.02)
    energy = 20 * np.log10(np.array([np.sqrt(np.mean(x[i:i+frame]**2)) + 1e-9
                                     for i in range(0, len(x) - frame, frame)]))
    on = energy > thresh_db
    segs, start = [], None
    for i, v in enumerate(on):
        t = i * 0.02
        if v and start is None: start = t
        if not v and start is not None and (t - start) > 0:
            if segs and start - segs[-1][1] < min_gap: segs[-1] = (segs[-1][0], t)
            else: segs.append((start, t))
            start = None
    return segs
 
caller, sr = sf.read("call.caller.wav")   # one channel per party
agent, _ = sf.read("call.agent.wav")
c, a = speech_segments(caller, sr), speech_segments(agent, sr)
for c_start, c_end in c:
    nxt = next((s for s, _ in a if s > c_end), None)
    if nxt is not None:
        print(f"turn latency {1000*(nxt - c_end):.0f} ms")

Plot the distribution per deployment and watch the p90, not the average. The average is fine on almost every system. The p90 is what callers remember.

Speech-to-speech models#

Models that go straight from audio to audio collapse most of the table above into one stage, and their turn latency can be excellent. We use them where the conversation is simple. The trade is control: with a single model doing everything, it is harder to insert tools, guardrails and business rules in the middle of a turn. Our production pattern is a hybrid, with a streaming pipeline for anything that touches a system of record and speech-to-speech for the conversational glue around it.

Whatever the architecture, the budget is the same, and the caller's ear is the only judge that counts.