Measure Voice AI Latency Across a Real Call

To measure voice AI latency usefully, begin with the experience you want to improve. The delay before a greeting, the pause after a caller finishes and the time spent waiting for a tool are different measurements. Combining them into one “latency” number makes comparisons difficult and can send optimisation in the wrong direction.

For ordinary conversation, the most relevant observation is usually the interval between the caller finishing an utterance and hearing the assistant begin a meaningful response. Internal timing helps explain that interval, but it does not automatically measure the full experience.

Draw a timeline for one turn

Record the caller's last relevant speech, the system's decision that the turn ended, the beginning of model work, the first generated audio and the first audible response where observable. Add a tool request and result if that turn needs external information.

Use one turn identifier or a carefully aligned timestamp trail. Do not combine the fastest transcription from one turn with the fastest synthesis from another and present the sum as a typical call.

LiveKit exposes per-turn and component timing through its observability documentation. Those fields help diagnose a pipeline, but their availability differs between a speech pipeline and a realtime model.

Keep first audio separate from playback

A text-to-speech provider's first audio byte arrives before that sound necessarily reaches the caller. Network delivery, buffering and the output route still matter. LiveKit's MetricsReport reference also explains that its default room-output playback timing does not include delivery to the client.

This means a dashboard labelled “playback” may still need careful interpretation. Read the metric definition and identify its endpoints before attaching a user-experience claim to it. A small number is only meaningful when everyone agrees what clock starts and stops.

Burki's latency discussion provides broader context. Treat a speed target as a goal to measure on the intended route, not a guarantee inherited from an article title.

Use a representative turn set

A hypothetical evaluation could include a short factual answer, a longer explanation, a correction, an interruption and a request requiring a supported lookup. Keep tool-dependent turns separate from answers based on already available information.

Record the model, voice, route, caller region, device and test time. Compare configurations under similar conditions and repeat enough cases to see whether the result is stable. There is no universal sample size that makes a small pilot representative of every caller.

Report the distribution rather than only the best turn. A median describes a typical observation in the sample; a high percentile or a list of slow cases shows the tail. With a very small sample, raw observations are often clearer than a precise-looking percentile.

Be careful when comparing clocks from different machines. Unaligned timestamps can create an apparent delay or even a negative interval. Prefer measurements captured on one clock where possible, and document any synchronisation or alignment method used to combine server events with recorded audio.

Listen while reading the timing

A faster response can be worse if the assistant cuts off a caller who was thinking. Similarly, a short acknowledgement followed by a long unexplained pause may improve “first sound” timing without improving useful response time.

Mark premature replies, false interruptions and filler separately. If the assistant says “Let me check” quickly, measure the later arrival of the actual answer too. The caller's perception depends on whether the waiting is understandable and the result is useful.

Change the demonstrated bottleneck

If most delay occurs before turn completion, changing the speaking voice may do little. If a lookup dominates, shortening the model prompt may not resolve the pause. Use the timeline to choose one change, then repeat the same cases and check quality alongside speed.

Burki's TTS comparison can help with voice-provider evaluation after the relevant component is identified. Start with one real call and one well-defined interval. A measurement tied to an audible turn is more actionable than a headline benchmark without a route or method.