For voice agents, time to first token (TTFT) has become the default metric for selecting an inference API. It is also the metric most likely to lead teams astray. TTFT marks the moment a language model begins generating output, but a text-to-speech model cannot synthesize audio until a complete clause or sentence arrives. The gap between these two events determines whether an agent feels conversational or sounds like it is interrupting. This article benchmarks every layer of the voice stack, from speech-to-text and LLM inference through text-to-speech and speech-to-speech models, to reveal what truly matters: time-to-first-sentence (TTFS).
Why TTFT Is the Right Entry Point and the Wrong Finish Line
A voice agent is essentially a latency budget with a language model inside it. Every stage in the pipeline consumes milliseconds that the end user can hear. Time to first token is the interval between sending an inference request and receiving the first token back. IBM defines it as the moment a system transitions from idle to visibly active. For chat applications, TTFT captures nearly the entire user experience. For voice, it is only one term in a much longer sum.
The mechanical reason is simple: a text-to-speech model cannot synthesize half a word. It needs a complete clause or sentence before it can produce audio. LiveKit, the voice agent platform, has coined the term time-to-first-sentence (TTFS) to describe this critical metric. In its Gemma 4 deployment post, LiveKit argues that TTFS is what users actually perceive during a conversation. This gives developers two knobs rather than one: TTFT controls when generation starts, and tokens per second controls how quickly the first sentence finishes. A provider that excels at one but lags on the other will not feel fast in practice.
The Latency Budget: What One Voice Turn Actually Costs
LiveKit breaks down the typical voice turn into four stages: speech-to-text (STT) at roughly 100–200ms, LLM inference at 300–500ms with streaming, text-to-speech (TTS) at 100–200ms, and network transport over WebRTC at 50–150ms. The practical end-to-end target ranges from 700ms to 1.2 seconds. Kwindla Hultman Kramer, co-creator of the Pipecat voice agent framework, advises targeting an 800ms median voice-to-voice latency, with 1,500ms acceptable for a proof of concept. His rough arithmetic divides that budget into four equal parts of about 200ms each: transport and media processing, STT plus phrase endpointing, LLM inference, and TTS synthesis.
Daily, the real-time video and audio company, supplies the human baseline through its work on the world’s fastest voice bot. Typical human response time in conversation is around 500ms. Pauses beyond 800ms begin to feel unnatural. Daily’s February 2026 voice-agent LLM benchmark translates this into a concrete requirement: natural conversation needs voice-to-voice latency under 1,500ms, which works out to roughly 700ms of TTFT budget for a text-mode LLM used inside a transcription-to-LLM-to-voice harness. That 700ms number is the bar against which every provider should be measured.
How to Read a TTFT Benchmark Without Being Misled
Before diving into the tables, five methodology facts change what the numbers actually mean.
1. Workload shape dominates. Artificial Analysis, a leading independent benchmark provider, changed its default workload in March 2026 to 10,000 input token prompts rather than 1,000. Longer prompts raise both TTFT and output speed. LiveKit argues this shift is closer to reality for voice agents, because production systems front-load policy instructions, persona descriptions, escalation rules, retrieved data, and tool schemas.
2. Server location is baked in. Artificial Analysis tests from a virtual machine in Google Cloud’s us-central1-acodecodecodecodecodecode zone. The site explicitly states that TTFT includes network latency and may advantage or disadvantage providers based on where they serve their endpoints.
3. Reasoning tokens count. In the Artificial Analysis definition, TTFT for a reasoning model is measured as the first reasoning token, not the first answer token. These are reported in separate columns.
4. Measure from the receiving side. Daily notes that model providers sometimes quote TTFT internal to their inference stacks. Daily itself measures from request send to first usable token off the API.
5. Runs are not repeatable. Daily is blunt: TTFT varies substantially between benchmark runs, and providers frequently change inference stacks and sometimes even weights without changing model names.
Layer 1: LLM Time to First Token
The following figures are from the Artificial Analysis API providers leaderboard, retrieved August 30, 2026. The “first chunk” column is TTFT. Workload is 10,000 input tokens, single prompt, median over 72 hours.
Lowest Measured First-Chunk Latency
| Provider | Model | TTFT | Output speed |
|---|---|---|---|
| Baseten | gpt-oss-120b (high) | 0.23s | 266 tok/s |
| Baseten | gpt-oss-120b (low) | 0.24s | 271 tok/s |
| DeepInfra | Nemotron 3 Ultra | 0.28s | 371 tok/s |
| Cohere | North Mini Code | 0.32s | 104 tok/s |
| Cohere | Command A+ | 0.40s | 239 tok/s |
| Baseten | Inkling Small | 0.42s | 337 tok/s |
| Modular | Gemma 4 31B (NVFP4) | 0.44s | 243 tok/s |
| Nebius | GLM-5.3-Flash | 0.46s | 206 tok/s |
| Fireworks | Nemotron 3.5 Lightning | 0.46s | 501 tok/s |
| Together AI | Kimi K2.7 Code | 0.47s | 245 tok/s |
| Cerebras | gpt-oss-120b (high) | 0.49s | 1,697 tok/s |
figurefigurefigurefigurefigurefigure
The Throughput Trap
Silicon vendors often optimize for a different metric than voice agents need. High token throughput does not guarantee low time-to-first-sentence if the initial token takes too long.
| Provider | Model | TTFT | Output speed |
|---|---|---|---|
| Cerebras | gpt-oss-120b (high) | 0.49s | 1,697 tok/s |
| Celeris | Celeris-1 | 0.62s | 1,612 tok/s |
| Cerebras | Gemma 4 31B | 0.53s | 1,351 tok/s |
| Groq | gpt-oss-20b (high) | 0.82s | 957 tok/s |
| SambaNova | gpt-oss-120b (high) | 0.92s | 706 tok/s |
| Groq | gpt-oss-120b (low) | 0.69s | 473 tok/s |
| Inception | Mercury 2 | 3.07s | 770 tok/s |
figurefigurefigurefigurefigurefigure
Mercury 2 is the clearest illustration. It is a diffusion-based language model and generates 770 tokens per second. Its first chunk, however, arrives at 3.07 seconds—more than four times the entire LLM budget for a natural conversation. Cerebras and Groq present a different case: their TTFT is respectable and their throughput is exceptional. For TTFS specifically, that combination is strong, because the sentence completes almost immediately after the first token lands.
Frontier and Proprietary Endpoints
| Provider | Model | TTFT | Output speed |
|---|---|---|---|
| Amazon Bedrock | GPT-5.6 Luna (non-reasoning) | 0.59s | 181 tok/s |
| Amazon Bedrock | GPT-5.6 Terra (non-reasoning) | 0.72s | 103 tok/s |
| OpenAI | GPT-5.6 Luna (non-reasoning) | 0.74s | 113 tok/s |
| Gemini 3.7 Flash (low), AI Studio | 0.84s | 315 tok/s | |
| Anthropic | Claude 4.5 Haiku (non-reasoning) | 0.84s | 82 tok/s |
| Amazon Bedrock | Nova Micro | 0.86s | 264 tok/s |
| Gemini 3.5 Flash (minimal), AI Studio | 0.90s | 202 tok/s | |
| OpenAI | GPT-5.6 Sol (non-reasoning) | 1.06s | 71 tok/s |
figurefigurefigurefigurefigurefigure
Notice the same model on different hosts. GPT-5.6 Luna non-reasoning measures 0.59s on Amazon Bedrock and 0.74s on OpenAI’s own API. Hosting and routing matter as much as the weights themselves.
The Vendor-Measured Outlier
LiveKit publishes its own TTFT figures for its inference product. Gemma 4 31B on LiveKit Inference measured 192ms, compared to Gemini 2.5 Flash at 911ms, GPT-5.5 at 966ms, GPT-4.1 at 1,006ms, and the same Gemma 4 31B via OpenRouter at 1,876ms. LiveKit is transparent about the mechanism: it runs Gemma behind SGLang with speculative decoding and deliberately under-packs each GPU to keep queueing delay low. A warm request, it says, starts returning tokens in around 100ms. The tradeoff is cost at $1.20 per 1M output tokens.
The same post reports TTFS across full conversations: 354ms for Gemma 4 31B on LiveKit, 1,034ms for Gemini 2.5 Flash, 1,088ms for GPT-4.1, 1,267ms for Gemini 3.0 Flash, and 1,404ms for GPT-5.5. Capability numbers accompany this: on IFBench, independently scored by Artificial Analysis, Gemma 4 31B scores 75.6% against GPT-5.5 at 75.9%, GPT-4.1 at 43%, and Gemini 2.5 Flash at 39%. On τ²-bench, GPT-5.5 leads at 93.9% with Gemma 4 31B at 76.9%.
Layer 2: Speech-to-Text and Turn Detection
For voice agents, STT latency is not simply transcription speed. It is how long after the user stops talking the pipeline knows the user has stopped talking. Artificial Analysis measures two things on its streaming STT leaderboard, both starting from a SileroVAD-detected end of speech: time to first partial transcript and time to final transcript. Its AA-WER Streaming index draws on roughly eight hours of audio, weighted AA-AgentTalk 50%, VoxPopuli 25%, and Earnings-22 25%.
Vendor-Published Latency Figures
| Model | Claim | Source type |
|---|---|---|
| Deepgram Flux | ~260ms p50 end-of-turn detection at defaults | Vendor docs |
| Deepgram Nova-3 | Sub-300ms streaming latency | Vendor docs |
| AssemblyAI Universal-Streaming | ~300ms immutable word emission | Vendor |
| Cartesia Ink-2 | 100ms transcript latency | Vendor |
| Speechmatics Voice SDK | 0.451 ± 0.022s end-of-speech to finals | Vendor internal tool |
figurefigurefigurefigurefigurefigure
Deepgram Flux is the most architecturally interesting entry. It folds end-of-turn detection into the recognition model rather than bolting a VAD on top. Deepgram states this can cut agent response latency by 200–600ms versus a traditional STT-plus-VAD pipeline. It exposes eot_thresholdcodecodecodecodecodecode (0.5–0.9), eager_eot_thresholdcodecodecodecodecodecode (0.3–0.9), and an EagerEndOfTurncodecodecodecodecodecode event that lets you start the LLM early. That last capability matters more than the raw number: if you can begin generation on an eager signal, you move LLM TTFT off the critical path entirely when the prediction is correct.
AssemblyAI Universal-Streaming inverts the usual partials-then-finals model by emitting immutable transcripts. AssemblyAI reported 307ms median word emission against 516ms for Deepgram Nova-3 in its own 2025 measurement. Its docs also recommend using unformatted transcripts for voice agents, since formatting arrives later and rarely changes LLM behavior.
Accuracy claims here are contested and vendor-published. AssemblyAI reports Universal-3.5 Pro Realtime at 6.99% WER on the open Pipecat voice-agent benchmark, ahead of Google Chirp3 at 9.04%, ElevenLabs Scribe v2 at 9.76%, and Deepgram Flux at 15.58%. Run your own tests before treating any of these as settled.
LiveKit also documents preemptive generation, which starts the LLM on a partial transcript. The caveat: if the reply must be regenerated after the final transcript arrives, you burn tokens and save nothing.
Layer 3: Text-to-Speech Time to First Audio
This is where vendor numbers diverge most sharply from what users actually experience.
ElevenLabs states Flash v2.5 delivers approximately 75ms. Its own docs qualify that carefully: 75ms refers to model inference time only. The company’s latency concepts page goes further, listing network round-trip at typically 20–200ms depending on geography, and noting that most audio players buffer before playback, with 500ms buffering being common. It also states that Eleven v3 is not built for real-time use and recommends Flash v2.5, Flash v2, or Multilingual v2 for its Agents Platform.
Cartesia states sub-90ms TTS and 100ms transcript latency for Sonic-3.6 and Ink-2. Coverage of the Sonic-3.6 release flagged both as vendor-stated model latency, not measured end-to-end round trips. Cartesia previously claimed 82ms end-to-end time-to-first-audio for Sonic 3.5. Sonic runs on state space models rather than transformers, which scale linearly rather than quadratically with sequence length.
On quality, the Artificial Analysis Provider Voice arena provides blind-listener Elo scores, retrieved August 30, 2026:
| Model | Elo | Price per 1M chars |
|---|---|---|
| Cartesia Sonic 3.6 | 1,288 | $49.00 |
| SpeechifyAI Simba 3.2 | 1,243 | $10.00 |
| Alibaba Qwen-Audio-3.0-TTS-Plus | 1,243 | $27.60 |
| Inworld Realtime TTS-2 Flash (preview) | 1,228 | $10.40 |
| BreezeBlue Breeze TTS 2 (open weights) | 1,220 | $34.00 |
| ElevenLabs v3 Conversational | 1,215 | $50.00 |
| Google Gemini 3.1 Flash TTS | 1,210 | $18.30 |
| ElevenLabs Flash v2.5 | 1,083 | $50.00 |
figurefigurefigurefigurefigurefigure
The gap between Sonic 3.6 at 1,288 and Flash v2.5 at 1,083 represents the quality cost of the low-latency tier that most agents actually run on.
Layer 4: Speech-to-Speech Time to First Audio
Speech-to-speech models collapse STT, LLM, and TTS into a single pass. Fewer round trips should mean lower latency. LiveKit cautions that realtime models are not guaranteed to be faster in every case, and that a well-tuned pipeline can be highly competitive.
The data supports that caution. From the Artificial Analysis speech-to-speech leaderboard, TTFA measured on Big Bench Audio, retrieved August 30, 2026:
| Model | TTFA | Speech reasoning | Task success | S2S Index |
|---|---|---|---|---|
| Deepslate Opal | 0.44s | 85% | — | — |
| Gemini 2.5 Flash Native Audio Dialog | 0.63s | 69% | — | — |
| Grok Voice Think Fast 2.0 High | 0.70s | 97% | 94.7% | 79.0% |
| Grok Voice Fast 1.0 | 0.78s | 93% | — | — |
| Qwen3.5 Omni Flash Realtime | 0.79s | 59% | 29.1% | — |
| OpenAI GPT-Realtime-1.5 | 0.81s | 81% | 85.1% | 70.3% |
| OpenAI GPT Realtime Mini (Oct ’25) | 0.81s | 64% | 79.6% | 56.8% |
| OpenAI GPT-Realtime-2.1 Mini Minimal | 0.85s | 63% | 76.7% | 52.8% |
| Google Gemini 3.1 Flash Live Minimal | 0.96s | 71% | 74.6% | 63.9% |
| OpenAI GPT-Realtime-2.1 Minimal | 0.97s | 87% | 89.4% | 70.3% |
| Amazon Nova 2.0 Sonic (Mar 2026) | 1.14s | 88% | 57.1% | — |
| OpenAI GPT-Realtime-2 (High) | 1.14s | 97% | 89.8% | 73.6% |
| OpenAI GPT-Realtime-2.1 High | 1.21s | 96% | 91.5% | 73.9% |
| Google Gemini 3.1 Flash Live High | 2.99s | 97% | 71.8% | 71.5% |
| OpenAI GPT-Realtime-2.1 Mini High | 4.28s | 75% | — | — |
figurefigurefigurefigurefigurefigure
Grok Voice Think Fast 2.0 High is the standout: 0.70s TTFA with 97% speech reasoning and 94.7% task success. The reasoning-effort penalty is clearly visible within single model families. Gemini 3.1 Flash Live moves from 0.96s to 2.99s between Minimal and High settings. OpenAI’s GPT-Realtime-2.1 moves from 0.97s to 1.21s, buying 2.1 percentage points of task success.
OpenAI shipped gpt-realtime-2.1codecodecodecodecodecode and gpt-realtime-2.1-minicodecodecodecodecodecode in early July 2026, stating that improved caching cut p95 latency by at least 25% across its Realtime voice models. Tail latency is what makes a phone agent feel broken, so that claim is more useful than a median improvement would be.
The Capability Gap
Daily’s benchmark quantifies why most production agents still use cascaded pipelines. On its aiwf_medium_contextcodecodecodecodecodecode test, GPT Realtime scored 86.7% against GPT-4.1 at 94.9%. Ultravox 0.7 was, in Daily’s assessment, the first speech-to-speech model to perform well on long multi-turn conversations, and it is open weights.
Artificial Analysis also benchmarks four vendor default cascaded systems: Deepgram Voice Agent (Nova-3 + GPT-4o Mini + Aura-2), ElevenLabs Agents (Scribe v2 Realtime + Gemini 2.5 Flash + Eleven Flash v2), Cartesia Line (Ink + Gemini 2.5 Flash + Sonic), and Inworld Realtime (Inworld STT 1 + Gemini 2.5 Flash + Inworld TTS 1.5 Mini). Three of the four run Gemini 2.5 Flash — a revealing consensus.
Reference Budgets
Assembled from the verified component figures above, these planning estimates provide a framework for architecture decisions.
Aggressive cascaded pipeline, US-hosted, colocated:
| Stage | Budget |
|---|---|
| Transport and media (WebRTC) | 50–150ms |
| STT + end-of-turn (Flux at defaults) | ~260ms |
| LLM first chunk (sub-0.5s tier) | 230–500ms |
| Sentence completion at 250+ tok/s | ~100ms |
| TTS first audio + network | 150–300ms |
| Total | ~790ms–1.3s |
figurefigurefigurefigurefigurefigure
That lands at or slightly above the 800ms target, which matches Kwindla’s framing that 800ms is tight but achievable.
Speech-to-speech, single model:
| Stage | Budget |
|---|---|
| Transport and media | 50–150ms |
| Model TTFA (minimal reasoning tier) | 700ms–1.0s |
| Total | ~750ms–1.15s |
figurefigurefigurefigurefigurefigure
Comparable latency, but with less observability and, per Daily’s benchmark, a measurable capability gap on tool calling and instruction following.
What to Do With This Data
Pick the metric your architecture is bounded by. If a TTS model sits downstream, optimize TTFS, not TTFT alone. That means you need to consider both TTFT and tokens per second together.
Colocate before you optimize models. LiveKit rates agent-model colocation as very high impact, above model choice. If you use SIP, keep the trunk geographically close too.
Cap reasoning effort explicitly. It is the largest single lever visible in the tables above, and it is a configuration flag on most modern endpoints.
Budget for tool calls. Any turn with a tool call roughly doubles LLM latency. Limit max_tool_stepscodecodecodecodecodecode, consolidate external API calls, and consider playing a thinking sound so silence is not the user’s only feedback.
Instrument before you tune. The LiveKit Agents SDK exposes e2e_latencycodecodecodecodecodecode, LLM time to first token, and TTS time to first byte per turn. Pipecat exposes the equivalent through enable_metricscodecodecodecodecodecode and observers. Store the logs externally and watch for regression.
Measure p95, not just p50. OpenAI’s own main improvement in July 2026 was a tail-latency reduction, because that is where voice agents break in practice.
Watch for infrastructure footguns. LiveKit documents that self-hosted agents on AWS burstable instance types such as t3 or t4g can hit severe latency and turn-detection timeouts even at apparently low CPU usage.
Time-to-first-sentence is the metric that aligns engineering optimization with human conversation. The data is now in place to make that shift.
