The Direct Answer
A trustworthy speech-recognition latency benchmark must separate four clocks: audio capture, network transport, server processing, and display or downstream execution. “Real time” by itself is not a measurement, and an 80 ms vendor claim does not establish that an application will produce useful text within 80 ms. The most useful result reports time to first non-empty partial transcript, time to stable final text, endpoint-detection delay, and throughput under a defined number of concurrent sessions. It should also report word error rate, because a fast system that changes words is not automatically better than a slightly slower one.
Also worth reading: How do you accurately benchmark word error rate for AI transcription services in 2026? · How Do You Design a Low-Latency Speech Recognition Architecture for Real-Time Voice Applications? · What Are the True Real-Time ASR Accuracy Tradeoffs in 2026?
For interactive voice applications, test a p95 first-partial latency below about 500 ms and a p95 finalization delay below roughly 1 second after speech stops. A human conversation often turns every 150–300 ms, but turn boundaries vary, and a target of 200 ms would be exceptionally strict for most remote streaming systems. Captioning, note-taking, and call transcription can tolerate longer delays than a live voice agent, so one universal threshold would be misleading. As of September 24, 2026, model announcements can make streaming recognition, diarization, and endpointing appear to be one capability, but benchmark reports should still identify which component each timing value measures.
What Streaming ASR Latency Actually Measures
Latency starts when a sample leaves the microphone and ends when the application displays text or acts on it. If a phone captures 20 ms audio frames but buffers 200 ms before uploading them, model inference cannot begin immediately. Network transit, authentication, request queuing, batching, decoding, client rendering, and screen refresh each add time. For a voice agent, speech-to-text latency also includes the time required to decide that the speaker has finished, which is often more important than the raw duration of one neural-network pass.
Time to first token, time to first partial, and time to final output describe different events. A system may emit an incorrect partial within 80 ms, revise it several times, and finalize the utterance 700 ms after endpoint detection. It may also deliver a correct transcript in 300 ms but wait 1.5 seconds to detect the end of the turn. These are not equivalent metrics, and marketing pages often use “latency” without defining the event, percentile, endpoint, or included overhead.
Useful measurements should distinguish algorithmic processing from wall-clock service latency. Algorithmic processing may exclude capture, upload, or rendering, while wall-clock timing follows a timestamped signal through the complete path. A benchmark should report both when possible, along with hardware, model version, audio format, transport protocol, region, and concurrency. Without those conditions, a number cannot be reproduced or used to compare products fairly.
A Reproducible Benchmark Protocol
Prepare a fixed corpus and freeze it before testing vendors. Include at least 30 minutes per major language, 10 hours for production-level comparisons, and edge cases such as silence, interruptions, telephone bandwidth, background noise, accents, proper names, and mixed languages. Record the same lossless PCM audio for every candidate; common streaming input is 16 kHz, 16-bit, mono PCM, but a vendor’s supported format should not be sacrificed just to match that convention. Generate timestamps for every audible speech boundary and annotate expected text independently of the systems being evaluated.
Run each system in its normal production configuration. Test cold starts separately from warmed sessions, because first-request initialization can add seconds. Capture repeated requests for at least 10 runs per configuration, use multiple network conditions, and calculate p50, p95, and p99 rather than reporting only an average. One slow request among 100 can be operationally important, especially in a call center, while a mean can conceal that tail. Preserve packet traces, server request identifiers, and transcript revisions so that a suspicious timing can be traced to a particular stage.
For accuracy, compute word error rate using word-level substitutions, deletions, and insertions, then report normalized text matching so that punctuation or capitalization conventions do not distort the result. Character error rate is useful for languages, names, and code-switching, but it can overstate or understate human-perceived mistakes. At least 1,000 independent utterances and 1,000 bootstrap resamples can provide more stable comparisons than a few scripted conversations, although no statistical method repairs a corpus that does not resemble real traffic. A 2% absolute WER difference matters less if both candidates score around 30% on a difficult language than if they score 5% and 7% on a clean English test.
Metrics and Comparison Table
The comparison should treat speed and quality as two axes rather than combining them into one score. Batch transcription has excellent throughput and low per-audio cost, but it does not provide partial text or natural turn-taking. A streaming endpoint can feel responsive while producing unstable text, and a diarization model can label speakers quickly without performing recognition at all. The table below shows how several benchmark labels should be interpreted.
| Feature | Batch or offline ASR | Streaming speech-to-text service | How to validate it |
|---|---|---|---|
| Time to first text | Usually minutes or hours | Target p95: 200–800 ms for interactive use | Timestamp microphone capture through visible text |
| Delay after speech ends | Often not measured | Target p95: about 100–500 ms after a valid endpoint | Compare audio boundary with final transcript |
| Partial-transcript stability | Not applicable | At least 2–5 revisions per normal sentence | Count edits from first partial to final text |
| Accuracy | WER or CER on fixed corpus | WER or CER plus revision cost | Publish language, sample size, and normalization rules |
| Throughput | Measured in audio-hours per GPU-hour | Measured in sessions and audio-minutes per second | Test at 1, 10, 50, and target concurrency |
| Reported 80 ms claim | Usually irrelevant | May cover only model or a special endpoint | Ask whether capture, network, queueing, and rendering are included |
Interpreting Current Models and Vendor Claims
Recent systems increasingly combine streaming recognition, speaker attribution, and turn detection. Meta’s Muse Voice Transcribe announcement, coverage citing an 80 ms figure, Nemotron’s real-time eight-speaker diarization model, and Sarvam’s multilingual Saaras V4 releases illustrate why buyers now face more complex tradeoffs. These announcements demonstrate active development, but press coverage is not equivalent to a controlled benchmark. An 80 ms result may refer to a model segment, a particular accelerator, a batch of cached audio, or a lower-level estimate rather than end-to-end user latency.
The Hugging Face open ASR leaderboard is useful for examining accuracy across public models and datasets, but leaderboard scores are not automatically live-streaming measurements. Offline WER can improve while streaming latency worsens if the model requires long context or a large chunk before decoding. Conversely, an architecture optimized for low delay can use less future context and lose accuracy. The Informer-associated state-space-model research is relevant to efficient sequence processing, yet an academic architecture result should not be compared directly with a managed service’s p95 unless hardware and task scope align.
Consumers should request current product documentation and a private test using their own traffic. A dated launch post is a starting point, not proof of present performance, because model revisions, regional capacity, batching policies, and API defaults can change. Benchmark at least the two leading candidates on the same day when possible. If a provider will not disclose the timing boundary, unit, percentile, or concurrency, treat its claim as marketing evidence rather than procurement data.
Common Benchmark Mistakes
The most frequent mistake is measuring text generation without measuring audio buffering. Another is averaging latency across a quiet demo and difficult speech, which makes the result look healthier than most sessions experience. Testers also use unrealistic pauses, perfectly clean recordings, one accent, and short filenames that behave differently from real vocabulary. They may ignore diarization errors even when overlapping speakers are essential, or they may credit a streaming model for sending partial text that repeatedly reverses earlier words.
Percentiles are often replaced with averages, and the sample is too small to estimate p99. Twenty requests can demonstrate that a service functions, but they cannot establish reliable tail behavior. Timing only successful responses hides timeouts, rate-limit waits, and reconnect delays, all of which affect customer experience. Comparisons can also be biased by selecting different languages, audio preprocessing, spell-checking rules, or language-model enhancement, so the evaluated system rather than the advertised model may be doing different amounts of work.
Cost mistakes follow the same pattern. Comparing the lowest listed price per minute ignores the number of audio channels, retries, partial emissions, or per-request fees. The arithmetic is straightforward: a $10 test covering 600 minutes costs about $0.0167 per minute before retries, but 15% retransmission raises effective input cost to roughly $0.0192 per successfully transcribed minute. This is an example rather than a market quote; vendors such as OpenAI, Meta, Mistral, Sarvam, and others may change prices or offer different packaging. Confirm current rates and free-tier limits on the provider’s official pricing page.
When to Act and What Thresholds to Set
Set thresholds according to the consequence of delay. In a live voice agent, p95 time to first partial above 800 ms is likely to feel sluggish, while a p95 endpoint delay above 1 second can produce overlaps or awkward interruptions. For live captions, a first partial around 1–2 seconds may be acceptable, but final text should follow a detected endpoint promptly. For interview transcription, 2–5 seconds can still be usable if users see progress and can correct later, whereas offline jobs should be evaluated mainly on throughput, WER, and cost.
Run an initial bake-off when changing providers, redesigning a mobile client, moving infrastructure regions, or adopting a new model. Do not respond to every latency fluctuation: a single noisy measurement is not enough. Establish regression limits such as a 10% increase in p95 or a 1-point absolute WER increase, then investigate changes larger than the ordinary variation of repeated runs. Production monitoring should track p50, p95, and p99 separately, because the average user may be fine while the slowest 1% drives complaints.
Accuracy gates should be language-specific. For a clean English call-recording system, WER near 5% may be strong; for noisy multilingual audio, 15% might reflect a difficult operating point rather than a defective model. Compare both WER and partial stability, and set business thresholds from the cost of corrections, not from a generic leaderboard. A provider that achieves 350 ms p95 first text with 8% WER may outperform one at 180 ms with 20% WER for archival work, while the latter could still win for a real-time assistant that mostly matches simple commands.
The Definitive Evaluation Method
The definitive benchmark is a versioned, end-to-end test built from representative audio and frozen expected transcripts. It measures microphone-to-first-partial, endpoint-to-final, p50/p95/p99 latency, WER or CER, partial revision count, diarization performance, failure rate, throughput, and total billed cost. It runs warm and cold, at realistic concurrency, under controlled network conditions, and with at least two independent repetitions. Its conclusions distinguish published claims from observed results and preserve the date, model identifier, region, hardware, and software configuration.
No single number settles the question. An 80 ms claim can be excellent if verified, irrelevant if it excludes buffering and network delay, or misleading if obtained only on cached speech. Real-time ASR should be selected through a reproducible workload, an accuracy floor, and product-specific delay limits. Teams that follow that method can defend the decision, detect regressions, and avoid purchasing a fast demonstration that fails under ordinary conversational conditions.