Whisper model quantization benchmarks: how do Q4, Q5, Q6, Q8, and FP16 compare in accuracy, speed, and memory for ASR?

What "Whisper quantization" actually means

Whisper is OpenAI's open-weights automatic speech recognition (ASR) family, ranging from the 39M-parameter tiny model to the 1.55B-parameter large-v3 model. Because the model weights are publicly available, the open-source community has produced GGML, GGUF, and ONNX-converted variants that quantize weights from 16-bit floating point (FP16) down to 4-bit integers. The naming convention is roughly Q4_0, Q4_K_M, Q5_0, Q5_K_M, Q6_K, and Q8_0, where the digit is the average number of bits per weight and the suffix indicates the quantization scheme (legacy 0, K-quant, or K-means K-quant). Lower-bit variants shrink VRAM use and accelerate inference, especially on CPUs and Apple Silicon, at the cost of some accuracy loss. As of the MLPerf Inference v5.1 results published in 2025 and the IBM Granite 3.3 speech release, the wider ML community treats Whisper-large-v3 in Q5_K_M or Q6_K as a sensible default, because it preserves most of the FP16 word error rate (WER) while cutting model size by roughly 60–70%.

Also worth reading: How Do You Choose the Best Whisper.cpp Quantization for Local Transcription? · Can Voice AI Transcription Accuracy Benchmarks Keep Pace With Real-Time Multilingual Models? · How Do Medical Speech Recognition Benchmarks Evaluate Clinical Accuracy?

The benchmark methodology that matters

Comparing quantization benchmarks is harder than it looks because Whisper accuracy depends on three variables that must be held constant: the audio dataset, the language, and the decoding parameters. LibriSpeech test-clean and test-other remain the most cited English benchmarks, with large-v3 FP16 typically scoring around 2.7% WER on test-clean and 5.2% on test-other. For multilingual work, the FLEURS benchmark is more representative. The third variable, decoding, matters because greedy decoding hides quality losses that beam search exposes, and temperature fallback ("compression_ratio_threshold" and "logprob_threshold" from whisper.cpp) can mask hallucinations that quantization tends to introduce on low-bit variants. Any benchmark that does not report all three of these should be treated with skepticism.

Quality vs. size: the core tradeoff

The headline finding across the whisper.cpp, Hugging Face transformers Optimum, and llama.cpp-style benchmarks is that Q8_0 is effectively lossless for ASR, typically within 0.05–0.1 percentage points WER of FP16 on LibriSpeech. Q6_K and Q5_K_M are where most practical deployments land, because they add roughly 0.2–0.4 WER points while shrinking the 1.55B-parameter large-v3 model from about 3.1 GB (FP16) to roughly 1.2 GB (Q5_K_M). Q4_K_M is the first level where accuracy degradation becomes audible on noisy audio, with WER regressions of 0.8–1.5 points on test-other. Below Q4, the IQ (importance-aware) quants such as Q4_0 and Q3_K_S start hallucinating entire sentences on long-form audio, which is unacceptable for production transcription. In short, the sweet spot for ASR quality is Q5_K_M to Q8_0, and Q4 is acceptable only when memory pressure dominates.

Speed and throughput on different hardware

Quantization is not just about accuracy; it is also about real-time factor (RTF), defined as minutes of audio processed per minute of wall-clock time. On Apple M3 Max, HackerNoon's 2025 Whisper benchmark roundup and SitePoint's Mac M3 Max vs RTX 4090 comparison both show large-v3 Q5_K_M running about 1.4–1.7× faster than FP16 in whisper.cpp, because the SIMD path on Apple AMX is bandwidth-bound and lower-bit weights fit more easily in the on-chip cache. On consumer NVIDIA RTX 4090, the speedup is smaller, because FP16 tensor cores already saturate; the win is more about fitting larger batch sizes in 24 GB. On the Willow Inference Server's Vulkan back end, the gap widens further, because integrated GPUs lack FP16 throughput and benefit disproportionately from 8-bit integer paths. Across all these platforms, Q4 is only marginally faster than Q5 on GPU, so there is little reason to drop below Q5 unless you are memory-constrained.

Memory footprint and the practical deployment math

The actual VRAM or RAM cost of a Whisper model is the weights plus the activations, plus a large encoder mel-spectrogram buffer. For large-v3, the rule of thumb in 2026 deployments is approximately 1.0× the model file size for weights, plus 0.4–0.6 GB of working memory, plus 1–2 GB for the KV cache on long-form transcription. That means Q5_K_M (≈1.2 GB weights) needs roughly 2.5–3.0 GB total, while FP16 (≈3.1 GB) needs about 4.5–5.0 GB. For an 8 GB edge device, this is the difference between running large-v3 and being forced down to small.en. For batched server workloads, Q4 can lift concurrent streams by 30–50% per GPU, which is why some transcription APIs (LocalAI, Willow) default to Q4_K_M for their hot path and reserve Q8 for re-scoring.

The three stacks that most teams compare in 2026 are whisper.cpp (GGML/GGUF, CPU-first), Hugging Face Transformers with Optimum (GPU-first, supports ONNX and TensorRT), and server-style wrappers like LocalAI, the Willow Inference Server, and Faster-Whisper (CTranslate2 back end). whisper.cpp is the only stack that natively exposes the full Q2–Q8 quant ladder and the only one that has published apples-to-apples WER numbers across all levels. Faster-Whisper / CTranslate2 supports INT8 and FP16 but not the 4-bit and 5-bit GGUF quants, so it sits in the Q8-equivalent accuracy tier while running roughly 4× faster than whisper.cpp on a single CPU. For GPU deployments, TensorRT engines generated by Optimum consistently beat both on RTX hardware but require a per-GPU build and a separate quantization pass. The practical takeaway is that if you want to pick one quant level and stay portable, Q5_K_M in whisper.cpp is the most widely tested combination.

The NVIDIA Blackwell and MLPerf context

It is worth noting what the wider MLPerf Inference v5.1 results (2025) and the NVIDIA Blackwell Ultra debut tell us about the quantization picture, even though those benchmarks are dominated by LLMs rather than ASR. The pattern is consistent: INT8 is treated as effectively free in accuracy terms, FP8 (E4M3/E5M2) is the new sweet spot for inference, and 4-bit is acceptable for high-volume serving where the last 1–2% of quality is recoverable with a re-ranker. Whisper-specific quantization sits one notch below the LLM frontier because speech models are smaller, more memory-bound, and less sensitive to weight precision than 70B-parameter LLMs. So while the industry is racing toward FP8 and 4-bit for Llama-class models, Whisper is comfortably settled at INT8/Q5 as the production default in 2026.

Common mistakes when reading or running these benchmarks

The most frequent error is comparing WER numbers across runs that used different beam sizes, because beam search amplifies quantization noise. A second is reporting numbers on test-clean only; quantization tends to hurt more on accented speech and noisy audio, so test-other and FLEURS are the more honest indicators. A third mistake is assuming the Q4 speedup seen on a Mac M-series chip will transfer to an NVIDIA GPU, where memory bandwidth is less of a bottleneck. Finally, many blog posts quote file sizes without subtracting the fact that GGUF containers include metadata and a tokenizer that can add 50–80 MB to the model weight total, which matters on devices with 8 GB of RAM.

When to choose which quant level

For a self-hosted transcription service serving English call-center audio on a 16 GB GPU, Q6_K with Faster-Whisper as a fallback gives the best cost-quality ratio. For multilingual podcasts transcribed on an M-series Mac, Q5_K_M in whisper.cpp is the standard. For embedded or Raspberry Pi-class devices, Q4_K_M with the small or medium Whisper model is the realistic ceiling, and below that you should consider Granite-Speech 3.3 (IBM) or a streaming model like Moonshine, because large-v3 at Q3 is unreliable. For archival re-transcription where you can afford to wait, FP16 or Q8_0 with beam=5 gives the best WER and is worth the extra compute. As MLPerf v5.1 results indicate, infrastructure is moving toward FP8 for new builds, but for the installed base of Whisper users, Q5_K_M remains the right default through 2026.

What the competitive landscape looks like

Whisper is no longer the only credible open ASR. Voxtral (Mistral), Granite-Speech 3.3 (IBM), and Parakeet/Canary (NVIDIA) all post competitive WERs on LibriSpeech and FLEURS, and several support their own quantization paths. Voxtral in particular is interesting because it is a speech-aware LLM, so the same INT4/INT8 quant techniques that work for Llama-class models apply directly. For a 2026 deployment decision, the practical question is no longer "which Whisper quant?" but "Whisper vs. a 7B speech-LLM at INT4?"; in many cases the speech-LLM wins on accuracy but loses on latency and memory, so Whisper Q5/Q6 remains the workhorse for high-volume transcription APIs.

Bottom line

If you need one number to remember: Whisper large-v3 Q5_K_M in whisper.cpp delivers roughly 95–97% of FP16 quality at 40% of the file size, and runs 30–50% faster on CPU. Q8_0 is the safest choice when accuracy dominates. Q4_K_M is acceptable only when memory or batch density is the binding constraint. Below Q4, Whisper is no longer trustworthy for production ASR in 2026.