African speech recognition: FLEURS 102, Whisper large-v3 vs Meta MMS-1B, lower WER?

Premium Deals
Mighty Travels Premium
Travel in style,
save up to 90%

On flights and hotels worldwide by booking the best deals when they appear.

See Deals

Sponsored

For African-language ASR in 2026, neither Whisper large-v3 nor Meta MMS-1B wins FLEURS 102 outright: the winner flips by language and audio domain, so the decision procedure below runs both models on the FLEURS test split for your target language and then on 30+ minutes of your real audio, with real-audio WER as the deciding metric.

sunlit recording booth Nairobi with acoustic foam panels
sunlit recording booth Nairobi with acoustic foam panels

How Whisper and MMS Differ Mechanically

Whisper large-v3 and Meta’s MMS-1B arrive at African-language transcription through fundamentally different training pipelines, and that difference explains why their errors diverge in practice. Whisper large-v3 is a sequence-to-sequence Transformer trained by OpenAI on 680,000 hours of supervised (audio, transcript) pairs. Because it learns a direct mapping from audio to text, it decodes full transcripts autoregressively, but its competence in any given African language is bounded by how much labeled data that language contributed to the supervised mix. Before assuming coverage, check the per-language word error rate (WER) breakdown in the openai/whisper repository’s large-v3 versus large-v2 comparison figure for your target language.

MMS-1B, introduced by Meta in 2023 in “Scaling Speech Technology to 1,000+ Languages,” takes the opposite route. It pretrains a wav2vec 2.0 encoder self-supervised on more than 1 million hours of unlabeled speech spanning over 1,100 languages, learning acoustic representations without requiring transcripts. A language-specific connectionist temporal classification (CTC) head is then fine-tuned on labeled data to map those representations to output tokens. This separation means broad acoustic exposure is established during pretraining, while transcription behavior is shaped later by the fine-tuning data available for each language adapter.

This architectural split—Whisper’s 680,000-hour supervised multilingual training versus MMS’s 1,000,000+ hour self-supervised wav2vec 2.0 pretraining with a fine-tuned CTC head—is the core reason the two models fail differently. Whisper’s autoregressive decoder conditions each token on previously generated text, so when supervised evidence for a language is thin it can produce fluent but ungrounded output or drift toward a higher-resource language. MMS’s CTC head emits frame-level predictions independently, so its failures more often surface as local substitutions, deletions, or fragmented tokens tied to acoustic mismatch or the quality of the fine-tuned adapter rather than coherent hallucinations.

Use this distinction as a diagnostic lens when inspecting errors: fluent, plausible-sounding mistranscriptions point to a decoder-side coverage gap typical of supervised seq2seq models, while choppy or acoustically plausible character errors point to representation or adapter limitations typical of a self-supervised encoder with a CTC head. Matching the observed failure pattern to the underlying mechanism tells you whether the bottleneck is labeled-language coverage in training or acoustic and adapter fit for your target speech.

dusty Sahelian courtyard dusk with terracotta walls woven
dusty Sahelian courtyard dusk with terracotta walls woven

Run Your Own FLEURS Eval Correctly

Use the FLEURS test split, not dev, for the headline number you compare across models. FLEURS covers 102 languages, including many African languages such as Yoruba, Amharic, Swahili, Hausa, Xhosa, and Zulu, and it is loadable through Hugging Face datasets, including nusantara/fleurs. Before planning an evaluation around a language, verify that its ISO code is actually present in the dataset; if it is missing, switch to a language-specific corpus or collect your own test set rather than assuming coverage.

Score every hypothesis with jiwer's wer(reference, hypothesis), but only after applying one normalization function to both sides: lowercase, strip punctuation, and collapse whitespace. Apply that exact function to references and hypotheses from both models, keep it in version control, and report it with the number. Inconsistent normalization is the most common reason two teams report different WERs for the same model and split.

Raw WER already counts insertions, but it can still understate hallucination risk when a model produces fluent, repeated, or overlong text that happens to overlap with reference words. Add a hallucination penalty to the reported score: flag utterances where the hypothesis is materially longer than the reference or contains repeated spans, then report raw WER and penalized WER side by side. If the model ranking changes under the penalized score, treat the penalized ranking as the conservative one for deployment.

Run the same protocol on at least 30 minutes of your own audio, with the same normalization and penalty, and choose the model by WER on that real audio rather than by the FLEURS number alone. A model can win FLEURS for your language and still lose on your microphones, speakers, code-switching, or domain vocabulary.

Keep the evaluation reproducible: record dataset version, split, ISO code, normalization code, jiwer version, and whether the penalty was applied. If you later rerun the eval, change one variable at a time so a WER movement tells you whether the model, the data, or the scoring changed.

Run Your Own FLEURS Eval Correctly — African speech recognition

Model Comparison: What Each Wins

Treat model selection as a conditional decision, not a leaderboard. The table screens the three candidates by coverage, failure mode, and deployment cost; the final choice still belongs to your own WER on real audio, not to a published average.

ModelCoverageTypical failure modeDeployment costConditional winner
Whisper large-v3Strongest where labeled training data existsHallucination on out-of-domain audioHighest of the threeIn-domain, well-resourced languages
Whisper large-v3-turboSame encoder coverage as large-v3Same hallucination risk with less decoder capacityLower than large-v3Only after it matches large-v3 WER on your target language
Meta MMS-1BBroadest language coverage via self-supervised pretrainingCTC decoding and per-language head qualityNot the heaviest optionCoverage or domain-shift constraints

Whisper large-v3 is the strongest candidate where labeled training data exists, but its known hallucination risk on out-of-domain audio means a strong benchmark score can still hide bad behavior on your recordings. Before committing, confirm serving capacity on your own hardware: measure peak VRAM and audio-minutes per second as described in the compute section rather than assuming from parameter counts.

Whisper large-v3-turbo keeps the same encoder as large-v3 while cutting the decoder from 32 layers to 4. Per the Whisper Notes guide, that suits most transcription use cases, but verify its African-language WER against large-v3 yourself before substituting it to save compute.

MMS-1B offers the broadest language coverage through self-supervised pretraining and decodes with CTC. That breadth helps when your language is poorly represented in supervised data, while CTC decoding and per-language head quality become the main failure modes to inspect.

When you inspect failures, separate hallucinated words from missed words: Whisper tends to invent plausible text, while MMS tends to drop or mangle rare words. That distinction tells you which post-processing or vocabulary support to add.

Deployment cost is a screening factor, not a substitute for accuracy: if a cheaper model raises WER on your audio, the saved compute is lost in review time. Keep the cost column in the table, but let accuracy on your recordings break ties.

Conditional winner: choose large-v3 for in-domain, well-resourced languages; choose MMS-1B when coverage or domain shift is the binding constraint; promote turbo only after it matches large-v3 on your target language. The table is a screen, not a verdict.

Model Comparison: What Each Wins — African speech recognition

Compute and Serving Costs That Matter

Budget the comparison before you budget the deployment. Running both models on 30 minutes of audio is a one-time GPU cost on a single consumer-class card — expect on the order of the audio's duration for Whisper large-v3 and less for turbo and MMS-1B, but verify on your own hardware by logging wall-clock GPU time as described below. That one-time cost is trivial next to choosing the wrong model for a production pipeline.

The bigger number is serving. MMS-1B's ~1B parameters and CTC head make it cheaper to serve per hour than Whisper large-v3, whose encoder-decoder generates tokens autoregressively and sequentially. If you serve high volume, that decoding difference compounds across every hour of audio.

Parameter count is a proxy, not a bill. A ~1B-parameter CTC model can still be memory-bound by batch size and sequence length, while a larger autoregressive model can be throughput-bound by token generation. Measure peak VRAM and audio-minutes per second on your hardware rather than assuming the smaller model always fits or the larger model always wins.

Benchmarking is not just transcription time. Include model loading, audio preprocessing, and repeated runs for variance; keep the audio fixed, log wall-clock GPU time, and divide by audio minutes to get a per-hour serving estimate. That turns a one-off eval into a number you can compare with production load.

For serving, separate cold-start from steady state. A model that loads slowly but decodes cheaply may be fine for batch jobs and poor for interactive transcription; conversely, an autoregressive model with higher per-hour cost may still be acceptable if latency requirements are loose and volume is low.

Treat the eval as a fixed cost and serving as the variable cost. Run both models on the FLEURS test split and on 30+ minutes of your own audio, record WER and GPU-minutes per hour, then choose by real-audio WER unless serving cost would make the better-WER model uneconomic at your volume.

Compute and Serving Costs That Matter — African speech recognition

What Published Numbers Do Not Tell You

Published FLEURS scores measure a narrow slice of speech: the FLEURS dataset, released by Google Research, consists of read sentences recorded in controlled conditions. That makes it a strong benchmark for comparing models on clean, scripted utterances, but a weak proxy for how those same models behave on the audio you actually need to transcribe. The gap between benchmark conditions and production conditions is where a FLEURS-based choice can mislead you.

Condition When the FLEURS-based rule breaks When it still wins
Speech style You decide from FLEURS alone while your target audio is conversational, spontaneous, or interview-style. You re-test both models on 30 or more minutes of your real conversational audio and choose by that result.
Code-switching You rely on monolingual FLEURS scores for audio where speakers embed English in Swahili, Yoruba, or another target language. You include representative mixed-language clips in your 30-minute real-audio test and score them explicitly.
Acoustic environment You extrapolate from clean, read FLEURS recordings to noisy field audio, distant microphones, or overlapping speakers. Your real-audio sample reflects the same noise, microphone placement, and overlap you expect in production.
Domain and vocabulary Your use case depends on specialized terms, names, or phrasing absent from FLEURS prompts. Your real-audio test covers the domain-specific vocabulary and speaking patterns your system must handle.

Code-switching deserves explicit attention because it is a common failure point that published benchmarks do not resolve. English embedded in Swahili or Yoruba, or similar mixes in other African-language contexts, is underrepresented in the African-language training data available to both models. No published FLEURS number establishes code-switch performance, so a leaderboard advantage on monolingual read speech cannot be treated as evidence of mixed-language reliability. The only defensible check is to assemble mixed-language clips that mirror your real usage and score transcripts against references for those clips.

Conversational delivery and noisy field conditions can flip the winner for the same reason: they introduce disfluencies, interruptions, background noise, and acoustic variability that read audiobook-style speech does not capture. A model that leads on FLEURS for a given language may trail on call-center recordings, community interviews, or outdoor field notes in that same language.

The practical rule is narrow but robust: FLEURS-based comparisons break only when you decide from FLEURS alone. They remain useful when you treat them as a starting point and re-test on 30 or more minutes of your real audio, including any code-switching, noise, and domain vocabulary your deployment will encounter.

What Published Numbers Do Not Tell You — African speech recognition

Picking for One Language

For Yoruba, treat the model choice as a three-checkpoint reproduction rather than a pick from a published leaderboard. Start by loading nusantara/fleurs, filtering to the yor language code, and confirming the test split clip count as soon as the dataset loads. If yor is absent, stop there and choose a different evaluation corpus for Yoruba; do not borrow results from a related language or assume a nearby code will behave the same.

Checkpoint two is the head-to-head on that test split. Run Whisper large-v3 and MMS-1B with identical normalization, then record WER_A for Whisper, WER_B for MMS, and a hallucination count for each model. Use the worksheet below as the single place where those values live, and keep RTF and notes in the same row so the comparison stays tied to one run.

ModelWERHalluc. clipsRTFNotes
Whisper large-v3WER_A
MMS-1BWER_B

Checkpoint three moves the comparison off the benchmark and onto your own audio. Run BOTH models — not just the checkpoint-two winner — on at least 30 minutes of real Yoruba audio, then record each model's WER and hallucination count in the same worksheet. This matches the canonical rule: if the real-audio ranking differs from the FLEURS ranking, or if either model's hallucination count is unacceptable for your use case, the real-audio WER decides rather than the FLEURS number.

The defensible Yoruba choice is the model with the better real-audio WER after all three checkpoints, with WER_A, WER_B, and the real-audio value all reproduced in your own run. Keep the worksheet with the model artifact so a later re-evaluation can start from the same baseline.

Because the winner can flip by audio domain, keep the FLEURS test result and the real-audio result separate in the worksheet; a model that wins the benchmark but loses on your audio should not be promoted without a documented reason.

Worked Example: Run the Numbers

Illustration only, not a benchmark: during one week in 2026, a product team must choose between Whisper large-v3 and MMS-1B for Swahili customer-support calls. The inputs are 30 minutes of real call audio, represented in this illustration by 4,500 reference words; the FLEURS test split for Swahili as the secondary check; and the same word-error-rate definition for both models. The decision rule for this example is that the lower WER on the real audio wins, while FLEURS is used only to break a tie.

ModelSubstitutionsDeletionsInsertionsTotal errorsReference wordsWER
Whisper large-v312045151804,5004.0%
MMS-1B10075502254,5005.0%

Calculation steps: for Whisper large-v3, 120 + 45 + 15 = 180 errors, and 180 / 4,500 = 0.040, or 4.0% WER. For MMS-1B, 100 + 75 + 50 = 225 errors, and 225 / 4,500 = 0.050, or 5.0% WER. The gap is 45 errors, which equals 1.0 percentage point on this 4,500-word sample.

Winner for this illustration: Whisper large-v3, because 4.0% is below 5.0% on the real-audio sample. Break-even trigger: MMS-1B would need 180 or fewer total errors to tie, which is 45 fewer than its illustrated 225; it would need 179 or fewer, or 46 fewer, to win outright. If the two real-audio WERs are equal, rerun the comparison on the FLEURS test split and choose the lower FLEURS WER.

A 45-error gap on 4,500 words is small enough that normalization choices, speaker overlap, or a different 30-minute sample could change the result, so the break-even trigger should be checked on a second 30-minute sample before committing. This example is only an arithmetic illustration: it does not claim published WERs for either model, and a real decision still requires your own FLEURS test-split run plus your own 30-plus minutes of audio.

Decision Rules for African-Language ASR

Start with a default, then let evidence override it. If Whisper's published per-language WER for your language is missing or clearly poor, begin with MMS-1B and switch only when your own evaluation shows Whisper winning on your target language and audio. Treat a missing published number as a warning, not as neutral evidence, because it usually means the language was not represented well enough in the benchmark to trust a headline comparison.

If the two models' WERs are within 2 points, do not chase the smaller headline number. Break the tie by hallucination rate first, then by real-time factor on your own hardware; a model that hallucinates less or decodes faster on your stack is the better production choice even when its WER is marginally higher. Record both the WER gap and the hallucination gap in your decision log so a later reviewer can see why the larger number was accepted.

If your audio is conversational, code-switched, or noisy, FLEURS alone is not sufficient for the final decision. Run the mandatory 30-minute real-audio re-test, or 30+ minutes if your deployment has longer utterances, and make the final choice on that WER rather than on the benchmark number. Keep the FLEURS result as a screening check, but let the real-audio result decide deployment.

For a speed-focused deployment, treat Whisper large-v3 turbo as a substitute candidate, not an automatic upgrade: it must pass the same FLEURS and real-audio checks before it replaces large-v3 in any of the rules above. If turbo wins on real audio but loses on hallucination rate, apply the same tiebreakers rather than assuming the faster model is safer.

When hallucination rates are also close, keep the model with the lower real-time factor on your hardware and document the tie; if neither model clears your error budget, revisit data collection or language coverage before changing the rule. Do not relax the real-audio requirement to make a model pass; instead, collect more representative audio or narrow the supported language set.

Apply these rules in order: missing or poor Whisper evidence favors MMS-1B, a close WER favors the lower-hallucination or faster model, and real-world audio takes precedence over the benchmark. If two rules conflict, the rule tied to your actual audio domain wins, because production errors are measured there.

What to do next

StepActionWhy it matters
1Use FLEURS 102 only as a shortlist filter for your target language, noting that neither Whisper large-v3 nor Meta MMS-1B wins outright.Published FLEURS 102 position alone does not settle African-language ASR.
2Run both Whisper large-v3 and Meta MMS-1B on the FLEURS test split for your target language and record each WER.This gives a controlled per-language comparison before real-audio testing.
3Collect 30+ minutes of your own target-language audio from the real deployment domain.Benchmark audio may not match your acoustic conditions or speaking style.
4Transcribe that 30+ minutes with both models and score WER against reference transcripts.Real-audio WER is the deciding metric in the canonical rule.
5Commit to the model with lower WER on your real audio; keep the other as a documented fallback if scores are close.Prevents choosing by benchmark position when the winner flips by language and domain.
6Repeat both evaluations when the target language, domain, or audio conditions change.FLEURS 102 is a starting point, not a final verdict.

Frequently Asked Questions

Which result should decide model choice if FLEURS 102 and my real audio disagree?

The decision procedure runs both models on the FLEURS test split for your target language and then on 30+ minutes of your real audio, with real-audio WER as the deciding metric.

Why can Whisper large-v3's African-language accuracy vary by language?

Its competence in any given African language is bounded by how much labeled data that language contributed to the supervised mix.

What should I inspect before assuming Whisper large-v3 covers my target language?

Check the per-language WER breakdown in the openai/whisper repository’s large-v3 versus large-v2 comparison figure for your target language.

Is there an outright FLEURS 102 winner between Whisper large-v3 and MMS-1B?

Neither Whisper large-v3 nor Meta MMS-1B wins FLEURS 102 outright because the winner flips by language and audio domain.

How does MMS-1B learn acoustic representations differently from Whisper?

MMS-1B pretrains a wav2vec 2.0 encoder self-supervised on more than 1 million hours of unlabeled speech spanning over 1,100 languages.

How does Whisper large-v3 generate transcripts mechanically?

Whisper large-v3 is a sequence-to-sequence Transformer that learns a direct mapping from audio to text and decodes full transcripts autoregressively.

Quick answers

Does either Whisper large-v3 or Meta MMS-1B win FLEURS 102 outright for African-language ASR?Neither Whisper large-v3 nor Meta MMS-1B wins FLEURS 102 outright; the winner flips by language and audio domain.
What is the deciding metric when choosing between Whisper large-v3 and Meta MMS-1B for a target language?Real-audio WER is the deciding metric after running both models on the FLEURS test split for your target language and then on 30+ minutes of your real audio.
How does Whisper large-v3 generate transcripts?Whisper large-v3 is a sequence-to-sequence Transformer that decodes full transcripts autoregressively.
What limits Whisper large-v3's competence in a given African language?Its competence in any given African language is bounded by how much labeled data that language contributed to the supervised mix.
How does MMS-1B learn acoustic representations?MMS-1B pretrains a wav2vec 2.0 encoder self-supervised on unlabeled speech spanning over 1,100 languages, learning acoustic representations.

Also worth reading: Whisper large-v3 Fine-Tuning: 18% WER Cut on Indian English Calls: Whisper large-v3 Fine-Tuning: 18% WER · Whisper's 2026 WER: Evidence, Decision Matrix, and Variance: Whisper's 2026 WER: Evidence, Decision · Whisper-large-v3 vs Scribie: Two Scoreboards, No Winner: Whisper-large-v3 vs Scribie: Two Scoreboards,

Premium Deals
Mighty Travels Premium
Travel in style,
save up to 90%

On flights and hotels worldwide by booking the best deals when they appear.

See Deals

Sponsored

Research Methodology & Editorial Standards

We begin by defining the specific objectives the reader needs to accomplish. Primary product documentation and authoritative secondary sources are assembled into a verified research corpus; drafting occurs only after this foundation is in place.

Every quantitative claim is subjected to dual-source verification. Any figure that cannot be independently corroborated is either qualified or omitted.

Published · Last reviewed · Owned by the Transcribeall editorial desk (About, Contact, Privacy).

Related answers