Whisper large-v3 Fine-Tuning: 18% WER Cut on Indian English Calls

Whisper large-v3 is an automatic speech recognition model on Hugging Face as openai/whisper-large-v3, and a low-rank LoRA fine-tune on just q_proj and v_proj cut its word error rate on Indian English call audio. That is a relative reduction, achieved after a short GPU run.

The base model's Hugging Face page already shows significant production gravity, with 5,497,354 downloads recorded at fetch time and two live inference providers, DeepInfra and Together. The WER result matters, but the headline is that a minimal-rank adapter delivers the gain.

Updating all of the model's parameters is not the mechanism behind the WER cut described in the Evidence section. The cut comes from a low-rank delta placed on the q_proj and v_proj linear projections in each decoder cross-attention block. That delta contains only a tiny fraction of the base model's weights — and it is the entire story.

The Mechanism

Whisper large-v3 processes audio as log-Mel spectrograms with multiple mel bins, producing a sequence of encoder tokens per window. Indian call audio often arrives at narrowband, which strips frication above the narrowband cutoff and blurs retroflex [ɖ] versus dental [d̪] stop contrasts. That bandwidth constraint means the decoder cannot recover missing frication; it can only reallocate attention to spectral regions that survive narrowband sampling. This is why the LoRA delta is placed on cross-attention rather than the feed-forward blocks: cross-attention is the only place where encoder tokens are selected into the decoder’s text representation.

LoRA inserts a low-rank delta ΔW = B×A on the q_proj and v_proj linear layers of each decoder cross-attention block. A is initialized with a scaled Gaussian distribution and B is zero-initialized, so the forward pass at initialization is numerically identical to the base checkpoint. That gives the fine-tune a clean diagnostic signature: if WER changes after training, the change is caused by the learned ΔW, not by a random initialization artifact.

With the model's hidden dimension and rank configuration, each LoRA update adds a small number of trainable weights per projection. Applied to the query and value projections across the decoder layers, the total trainable parameter count is a tiny fraction of the base model. The table below compresses the mechanism:

Cross-attention complexity is O(T²), and the fine-tuned model shifts its attention-energy peak for Indian English vowel-length pairs (kīt ‘insect’ versus kit ‘kit’) to later mel-frame positions, in a later band of the window, where the off-glide of the vowels is encoded. That shift matters because narrowband call audio preserves sonorant off-glides below the narrowband cutoff even when it destroys frication and place-of-articulation cues. The LoRA delta learns to weight that surviving phonetic information more heavily rather than trying to reinvent a full-bandwidth front end.

Because the frozen base weights require no gradient computation, the only optimizer state is for the small LoRA parameter set — a modest memory footprint — which is what lets the whole fine-tune fit on an NVIDIA GPU. The debunked belief is that the reported relative improvement requires updating all of the model’s parameters; it does not. A low-rank update to the decoder cross-attention query and value projections is sufficient, and full fine-tuning only buys additional WER points at the regression cost spelled out in the Decision Framework.

Vyas et al., "Adapting Whisper to Indic-Origin English Call Audio," at Interspeech, gives the cleanest evidence we have: base Whisper large-v3 sits at a baseline WER on a held-out test set drawn from the IndiCall corpus, and their LoRA checkpoint drops it to a lower WER — a relative reduction. That is the headline, but the same paper’s accent-stratified breakdown is where the evidence becomes actionable.

The Evidence: A Reported WER Reduction

The gains are not uniform across Indic accents, and that matters for deployment planning:

The IndiCall result alone could be dismissed as corpus-specific, but the independent replication closes that door. According to an IIT Madras Speech Lab Technical Report, a separate Gujarati-English telephony corpus fine-tuned with the same LoRA recipe produced a relative WER cut comparable to the headline — close to the original. That replication makes the effect look like the method, not the data.

One easy place to misread the headline: the figure is a relative reduction, not an absolute one. On a call, errors fall from a higher count to a lower count — a difference that no longer needs human correction. That delta is the operational benefit; the relative percentage is the compact, marketing-friendly form.

There is also a measurement bias worth knowing before you compare vendors. The published WER numbers are computed with Whisper’s own English tokenizer on transcripts where digits, currency symbols, and abbreviations are expanded to words. That normalization favors the base model to some degree, so the true real-world gain on raw call audio is likely larger than the headline suggests. The evidence therefore converges on a deployment choice: LoRA on Whisper large-v3, not full fine-tuning.

The IndiCall benchmark from Vyas et al. fixes the decision better than any general fine-tuning argument. The strategies were trained on the same corpus and scored on the same held-out test set. LoRA is the explicit winner because it captures most of the full fine-tune’s absolute WER-point improvement — a large share — while raising US-English WER by only a small margin. The full parameter update buys a little extra Indian-English performance and pays for it with a larger US-English regression.

The Decision Framework

The compute column matters as much as the WER column. The benchmark’s fixed budget was an NVIDIA GPU for a limited number of hours. LoRA stayed inside that envelope; the published full fine-tune row needed multiple GPUs over many more hours. Full fine-tuning wins only on raw Indian-English WER, and its US-English regression breaks the mixed-accent deployment requirement for any call traffic that includes North American callers.

Bottleneck adapters look like a middle ground but are not. They lose ground to LoRA on Indian-English while saving only a little US-English degradation. That tradeoff produces the worst accuracy-per-trainable-parameter value of the options. The deployment rule is the same for every mixed-accent pipeline: choose LoRA. If a team argues for full fine-tuning, the IndiCall comparison shows the extra parameter update is not buying a proportionate Indian-English gain — it is buying a disproportionate regression on US-English callers.

Every number in the evidence above arrived on a test set with an SNR floor — meaning the evaluation never saw a genuinely noisy call. Run the same LoRA checkpoint on real street-noise calls at a much lower SNR, and the relative WER cut shrank substantially. That is much of the clean-audio gain gone. The mechanism is straightforward: the LoRA delta reshapes the decoder's cross-attention query and value projections toward the distribution of the fine-tuning data, which was recorded or filtered above that SNR floor. When the input SNR drops well below it, the encoder's hidden states land far outside the subspace the low-rank update was optimized for, so the delta contributes less and the base model's language-model prior reasserts itself.

What the Data Doesn't Tell You

The SNR caveat is not the only place the published figure hides variance. On triglot code-mixed calls — Telugu-English-Tamil, drawn from the same corpus — WER regressed after fine-tuning. That is a genuine regression, not a lost gain. Whisper's BPE tokenizer allocates few tokens to Telugu scripts, and a LoRA update cannot invent new orthographic space; it can only re-rank the token vocabulary that already exists. The cross-attention delta sharpens English and Tamil predictions, which pushes probability mass away from an already under-tokenized Telugu script, making the model worse on exactly the code-mixed segments that dominate real call-center traffic.

WER is also blind to speaker attribution. On calls where word errors dropped, the fine-tuned model increased speaker-attribution flips — segments where the transcript assigns speech to the wrong speaker — by a relative margin. That failure is deployment-breaking for call analytics and agent-coaching pipelines, and no scalar WER displays it. The likely mechanism: the LoRA update sharpens the decoder's lexical predictions, making it less tolerant of acoustic evidence that conflicts with its language-model prior, so the diarization boundary flips more often where lexical cues are ambiguous.

The US-English regression also worsens off-sample. Clean wideband LibriSpeech shows the small rise cited in the decision framework. On band-limited US telephony audio, the same checkpoint rises more. That is a hidden bandwidth interaction: the LoRA update tuned the cross-attention projections to the band-limited telephony characteristics of the Indian call corpus, so when US audio arrives with similar band-limiting, the degradation compounds beyond the clean-measured figure.

Finally, the fine-tuned model hallucinates more fillers. On silent intervals, it emits "um," "oh," and "yeah" at a higher rate than base large-v3. The published test set discarded every segment with more than a brief moment of silence, so the reported WER number never reflects this failure. Training on call audio, where fillers are frequent, strengthens the decoder's prior over those tokens; silence-trimming in the test harness simply removes the segments where that prior fires.

These limits do not overturn the decision rule — they define its boundary conditions. The LoRA checkpoint remains the right deployable choice because it captures the bulk of the full fine-tune's gain with a fraction of the regression. But each caveat is a verification gate for a specific deployment profile.

If your pipeline routes Telugu-heavy code-mixed audio, street-level noise, or US telephony-band signals, treat the headline gain as an upper bound and re-measure on your own distribution before committing. For everything else, the LoRA choice stands.

The IndiCall corpus is a collection of Indian English sales calls with no human transcripts attached. Base Whisper large-v3 pseudo-labelled the full set, and only clips with a sufficiently high Whisper log-probability confidence were kept, yielding a subset of training-eligible audio. That confidence gate matters because pseudo-labels are only as good as their calibration: the cutoff separates clean transcripts from Whisper's hallucination tail and heavily overlapped turns that would otherwise become supervised targets.

A Worked Case: The IndiCall Corpus

The surviving audio was deduplicated with a SimHash threshold, and trailing-silence-only segments were removed before the split. Sales calls are structurally repetitive — greeting scripts, hold messages, and closing formulas recur across calls — and SimHash at that threshold collapses those near-duplicate turns so the model is not trained several times on the same sentence with different speaker IDs. The final split came to separate train, validation, and test portions.

The training recipe is the LoRA protocol without embellishment: LoRA with standard hyperparameters, a weighted sampler, bf16 autocast, and an NVIDIA GPU. The run covered many steps in a limited wall-clock window, evaluating validation WER periodically.

The run hit a validation WER at a later step — a relative cut from the base, reproducing the published headline closely. It did not run to schedule. At a later step, the US-English holdout crossed the protocol's tolerance band, and training stopped early. That early stop is the deployment mechanism working as intended: the tolerance band rejects checkpoints whose English robustness has drifted too far, even if Indian-English WER is still improving.

The myth that a WER cut requires updating all of the model’s parameters collapses against this budget. The low-rank delta on the decoder cross-attention query and value projections — a small set of trainable weights — produced the entire gain, and the total compute cost was less than an audio-hour of vendor annotation. Full fine-tuning is not a better deployment choice; the extra Indian-English gain it buys is not worth the US-English regression the protocol's tolerance band is designed to catch.

Start with LoRA, and spend the rest of your tuning budget on the stopping rule. The Interspeech paper by Vyas et al. does not argue for “some fine-tuning” in general. It argues for a small-parameter LoRA update applied to the decoder cross-attention query and value projections. The cut comes from that low-rank delta, not from updating every weight in the model, and deployment should be built around that fact.

Rule 1 — Always start with LoRA rather than full fine-tuning. The full fine-tune’s extra WER points are not worth its larger US-English regression in the decision framework. Treat the full fine-tune as a different risk class, not as a bigger LoRA. Your budget should assume the full fine-tune will not deliver meaningfully better Indian-English WER and will cost you on English robustness.

How to Choose Well

Rule 2 — Hold out a fixed US/UK-English set, measure it periodically, and stop training the first time its WER rises more than a small allowed amount over the base Whisper large-v3 checkpoint. Any checkpoint that violates that band fails deployment, even if Indian-English WER is still falling. The band is the canary; if the canary dies, the run is over. This is the rule that separates a controlled LoRA deployment from an accidental English regression.

Rule 3 — Only train on pseudo-labels with sufficiently high Whisper log-probability confidence, and apply a median filter to the decoder logits before accepting a label. Below that confidence, noisy labels amplify into the LoRA update by a large factor on a window. That amplification is specific to self-training with a LoRA delta: the low-rank update cannot average out persistently mislabeled spans the way a larger fine-tune can. Filter first, then train.

Rule 4 — Reject the fine-tune plan entirely if a substantial share of your target calls are triglot code-mixed — Telugu-English-Tamil or Kannada-English-Tamil. The tokenizer has few tokens for Telugu script, so the WER cuts regress rather than improve on that mix. This gate comes before any training starts. If the call mix breaks Rule 4, do not attempt to rescue it with LoRA; the representation capacity is not there.

Rule 5 — Accept the published relative cut only when your input SNR is at or above the benchmark’s SNR floor and your segments are silence-trimmed to the same protocol. For low-SNR noisy calls, run front-end speech enhancement first, because the LoRA fine-tune alone reduces its benefit. The same checkpoint that looks great on clean trimmed segments will underdeliver on raw street-noise audio.

Run the gates in order. If the corpus passes Rule 4 and the labels pass Rule 3, the only remaining failure mode is audio mismatch — which is why Rule 5 is the gate you verify last, on the exact deployment channel, before accepting the published relative reduction as your validation target. The public source record does not include training-dataset composition, hyperparameters, or evaluation-set details for the fine-tuning run, so treat these gates as the reproducibility layer the paper itself leaves open.

Rule 5 — Accept the published relative cut only when your input SNR is at or above the benchmark’s SNR floor and your segments are silence-trimmed to the same protocol. For low-SNR noisy calls, run front-end speech enhancement first, because the LoRA fine-tune alone reduces its benefit. The same checkpoint that looks great on clean trimmed segments will underdeliver on raw street-noise audio.

Run the gates in order. If the corpus passes Rule 4 and the labels pass Rule 3, the only remaining failure mode is audio mismatch — which is why Rule 5 is the gate you verify last, on the exact deployment channel, before accepting the published relative reduction as your validation target. The public source record does not include training-dataset composition, hyperparameters, or evaluation-set details for the fine-tuning run, so treat these gates as the reproducibility layer the paper itself leaves open.

What to do next

Frequently Asked Questions

Which exact projection matrices does the LoRA adapter modify in Whisper large-v3?

The LoRA delta is placed on the q_proj and v_proj linear layers of each decoder cross-attention block.

How many Hugging Face downloads did openai/whisper-large-v3 have when the page was fetched?

Hugging Face shows 5,497,354 downloads for openai/whisper-large-v3 at fetch time.

What happens to the LoRA checkpoint's relative WER cut on street-noise calls below the evaluation SNR floor?

Run the same LoRA checkpoint on real street-noise calls at a much lower SNR, and the relative WER cut shrank substantially.

Why can the LoRA fine-tune be sure the WER change comes from the learned delta and not initialization?

A is initialized with a scaled Gaussian distribution and B is zero-initialized, so the forward pass at initialization is numerically identical to the base checkpoint.

What is the US-English regression tradeoff if you choose full fine-tuning instead of LoRA?

The full parameter update buys a little extra Indian-English performance and pays for it with a larger US-English regression.

Which accent group saw the weakest WER improvement in the IndiCall breakdown?

Punjabi-English showed the weakest gain, so do not assume uniform lift.

Quick answers

Sources: Reddit, Reddit, arXiv, arXiv, Reddit

Also worth reading: How to turn your voice recordings into accurate text without typing a single word: How to turn your voice · Achieve seamless sound effects in your audio creations: Achieve seamless sound effects in · Stop typing your notes convert audio to text instantly: Stop typing your notes convert