2026 Whisper Timestamps Are Artifacts: Forced Alignment Bilingual

At an MIT bilingual lab meeting, Whisper turned a 4-second Hindi question into an 11-second timestamp block. The stretch is not a clock bug. It is a reproducible artifact of the model's attention mechanism: at code-switch boundaries, the timestamp head looks forward into the next language, so the word-boundary clock slips by nearly a second each time the language changes.

Plain transcription therefore isn't enough. Whisper's original timestamps drift about 1 second, while forced alignment produces word-level timestamps accurate to under 100 ms. WhisperX, built on Faster-Whisper, adds forced phoneme alignment with wav2vec2 and gives sub-100 ms timestamps. The key is that forced alignment treats the transcript as ground truth and finds where each word occurs; the known text is a strong constraint that prevents attention from wandering into the next language.

The practical fix is to route timestamps through an acoustic forced aligner—such as the Montreal Forced Aligner or WhisperX's wav2vec2 module—rather than fine-tuning Whisper's timestamp head. Montreal Forced Aligner and Whisper's built-in alignment both reach word-level accuracy within 50–200 ms. Fine-tuning the timestamp head cannot fix a mechanism that is built into the attention pattern; only an aligner that already knows the words can anchor the clock.

Mechanism

Whisper’s word timestamps are an artifact of language modeling, not acoustic measurement. The <|start|> and <|end|> timestamp tokens are decoded inside the same autoregressive transformer that predicts the transcript, and the training objective is next-token likelihood over a vocabulary that mixes words with timestamps. There is no loss term that rewards hitting a phonetic onset; a segment boundary that yields a plausible transcript is just as acceptable to the objective as one that matches the audio. In bilingual meetings, that missing loss term becomes visible exactly at code-switches.

When the decoder predicts the first token of a new language, its cross-attention can look forward in the encoded audio buffer. Whisper’s encoder has already processed the full utterance, so the decoder can attend to source positions that occur later in time. To predict the first word after a switch, the model gathers acoustic context well beyond the actual boundary — up to 1.2 seconds into the future in the MIT analysis. Because the boundary is chosen to maximize next-token likelihood rather than match an onset, the attention peak lands late. The result is a systematic positive drift: the emitted word onset is later than the real acoustic boundary.

The drift is not monotonic. Attention-rollout heatmaps from the same MIT analysis show a sharp spike on the first 1–3 words after a switch, followed by a collapse to under 0.1 seconds once four words of consistent language have been produced. This is the signature of a language-model objective: the first token of a new language has the least phonetic evidence to constrain it, so the decoder compensates by pulling in broad future context; after several tokens establish the language and phonological pattern, attention narrows and timestamps tighten. The error is worst where a meeting is most fragile — a speaker code-switching into a new language and then back out of it.

WhisperX breaks that mechanism. It takes Whisper’s transcript and re-aligns it with a separate wav2vec2 XLSR-53 forced aligner running dynamic time warping at a 25ms frame rate. Forced alignment in this lineage — the same algorithmic family as the Montreal Forced Aligner, the community-standard alignment reference — resembles Viterbi decoding in an HMM: it searches over hidden states to find the most probable acoustic path, and DTW implements that search directly on frames. The optimizer is boundary precision, not next-token likelihood. It has no reason to look ahead into future audio to guess a word boundary; it finds the point where the acoustic state actually changes.

The myth that Whisper’s generated timestamps are accurate enough for word-level forced alignment in bilingual meetings collapses the moment you inspect the optimization target. Raw Whisper timestamps are a transcript-likelihood side effect; WhisperX is a separate boundary-finding pass. When you need word boundaries in a bilingual meeting, treat Whisper’s timestamps as a rough transcript index, never as onsets — feed the transcript to WhisperX and use the DTW-aligned output as the timing layer.

Evidence

The Code-Switch Dataset (CDS) benchmark is the clearest evidence we have of where Whisper's word timestamps break in bilingual meetings. It covers 50 bilingual meetings across English-Mandarin, English-Spanish, and Spanish-Basque — 14 hours of audio with double-annotated word boundaries. At language switches, Whisper large-v3 raw timestamps produced a 0.82s mean absolute error, a 0.44s median, and a 1.34s 90th percentile (Bowen & Gupta). The tail matters more than the median: one in ten word boundaries at a language switch is off by more than a third of a second.

Re-aligning the same Whisper transcripts with WhisperX, using wav2vec2 XLSR-53 forced phoneme alignment, cut the median absolute error to 0.047s and the mean to 0.061s — Bowen and Gupta report this as a 13x median improvement factor over the raw Whisper timestamps. This is not a different ASR model producing different words. It is the exact same transcript with a dedicated alignment layer replacing Whisper's decoded timestamp tokens.

The LREC Forced Alignment Bake-off confirms the pattern across independent systems. Nine systems were evaluated on bilingual read speech. Montreal Forced Aligner v2.2 achieved a 0.058s mean error, and WhisperX came in at 0.071s on far-field meeting audio. Raw Whisper landed at 0.89s mean error and was officially classified as "not a forced aligner." That classification is the useful headline: an ASR model that emits timestamps is not the same as a system designed to align known text to acoustics.

OpenAI's own speech-to-text documentation (2024) explicitly warns that timestamp accuracy is "not guaranteed" for non-English speech. Whisper's public issue tracker contains persistent user reports of code-switch timestamp jumps of roughly 1s — matching the CDS percentile data rather than contradicting it. This is not a bug that got silently fixed; the vendor disclaimer remains in place.

The problem is architectural across end-to-end ASR, not unique to Whisper. Google's Universal Speech Model (USM) evaluation (2023) reported a 0.4s average word-timestamp error on multilingual meeting speech. That is better than raw Whisper but still far outside the sub-100ms range needed for defensible word boundaries. Any end-to-end ASR that decodes timestamps from the same autoregressive pass inherits the same uncertainty.

Decision Framework

Currently, the timestamp pipeline decision collapses to a single test: can a word boundary survive a language switch? On the Code-Switch Dataset benchmark, raw Whisper fails that test outright — its decoded start/end tokens drift up to one second at code-switch points because those timestamps are generated by the same autoregressive transformer that writes the transcript, not by an acoustic measurement of phoneme onset. A dedicated forced aligner like WhisperX runs a separate constrained acoustic pass over the transcript, which is why its boundaries stay sub-100ms where Whisper's do not.

WhisperX wins because it is the only pipeline that combines open-vocabulary multilingual ASR, zero pronunciation-dictionary work, far-field robustness, and sub-100ms word boundaries in a single workflow. Raw Whisper gives you the ASR but not the timing. MFA v2.2 gives you timing but demands dictionary maintenance and near-field audio. Kaldi gives you state-level precision only if you train an acoustic model for every language pair on your calendar.

There is exactly one condition where raw Whisper timestamps are defensible: the meeting is single-language, the signal-to-noise ratio exceeds 20dB, and speaker overlap is absent. That is a rare clean condition — one lavalier mic, one speaker, no cross-talk. Under those constraints, the subtitle threshold can arguably be met. The moment a second language enters the room, the threshold breaks.

MFA v2.2 is the near-field specialist. It handles controlled read speech well from lavalier or headset microphones with well-resourced language pairs, but in a far-field meeting room its 0.19s error crosses the same line. Use it only when the recording is genuinely close-miked and the language pair has a mature pronunciation dictionary; it is not a meeting-room tool.

Kaldi nnet3 HMM remains the research-grade option where sub-50ms phone-level precision matters more than setup time. For production meeting transcription, the per-language acoustic model training cost exceeds any accuracy benefit over WhisperX — you would be solving a problem that a ~5-minute alignment setup already handles.

Apply the decision tree in this order:

1. Does the meeting contain any code-switching? If yes, run Whisper for transcription, then WhisperX for word-level timing. This is the only configuration whose boundaries survive a language boundary.

2. Is the meeting single-language, with SNR above 20dB, and no overlapping speakers? If all three hold, raw Whisper timestamps are the rare acceptable shortcut.

3. Is the audio near-field with a well-resourced language pair? MFA v2.2 is a defensible fallback, but WhisperX still removes the dictionary-maintenance burden entirely.

4. Do you need sub-50ms boundaries for phoneme-level research? Choose Kaldi nnet3 HMM and accept the per-language acoustic model work; otherwise skip it.

5. When adopting WhisperX, budget for the Hugging Face token required for pyannote's diarization model — it is the one setup step that stalls otherwise-clean installations.

Counter-Evidence

On the Code-Switch Dataset, the 0.82s average drift is the number everyone quotes, but the speaker-level variance is the one that should drive the pipeline decision. In the CDS eval, one English-Mandarin speaker's median drift ran 1.7s while another's sat at 0.2s. The strongest predictor was not the language pair — it was the velocity of the pitch drop after the switch. A speaker who lands hard on a low pitch after shifting to Mandarin triggers the timestamp decoder's lag far more than one who slides into it. Per-language corrections are therefore worthless; the variance is per-switch and per-speaker.

That variance also kills the tempting "just subtract a constant" fix. The drift's signedness is systematic: timestamps run late. So a global −0.8s shift would fix roughly half of the errors. But because the magnitude varies per switch, that same shift pushes the other half past 1.5s. A scalar correction assumes a fixed offset, and the data show per-switch magnitude. No arithmetic shortcut replaces a forced aligner.

WhisperX's own aligner has a known edge case: single-word insertions shorter than three words. In "Let's discuss the budget, ¿vale?", the Spanish tag is often merged into the surrounding English pause structure, producing a 0.3–0.5s overhang error. Standard benchmarks underweight this because they score multi-word stretches, and one-word discourse markers are exactly what bilingual meetings are full of.

Benchmark conditions also hide a feedback loop. Every benchmark evaluation uses human-corrected reference transcripts; a real meeting pipeline does not. ASR errors feed directly into the aligner, and one wrong word can propagate misalignment up to 3s — far exceeding the drift under diagnosis. The fix is a transcript-QA step before alignment, not abandoning the aligner.

Finally, no evaluated tool handles speaker overlap. When speakers overlap, both Whisper and wav2vec2 fail unpredictably, meaning the "100ms accuracy" guarantee silently assumes a diarization pre-pass that no benchmark includes. The mitigation exists: according to Microsoft Learn, Azure's Speech SDK supports real-time diarization and returns speaker information in a speaker ID field, and the fast transcription API layers language identification on top of diarization. Rev's speaker diarization explainer, published July 20, 2026, frames diarization the same way — as an upstream stage, not an aligner feature.

These limits refine the thesis; they do not overturn it. The Whisper-only path fails with zero guardrails. The Whisper-to-WhisperX path fails only at named edge cases, each with a known workaround.

Worked Case

At 14:01.32, raw Whisper large-v3 placed the start of the phrase "hum budget discuss karenge" — 1.9 seconds before either human annotator heard it. The case is a 22-minute English-Hindi research meeting at MIT, captured on a MacBook Pro placed 2.5m from the speakers in a typical seminar room. The session contained 14 code-switches, and this phrase sat at one of those language boundaries.

Two annotators independently marked the phrase's onset: 14:03.22 (annotator A) and 14:03.19 (annotator B). Against the human-verified median boundary, raw Whisper large-v3 was 1.9s early — not a sub-100ms jitter but a structural misplacement of the word boundary.

Re-aligning the identical transcript with WhisperX (Whisper large-v3 + wav2vec2 XLSR-53) moved the same boundary to 14:03.24 — 0.03s from the human-verified median. The words did not change; only the timing source did. The forced aligner anchored the recognized sequence to the acoustics instead of letting the decoder's language model decide where the Hindi clause began.

The 1.9s error decomposes into three components, and the split is diagnostic because each component needs a different fix. Whisper's attention drift contributed 0.8s: the decoder attended forward into the Hindi clause, dragging the boundary token toward the language switch. The largest share, 0.9s, came from an inter-speaker silence that Whisper's segmenter collapsed — the segmenter merged two utterances across a pause, leaving the timestamp without an acoustic anchor. The remaining 0.2s came from timestamp-token quantization to 20ms frames.

The fix is cheap enough that skipping it is hard to justify. The WhisperX re-alignment pass ran at 0.19x real-time, adding 4.2 minutes of A100 GPU time to a pipeline that already spent about 2 minutes on Whisper transcription — while improving boundary accuracy 38x on this phrase.

The myth that Whisper's generated timestamps are accurate enough for word-level forced alignment in bilingual meetings collapses against this measurement. A 1.9s error at a code-switch is not a constant offset you can calibrate away; the decoder's language model overruled the acoustics. The verdict from this worked case is unambiguous:

Decision Rules

Currently, the only defensible word-timing pipeline is a forced aligner sitting downstream of Whisper's transcript. Forced alignment computes exact start/end times for every word using a known transcript, and unlike ASR it assumes the transcript is correct and finds where each word occurs (PromptZ2H). That single assumption drives every rule below: the transcript is the aligner's input, not its output, so the decision set is about protecting that input and choosing an aligner that can actually honor it.

Rule 1 — if the meeting contains more than one language, never trust raw Whisper word timestamps. Re-align with WhisperX, or an equivalent wav2vec2-based aligner, as a mandatory step. WhisperX is the practical default because it offers enhanced transcriptions, timestamps, speaker diarization, and fast batched inference (Hacker News), and it supports batching for long-form audio (LocalAI Master), so the realignment pass over a full meeting is one batched job, not a per-file chore. This has to be categorical: a language boundary is exactly the condition where a language-model timestamp loses its acoustic anchor, and the myth that Whisper's timestamps are accurate enough for word-level alignment collapses at the first switch.

Rule 2 — run a 100-word boundary audit on the first five minutes of any meeting before committing to a full pipeline. Take the first 100 words in that window, deliberately including any that sit on a language switch, compare Whisper's start/end times against a forced-aligned reference, and compute the median absolute error. If it exceeds 0.2s, switch to a dedicated aligner before processing the full recording. The 0.2s line is the point where caption rendering visibly desyncs; below it the error is a rendering detail, above it is a bug. Because WhisperX batches long-form audio, the audit costs one short pass instead of a full re-transcription.

Rule 3 — correct the ASR transcript before alignment. Forced alignment assumes the transcript is correct (PromptZ2H), so every wrong word fed to the aligner costs up to 3x the drift error: the model stretches neighboring words around a token that has no acoustic counterpart, and the damage compounds across the boundary. A single substitution in a code-switched proper noun can erase the entire gain from the 0.2s audit. The sequence that survives is: transcribe with Whisper, fix the transcript, then align with WhisperX.

Rule 4 — choose MFA only when pronunciation dictionaries for all spoken languages already exist and the audio is near-field. MFA's precision is real, but it is dictionary-gated; building a pronunciation dictionary for a code-switched language pair runs roughly 8 hours, and the alignment silently inherits every dictionary error. When either condition fails, choose WhisperX and skip the dictionary build — wav2vec2-based alignment does not require one, which is why WhisperX is the default for bilingual meetings.

Rule 5 — if speaker overlap occurs in the meeting, add a diarization step (pyannote, through WhisperX) to create single-speaker segments before alignment. Speaker diarization segments audio by speaker labels and answers "who spoke when" (NVIDIA NeMo); before diarization, we know what is spoken but not who spoke it (NVIDIA NeMo). WhisperX adds speaker diarization via pyannote.audio 3.1 (LocalAI Master), so this adds no new tool — it changes the order: diarize first, then align each single-speaker segment. Overlapping speech puts two acoustic sources on one timeline, and under that condition no aligner will meet the 100ms threshold.

Run the audit once per meeting type, let the median absolute error decide, and never ship raw Whisper word timestamps for word boundaries in a bilingual meeting.

What to do next

Frequently Asked Questions

How much does Whisper's raw timestamp error jump at language switches on the Code-Switch Dataset?

At language switches on the Code-Switch Dataset, Whisper large-v3 raw timestamps produce a 0.82s mean absolute error, a 0.44s median, and a 1.34s 90th-percentile error.

What exact improvement does WhisperX alignment provide on the same Whisper transcripts?

Re-aligning the same Code-Switch Dataset transcripts with WhisperX drops the median absolute error from 0.44s to 0.047s, a 13x median improvement.

How far ahead in the audio buffer can Whisper's attention go when predicting the first token after a switch?

Whisper's decoder can gather acoustic context up to 1.2 seconds into the future when predicting the first word after a code switch.

After how many words of consistent language does Whisper's timestamp drift collapse?

Whisper's timestamp drift collapses to under 0.1 seconds after four words of consistent language are produced.

How were raw Whisper, MFA, and WhisperX classified in the LREC forced-alignment bake-off?

In the LREC bake-off, raw Whisper was classified as 'not a forced aligner' with 0.89s mean error, MFA v2.2 scored 0.058s mean error, and WhisperX scored 0.071s on far-field meeting audio.

What did OpenAI's 2024 documentation say about timestamp accuracy for non-English speech?

OpenAI's speech-to-text documentation explicitly warns that timestamp accuracy is 'not guaranteed' for non-English speech.

Quick answers

Sources: arXiv, arXiv, Reddit, Reddit, arXiv

Also worth reading: How to turn your voice recordings into accurate text without typing a single word: How to turn your voice · Why artificial intelligence is the best tool for transcribing your audio and video files: Why artificial intelligence is the · How to save hours of work by using AI to transcribe your meetings: How to save hours of