whisper-large-v3 DFlash draft

A DFlash block-diffusion speculative-decoding draft for openai/whisper-large-v3. The draft proposes 15 tokens per step in one forward pass, and Whisper verifies them under greedy decoding. It was trained on AWS Trainium2 using only openly licensed speech data. The training code is at github.com/jimburtoft/whisper-dflash.

Format: the public DFlash checkpoint format (DFlashDraftModel, Qwen3-style layers), plus two tensors for the audio context, described below.
Size: 5 draft layers, hidden 1280, 20 query heads x 64, 4 KV heads, SwiGLU 5120. 132.7M parameters (BF16, 265 MB). The token embedding / LM head is not shipped. The draft uses Whisper's own tied embedding from the target checkpoint.
Block size 16 (num_speculative_tokens = 15).
Target features: outputs of Whisper decoder layers [1, 8, 15, 22, 29], before the final layer norm.
Audio context (encoder-decoder extension): the Whisper decoder's hidden states only reflect the audio that has already been decoded. So the draft also attends to the target's encoder output: 1500 frames, stacked 4 at a time into 375 positions, projected by audio_fc (5120 -> 1280) and audio_norm. These become keys and values without RoPE that every block query can see. They depend only on the audio, so a server computes them once per request. Without this context the draft reached a held-out tau of about 1.5. With it, tau was 2.7 after the same short pilot run. dflash_config.audio_context = "own_fc", audio_stack = 4.
Mask token: id 50361 (<|startoflm|>). It is in Whisper's suppress_tokens, so the model never generates it.

Results

All numbers come from real audio, greedy decoding and held-out sets that were never used in training. tau = mean tokens per verify step: accepted draft tokens plus the target's bonus token.

Acceptance (offline, reproducible with the released code)

The evaluation uses 64 held-out clips per set (<= 30 s), with an anchor at every response position, under the vLLM DFlash greedy accept rule. The second column runs distil-whisper/distil-large-v3 as a K-token autoregressive draft on the same anchors, for reference.

set	DFlash tau (block 16)	distil-large-v3, K=10 / K=15
LibriSpeech test-clean	7.57	8.22 / 10.11
LibriSpeech test-other	7.52	8.05 / 9.92
VoxPopuli en test	7.77	8.45 / 10.65
FLEURS en_us test	6.39	7.68 / 9.21
AMI ihm test	5.60	6.61 / 7.37
Common Voice 17 en test	4.13	5.58 / 5.96
all (7,676 anchors)	6.77	7.67 / 9.25

distil-large-v3 accepts more tokens per verify step. DFlash still serves faster, because it drafts a whole block in one pass instead of K sequential passes (see below).

End-to-end serving (vllm-neuron, Neuron SDK 2.33, Trainium2)

Environment: one trn2.3xlarge (one Trainium2 device), the public Neuron SDK 2.33 DLAMI vLLM venv (vllm 0.29.0, vllm-neuron 0.29.0.1.2.0), with the public Whisper branch jimburtoft/vllm-neuron whisper/0.29-public installed on top, at commit aa283ad. That commit includes the DFlash-for-Whisper change. Each request goes through /v1/audio/transcriptions, one request at a time, in the latency layout (TP=4, LNC=2). Each figure is the mean of the per-clip minimum over 2 passes after 1 warm-up. The latency set is the public 18-clip pub18-ls-e22 set: 3 clips per 5 s bucket from 0-30 s, drawn from LibriSpeech test-clean / test-other and Earnings22-Cleaned-AA-chunked.

config	mean	p50	p90	tokens per verify step
no draft	249.9 ms	191.9 ms	476.9 ms	1
distil-large-v3 draft, K=10 (async)	95.3 ms	85.0 ms	155.8 ms	8.80
this DFlash draft, block 16 (async)	89.6 ms	71.5 ms	139.7 ms	7.09

On these 18 clips the output tokens are identical to the no-draft server, and also on 48 LibriSpeech test-other clips. The verifier decides every token.

WER (%, open_asr_leaderboard normaliser), full sets:

set	no draft	distil-large-v3	this draft	tokens per verify step (this draft)
Earnings22-Cleaned-AA-chunked (341 chunks)	8.52	8.65	8.65	4.48
LibriSpeech test-clean (2,620)	1.47	1.47	1.47	9.09
LibriSpeech test-other (2,939)	3.12	3.12	3.13	7.66

On these larger sets a small share of transcripts differ from the no-draft server: 23 of 341, 18 of 2,620 and 39 of 2,939. The cause is bf16 near-ties in the multi-token verify step. The distil draft shows a similar number.

Throughput layout: in the 8-replica layout (TP=1 x DP=8, LNC=1) the draft runs and gives correct output. Its throughput is lower than serving without a draft, because at TP=1 the draft graph takes about 30 ms per step. For throughput serving, use the distil-large-v3 draft (K=4) or no draft. Use this draft for single-request latency.

Usage

vllm-neuron on Trainium2 (latency layout)

Install the public Whisper branch into the SDK 2.33 DLAMI vLLM venv as described in NeuronStuff whisper-vllm-neuron. Then start the lat configuration from that guide with:

  --speculative-config '{"method": "dflash", "model": "jburtoft/whisper-large-v3-DFlash", "num_speculative_tokens": 15}'

Other servers

The serving side needs two things. First, the target must expose auxiliary hidden states (the input of decoder layers target_layer_ids + 1). Second, the draft's audio K/V must be prepended. Upstream vLLM's DFlash proposer (qwen3_dflash.py) loads the decoder side of this checkpoint, but it has no audio-context input. It would reject audio_fc / audio_norm, and running it without them would give a much weaker draft.

Reference implementation (PyTorch)

The training code repository contains:

model/dflash.py: the draft, including the audio context;
model/release.py: load and save;
eval/acceptance_eval.py: offline acceptance.

Without the audio context, the block outputs match the public z-lab/dflash reference to fp32 rounding.

Training

Full reproduction steps are in the code repository README.

Data: 175,000 clips (464 h, 1-30 s) from LibriSpeech, LibriTTS-R, Multilingual LibriSpeech (en), Common Voice 17 (en), VoxPopuli (en), The People's Speech (clean config, CC BY / public-domain sources only) and AMI. Eval speakers, eval meetings and transcripts that overlap eval sets were removed from training. FLEURS and Earnings22 were used for evaluation only.
Labels: Whisper's own greedy transcripts of the real audio (English, transcribe, no timestamps, with Whisper's token suppression). Whisper's decoder hidden states and encoder output were computed on the fly during training.
Objective (paper recipe):
- random anchors in the response; each block is the anchor plus 15 mask tokens;
- bidirectional attention within a block, to the target features before the anchor, and to the audio context;
- position weights that decay exponentially (gamma 7).
Schedule: 150,000 steps of 8 clips x 16 blocks; AdamW at 6e-4 with a cosine schedule (4 % warmup); bf16 autocast with fp32 master weights; gradient clip 1.0. Held-out tau levels off from about 130k steps.
Compute: AWS Trainium2 (one trn2.3xlarge) with PyTorch on Neuron (torch.compile(backend="neuron")), on the public SDK 2.33 DLAMI torch-neuronx venv. Three cores ran the frozen target and one trained the draft. Labelling took about 5 h and training about 19 h. The target is the stock transformers Whisper model.

Limitations

English transcription only (<|en|><|transcribe|><|notimestamps|>). Translation, timestamps and other languages were not trained.
Clips of up to 30 s (one Whisper window).
Acceptance is lower on short crowd-sourced clips (Common Voice) and on earnings-call speech, which does not appear in the training data. On Earnings22 chunks the draft accepts about 4.5 tokens per verify step, against about 9 on LibriSpeech.
Trained for block size 16.

License and attribution

The draft weights and training code are released under Apache-2.0. The draft is derived from the outputs, decoder hidden states and encoder outputs of openai/whisper-large-v3 (Apache-2.0, OpenAI). It was trained on the datasets listed above. See NOTICE for the required attributions and ATTRIBUTION_peoples_speech.csv for per-source credits for The People's Speech.

Users must not attempt to identify speakers in Common Voice, LibriSpeech or Multilingual LibriSpeech. This model distributes no audio and no speaker identifiers.

Citation

@article{chen2026dflash,
  title   = {DFlash: Block Diffusion for Flash Speculative Decoding},
  author  = {Chen, Jian and Liang, Yesheng and Liu, Zhijian},
  journal = {arXiv preprint arXiv:2602.06036},
  year    = {2026}
}
@article{radford2022whisper,
  title   = {Robust Speech Recognition via Large-Scale Weak Supervision},
  author  = {Radford, Alec and Kim, Jong Wook and Xu, Tao and Brockman, Greg and McLeavey, Christine and Sutskever, Ilya},
  journal = {arXiv preprint arXiv:2212.04356},
  year    = {2022}
}