hhhhhhy5/MFD-Qwen3-8B

MFD drafter for Qwen3-8B (1.42M prompts)

A Multi-Future Drafting (MFD) draft model for Qwen/Qwen3-8B, from Multi-Future Drafting: Mixture Block Drafters for Tree-Verified Speculative Decoding (arXiv:2611.13017; code: https://github.com/hhhhhhy5/multi-future-drafting; project page: https://hhhhhhy5.github.io/multi-future-drafting/).

MFD is a DFlash block drafter with K = 8 learned mode embeddings at its input. All 8 modes draft the block in the same forward pass, each one coherent future. A mixture draft tree then keeps the B prefixes with the highest best-mode log-probability, and the target verifies the tree in one pass. Verification is the standard tree verification, so decoding is lossless: the output is the target's own.

Model

Target	Qwen/Qwen3-8B (no thinking)
Architecture	DFlash drafter: 5 layers of width 4096, reading target layers [1, 9, 17, 25, 33]; block of 16 (15 drafted positions); 8 mode embeddings (mode_embed, 8 x 4096) and a linear gate (mode_gate, unused at inference)
Parameters	1049M, bf16; the target's embedding and LM head are used at run time and are not stored here
Initialization	z-lab/Qwen3-8B-DFlash-b16 (MIT)
Training data	the 1.42M prompts of mlabonne/open-perfectblend (the data of Domino and DARTree), with responses regenerated by Qwen3-8B without thinking (T = 0.7, top-p 0.8, top-k 20, at most 2048 new tokens)
Training	3 epochs, AdamW, learning rate 1e-4 (4% warmup, cosine), gradient clipping 1, sequences up to 3072 tokens, 36 sequences per step, 64 anchor blocks per sequence x 8 modes
Objective	relaxed winner-take-all over the 8 modes (the mode with the largest soft coverage of a block gets weight 0.9, the others 0.1 / 7), each mode trained with the D-PACE loss (smoothing 0.5); gate cross-entropy with weight 0.1

Results (from the paper)

Hugging Face Transformers, batch size 1, one NVIDIA RTX PRO 6000 Blackwell. Each cell is the speedup over autoregressive decoding, with the acceptance length tau in parentheses, as the mean over GSM8K, MATH-500, HumanEval, MBPP and MT-Bench. Every tree uses B = 128, the best budget on this target (Table 1):

Method	T = 0	T = 1
DFlash chain (released drafter)	4.26x (5.99)	3.48x (4.88)
DDTree (released drafter)	5.73x (8.18)	5.00x (7.13)
D-PACE + DDTree, same data, 3 ep	6.43x (9.18)	5.26x (7.48)
This drafter (MFD tree)	6.98x (10.18)	5.88x (8.52)

SGLang 0.5.22 with the MFD patch, batch size 1, the same 5 tasks, B = 63, 64K draft vocabulary (Table 2):

Method	T = 0	T = 1
DFlash chain (released drafter, SGLang native)	4.39x (5.90)	3.62x (4.88)
DDTree (released drafter)	5.27x (7.79)	4.56x (6.74)
D-PACE + DDTree (1.42M prompts)	5.99x (8.84)	4.82x (7.15)
This drafter	6.43x (9.72)	5.26x (7.96)

Usage

The drafter runs with the code of https://github.com/hhhhhhy5/multi-future-drafting. Hugging Face inference (from inference/hf, HF inference environment):

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from model import DFlashDraftModel
from ddtree import ddtree_generate, enable_cpp_tree, enable_mode_attention

target = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", attn_implementation="sdpa", dtype=torch.bfloat16).cuda().eval()
draft = DFlashDraftModel.from_pretrained("hhhhhhy5/MFD-Qwen3-8B", attn_implementation="sdpa", dtype=torch.bfloat16).cuda().eval()
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")
text = tok.apply_chat_template([{"role": "user", "content": "What is 17 * 23?"}], tokenize=False,
                               add_generation_prompt=True, enable_thinking=False)
ids = tok.encode(text, return_tensors="pt").cuda()
enable_cpp_tree(True, fused_logits=True)
enable_mode_attention(True)
out = ddtree_generate(model=draft, target=target, input_ids=ids, mask_token_id=draft.mask_token_id,
                      max_new_tokens=512, block_size=draft.block_size, tree_budget=128,
                      stop_token_ids=[tok.eos_token_id], temperature=0.0)
print(tok.decode(out.output_ids[0, ids.shape[1]:]))

SGLang serving (serving environment of the code repository):

bash inference/sglang/launch_server.sh --model Qwen/Qwen3-8B --draft hhhhhhy5/MFD-Qwen3-8B \
    --draft-vocab inference/sglang/vocab/draft_vocab_qwen3-8b_64k.json \
    --engine-config inference/sglang/configs/mfd-b63.env --port 30000

Limitations

• The mixture tree pays off at batch sizes 1-2. In SGLang, single-mode trees serve faster from batch 8 on; an MFD-D drafter keeps one engine fast at every batch size.
• The drafter is tied to its target: it reads the target's hidden states and uses its embedding and LM head.

License and attribution

License: Apache-2.0. This drafter is fine-tuned from z-lab/Qwen3-8B-DFlash-b16 (MIT), whose notice applies to the weights it started from. The training responses were generated by Qwen/Qwen3-8B (Apache-2.0). Prompt source: mlabonne/open-perfectblend (Apache-2.0).

Citation

@article{huang2026mfd,
  title={Multi-Future Drafting: Mixture Block Drafters for Tree-Verified Speculative Decoding},
  author={Huang, Haoyu and Kang, Jialiang and Yang, Linlin and Yang, Xiao and Zhang, Baochang},
  journal={arXiv preprint arXiv:2611.13017},
  year={2026}
}