MFD drafter for Qwen3.5-4B (1.42M prompts)

A Multi-Future Drafting (MFD) draft model for Qwen/Qwen3.5-4B, from Multi-Future Drafting: Mixture Block Drafters for Tree-Verified Speculative Decoding (arXiv:2611.13017; code: https://github.com/hhhhhhy5/multi-future-drafting; project page: https://hhhhhhy5.github.io/multi-future-drafting/).

MFD is a DFlash block drafter with K = 8 learned mode embeddings at its input. All 8 modes draft the block in the same forward pass, each one coherent future. A mixture draft tree then keeps the B prefixes with the highest best-mode log-probability, and the target verifies the tree in one pass. Verification is the standard tree verification, so decoding is lossless: the output is the target's own.

Model


Target	Qwen/Qwen3.5-4B (thinking mode)
Architecture	DFlash drafter: 6 layers of width 2560, reading target layers [1, 5, 9, 13, 17, 21, 25, 29]; block of 16 (15 drafted positions); 8 mode embeddings (`mode_embed`, 8 x 2560) and a linear gate (`mode_gate`, unused at inference)
Parameters	634M, bf16; the target's embedding and LM head are used at run time and are not stored here
Initialization	z-lab/Qwen3.5-4B-DFlash (Apache-2.0)
Training data	the 1.42M prompts of mlabonne/open-perfectblend (the data of Domino and DARTree), with responses regenerated by Qwen3.5-4B in thinking mode (T = 0.7, top-p 0.8, top-k 20, at most 3072 new tokens, responses cut at that length kept)
Training	3 epochs, AdamW, learning rate 1e-4 (4% warmup, cosine), gradient clipping 1, sequences up to 3072 tokens, 36 sequences per step, 64 anchor blocks per sequence x 8 modes
Objective	relaxed winner-take-all over the 8 modes (the mode with the largest soft coverage of a block gets weight 0.9, the others 0.1 / 7), each mode trained with the D-PACE loss (smoothing 0.5); gate cross-entropy with weight 0.1

Results (from the paper)

Hugging Face Transformers, batch size 1, one NVIDIA RTX PRO 6000 Blackwell. Each cell is the speedup over autoregressive decoding, with the acceptance length tau in parentheses, as the mean over GSM8K, MATH-500, HumanEval, MBPP and MT-Bench. Every tree uses B = 256, the best budget on this target (Table 1):

Method	T = 0	T = 1
DFlash chain (released drafter)	4.85x (6.25)	3.29x (4.23)
DDTree (released drafter)	6.53x (8.78)	4.77x (6.51)
D-PACE + DDTree, same data, 3 ep	7.31x (9.82)	4.74x (6.46)
This drafter (MFD tree)	7.78x (10.72)	5.18x (7.23)

Usage

The drafter runs with the code of https://github.com/hhhhhhy5/multi-future-drafting. Hugging Face inference (from inference/hf, HF inference environment):

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from model import DFlashDraftModel
from ddtree import ddtree_generate, enable_cpp_tree, enable_mode_attention

target = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-4B", attn_implementation="sdpa", dtype=torch.bfloat16).cuda().eval()
draft = DFlashDraftModel.from_pretrained("hhhhhhy5/MFD-Qwen3.5-4B", attn_implementation="sdpa", dtype=torch.bfloat16).cuda().eval()
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-4B")
text = tok.apply_chat_template([{"role": "user", "content": "What is 17 * 23?"}], tokenize=False,
                               add_generation_prompt=True, enable_thinking=True)
ids = tok.encode(text, return_tensors="pt").cuda()
enable_cpp_tree(True, fused_logits=True)
enable_mode_attention(True)
out = ddtree_generate(model=draft, target=target, input_ids=ids, mask_token_id=draft.mask_token_id,
                      max_new_tokens=4096, block_size=draft.block_size, tree_budget=256,
                      stop_token_ids=[tok.eos_token_id], temperature=0.0)
print(tok.decode(out.output_ids[0, ids.shape[1]:]))

Hybrid Gated-DeltaNet targets are served by the Hugging Face path only; it verifies trees exactly with the tree kernels of inference/hf/qwen35_tree.py.

Limitations

Measured at batch size 1 in the Hugging Face harness, the only path in the code repository that serves this hybrid target.
The drafter is tied to its target: it reads the target's hidden states and uses its embedding and LM head.

License and attribution

License: Apache-2.0. This drafter is fine-tuned from z-lab/Qwen3.5-4B-DFlash (Apache-2.0), whose notice applies to the weights it started from. The training responses were generated by Qwen/Qwen3.5-4B (Apache-2.0). Prompt source: mlabonne/open-perfectblend (Apache-2.0).

Citation

@article{huang2026mfd,
  title={Multi-Future Drafting: Mixture Block Drafters for Tree-Verified Speculative Decoding},
  author={Huang, Haoyu and Kang, Jialiang and Yang, Linlin and Yang, Xiao and Zhang, Baochang},
  journal={arXiv preprint arXiv:2611.13017},
  year={2026}
}