RedHatAI/Qwen3.5-397B-A17B-FP8-dynamic-speculator.dflash

This is a DFlash speculator model for RedhatAI/Qwen3.5-397B-A17B-FP8-dynamic.

Training Details

This model was trained using the Speculators library on a subset of Magpie-Align/Magpie-Llama-3.1-Pro-300K-Filtered and the train_sft split of HuggingFaceH4/ultrachat_200k. Responses were regenerated by RedHatAI/Qwen3.5-397B-A17B-FP8-dynamic and stored at Qwen3.5-397B-A17B-FP8-dynamic-responses

Commands

Using the Speculators library and the helper scripts provided in the repo.

Prepare data

# In virtual environment with speculators installed
python scripts/prepare_data.py \
  --model RedhatAI/Qwen3.5-397B-A17B-FP8-dynamic
  --data ./regenerated_data.jsonl \
  --assistant-pattern "<\|im_start\|>assistant\s*([\s\S]*?)<\|im_end\|>" \
  --output ./output \
  --seq-length 16384

Launch vLLM

# In (separate) virtual environment with [vLLM](https://github.com/imargulis/vllm/tree/fix/extract-hidden-states-hybrid-block-size) installed
CUDA_VISIBLE_DEVICES=0,1,2,3 vllm_venv/bin/python scripts/launch_vllm.py \
  RedhatAI/Qwen3.5-397B-A17B-FP8-dynamic \
  --target-layer-ids 3 15 23 35 47 \
  --max-model-len  32000 \
  --reasoning-parser qwen3 \
  --language-model-only \
  --kv-cache-dtype fp8 \ 
  --max-num-batched-tokens 32768\
  --tensor-parallel-size 4 \ 
  --async-scheduling

Launch training

Must be run once vLLM has finished launching and is running in the background.

# In virtual environment with speculators installed
CUDA_VISIBLE_DEVICES=4,5,6,7 torchrun \
  --standalone \
  --nproc_per_node 4 \
  scripts/train.py \
  --verifier-name-or-path RedhatAI/Qwen3.5-397B-A17B-FP8-dynamic \     
  --data-path ./output \    
  --on-missing generate \    
  --on-generate delete \    
  --scheduler-type cosine \    
  --draft-vocab-size 24576 \    
  --max-anchors 3072 \    
  --target-layer-ids 3 15 23 35 47 \
  --speculator-type dflash \     
  --logger trackio  \    
  --lr 0.0006 \    
  --epochs 5 \ 
  --draft-config 397_config.json

Model Specifications

Base Model	RedhatAI/Qwen3.5-397B-A17B-FP8-dynamic
Chat Template	RedhatAI/Qwen3.5-397B-A17B-FP8-dynamic (use /chat/completions endpoint)
Format	Safetensors
License	Apache 2.0
Validation Hardware	Nvidia H200

Deployment

# Install vLLM from the required PR
pip install git+https://github.com/vllm-project/vllm.git  
                                                                                                                                                                                                                                                                                                          
# Deploy with speculative decoding                                                                                                                                                                                                                                                                        
vllm serve RedhatAI/Qwen3.5-397B-A17B-FP8-dynamic \                                                                                                                                                                                                                                                                                
    --tensor-parallel-size 8 \                                                                                                                                                                                                                                                                            
    --max-num-batched-tokens 32768 \
    --attention-backend FLASH_ATTN \ 
    --speculative-config '{                                                                                                                                                                                                                                                                               
        "model": "RedHatAI/Qwen3.5-397B-A17B-FP8-dynamic-speculator.dflash",                                                                                                                                                                                                                                                   
        "num_speculative_tokens": 7,                                                                                                                                                                                                                                                                   
        "method": "dflash"                                                                                                                                                                                                                                                                                
    }'

Preliminary Evaluations

Per-position token acceptance rates across datasets:
(with reasoning enabled)

Latency Speedup

Speedup comparisons of DFlash speculative decoding vs. baseline (no speculation) at varying request rates on Nvidia H200:

References

Paper: DFlash: Block Diffusion for Flash Speculative Decoding