jiapingW/Qwen3-8b-dspine-SpecForge

DSpine · Qwen3-8B draft · step 66582

Device label: RTX PRO5000. Target model: Qwen3-8B.

Core files

File	Contents
model.safetensors, config.json	Draft weights and inference configuration for the selected checkpoint
training/launch.yaml	Core training parameters; supply deployment settings for your environment
training/draft_config.json	Model architecture and algorithm parameters used during training
tensorboard/	Logs from the entire training run, including steps beyond the selected checkpoint where applicable
bench/results.json	Per-benchmark and aggregate results for both verification widths, at full numerical precision
README.md	Training method and benchmark results

Training

Data source: Regenerated PerfectBlend training data (JSONL).

Training uses PerfectBlend data regenerated by Qwen3-8B at temperature 0 with thinking disabled, formatted with the qwen3-instruct template and no system prompt. The target model supplies hidden states and the information required for distillation, and the draft model trains on the extracted features. The 32 training ranks run separately from the 8 target inference services, with Mooncake transferring features between them.

This model was trained with max_length 3072, while DFlash, Domino, and DSpark in the same comparison used 8192. Training was planned for 6 epochs; the selected checkpoint steps and training sequence lengths differ across methods.

Parameter	Setting
Planned total steps / selected checkpoint	66582 / 66582
block_size / max_length	16 / 3072
Draft layers / hidden size / FFN size	5 / 4096 / 12288
attention heads / KV heads	32 / 8
Target feature layer indices	1, 9, 17, 25, 33
Microbatch per rank / gradient accumulation	2 / 2
Effective global batch size	32 × 2 × 2 = 128
Optimizer	AdamW with FP32 master parameters, weight_decay=0
Peak learning rate / schedule	6e-4 / cosine
LR warmup / gradient clipping	4% of total steps / max_grad_norm=1.0
num_anchors / loss_decay_gamma	512 / 7.0
objective_chunk_blocks / seed	128 / 42
Parameter dtype / attention	BF16 / FlexAttention
Gradient strategy / kernel	NO_SHARD / DDP; Liger

DSpine adds token information transfer and injection across layers within the parallel draft backbone, then rescores candidates conditioned on the preceding token. The transfer rank is 1024, the message size is 512, and top-k is 16. Both the backbone and refinement use 0.1 × CE + 0.9 × L1, with an additional cosine alignment term. The alignment coefficient warms up over 146 steps to 0.5, and the alignment term is scaled by the detached CE value. The target probability mass threshold for the candidate set is 1e-4; refinement L1 is included only when the mass exceeds this threshold.

For the first 3 layers, the teacher replacement probability stays at 0.5 for the first 1/6 of the planned steps, then decreases linearly to 0 by 1/3 of the planned steps, or step 22194. Distillation supervision from the target model continues afterward; inference does not use label replacement. Training completed at step 66582, and this model is the final checkpoint.

See training/launch.yaml for the core training parameters; supply deployment settings for your environment. The learning rate schedule and training phases defined as fractions of the run use the planned total step count.

Benchmark (temperature=0)

Each mode contains 2400 requests: GSM8K 1319, MATH500 500, HumanEval 164, MBPP 257, and MT-Bench with 80 two-turn conversations (160 requests). Both draft16/verify16 (D16/V16) and draft16/verify8 (D16/V8) use a draft block size of 16; only the target verification width changes. The verification width includes the anchor.

Benchmark	Requests	D16/V16 request macro τ	D16/V16 micro τ	D16/V8 request macro τ	D16/V8 micro τ
GSM8K	1319	10.207611	9.719553	6.595816	6.484690
MATH500	500	9.862971	9.174848	6.495932	6.341415
HumanEval	164	7.797521	7.684850	5.821919	5.820436
MBPP	257	7.266683	6.795693	5.580626	5.411276
MT-Bench	160	5.560371	4.421689	4.387794	3.884965
All requests, pooled	2400	9.346381	8.301265	6.266212	5.981371

Aggregate	D16/V16	D16/V8
Equal-weight mean of request macro τ across the five benchmarks	8.139031	5.776417
Equal-weight mean of micro τ across the five benchmarks	7.559327	5.588556
pooled completion_tokens	1034404	1029035
pooled verify_count	124608	172040

See bench/results.json for results and counts at full numerical precision.

For each request, τ = completion_tokens / verify_count, including target correction / bonus tokens. Request macro is the arithmetic mean of request-level τ; micro is the total token count divided by the total number of verification rounds. Pooled metrics aggregate all requests. The equal-weight benchmark means first compute each benchmark metric, then average across the five benchmarks.

Evaluation uses SGLang 0.5.21 with the corresponding algorithm adapter, BF16, TP=1, FlashInfer, and context_length=16384; temperature=0, thinking disabled, no system prompt, and max_new_tokens=2048. Each replica uses max_running_requests=1, with CUDA graphs and the Radix cache disabled. Each round generates a draft block of 16, and verification selects 15 or 7 draft candidates plus the anchor. The tables measure acceptance length, not task accuracy or throughput.

Benchmark (temperature=1)

Target temperature is 1, top_p=1 and top_k=-1 (no truncation). Thinking is disabled, the system prompt is empty, and max_new_tokens=2048. Each mode contains 2400 requests on the same five benchmarks as the temperature=0 evaluation.

Native draft policy: native temperature-conditioned sampling. DFlash and Domino keep greedy proposals; DSpark and DSpine retain their native temperature-conditioned proposal sampling. These are native-policy comparisons, not a claim of identical draft distributions across methods.

Both modes compute the complete draft16 before verification. Verify16/8 includes the anchor, so 15/7 candidates are verified. DSpark computes its native 16 candidates before truncation. Sampling verification uses the actual proposal distribution, including the sparse candidate distribution for DSpine.

Benchmark	Requests	D16/V16 request macro τ	D16/V16 micro τ	D16/V8 request macro τ	D16/V8 micro τ
GSM8K	1319	9.267633	8.636068	6.286889	6.122255
MATH-500	500	8.374978	7.306807	5.963666	5.604760
HumanEval	164	6.920118	6.795118	5.386519	5.348690
MBPP sanitized	257	6.791953	6.303459	5.365457	5.146649
MT-Bench	160	4.923670	3.948304	4.179546	3.684266
All requests, pooled	2400	8.366548	7.119279	5.918866	5.495062

Aggregate	D16/V16	D16/V8
Equal-weight mean of benchmark macro τ	7.255670	5.436415
Equal-weight mean of benchmark micro τ	6.597951	5.181324
Pooled completion_tokens	1048741	1047738
Pooled verify_count	147310	190669

Macro averages per-request completion_tokens/verify_count; micro divides total completion tokens by total verification rounds. Correction/bonus tokens are included. Pooled metrics are request-weighted; equal-weight benchmark means are listed separately.

This is one stochastic benchmark run per setting, not a multi-seed confidence interval. Request seeds follow 42 + uint32_le(SHA256(dataset:sample_id:turn)[:4]); RNG consumption and generated histories can differ by model, width, and replica assignment. Small differences should not be interpreted as statistically established improvements.

Runtime: SGLang 0.5.21, BF16, TP=1, two independent replicas per mode, one request per replica, context_length=16384, CUDA graphs and Radix cache disabled. A process-local overlay enables sampled split verification without changing the original installed runtime. Synthetic distribution/acceptance tests and real-model structural smoke audits passed before the full benchmark; these checks are not a proof of bitwise equivalence to target-only decoding.

The original temperature=0 values remain unchanged. Full-precision temperature=1 metrics, counters, model hashes and runtime provenance are under temperature_1 in bench/results.json. These measurements are acceptance lengths, not answer accuracy or throughput.

Additional checkpoints

ckpt_42k/ contains the step 42000 weights, configuration, README, and its own benchmark results. This root directory remains the step 66582 model.