satgeze/Qwen3.5-0.8B-DSpark

Qwen3.5-0.8B DSpark drafter (v0.1, experimental)

Card status (2026-07-25, corrected): all numbers below are measured. The matched-vs-generic A/B has landed (see its section) and the vLLM smoke test passed (see How to run). Correction: an earlier version of this card described a "quant tax" and claimed a bf16-trained head loses most of its acceptance on aggressive quants. Follow-up work refuted that: acceptance tracks whether the target is still coherent, not the precision gap between head and target. The affected sections are rewritten below, with the evidence.

A DSpark speculative-decoding draft head for Qwen/Qwen3.5-0.8B, trained from scratch with the DeepSpec toolkit via our fork branch satindergrewal/DeepSpec:qwen35-lane (qwen3.5-family target support, online-training mode merged from PR #23 by Ofir408, warm-start hook). As far as we know this is the first published DSpark head for a vanilla Qwen3.5 model (see DeepSpec issue #22, an open request for exactly these heads).

Status: v0.1 = pipeline validator, honestly below the deployment bar. Trained on 12K self-distilled samples from scratch. The training pipeline, conversion path, and llama.cpp serving all work end to end; acceptance is data-limited exactly as the community scaling curve predicts. v0.2 targets 50-100K samples. Read this card for the measured behavior of a small cold-start head: the datasets here (economics ladder, target-quant ladder, per-dataset eval) are all measured on this exact head and target, and they are more useful than the head itself.

How to run it

llama.cpp (tested, all numbers on this card were measured with this)

DSpark runtime support is in PR #25173 plus this model family's support on our branch. Until both are in mainline, build the branch:

git clone --branch dspark-qwen35 https://github.com/satindergrewal/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON   # Metal is the default on Apple Silicon
cmake --build build -j --target llama-server

Get the head and a target:

hf download satgeze/Qwen3.5-0.8B-DSpark Qwen3.5-0.8B-DSpark.gguf --local-dir .
hf download unsloth/Qwen3.5-0.8B-GGUF Qwen3.5-0.8B-BF16.gguf --local-dir .

Serve with speculative decoding:

./build/bin/llama-server \
  -m Qwen3.5-0.8B-BF16.gguf \
  -md Qwen3.5-0.8B-DSpark.gguf \
  --spec-type draft-dspark --spec-draft-n-max 7 \
  -c 4096 -ngl 99 -fa on --jinja --port 8080

--spec-draft-n-max 7 matters: llama.cpp defaults to drafting 3 tokens, but this head was trained for a block of 7, so the default discards part of the (already marginal) speedup. Set it to the head's block size, printed at startup as block_size=.

Any OpenAI-compatible client works against http://localhost:8080/v1; the server log prints draft acceptance per request. Note the economics section below before deploying this pair: on fast hardware this 0.8B target outruns its own drafter.

vLLM (smoke-tested: works)

vLLM has DSpark support in mainline since #46995 (merged 2026-07-01, DeepSpec-format heads) and #47093 (merged 2026-07-02, speculators-format checkpoints). Requires vLLM >= 0.25 (0.24.0 predates the merges and rejects "method": "dspark"). This repo's safetensors are DeepSpec-format with architectures: ["Qwen3DSparkModel"], the exact key vLLM's registry maps for DSpark drafters.

Verified 2026-07-24 on vLLM 0.25.1: the head loads directly and speculates (coherent output; spec-decode metrics active, position-0 acceptance ~0.30 on the code probe, decaying by position as expected):

vllm serve Qwen/Qwen3.5-0.8B \
  --speculative-config '{"method": "dspark", "model": "satgeze/Qwen3.5-0.8B-DSpark", "num_speculative_tokens": 7}'

Setup notes from our smoke test (RTX Pro 6000 Blackwell, sm_120a): flashinfer's device detection can fail in the engine subprocess and raise a misleading "requires sm75 or higher"; fix with FLASHINFER_CUDA_ARCH_LIST="12.0a". First run JIT-compiles kernels (needs ninja and nvcc on PATH). With a 248K-vocab model, default warmup can OOM even on 96GB; lower --gpu-memory-utilization / --max-num-seqs if it does.

SGLang

DSpark support is on SGLang's roadmap: watch the tracking issue sgl-project/sglang#30344 for the landing PRs and the up-to-date run method. We will add the command here once it ships; everything on this card runs today with the llama.cpp commands above.

Measured results (llama.cpp, RTX Pro 6000 Blackwell, greedy, 200-token completions)

metric	value
draft acceptance (code prompt)	0.116
draft acceptance (counting)	0.152
draft acceptance (no-think prompt)	0.158
target baseline decode	~500 t/s
with DSpark	198-250 t/s (net slowdown)

Apple Silicon (M3 Max 128GB, Metal, same GGUFs, same probes)

metric	value
draft acceptance (code prompt)	0.1159 (88/759, identical accepted count to CUDA)
draft acceptance (counting)	0.1518
target baseline decode	146-147 t/s
with DSpark	74-84 t/s (net slowdown)

Acceptance on Metal matches CUDA to the third decimal on byte-identical GGUFs: the drafter's behavior is backend-independent. Only the economics change per platform.

Economics across the full target-quant ladder (measured, M3 Max Metal)

target rung	baseline t/s	with DSpark t/s	net
CUDA Pro 6000, bf16	~500	198-250	0.45x
Metal bf16	142-146	74-84	0.55x
Metal Q8_0	185-187	74-86	0.43x
Metal Q4_K_M	213-217	65-83	0.34x
Metal Q2_K	~219	46-69	0.26x
Metal IQ1_S (imatrix)	208-223	44-50	0.22x

SPEED-Bench (qualitative set, coding+writing, temperature 0, concurrency 1, M3 Max Metal, bf16 target)

output len	coding spec / base	writing spec / base	ratio
256	79.1 / 138.3 t/s	77.2 / 138.1 t/s	0.57x / 0.56x
1024	80.6 / 140.2 t/s	51.3 / 140.4 t/s	0.57x / 0.37x

Aggregate draft acceptance across the diverse SPEED-Bench prompts: 0.126-0.139, consistent with the greedy probes. The verdict below holds on varied real prompts, not just narrow probes.

A 0.8B target is structurally a poor speculation host: the smaller and faster the target gets, the worse speculation pays, because draft+verify overhead is roughly constant while the baseline accelerates. Break-even for this pair on Metal sits around 0.47 acceptance, which no 12K-sample head reaches. Speculation value starts around 9B-class targets (we measure 1.62x overall / 2.19x coding on a 9B with a fully-trained head, and 2.54-2.67x on a 27B: satgeze/Qwen3.6-27B-DSpark).

Loop rate: this target degenerates on long greedy generation, at every precision

Acceptance says how fast the drafter makes the target go. It says nothing about whether the target's long-form output is usable. Measured separately with an independent harness (8 prompts, greedy, 1500 tokens each, real cycle detection, prompt set v1, M3 Max Metal):

target rung	loop rate (greedy, 1500 tok)
bf16 (16 bpw)	7/7 (100%)
Q8_0 (8.5 bpw)	6/7 (86%)
Q4_K_M (4.8 bpw)	7/8 (88%)
Q2_K (2.6 bpw)	8/8 (100%)
IQ1_S (1.6 bpw)	6/6 (100%)

This is flat, and it is flat at the ceiling. The model loops on essentially every long greedy generation even at full bf16 precision, so here looping is a property of the 0.8B model under long greedy decoding, not a quantization effect. For contrast, the same harness on a 27B target finds zero loops at every rung down to 2.9 bpw, and only breaks below 2 bpw. Model scale, not bit-width, dominates this failure mode.

Mitigation, measured on the same rungs: enabling the DRY sampler (temperature 0.7) cuts the bf16 loop rate from 100% to 38%, but leaves Q4_K_M at 88%. So DRY helps and is worth enabling, but it does not make this target reliable for long free-form generation. Use this head for what the rest of the card describes (short completions, pipeline validation, quantization studies), not for long-form or agentic runs.

(Loop rates are only comparable within the same prompt set, token budget and decoding settings, and were all measured on Metal. Do not compare them against rates measured on another backend. Two further scoping notes: these are raw /completion probes, deliberately worst-case; chat-template serving with a repetition sampler behaves measurably better, so state which mode you measured. And if you measure a reasoning model, count reasoning tokens: a loop can live entirely inside the think field while visible content stays short, and a metric that reads only content scores a looping model as clean.)

Target quantization ladder, measured on this head

Same head (trained vs the bf16 target), same probes, only the target quantization changed. All rungs quantized from the same bf16 GGUF; the IQ1_S rung uses an imatrix computed on this model's own regeneration data (600 samples), measured on M3 Max Metal:

target	code probe acc	counting probe acc
bf16 (16 bpw)	0.1159	0.1518
Q8_0 (8.5 bpw)	0.1159	0.1570
Q4_K_M (4.8 bpw)	0.1281	0.2010
Q2_K (2.6 bpw, no imatrix)	0.0472	0.1358
IQ1_S (1.6 bpw, imatrix)	0.0493	0.0256

The bf16-trained head is effectively free down to ~4-bit targets. Below that, this 0.8B target itself stops being a coherent model and acceptance falls with it. The Q2_K row here is uncalibrated: on our 27B ladder, adding an own-distribution imatrix at the same bit-width moved acceptance from 0.149 to 0.251, and a calibrated 1.6-bpw IQ1_S 27B target held Q8-level acceptance (0.293) with the same bf16-trained-head recipe. So what collapses at the bottom of this table is the target, not the head. The same law, measured on the official deepseek-ai/dspark_qwen3_4b_block7 head in the same runtime, for reference:

target precision (4B head, reference)	draft acceptance
Q8_0 (near-lossless)	0.365
Q2_K	0.220
IQ1_S (~1.5-bit, imatrix)	0.074

Practical consequences:

1. At aggressive quantization, acceptance falls when the target itself degrades. This is a small-model effect: a 0.8B at 2.6 bpw and a 4B at 1.6 bpw are broken models, and no drafter can predict a broken model. A 27B at 1.6 bpw with a calibrated imatrix keeps Q8-level acceptance using this same bf16-trained-head recipe. Before blaming precision, check whether the target is still coherent (perplexity and loop rate both help).
2. The fix is to self-distill and train the head against the exact deployed artifact (the quantized GGUF actually served), not the full-precision base. Our llama.cpp capture tool + DeepSpec-format cache builder exist for exactly this (validated at cosine >= 0.9998 against the HF capture pipeline; full how-to guide); matched-vs-generic A/B numbers land here as they are measured. The same applies to modified variants (e.g. abliterated models): any distribution shift in the target taxes a mismatched head.
3. When evaluating published heads, compare targets at the precision the head was trained against before concluding anything about the head or the runtime.

Matched-vs-generic A/B: does capture precision matter? (measured)

We trained two additional heads with identical config/seed/steps, differing ONLY in where the training features came from: hidden states captured from the bf16 GGUF vs from the Q8_0 GGUF, both captured with our llama.cpp capture tool (this model is the testbed; caches validated at cosine >= 0.9998 against the HF pipeline). Acceptance matrix (code probe / counting probe):

head trained on	vs bf16 target	vs Q8_0 target
bf16-captured features	0.126 / 0.152	0.123 / 0.124
Q8-captured features	0.116 / 0.150	0.116 / 0.134

Two results:

1. Pipeline validation: the bf16-GGUF-captured head reproduces the original HF-cache-trained v0.1 head (0.126/0.152 vs 0.116/0.152) within probe noise, and their final training losses are statistically identical (1.646 vs 1.63). Training from llama.cpp-captured features is a full substitute for the HF capture pipeline, proven through training, not just feature cosine.
2. No matched-head advantage at Q8: all four cells sit within probe variance. This is the expected result: Q8 features drift from bf16 by cosine ~0.999, so there is nothing for a matched head to learn. The 27B ladder later closed this out further: at 1.6 bpw with a calibrated imatrix, the same bf16-trained recipe still holds Q8-level acceptance, so the "matched training pays off at low bit-width" half of this remains a hypothesis we have not confirmed. Published matched-vs-unmatched A/Bs elsewhere measure 0-2% (arXiv 2607.04244 Table 4: 4.92 vs 4.97 accepted length; SpecForge PR #318: 7.10 vs 7.11).

Cross-stack controlled A/B: the same head in llama.cpp and vLLM (measured)

One trajectory-matched comparison, everything pinned that can be pinned: the same 8 prompts (code, math, prose), temperature 0, max_tokens 256, sequential single requests, speculative depth 7 on both stacks, bf16 target weights on both sides (HF safetensors in vLLM, the GGUF conversion of the same weights in llama.cpp), and this same step-450 head (safetensors in vLLM's --speculative-config '{"method": "dspark", ...}', the GGUF conversion via -md in llama.cpp). Acceptance computed from each stack's own raw counters (llama.cpp per-request draft acceptance log line; vLLM /metrics spec-decode counters).

stack	accepted / drafted	acceptance per drafted token	accepted per verify step
llama.cpp (--spec-type draft-dspark, RTX Pro 6000)	1,064 / 6,734	15.8%	~1.11 (mean accepted len 1.66-3.23 across prompts)
vLLM 0.25.1 (method: dspark, same GPU)	472 / 10,990	4.3%	0.30 (per-position accepts 356/100/12/2/2/0/0)

Same weights, same proposals available, roughly 3.7x more drafted tokens survive verification in llama.cpp, on either denominator. Read this carefully for what it is:

• It is not a head-quality claim. Both stacks ran the identical head. It is a runtime-semantics difference: how each stack conditions the drafter (anchor layout, markov head usage, block handling) and verifies its proposals. It is consistent with the pattern documented on the 27B card, where speculators-lineage heads that score well in vLLM collapse in llama.cpp: the two runtimes are not interchangeable executors of the same artifact.
• One model, one head, one prompt set. Do not generalize the 3.7x multiplier; do expect the direction to hold, and measure your own stack pair before committing to serving numbers.
• Throughput is deliberately not compared here. At 0.8B, speculation is a net slowdown in wall-clock terms regardless of stack (see the economics section); this A/B isolates the acceptance mechanics only.
• Reproduction note: in llama.cpp, --spec-type draft-dspark is required. Without it the server loads the draft model and then silently serves without speculation; the only visible differences are a missing draft acceptance line in the per-request timings and faster-than-expected raw decode.

Official DeepSpec eval (per-proposal acceptance, temperature 0, step 450)

dataset	acceptance		dataset	acceptance
gsm8k	0.291		livecodebench	0.164
math500	0.278		mt-bench	0.129
aime25	0.239		alpaca	0.114
humaneval	0.179		arena-hard-v2	0.104
mbpp	0.176

Confidence head calibration is notably good even at this data scale: AUC 0.94, ECE 0.018 (mt-bench).

How this head was trained

The training signal is self-distillation: the 12K prompts (mlabonne/open-perfectblend) were answered by the target model itself (served with llama.cpp, thinking disabled, temperature 0.7), so the drafter learns the exact distribution it will draft for at inference. Hidden states were captured at layers [1, 6, 11, 16, 21] of 24 with DeepSpec's prepare_target_cache (11,577 / 11,722 samples valid), then the drafter (block size 7, 5 layers, markov_rank 256, confidence head) trained for 5 epochs / 450 steps at global batch 128 on a single 96GB GPU, loss 2.86 to 1.63. Cold start: unlike our 27B head (warm-started from a z-lab DFlash checkpoint, mean accepted length 5.4), this head was initialized from scratch, and at 12K samples that is the dominant limitation: mean accepted length ~1.8-2.05, exactly where the community scaling curve predicts a cold-start 12K head to land.

Two honest findings worth more than the head itself:

1. From-scratch heads need far more than 12K samples. The community curve (each point = that head on its own target): 2.4K samples = 1.08 accepted length (failed), 10K warm-started = 1.51, 250K = 3.34, ~1M = 4.6-4.8. v0.1 lands where the curve predicts for cold-start 12K. Warm-starting is the highest-leverage fix (our 27B head proves it at the same data scale).
2. Small targets are poor speculation hosts on fast hardware (see the economics ladder above). Small-target heads make sense on slow/edge hosts, not on datacenter GPUs.

Recipe (reproducible)

• Data: 12,000 prompts from mlabonne/open-perfectblend, responses regenerated by the target itself (llama-server bf16 GGUF, temperature 0.7, top-p 0.8, top-k 20, thinking disabled, max 2048 tokens).
• Cache: DeepSpec prepare_target_cache (capture layers [1, 6, 11, 16, 21] of 24), 11,577/11,722 valid.
• Training: DeepSpec Qwen3DSparkTrainer, block_size 7, 5 draft layers, markov_rank 256, lr 6e-4, global batch 128, 5 epochs (450 steps), bf16, single GPU. Final loss 1.63 (from 2.86).
• Conversion: llama.cpp branch satindergrewal/llama.cpp:dspark-qwen35 (adds qwen3.5-family DSpark target+drafter support on top of PR #25173), then served with --spec-type draft-dspark --spec-draft-n-max 7.

Files: model.safetensors (drafter, bf16), Qwen3.5-0.8B-DSpark.gguf (converted, ready for llama.cpp -md), training config, this card.

Credits

DeepSeek (DeepSpec toolkit and DSpark method), wjinxu (llama.cpp DSpark PR #25173), ruixiang63 (DFlash/EAGLE3 foundation in llama.cpp), Ofir408 (DeepSpec online-training mode PR #23), the community heads whose published numbers anchor the scaling curve (dbirks, fal, stanleyphoong, deepseek-ai official), Qwen team (target model), mlabonne (open-perfectblend prompts).