singulared/DeepSeek-V4-Flash-0731-DSpark-GGUF

DeepSeek-V4-Flash-0731 DSpark drafter (GGUF, for mainline llama.cpp)

A DSpark speculative-decoding drafter extracted from the official DeepSeek-V4-Flash-0731 weights, in the dflash architecture that mainline llama.cpp uses.

It pairs with any 0731 target GGUF. The drafter is a separate model, so a target quantized before MTP support existed - which is most of them, including files with no mtp.* tensors at all - still gets speculation with no rebuild.

Target model used for testing

All measurements below use antirez/deepseek-v4-gguf, specifically DeepSeek-V4-Flash-Layers37-42Q4KExperts-OtherExpertLayersIQ2XXSGateUp-Q2KDown-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-fixed-0731.gguf (90.9 GiB). It is a good reference target for two reasons:

• Its importance matrix was computed on the 0731 weights themselves - 202,100 chunks of chat-template-rendered prompts, covering the routed expert tensors.
• It embeds a chat template. The 0731 release ships no Jinja template at all (only the encoding_dsv4 Python scripts), so quants that rely on a converter's bundled template may behave differently for tool calls and reasoning.

That repo also has a plain IQ2XXS-...-0731 variant (80.8 GiB) and a larger Q4KExperts-F16HC-F16Compressor-F16Indexer-...-0731 (153 GiB) if you have the memory.

Note this target contains no mtp.* tensors - it was built before MTP export existed. That is precisely the case this drafter is for: speculation without rebuilding the target.

Requirements

Mainline llama.cpp with commit 596a579 ("DeepseekV4 MTP + DSpark", #25784, merged 2026-08-02) or newer. Nothing else is needed to use this file; the extra PR below was only needed to build it.

Usage

llama-server \
  -m DeepSeek-V4-Flash-...-fixed-0731.gguf \   # e.g. from antirez/deepseek-v4-gguf
  -md DeepSeek-V4-Flash-0731-DSpark-BF16.gguf \
  --spec-type draft-dspark --spec-draft-n-max 2 \
  -ngl 99 -ngld 99 -fa on

Measured results

AMD Radeon 8060S (gfx1151, Strix Halo), 128 GB unified memory, ROCm 7.2.4, HIP backend, mainline llama.cpp 0b14b87. Target as described above. Temperature 0, f16 KV cache. Swap disabled, one discarded warmup request per server, distinct filler text per depth so no prompt-cache prefix is shared between measurements.

Speedup vs context depth (--spec-draft-n-max 2)

Decode, tokens/s. Every cell measured on the same machine and target, swap disabled, one discarded warmup request per server, and distinct filler text per depth so no prompt-cache prefix is shared between measurements. -c 49152 for the first three rows, -c 131072 for the last two.

prompt tokens	mainline	mainline + drafter	gain	hipCUB [1]	hipCUB + drafter
cached (4)	13.61	19.47	+43%	13.50	19.31
11,929	9.84	14.18	+44%	12.14	16.30
24,530	9.38	13.65	+45%	11.34	15.04
39,628	9.48	13.35	+41%	10.67	13.12
84,829	7.45	10.95	+47%	9.05	10.95

The drafter gives +41-47% at every depth tested, and the gain does not fade as context grows

• at 85K it is the largest measured. Draft acceptance was 0.67-0.76 across the range.

Note the cached row: 13.61 vs 13.50 without the drafter, 19.47 vs 19.31 with it. With a prompt that short the hipCUB code path is never reached, so those columns are a control - they should be identical, and they are.

About the hipCUB columns

They are a separate, unofficial AMD patch [1], included because the interaction turns out to be depth-dependent:

• Without the drafter it helps everywhere: +23% at 12K, +21% at 24K, +13% at 40K, +22% at 85K.
• With the drafter it helps only up to ~32K: +15% at 12K and +10% at 24K, but -2% at 40K and 0% at 85K (10.95 either way).

The likely reason is that speculation cuts the number of target forward passes, so the indexer's CPU fallback stops being the limiting factor at depth and memory bandwidth over the 90 GiB of weights dominates instead. If you run this drafter, the patch is worth having for mid-length contexts and is neutral beyond that; if you run the target without a drafter, it helps throughout.

---

[1] argsort/top_k above 1024 columns have no GPU path on HIP, so they fall back to the CPU backend - for DeepSeek-Sparse-Attention models that puts V4's per-token lightning-indexer top_k on the CPU, and the cost grows with context depth (visible as ~416% CPU on mainline at depth versus ~100% with the patch, the latter being the backend's normal busy-wait). Enabling hipCUB (rocPRIM's CUB-compatible API) restores the GPU path. Branch: https://github.com/singulared/llama.cpp/tree/hipcub-port - not upstream, feature-detected so builds without hipcub-dev are unchanged, and -DGGML_HIP_NO_HIPCUB opts out.

Choosing --spec-draft-n-max (short prompt, -c 32768)

n-max	decode	draft acceptance
none (baseline)	14.03 t/s	-
1	17.73 t/s	0.801
2	20.43 t/s	0.759
3	16.04 t/s	0.591
4	16.10 t/s	0.540
6	13.75 t/s	0.439

n-max 2 is the knee. Acceptance falls monotonically as drafting goes deeper, and past 2 the extra verification costs more than the accepted tokens save; by n-max 6 the gain is gone.

KV cache: f16 measured fastest here (20.61 vs 17.06 q8_0 vs 17.76 q4_0 t/s on one prompt), and V4's MLA already keeps the cache small, so there is little to save by quantizing it. Those three figures are from a single prompt at one context length - treat them as provisional.

Memory, measured at -c 131072 on a 128 GB machine: the target alone occupies 92 GiB of GTT (21 GiB free), and target + drafter 104 GiB (11 GiB free). It fits, but with little room - run with swap disabled if you are benchmarking, otherwise memory pressure degrades throughput silently rather than failing.

Contents

81 tensors, 3 blocks, 10.15 GiB. The converter fixes the layout - MXFP4 experts, Q8_0 attention and shared experts, BF16/F32 router, norms, sinks and hyper-connections - so it is not re-quantizable (llama-quantize rejects mxfp4 input). Precision outside the experts is deliberately high: drafter precision drives acceptance.

How it was built

python convert_hf_to_gguf.py <fp8-checkpoint-dir> \
  --dspark --target-model-dir <same-dir> --outtype bf16 \
  --outfile DeepSeek-V4-Flash-0731-DSpark-BF16.gguf

--target-model-dir is mandatory: the drafter needs the target's tokenizer.

--dspark was added by PR #26452, merged as 3581ba0, so current master has it. It is needed because the 0731 checkpoint ships three mtp.* blocks (matching dspark_target_layer_ids: [40, 41, 42]) while config.json declares num_nextn_predict_layers: 1 - so plain --mtp fails with ValueError: Unexpected DeepSeek-V4 MTP layer 1.

Credits

DeepSeek for the weights (MIT), am17an for the llama.cpp DSpark implementation and the --dspark export path, antirez for the target quant and its imatrix, used for every measurement here.