kodelow/Kimi-K3-DSpark-NVFP4

Kimi-K3-DSpark — NVFP4 (weight-only, NVFP4A16)

⚠️ Not recommended for serving. Use Kimi-K3-DSpark-FP8 instead.

This checkpoint is published as a reference artifact so the 4-bit measurement below is reproducible, not as a deployment option. It is a valid, working NVFP4 quantization — it is simply the wrong format for a speculative drafter, and the arithmetic below shows why no kernel speedup can recover it.

A 4-bit NVFP4 (weight-only, NVFP4A16) quantization of Inferact/Kimi-K3-DSpark, the MLA-native DSpark draft model for Kimi-K3, produced with qstream.

The original model card follows in full below.

Size	3.71 GiB (vs 4.6 GiB FP8, 6.6 GiB BF16)
Format	compressed-tensors nvfp4-pack-quantized, group-16, E4M3 group scales, weight-only (no input_activations → vLLM's a16 path)
Status	reference artifact — see the warning above

Considerations

The reason to quantize a drafter is draft-pass latency on the critical path at bs=1, not footprint — 0.9 GiB beside a ~1 T-param target is noise. So the trade has to be evaluated on speed, and on that axis it fails decisively.

Speculative throughput goes as a / (d + T) — accepted tokens per iteration over draft-pass cost plus target-verify cost. Against the FP8 build:

ratio = (a_nvfp4 / a_fp8) × (d_fp8 + T) / (d_nvfp4 + T)

The acceptance term is measured below: 3.29 / 5.70 = 0.577, so NVFP4 must win 1.73× on the draft pass merely to break even.

It cannot. NVFP4 removes ~19% of the draft bytes, so d_nvfp4 ≈ 0.81 · d_fp8, and as d/T → ∞ the factor (d_fp8+T)/(0.81·d_fp8+T) tops out at 1.23 — below 1.73 even in the limit where the draft pass dominates the target entirely. Being generous and assuming the kernel beats what bytes predict (native NVFP4 tensor cores on SM100/SM103 at 2× FP8, so d_nvfp4 ≈ 0.5 · d_fp8), break-even would require d_fp8 ≥ 5.5·T: the 3.5 B dense draft pass costing five times a ~1 T MoE verify. Realistically d/T ≈ 0.14, giving a ~1.03× draft win against a 1.73× hole.

No achievable draft-pass speedup pays for the acceptance loss. That conclusion needs no latency benchmark — it follows from the acceptance measurement plus an upper bound on the speedup.

Why acceptance is so sensitive

Acceptance length is a prefix-run-length statistic — a drafted block is accepted only up to its first mismatch — so per-token error compounds multiplicatively across all 7 positions. Weight error that would be invisible in a standalone LM turns directly into lost acceptance. NVFP4's 20.4 dB weight SNR is perfectly healthy for 4-bit. It is still not enough here.

Measurement

Full methodology, including why the reference is FP32 with BF16 measured as the noise floor, is documented on the FP8 card. In brief: drift against an FP32 reference on GSM8K token streams with the real Kimi-K3 tokenizer and the real target lm_head; the unobtainable aux hidden states swept over three distributions (boot = drafter's own layer outputs, the realistic one; heavy, gauss = synthetic stress). 1024 blocks per regime, 7168 drafted tokens, 95% CI ±0.7–0.9 pp.

Drift-limited acceptance ceiling (of 7)

Variant	boot	heavy	gauss
FP32 reference	7.00	7.00	7.00
BF16 (served baseline — noise floor)	6.25	6.04	6.13
FP8 (recommended)	5.70	4.91	5.04
NVFP4 (this model)	3.29	2.34	2.25
MXFP4	3.08	2.28	2.15

For scale, the base model's own measured acceptance ranges from 5.64 (GSM8K) down to 2.72 (AIME 2026). NVFP4's ceiling on GSM8K — the easiest, lowest-entropy benchmark — is roughly where the BF16 drafter lands on its hardest ones. A ceiling is an upper bound on preserved behaviour, so real acceptance sits below it; extrapolating to high-entropy workloads puts it near ~1.5–2, where a 7-token block-diffusion pass plus sequential Markov rollout plus verification stops paying for itself. That low-end figure is extrapolation — the measurement is GSM8K only, and drift is not acceptance.

NVFP4 does beat MXFP4 consistently across all three regimes, which is why it is the 4-bit build published here and MXFP4 is not.

Detail

Metric	BF16	FP8	NVFP4	MXFP4
Weight SNR	exact	31.5 dB	20.4 dB	19.0 dB
Greedy agreement (boot)	—	98.03%	91.43%	90.42%
Sequential rollout (boot)	—	93.30%	73.67%	71.53%
1−TV coupling (boot)	—	98.41%	93.36%	92.59%
KL mean (boot)	—	0.00100	0.01692	0.02063
KL p99 (boot)	—	0.00387	0.05266	0.06611
Hidden-state rel err (boot)	—	1.90%	8.79%	9.42%
Confidence r (boot)	0.9999	0.9998	0.9958	0.9941

Per-position sequential agreement (boot), positions 1→7:

Variant	1	2	3	4	5	6	7
BF16	98.6	98.0	97.1	96.4	95.7	94.6	93.5
FP8	98.1	96.0	94.5	92.8	92.0	90.4	89.3
NVFP4	91.6	83.8	78.3	71.3	67.2	64.0	59.6

The steep decay is the compounding that prefix-run-length acceptance punishes. 1−TV is the sampling-aware figure, relevant under draft_sample_method: probabilistic.

On the confidence head: correlation degrades consistently with 4-bit (r ≈ 0.977–0.996 versus FP8's 0.9997), but it stays high, and the thresholded scheduling-decision flip rate does not separate the formats where inputs are realistic — under boot it is FP8 12.4%, NVFP4 10.0%, MXFP4 9.3%, against a BF16 floor of ~7%. Larger NVFP4 flip rates appear only in the synthetic regimes and do not reproduce under boot. Scheduling stability is not a reason to prefer FP8 here; acceptance is.

What is quantized to what

Component	Precision
layers.*.mlp.{gate,up,down}_proj, self_attn.{o_proj,q_b_proj}, context_proj	NVFP4
embed_tokens, markov_head.markov_w{1,2}	BF16
q_a_proj + kv_a_proj_with_mqa (fuse into fused_qkv_a_proj; compressed-tensors raises if fused shards disagree on scheme)	BF16
kv_b_proj (MLA weight absorption), all norms, confidence_head	BF16

2.19 B of 3.56 B params quantized (61%). Identical tensor selection to the FP8 build, so the two are directly comparable.

Limitations

• Acceptance is not measured. Everything here is drift against a high-precision reference, not agreement with Kimi-K3 — that needs the ~1 T target in the loop. Drift bounds acceptance impact; it does not equal it, and the ceiling is conservative (a divergence past the point the target would have rejected anyway costs nothing).
• Aux hidden states are synthetic, swept over three distributions. The ranking is stable across all three, but none is Kimi-K3's actual distribution.
• Long-context (YaRN) behaviour untested. A position sweep through the YaRN regime was planned and not completed.
• No latency benchmark. The break-even argument above is an upper bound on the achievable speedup, not a measurement. Whether vLLM's per-channel weight-only FP8 path is itself fast for these shapes is a separate open question.
• Not load-tested in vLLM. The config is verified structurally — single nvfp4-pack-quantized group, weight-only, 7 ignored modules, no target/ignore overlap, fused-module constraint respected — but has not been through a real loader.

How it was made

qstream-quantize \
  --model_dir <Inferact/Kimi-K3-DSpark> \
  --output_dir ./Kimi-K3-DSpark-NVFP4 \
  --quant_format nvfp4 \
  --include_layers "mlp.gate_proj" "mlp.up_proj" "mlp.down_proj" \
                   "self_attn.o_proj" "self_attn.q_b_proj" "context_proj" \
  --device cuda --workers 1

Built with qstream @fcfe8a2.

License

Inherits the Kimi-K3 license from the base model. This is a derivative (quantized) work of Inferact/Kimi-K3-DSpark.

---

Original model card

Model Overview

Inferact/Kimi-K3-DSpark is an MLA-native DSpark draft model that accelerates Kimi-K3 on vLLM, served through vLLM's native dspark speculative method.

The draft trains on target hidden states extracted from vLLM itself — the same engine that serves it, so the numerics it learns from are the numerics it meets at inference. TorchSpec provides the loop, streaming those hidden states from live target inference into concurrent FSDP draft training.

DSpark = a block-diffusion backbone of 5 dense layers with non-causal attention, drafting 7 tokens in a single parallel pass, a low-rank sequential Markov head supplying the intra-block dependency, and a confidence head for resource-aware scheduling. Mirroring Kimi-K3's own MLA attention means draft and target share one KV layout (a compact 576-element latent per token), so the draft's pages unify with the target's cache — KV offloading and P/D disaggregation work with no separate page format.

---

Performance

Peak bs=1 decode: 464 tok/s

Under low-entropy real reasoning workload — Kimi-K3 + DSpark on vLLM can achieve 464 tok/s using the public vllm/vllm-openai:kimi-k3 image on 4 × GB300 at bs=1 & tensor-parallel-size=16.

Speculator acceptance

Speculators do best on predictable, low-entropy work like the reasoning workload behind the 464 tok/s above, and worst on open-ended, high-entropy generation. So we measured 14 benchmarks spanning math, code synthesis, real-world software engineering, multi-turn chat, RAG and QA, multilingual text, creative writing and long-context generation — all with the Kimi-K3 chat template enabled and production sampling parameters, with temperature=0 reported alongside for reproducibility. Acceptance stays strong at long context too, verified on AA-LCR's ~95k-token multi-document prompts. That is the most comprehensive view we can give of how the speculator performs on real-world workloads.

Acceptance length, with 7 speculative tokens:

benchmark	temperature=0	temperature=1.0, top_p=0.95	prompts
GSM8K	5.64	5.44	1319
HumanEval	5.34	5.07	164
MBPP	4.44	4.31	256
SPEED-Bench · coding	4.38	4.22	80
SPEED-Bench · multilingual	4.21	4.10	80
SPEED-Bench · RAG	4.11	3.97	80
MATH-500	3.82	3.77	500
SPEED-Bench · low-entropy, 10k input	3.72	3.66	512
SWE-bench Pro	3.35	3.11	128
AA-LCR · ~95k input	3.19	3.23	100
MT-Bench	3.14	3.06	80
SPEED-Bench · QA	3.07	2.98	80
SPEED-Bench · writing	2.79	2.69	80
AIME 2026	2.72	2.64	30
mean	3.85	3.73

Benchmarks: GSM8K, MATH-500, AIME 2026, HumanEval, MBPP, SWE-bench Pro, MT-Bench, AA-LCR — 100 multi-document prompts of 71k–115k tokens — and six splits of NVIDIA SPEED-Bench: its throughput_16k low-entropy split at 10k-token input, plus five qualitative categories.

---

Training

Data — all responses regenerated on-policy by Kimi-K3 itself, so the draft learns the target's own reasoning traces and chat formatting. Prompts come from public datasets:

• lightseekorg/kimi-mtp-dataset — general instruction prompts
• nvidia/OpenCodeInstruct — single-turn coding
• A nine-category prompt pool from the NVIDIA Nemotron SFT/RL collections and CohereLabs/aya_dataset, spanning chat, code, multilingual, RAG/QA, math, structured output and safety. Evaluation-set prompts are excluded.

Method: DSpark with block_size=7, trained on a combined CE + L1 distribution-distillation objective against the target's post-final-norm hidden state, bf16. The draft consumes Kimi-K3 auxiliary hidden states from target layers (2, 23, 47, 71, 89) of 93. Roughly two epochs in total, on GB300 nodes.

Draft architecture, block size, sequence length, and loss weights are YAML-configurable — see the TorchSpec repo.

---

Quick Start

Requirements

For serving Kimi-K3 itself — hardware, parallelism and engine flags — follow the official vLLM recipe: recipes.vllm.ai/moonshotai/Kimi-K3.

Enable the draft

Add to your Kimi-K3 vllm serve command:

--speculative-config '{"method": "dspark", "model": "Inferact/Kimi-K3-DSpark", "num_speculative_tokens": 7, "attention_backend": "FLASHINFER_MLA", "draft_sample_method": "probabilistic", "rejection_sample_method": "block"}'

Sampling options

vLLM offers two knobs on top of the defaults, both used in the temperature=1.0 column above:

• draft_sample_method — probabilistic samples the draft from its own distribution instead of taking its argmax. Pair it with a sampling client; use greedy when serving at temperature=0 so the draft matches the client.
• rejection_sample_method — block verifies the drafted block as a unit rather than token by token. It is a no-op under greedy decoding (at temperature=0 verification reduces to a deterministic argmax match), so it only applies to the sampling configuration.

The temperature=0 column was produced with {"draft_sample_method": "greedy"} and no rejection_sample_method.