doth4580/Qwen3.8-27B-DFlash2-NVFP4-RTNcal-FIXED

Qwen3.8-27B-DFlash2-NVFP4-RTNcal — FIXED (veloGB10 engine format)

This is a derived work, not a mirror. The quantization is maurienne-ai/Qwen3.8-27B-DFlash2-NVFP4-RTNcal: a calibrated RTN NVFP4 quantization of the DFlash 2 block-diffusion draft for Qwen3.8-27B. This repository re-packages that checkpoint in the file layout the veloGB10 engine loads — hence FIXED. Upstream ships the NVIDIA ModelOpt layout, which SGLang loads natively but this engine does not. Full credit for the calibration and the quantization belongs to the upstream authors; their model card is reproduced in full at the bottom of this file.

Weights differ from upstream (different container, weight-only instead of W4A4) — see below. License: Apache-2.0, same as the base DFlash 2 draft and Qwen3.8-27B; a copy ships in this repository as LICENSE.

What this is

A DFlash 2 draft (speculative decoding) model for Qwen3.8-27B, in NVFP4, packed for the veloGB10 engine. It is not a standalone language model: it runs beside the target model and drafts a block of tokens for the target to verify, which makes it lossless with respect to the target — greedy output matches the target exactly, sampling preserves its distribution.

Intended companion in the veloGB10 engine: our NVFP4 Qwen3.8-27B target, doth4580/Qwen3.8-27B-NVFP4-FULL.

What "FIXED" means — the difference from upstream

Upstream is packaged for SGLang's ModelOpt FP4 path (--speculative-draft-model-quantization modelopt_fp4, a W4A4 scheme with static activation scales in a quantization_config block). The veloGB10 engine's drafter loader keys on a df2_quant field in config.json and reads a weight-only packed sidecar. Same calibrated numbers, different container:

	upstream …RTNcal	this repo …RTNcal-FIXED
weights	model.safetensors, 1.55 GB, 186 tensors (ModelOpt)	nvfp4.safetensors, 1.01 GB — 46 packed linears — plus model.safetensors, 0.62 GB of BF16 tensors
activation quant	W4A4, static input_scale per tensor	none — weight-only; activations stay BF16
config	quantization_config + ModelOpt ignore list	df2_quant, df2_quant_recipe, df2_quant_source_sha256
loads in	SGLang	veloGB10 engine
KV cache dtype knob	FP8 KV via SGLang flags	n/a (set on the target model, not the draft)

Three details worth knowing before you serve it:

• Both safetensors files are required; the directory is loaded as a unit. The BF16 model.safetensors is not a spare copy of the draft. It holds the tensors that deliberately stay high-precision — the candidate-selector codebooks and hidden projection, all RMSNorms, the dynamic-convolution base kernels — plus the BF16 twins of fc, k_proj and v_proj. The prompt-prime path runs at large batch (M up to 8192) on the BF16 GEMM, while the block steps run low-M through the FP4 GEMM. Those twins are the only duplicated bytes (~367 MB).
• Block-agnostic. The packed tiles depend only on weight shapes, so the block-8 and block-16 draft rounds both read this same directory — no re-bake, no conversion.
• The pin is honored by default. df2_quant_source_sha256 is 67fc76d68dc5a9415511a4f394ef744d67510cd20e93b37cc2cc7d28e4bab65c, which is the veloGB10 engine's published sha256 for the BF16 DFlash 2 draft artifact — the checkpoint this bake consumed (--df2-bake-nvfp4 <src> <dst>). Serving normally needs no --sha256 flag.

Files

File	Bytes	sha256 (first 16)	Contents
config.json	1,405	3c082fde97b8f803	DFlash 2 draft config + the three df2_quant* fields
model.safetensors	624,442,648	76e6bfff5c946abd	46 BF16 tensors: selector codebooks/hidden_projection, norms, conv base kernels, and the fc / k_proj / v_proj BF16 twins
nvfp4.safetensors	1,010,090,080	269ae9ca7b1c76ac	46 quantized linears × 3 tensors (138 total): weight_packed U8 (E2M1 nibbles), weight_scale F8_E4M3 (group 16), weight_global_scale F32

Quantized linears: fc, per-layer self_attn.{q,k,v,o}_proj, mlp.{gate,up,down}_proj, and the attention_conv / mlp_conv kernel projections — 46 in total across the 5 draft layers.

Draft geometry (from config.json): 5 layers, hidden_size 5120, 32 attention heads / 8 KV heads, head_dim 128, target_layer_ids [5,19,33,47,61], block size 8, max_position_embeddings 262144.

Usage with veloGB10

Put this directory next to the binary (any path works — it is passed with --draft-dir), download the target, and start the server. Full step-by-step instructions live in the veloGB10 repo (QWEN38_27B_SETUP.md).

One GB10:

./gb10_inference --server \
  --model-dir ~/veloGB10/3.8-27b-nvfp4-full-all \
  --draft-dir ~/veloGB10/Qwen3.8-27B-DFlash2-NVFP4-RTNcal-FIXED \
  --port 9000 \
  --max-seq-len 262144 \
  --max-batch 1 \
  --max-tokens 65536 \
  --prefix-cache on \
  --default-presence-penalty 1.5 \
  --mtp=auto \
  --spec-source dflash2-auto

TP=2 (one peer node running ./gb10_inference --node --port 29500): same as above plus

  --tp 2 --nodes <peer-ip>:29500

TP=4 (three peer nodes): same plus

  --tp 4 --nodes <peer-ip1>:29500,<peer-ip2>:29500,<peer-ip3>:29500

When the drafter is live the engine logs:

[df2] DFlash2 round RESIDENT (spec-source=dflash2-auto) — serving via the S4F integrated round

and per node, one READY line each — that is your confirmation that the nodes were configured correctly. Only models are shipped to the nodes; you never copy model files to them by hand.

Benchmarks

The upstream card's tables (below, verbatim) were measured with SGLang on an RTX 5090 using the W4A4 ModelOpt layout, which is not what ships here, and they are not veloGB10 measurements. Acceptance and throughput on the veloGB10 engine are workload- and topology-dependent; see the veloGB10 repository for engine-side numbers on GB10 (single, TP=2, TP=4).

License and attribution

Apache-2.0. Original quantization and calibration: maurienne-ai/Qwen3.8-27B-DFlash2-NVFP4-RTNcal (quantized 2026-08-29). Base draft: incoai/Qwen3.8-27B-DFlash2 (z-lab / Inco AI). Repackaged for the veloGB10 engine. Their model card body is reproduced in full and unmodified below; its YAML header is folded into this file's own front matter above.

---

Upstream model card — maurienne-ai/Qwen3.8-27B-DFlash2-NVFP4-RTNcal

Reproduced verbatim from the upstream repository. Their benchmark numbers describe the W4A4 ModelOpt layout served by SGLang on an RTX 5090, not the weight-only repack in this repository.

Qwen3.8-27B-DFlash2-NVFP4-RTNcal

Calibrated NVFP4 (W4A4) quantization of the DFlash 2 block-diffusion draft model for Qwen3.8-27B, packaged in the NVIDIA ModelOpt layout that SGLang loads natively.

Built for a single RTX 5090 (32 GB) serving Qwen3.8-27B in NVFP4 with SGLang: the draft shrinks from 3.53 GB (BF16) to 1.37 GB of VRAM, which turns into +44 % KV-cache context (90K → 130K tokens) at the same decode speed and the same acceptance rate as the BF16 draft. Output quality is unchanged by construction: the target model verifies every drafted token.

Draft	VRAM	KV context*	Decode (end-to-end)	Accepted / 8
BF16 (upstream)	3.53 GB	90K	215 tok/s	3.71
NVFP4, calibrated (this repo)	1.37 GB	130K	228 tok/s	3.60
NVFP4, round-to-nearest, uncalibrated	1.37 GB	130K	210 tok/s	3.26

* Target gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090, FP8 KV cache, --mem-fraction-static 0.90, HiCache on. Benchmark: 5 mixed prompts (code, math, French prose, JSON, essay) × 600 generated tokens, 3 runs each, temperature 0.7. Run-to-run noise on the acceptance length is about ±0.2, so BF16 and this checkpoint are at parity.

Long-context agentic benchmark

Coding-agent session on the SGLang source tree (system prompt + 6 tool schemas, tool results = real source files), thinking on, streaming; 3 turns per context size (cold analysis, follow-up on the cached prefix, tool call with tools), 2 passes. TTFT is identical (prefill is the target's work, ~4K tok/s; cached prefix → 0.04–0.2 s in both), so only decode is shown.

Context	Decode BF16 → NVFP4 cal. (turn 1, thinking)	Decode BF16 → NVFP4 cal. (turn 2)	Accepted/8 BF16 → NVFP4 cal.
8K	166 → 200 tok/s	202 → 183	3.12 → 3.03
32K	168 → 196	217 → 235	3.18 → 3.32
64K	174 → 180	203 → 229	3.44 → 3.19
85K	157 → 166	178 → 218	3.18 → 3.21
110K	exceeds BF16 context → 207	— → 226	— → 3.63

Tool calls were emitted correctly in 10/10 tool turns with this draft (6/8 with BF16, the misses being sampling noise at T=0.7).

What is quantized

• NVFP4 (E2M1, group size 16, FP8-E4M3 block scales, FP32 per-tensor global scale) — all 35 linear projections of the 5 draft layers: self_attn.{q,k,v,o}_proj, mlp.{gate,up,down}_proj. Fused shards (q/k/v, gate/up) share their weight_scale_2 and input_scale, as ModelOpt requires.
• BF16 (unchanged) — fc (target-feature projection), attention_conv.* / mlp_conv.* (dynamic depthwise convolutions), candidate_selector.* (codebooks + hidden projection), all RMSNorm weights. These are plain nn.Linear/parameters in SGLang's DFlash2DraftModel and cannot be quantized there.
• Activations are quantized at runtime by SGLang to NVFP4 with the static per-tensor input_scale stored here (W4A4).

Calibration — the part that matters

Round-to-nearest NVFP4 without calibration loses ~13 % acceptance (3.26 vs 3.71). The whole gap is recovered by calibrating the activation scales on the draft's real inputs:

• On-policy data: 460 conversations generated by the target model itself through SGLang (thinking enabled on half of them), 6 domains — English multi-turn chat (ultrachat), French (Wikipedia-based tasks + free prompts), code (glaive-code-assistant + bash/SQL/TS/Rust/Go/CUDA prompts), tool calling (glaive-function-calling-v2 with tool schemas), math (GSM8K, MATH-500), structured output (JSON/YAML/Markdown). 281,649 generated tokens.
• Capture: forward pre-hooks on the 20 quantizable linears of the BF16 draft while it served those conversations (prefill and decode, eager mode), ~621K input rows per layer. Per-tensor input_scale = amax / (6 · 448).
• Weights: round-to-nearest on the NVFP4 grid; global scale amax / (6 · 448), per-group FP8 scales.

Things that were tried and made no measurable difference on this draft (all within ±0.2 noise): GPTQ with full Hessians, keeping q/k/v in BF16, SmoothQuant folding (o_proj, gate_up, down only — the qkv fold is not function-preserving in SGLang because context K/V are materialized from hidden_norm(fc(features)) without the per-layer input_layernorm).

Usage with SGLang

Requires SGLang built from main at or after commit ecbadf0b (adds DFlash2DraftModel), CUDA 13, a Blackwell GPU (sm120) for the NVFP4 GEMMs, and the draft quantization flag:

sglang serve \
  --trust-remote-code \
  --model-path gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090 \
  --served-model-name qwen3.8-27b \
  --kv-cache-dtype fp8_e4m3 \
  --mem-fraction-static 0.90 \
  --attention-backend flashinfer \
  --max-running-requests 1 --cuda-graph-max-bs 1 \
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder \
  --mamba-full-memory-ratio 5.61 \
  --mamba-radix-cache-strategy extra_buffer_lazy --mamba-ssm-dtype bfloat16 \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path <this repo> \
  --speculative-draft-model-quantization modelopt_fp4 \
  --speculative-num-draft-tokens 8 \
  --chunked-prefill-size 1024 \
  --cuda-graph-bs-prefill 64 128 256 512 1024 \
  --enable-memory-saver \
  --enable-hierarchical-cache --hicache-ratio 2 --hicache-write-policy write_through \
  --host 0.0.0.0 --port 30000

Notes:

• --speculative-draft-model-quantization modelopt_fp4 is required; SGLang does not requantize custom draft models on the fly.
• --speculative-num-draft-tokens 8 is the DFlash 2 block size and cannot be raised.
• The ignore list in config.json (flat ModelOpt format) is what SGLang reads to leave fc & co. in BF16; keep it if you edit the config.
• --cuda-graph-bs-prefill 64 128 256 512 1024 and MAX_JOBS=2 keep host-RAM usage sane on a 64 GB box during graph capture / JIT.
• KV cache in FP4 (--kv-cache-dtype nvfp4) does not work with speculative decoding on this SGLang build (hybrid GDN model).

Files

• model.safetensors — 1,550,153,248 bytes (sha256 2228b9b2…), 186 tensors (35 uint8 packed FP4 weights, 35 FP8 block scales, 70 FP32 scales, 46 BF16).
• hf_quant_config.json, config.json (quantization_config with ignore), tokenizer-free (uses the target's tokenizer), README.incoai-original.md (upstream card).

Reproduce

Scripts (SGLang side): dflash_calib_hook.py (activation capture, temporary patch of sglang/srt/models/dflash.py), calib/build_prompts.py + calib/run_calib.py (on-policy calibration run), quantize-dflash2-gptq-nvfp4.py --rtn (export). Base weights: incoai/Qwen3.8-27B-DFlash2 (BF16). Quantized on 2026-08-29.

Changelog

• 2026-08-30 — config.json: quantization_config.producer is now ModelOpt's dict form (was a plain string, which crashed vLLM's draft ModelConfig construction with AttributeError: 'str' object has no attribute 'get'; SGLang ignored the key). Weights unchanged (same model.safetensors, sha256 2228b9b2…). Thanks to @joelafrite for the report and the vLLM numbers.
• 2026-08-29 — initial release.

License

Apache-2.0, same as the base DFlash 2 draft and Qwen3.8-27B.