morosystems/ThinkingCap-Qwen3.6-27B-NVFP4

ThinkingCap-Qwen3.6-27B — NVFP4

NVFP4 (4-bit) quantization of bottlecapai/ThinkingCap-Qwen3.6-27B — the token-efficient finetune of Qwen/Qwen3.6-27B that produces ~50% fewer thinking tokens on average (up to ~90% on individual examples) while preserving answer quality.

This quant makes ThinkingCap practical on memory-bandwidth-limited Blackwell hardware: weights shrink from ~55 GB (BF16) to ~20.6 GB, which on an NVIDIA DGX Spark (GB10, 273 GB/s unified memory) takes single-stream decode from 4.6 tok/s to a measured 26.0 tok/s (with the preserved-MTP speculative decoding, n=3) — a 5.7× speedup — or 62.5–64.1 tok/s (up to 13.9×) with an external DFlash drafter (n=12, BF16 KV; see version-compatibility notes below). Quantized by MoroSystems — on a DGX Spark itself.

What's inside

• Weights: NVFP4 (FP4 with FP8 block scales, group size 16), weights + activations (W4A4), produced with NVIDIA TensorRT Model Optimizer 0.45.0 (quant_method: modelopt).
• Kept in BF16 for quality: lm_head, every linear_attn.conv1d (gated-deltanet short conv), the full vision tower (model.visual*), and the MTP head (mtp*) — so native qwen3_next_mtp speculative decoding still works.
• Also kept in BF16 (more conservative than typical community NVFP4 quants of this family): embed_tokens and the deltanet linear_attn.in_proj_a/in_proj_b projections, consistent with NVIDIA's upstream arch fix for Qwen3.5/3.6 exports.
• Calibration: open datasets only, reasoning-flavored to match the model's use: cnn_dailymail, nvidia/OpenCodeReasoning, nvidia/OpenMathReasoning (171 samples each, 513 total).
• KV cache: FP8 kv_cache_scheme baked into config.json (same as NVIDIA's official Qwen3.6-27B-NVFP4). Note: vLLM 0.22.x applies this scheme even with --kv-cache-dtype auto, which breaks attention backends that lack FP8-KV support (e.g. flash_attn, required for external-drafter speculative decoding like DFlash). For BF16 KV, serve a local copy with quantization_config.kv_cache_scheme removed from config.json — the checkpoint contains no k_scale/v_scale tensors, so this is a clean config-only change.
• Multimodal checkpoint: the vision tower and processor are included, same as the original.

Serving with vLLM

Works out of the box (quantization auto-detected). Reference command, tuned for a single DGX Spark (GB10):

vllm serve morosystems/ThinkingCap-Qwen3.6-27B-NVFP4 \
  --served-model-name thinkingcap \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.85 \
  --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":3}' \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --trust-remote-code

Notes from our DGX Spark deployment:

• With speculative decoding + tool calling, set VLLM_ENFORCE_STRICT_TOOL_CALLING=0 — draft tokens crash xgrammar's structural-tag FSM otherwise (vLLM issue #44006); the qwen3_xml parser still extracts tool calls.
• Reasoning surfaces in the reasoning field of chat completions.
• Sampling: the model inherits Qwen3.6 recommendations (temperature=1.0, top_p=0.95, top_k=20, min_p=0), shipped in generation_config.json.

Measured on DGX Spark (GB10, 128 GB unified)

vLLM 0.22.x, --max-model-len 262144, fp8 KV, MTP n=3 spec-decode (2026-07-07):

Metric	BF16 original	This quant, MTP n=3 (fp8 KV)	This quant, DFlash n=12 (BF16 KV)
Weights on disk	~55 GB	20.6 GB	20.6 GB
Single-stream decode	4.6 tok/s	26.0 tok/s (5.7×)	62.5 tok/s (13.6×)
Draft acceptance	—	2.33/3	4.99/12
Prefill (8k / 32k prompt)	—	~1,617 / ~1,444 tok/s	~1,482 / ~1,372 tok/s
Aggregate @ 8 concurrent streams	—	162 tok/s	218 tok/s
KV pool	—	8.1× @262k	3.25× @200k

DFlash setup (2026-07-08): external DFlash drafter, num_speculative_tokens=12, --attention-backend flash_attn, BF16 KV — requires stripping quantization_config.kv_cache_scheme from a local copy of config.json (see KV cache note above). Short-thinking behavior does not hurt drafter acceptance (4.99/12 vs 4.81/12 on the base-model-family drafter's home turf).

vLLM version compatibility (measured 2026-07-08):

• MTP path: works out of the box on vLLM 0.22.x (26.0 tok/s) and 0.23+ (27.3 tok/s, CUTLASS NVFP4 GEMM auto-selected).
• DFlash path: vLLM ≥0.23 cannot run this drafter without PR #40898 (still open) — upstream raises NotImplementedError on its mixed sliding/full layer_types. On 0.22.x-era builds it works as-is (62.5 tok/s); on current main with the PR applied it measures 64.1 tok/s.

Quantization of reasoning finetunes can distort behavior, so we specifically checked that the short-thinking property survives: on a fixed seeded prompt set (math, logic, code, knowledge), the quant answers correctly with a median of ~175 estimated reasoning tokens — squarely in the BF16 ThinkingCap range for questions of that size, with no looping observed. Tool calling (qwen3_xml) and the reasoning field were also verified end-to-end.

vs. NVIDIA's official Qwen3.6-27B-NVFP4 (measured on DGX Spark)

nvidia/Qwen3.6-27B-NVFP4 quantizes the base model with a more conservative mixed-precision recipe (per its hf_quant_config.json): FP8 W8A8 on attention projections, NVFP4 W4A16 on MLPs, FP8 KV, MTP head excluded — totalling 21.9 GB vs this repo's 20.6 GB. We benchmarked both on the same DGX Spark with identical vLLM 0.22.x settings (MTP n=3, fp8 KV, 262k context, 2026-07-08):

Metric (think-off, 400 tok)	This quant (ThinkingCap)	nvidia/Qwen3.6-27B-NVFP4 (base)
Single-stream decode	26.0 tok/s	34.2 tok/s
Prefill 8k / 32k prompt	~1,482 / ~1,372 tok/s	~866 / ~1,018 tok/s
Thinking on: time to first answer token	7.4 s	12.7 s

The FP8-attention mix decodes ~31% faster with MTP (its FP8 GEMV/GEMM kernels are more efficient than the NVFP4 path on GB10 — and this persists on vLLM 0.23 with the CUTLASS NVFP4 kernel active, so it is inherent to the precision layout, not a fixable kernel gap), but it prefills ~40% slower (W4A16 MLPs), and as the base model it spends roughly twice the thinking tokens before answering — so ThinkingCap still reaches answers markedly sooner end-to-end. For maximum single-stream speed on this checkpoint, use DFlash speculative decoding instead of MTP: 62.5 tok/s measured (external drafter, n=12, BF16 KV — strip kv_cache_scheme from config.json as noted above).

Original model

All quality claims and benchmark numbers below are for the BF16 original, evaluated by BottleCapAI — see the original model card for the full tables and methodology. Headline results (macro averages, base Qwen3.6-27B → ThinkingCap):

Suite	Accuracy (base → ThinkingCap)	Thinking-token reduction
Out-of-domain (14 benchmarks)	81.5 → 80.7	↓ 45.8%
In-domain holdouts (7 benchmarks)	94.4 → 95.4	↓ 57.7%
Safety (guardrails preserved)	99.4 → 99.5 SAFE%	↓ 21.9%

Truncation failures (think trace hitting the token cap) drop from 2.9% to 0.4% out-of-domain.

Provenance & license

• Base model Qwen/Qwen3.6-27B: Apache-2.0.
• bottlecapai/ThinkingCap-Qwen3.6-27B declares no explicit license; this repository is published under Apache-2.0 (inherited from the base model) as a community quantization, with full attribution. If you are BottleCapAI and want this changed or taken down, open a discussion here and we'll comply.
• Quantization: TensorRT Model Optimizer 0.45.0 (hf_ptq.py --qformat nvfp4), PTQ run on a single DGX Spark. The full recipe is described above (calibration datasets/sizes, exclusion lists, KV scheme); the exact quantization_config/hf_quant_config.json in this repo are the authoritative record.

Citation

Please cite the original model:

@misc{,
  title     = {bottlecapai/ThinkingCap-Qwen3.6-27B},
  author    = {Lasocki, Karol and Osusky, Adam and Lindauer, Jan and Jirkovsky, Adam and Mihal, Filip and Platek, Ondrej and Herel, David and Ihnatchenko, Luka and Bartek, Vojtech and Jirak, Jiri and Mikolov, Tomas},
  year      = {2026},
}