morosystems

/

ThinkingCap-Qwen3.6-27B-NVFP4

like 7

Follow

MoroSystems s.r.o. 2

Summary

config.json advertises static FP8 KV-cache quantization, but the checkpoint contains zero
k_scale/v_scale tensors. vLLM therefore falls back to an uncalibrated scaling factor of 1.0
and emits accuracy warnings.

Because the setting comes from the checkpoint's own quantization_config, it cannot be
overridden or disabled from the serving side — so downstream users have no workaround.

Details

config.json declares:

"kv_cache_scheme": {"dynamic": false, "num_bits": 8, "type": "float"}

dynamic: false denotes static quantization, which requires precomputed scales stored in the
weights. Enumerating every *scale* tensor in model.safetensors.index.json (2,399 tensors):

The NVFP4 weight quantization is complete and correct. It is specifically the KV-cache
calibration that is declared in config but absent from the weights.

The model has 64 layers — 16 full_attention and 48 linear_attention — so exactly 16 k_scale
and 16 v_scale tensors are expected (only full-attention layers maintain a KV cache).

Observed behavior

WARNING [kv_cache.py] Checkpoint does not provide a q scaling factor. Setting it to k_scale.
WARNING [kv_cache.py] Using KV cache scaling factor 1.0 for fp8_e4m3. If this is unintended,
                      verify that k/v_scale scaling factors are properly set in the checkpoint.
WARNING [kv_cache.py] Using uncalibrated q_scale 1.0 and/or prob_scale 1.0 with fp8 attention.
                      This may cause accuracy issues.

Reproduced identically on two vLLM versions — this is not a version-specific artifact:

• vllm/vllm-openai:v0.19.1-cu130-ubuntu2404
• vllm/vllm-openai:vllm-x86_64-cu13-0.25.1 (v0.25.1)

With an unfitted scale of 1.0, the e4m3 dynamic range is not matched to the actual K/V activation
distribution, so available mantissa precision is not used effectively. The effect is expected to
compound over long contexts.

No downstream workaround

All of the following still yield fp8 KV with scale 1.0, because the checkpoint config takes
precedence over CLI flags:

• omitting --kv-cache-dtype entirely
• explicitly passing --kv-cache-dtype auto

Reference: a sibling quant of the same base model does this correctly

unsloth/Qwen3.6-27B-NVFP4 (same base model and architecture, compressed-tensors) ships:

16 matches the 16 full_attention layers exactly. Its kv_cache_scheme also records
"observer": "static_minmax", documenting how the scales were calibrated.

Suggested fix (either resolves it)

1. Run KV-cache calibration and ship the 16 k_scale + 16 v_scale tensors — preserves the
   FP8 KV memory savings. (Preferred.)
2. Remove kv_cache_scheme from config.json — a one-line change; serving stacks then use
   bf16 KV cache. Trades memory for accuracy.

Most likely cause: the kv_cache_scheme block was inherited/templated into the config during the
finetune-and-requantize step, while the KV calibration pass was not run.

Environment

• Model: morosystems/ThinkingCap-Qwen3.6-27B-NVFP4 (snapshot 656627c8)
• vLLM: 0.19.1 (cu130) and 0.25.1 — same behavior on both
• GPU: NVIDIA RTX PRO 6000 Blackwell (compute capability 12.0)
• Flags: --max-model-len 262144 --dtype bfloat16 --enable-prefix-caching