ThinkingCap-Qwen3.6-27B-FP8-attnbf16

FP8_DYNAMIC W8A8 quantisation of bottlecapai/ThinkingCap-Qwen3.6-27B — a dense 27B qwen3_5 token-efficiency fine-tune (multimodal vision tower + hybrid linear/full attention) with an MTP speculative-decoding head. Produced with llm-compressor.

Why this build exists

The upstream official FP8 (bottlecapai/ThinkingCap-Qwen3.6-27B-FP8) is a DeepSeek-style block-128 checkpoint (quant_method: fp8) that quantises three of the recurrent SSM projections: linear_attn.in_proj_qkv, linear_attn.in_proj_z and linear_attn.out_proj. Discussion #6 on that repo reports multi-turn agent instability — the model losing track of what it did in earlier turns — and asks for a build that keeps those layers in bf16.

This build keeps the entire linear_attn block in bf16. That has been our standing recipe for the Qwen3.5/3.6 family since well before that report: the Mamba/SSM block carries recurrent state across the sequence, and quantising it measurably degrades these models (on Qwen3.6-27B, NVFP4-ing it roughly doubled KLD and introduced a thinking-loop pathology).

Precision layout vs upstream, verified tensor-by-tensor on both checkpoints:

Module	This build	Upstream -FP8
linear_attn.in_proj_qkv	bf16	FP8
linear_attn.in_proj_z	bf16	FP8
linear_attn.out_proj	bf16	FP8
linear_attn.in_proj_a / in_proj_b / norm / conv1d	bf16	bf16
self_attn.{q,k,v,o}_proj, mlp.{gate,up,down}_proj	FP8	FP8
model.visual.*, mtp.*, lm_head	bf16	bf16

Scope of that claim — please read before choosing this build. We verified the precision layout of both checkpoints directly from their weights, so the table above is fact. But we then went looking for the reported effect and did not find it: turn-bucketed KLD over real multi-turn conversations shows divergence falling with conversation depth in the upstream build just as much as in ours (see "Multi-turn behaviour" below). We have not reproduced the instability in Discussion #6, and we cannot claim this build fixes it. What we can say is that it is the bf16-SSM configuration that report asks for. Two corrections to the report itself: conv1d is a Conv1d and is never a quantisation target in either pipeline, and on this architecture the projection is split in_proj_qkv + in_proj_z rather than Qwen3-Next's fused in_proj_qkvz.

This is also a compressed-tensors checkpoint rather than DeepSeek-format FP8, which means it loads under 🤗 transformers on Blackwell — the finegrained_fp8 path used by the upstream FP8 has no SM120 recipe and fails there. That is not a theoretical concern: measuring the upstream checkpoint for the table below required dequantising it by hand first.

Quantisation

• Scheme: FP8_DYNAMIC W8A8 — 8-bit FP8 (E4M3) weights with dynamic per-token FP8 activation quantisation (data-free; activation scales computed at inference) — MLPs only; the entire attention path stays in bf16.
• Tooling: llm-compressor 0.12.0 + compressed-tensors 0.17.1, transformers 5.8.1, torch 2.11+cu128, on a Blackwell RTX PRO 6000 (SM120).
• Quantised: the mlp (gate/up/down) projections of the 64 language-model layers only — no attention weight is quantised.
• Kept in bf16 (quantization_config.ignore): lm_head, all model.visual.* (vision tower), the entire linear_attn Mamba/SSM block, the entire self_attn block, and mtp.*. Also bf16 by construction: embed_tokens, every layernorm, and the q_norm/k_norm QK-RMSNorms.
• Recipe: recipe.yaml. Data-free — no calibration set, so no calibration-domain bias.

The attention path is entirely bf16

This variant additionally keeps every attention weight in bf16 — q_proj, k_proj, v_proj, o_proj on top of the linear_attn block. Only the MLPs are quantised.

That trade is unusually cheap on this architecture. Just 16 of the 64 layers are full_attention (the other 48 are linear_attn), so the whole self_attn block is only 1.68 B params: FP8-ing it saves ~1.56 GiB, while the MLPs (17.1 B params) deliver ~15.9 GiB. Giving up ~4.6% of on-disk size buys a fully bf16 attention path.

The specific thing this protects: this architecture sets attn_output_gate: true, which fuses the attention output gate into q_proj — that is why q_proj is [12288, 5120] rather than [6144, 5120] (2 × 24 heads × 256 head_dim). Half of that tensor is not queries at all; it is a multiplicative per-head gate controlling how much each attention head writes into the residual stream. Quantisation error on a multiplicative gate behaves worse than on an additive projection, and those 16 full-attention layers are the ones carrying long-range retrieval across a conversation.

Honesty about this claim: this is a mechanistic argument plus a cheap price, not a reproduced failure. The sibling build's aggregate distortion is already low, and we have not demonstrated that quantising q_proj causes any specific misbehaviour. Pick this build if multi-turn or long-context stability is worth 1.5 GiB to you; pick the sibling if it isn't.

Sibling build: huginnfork/ThinkingCap-Qwen3.6-27B-FP8 — the other variant of this quantisation. Same parent, same pipeline, same evaluation; they differ only in the ignore list.

Speculative decoding (MTP)

This repo ships the 15 mtp.* multi-token-prediction tensors. They are byte-identical to the Qwen/Qwen3.6-27B base MTP head (verified tensor-by-tensor) — the ThinkingCap LoRA merge did not retrain them. llm-compressor drops MTP during quantisation (transformers v5 doesn't instantiate the submodule), so they were regrafted in bf16 from the bf16 parent and are present in this repo's safetensors index for vLLM/SGLang to pick up.

Draft acceptance is not independently measured here. Two data points bracket what to expect from a base-inherited head: a light edit (heretic abliteration) retained 91.8% acceptance, while a heavy reasoning distill (Tess-4-27B) collapsed to 0%, where speculative decoding actively slowed generation. ThinkingCap is a merged LoRA (r=64) — closer to the light end — and the parent's card reports ≈3.34 drafts per verify step. If you enable --speculative-config, measure acceptance on your own workload and disable it if acceptance is low.

Quality

KLD is per-token KL divergence vs the bf16 ThinkingCap parent over 8 neuralmagic/calibration samples (max_seq=1024) — measured against the parent, so it isolates quantisation loss rather than the fine-tune delta. PPL is wikitext-2-raw (test), non-overlapping 2048-token chunks, identical tokenizer to the parent.

Model	KLD vs bf16 parent (nats)	PPL (wikitext-2-raw)	ΔPPL
bottlecapai/ThinkingCap-Qwen3.6-27B (bf16)	0	7.3226	—
this (FP8-attnbf16)	0.0132	7.3927	+0.96%

The KLD below is weight-only: the eval does not model FP8_DYNAMIC's per-token activation quant, so the served KLD is marginally higher.

All variants, including the upstream official FP8

Every row was measured by us on the same box, same scripts, same prompts. The upstream FP8 cannot be loaded by transformers on Blackwell at all, so it was block-128 dequantised to bf16 first (weight round-trip preserved, so its precision loss is captured — same weight-only caveat as our own FP8 rows).

Build	Attention path	Quantised	Size	KLD vs bf16 parent	PPL	ΔPPL
bottlecapai/ThinkingCap-Qwen3.6-27B (bf16 parent)	bf16	— nothing —	51.7 GiB	0	7.3226	—
bottlecapai/ThinkingCap-Qwen3.6-27B-FP8 (upstream official)	FP8	self_attn + mlp + 3 SSM projections	29.1 GiB	0.0148	7.3349	+0.17%
ThinkingCap-Qwen3.6-27B-FP8	FP8 (q/k/v/o_proj quantised)	self_attn + mlp	34.3 GiB	0.0222	7.4131	+1.24%
ThinkingCap-Qwen3.6-27B-FP8-attnbf16 ← this repo	bf16 (entire path)	mlp only	35.8 GiB	0.0132	7.3927	+0.96%

Read this table honestly: on average-case metrics the upstream FP8 is the strongest of the three, and it is also the smallest.

We do not have a verified explanation for that, and the obvious one is wrong. The two schemes differ in scale orientation, not simply granularity: llm-compressor emits one scale per output channel (covering a whole row — 5,120 elements here), while upstream uses a 2D 128×128 grid (16,384 elements per scale). By element count ours is the finer of the two, so "upstream has more scales" does not explain it. What upstream's grid does have, and per-channel scaling entirely lacks, is resolution along the input dimension. If large weights cluster in particular input columns — commonly reported for transformer weights — a single row-wide scale is inflated by them and costs precision for every other value in that row, whereas a 2D grid isolates them. That is a plausible mechanism we have not measured; treat it as a hypothesis.

Where our builds do win:

• Worst-case token divergence. Max single-token KLD is 13.47 nats for upstream vs 7.56 (our FP8) and 5.04 (our -attnbf16). Upstream is better on average but has a heavier tail — rare tokens where it diverges hard.
• It loads on Blackwell. Upstream is DeepSeek-format FP8 (quant_method: fp8); transformers' finegrained_fp8 path has no SM120 recipe, so it will not load on an RTX PRO 6000 / B200 without a manual dequantisation step. These are compressed-tensors, which load natively in both transformers and vLLM.
• The bf16 SSM/attention layout requested in Discussion #6 (see above).

Identical across our two builds: bf16 linear_attn, bf16 vision tower, bf16 lm_head / embed_tokens / all layernorms / q_norm+k_norm, and the 15 mtp.* tensors regrafted in bf16.

Multi-turn behaviour — measured, and it does not show drift

Because Discussion #6 describes a multi-turn failure, a single averaged KLD cannot settle it. We therefore measured per-token KLD bucketed by assistant-turn index over 20 real multi-turn conversations from allenai/WildChat-1M (≥6 messages, 2048-token window, 18,551 assistant tokens scored, KLD on assistant tokens only).

If quantisation corrupted recurrent state across turns, KLD would rise with turn depth. It does not — in any of the three builds:

Build	mean KLD (multi-turn)	turn 0	deepest turn	slope per turn
upstream -FP8	0.0077	0.0055	0.0020	−0.00055
our -FP8	0.0125	0.0084	0.0027	−0.00258
our -FP8-attnbf16	0.0122	0.0077	0.0026	−0.00242

All three slopes are negative: divergence shrinks as the conversation deepens, for the upstream build as much as ours.

What this does and does not prove. The measurement is teacher-forced — both models see identical ground-truth history, with context reset to truth at every token. It therefore tests representational drift, and finds none. It cannot detect trajectory divergence, where a small early deviation changes what the model conditions on and compounds. That is arguably the likelier mechanism for the behaviour in Discussion #6, and measuring it needs free-running generation, which we have not done. So: we found no evidence for the reported mechanism, and we are not claiming to have disproved the report.

Raw measurement JSONs are in this repo (kld_thinkingcap-fp8-attnbf16.json, ppl_thinkingcap-bf16.json, ppl_thinkingcap-fp8-attnbf16.json).

Inference

Dense checkpoint — runs under both 🤗 transformers and vLLM.

transformers

from transformers import AutoModelForImageTextToText, AutoProcessor
import torch

model = AutoModelForImageTextToText.from_pretrained(
    "huginnfork/ThinkingCap-Qwen3.6-27B-FP8-attnbf16", dtype=torch.bfloat16, device_map="auto",
)
proc = AutoProcessor.from_pretrained("huginnfork/ThinkingCap-Qwen3.6-27B-FP8-attnbf16")

vLLM

vllm serve huginnfork/ThinkingCap-Qwen3.6-27B-FP8-attnbf16 \
    --trust-remote-code --quantization compressed-tensors \
    --max-model-len 8192 --max-num-seqs 32

--max-num-seqs 32 is not optional on this architecture: every Qwen3.5/3.6 model has linear_attn layers, and vLLM aborts CUDA-graph capture when max_num_seqs exceeds the available Mamba cache blocks.

On Blackwell (RTX PRO 6000 / B200, SM120/SM100): FlashInfer's JIT arch probe mis-detects SM120 and aborts engine startup. Disable it:

TORCH_CUDA_ARCH_LIST=12.0+PTX VLLM_USE_FLASHINFER_SAMPLER=0 VLLM_USE_FLASHINFER=0 \
    vllm serve huginnfork/ThinkingCap-Qwen3.6-27B-FP8-attnbf16 --trust-remote-code --quantization compressed-tensors \
    --max-model-len 8192 --max-num-seqs 32

vLLM's NVFP4/FP8 Marlin path may print accuracy warnings about parallel-layer global scales; those concern the serving kernel, not these weights. The KLD/PPL above are measured in transformers.

Provenance

Quantised from the bf16 parent with no intermediate re-upload. mtp.* regrafted in bf16 from the same parent checkpoint. Pipeline and landmines documented at the source repo for this work.