Accept the ThinkingCap license
This repository is publicly accessible, but you have to accept the conditions to access its files and content.
These weights are a quantization of BottleCap AI's ThinkingCap-Qwen3.8-27B and are licensed under the PolyForm Small Business License 1.0.0 with BottleCap AI's additional personal-use permission (see LICENSE). Commercial use by organizations that are not a "Small Business" requires a license from BottleCap AI.
Log in or Sign Up to review the conditions and access this model content.
ThinkingCap-Qwen3.8-27B-MXFP4
BottleCap AI's ThinkingCap-Qwen3.8-27B โ a thinking-efficiency finetune of Qwen3.8-27B that reaches the base model's accuracy with roughly half the reasoning tokens โ quantized to MXFP4 (OCP microscaling FP4: e2m1 values, one power-of-two scale per 32 weights) in the compressed-tensors format.
Who this is for: GPUs with fast MXFP4 support but no NVFP4 โ in particular AMD RDNA4 (Radeon AI PRO R9700, RX 9070 series, gfx1201) with vllm-sly-radiance, and AMD Instinct MI350/MI355. On NVIDIA Blackwell use BottleCap's own NVFP4 build โ it is closer to bf16 than any MXFP4 version can be (see below).
Quick start
RDNA4 with vllm-sly-radiance (one R9700, 32 GB, 262k context, DFlash2 speculative decoding):
docker run --rm --device=/dev/kfd --device=/dev/dri --group-add video \
--security-opt seccomp=unconfined --ipc=host -p 8000:8000 \
-e RADIANCE_MXFP4=1 -e RADIANCE_MXFP4_W4A8=1 -e RADIANCE_MXFP4_WPERM=1 \
-e RADIANCE_LMHEAD_INT4=1 -e RADIANCE_EMBED_INT8=1 -e RADIANCE_EMBED_BITS=4 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
ghcr.io/slybase/vllm-sly-radiance:0.4.1-rocm10.0 \
--model slybase/ThinkingCap-Qwen3.8-27B-MXFP4 \
--max-model-len 262144 --kv-cache-dtype fp8 --gpu-memory-utilization 0.96 \
--speculative-config.method dflash --speculative-config.model syvai/Qwen3.8-27B-DFlash2-W4A16 \
--speculative-config.num_speculative_tokens 7 \
--limit-mm-per-prompt.image 0 --limit-mm-per-prompt.video 0 \
--reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_xml
The full production argument set is in the
vllm-sly-radiance README. Other platforms: any vLLM build
whose compressed-tensors MXFP4 scheme has a native kernel for your GPU (vllm serve slybase/ThinkingCap-Qwen3.8-27B-MXFP4). Stock vLLM on RDNA4 falls back to an emulation kernel and is
slow โ use the fork there.
Quality
Measured on one R9700 against BottleCap's NVFP4 checkpoint computed exactly (the bf16 model does not fit a 32 GB card): mean negative log-likelihood of 14,125 fixed tokens (Python code, legal prose, German and English Markdown), top-20 prompt logprobs, deterministic.
| build | NLL | ฮ vs NVFP4 | top-1 token agreement |
|---|---|---|---|
| BottleCap NVFP4, exact | 0.787 | โ | โ |
| this model (GPTQ MXFP4) | 0.823 | +0.036 ยฑ 0.013 | 91.2 % |
| MXFP4 round-to-nearest from bf16 | 0.819 | +0.032 ยฑ 0.013 | 90.8 % |
| NVFP4 requantized to MXFP4 at load | 0.881 | +0.094 ยฑ 0.012 | 91.1 % |
GSM8K (CoT zero-shot, 200 questions, greedy, vllm-sly-radiance): 0.860 ยฑ 0.025 (NVFP4 requantized at load: 0.845 ยฑ 0.026).
Weight error against bf16 (relative RMS, 48-layer sample): MXFP4 0.113, NVFP4 0.095. The gap is the format: MXFP4's power-of-two scale per 32 weights is coarser than NVFP4's e4m3 scale per 16, and GPTQ does not close it (round-to-nearest with a per-block MSE exponent scores the same). If your hardware runs NVFP4, use NVFP4.
Speed (one R9700, vllm-sly-radiance 0.4.1, 300 W)
| metric | value |
|---|---|
| step time (one sequence, DFlash2 k=7) | 33.7 ms |
| accepted tokens per step (12 chat prompts, greedy / T=0.7) | 4.14 / 3.96 |
| end-to-end decode, short chat answers | ~115 tokens/s |
| KV cache (fp8, 262k context, 0.96 memory) | 364,897 tokens |
Same kernels and step time as the AMD Quark MXFP4 build of the base model; ThinkingCap's shorter reasoning makes answers arrive sooner at the same tokens/s.
How it was made
- Source:
bottlecapai/ThinkingCap-Qwen3.8-27B(bf16, revisiond2b4e7a). - llm-compressor 0.14.0,
GPTQModifier(scheme="MXFP4A16", targets="Linear", actorder="static"), sequential (layer-by-layer) calibration on the GPU with disk offload, so the whole 54 GB model never sits in RAM. - Calibration: 128 samples, max 2,048 tokens โ 88 UltraChat dialogues rendered with the ThinkingCap chat template, 40 German technical documentation blocks.
- Quantized: every linear of the language model (attention, Gated DeltaNet projections incl.
in_proj_a/b, MLP). Kept bf16: embeddings,lm_head, norms, conv, vision tower, MTP head (vLLM skips the latter two unless enabled). - The recipe is in
recipe.yaml.
License
Weights and associated materials: PolyForm Small Business License 1.0.0 with BottleCap AI's additional
personal-use permission โ see LICENSE, NOTICE and LICENSE-PolyForm-Small-Business-1.0.0.txt.
Required Notice: Copyright 2026 BottleCap AI (https://bottlecapai.com). ThinkingCap is derived from
Qwen3.8-27B (Apache-2.0, see LICENSE-Apache-2.0-Qwen.txt). This repository only re-quantizes the
weights; all model credit belongs to BottleCap AI and the Qwen team.
- Downloads last month
- 47