tcclaviger/ThinkingCap-3.8-27B-PARO5

NOTICE: This checkpoint runs only on the tcclaviger/vllm image on AMD AI PRO R9700 (gfx1201) GPUs

Activation-aware ParoQuant int5 / FP8 quantization of ThinkingCap-Qwen3.8-27B — 25.2 GB

• MLP (gate / up / down, all 64 layers): ParoQuant int5, 128-channel groups, 8 pairwise rotations + channel scales per group, activation-aware — 5.25 bits per weight plus rotations
• Attention and Gated DeltaNet projections: FP8, 128x128 blocks
• Embeddings, lm_head, norms, GDN conv / gates, MTP block: BF16
• WikiText-2 perplexity: 6.8807 ± 0.0317 (BF16 source: 6.864 ± 0.0315)

End-of-prefill divergence vs the BF16 source

top-1 agreement	97.48%
mean KLD	0.0115 ± 0.0015
median KLD	0.0006
p99 / max	0.156 / 0.691
RMS Δp at base argmax	0.053

docker run --rm -it \
  --network host --shm-size 32g \
  --ulimit memlock=-1:-1 \
  --cap-add SYS_PTRACE --security-opt seccomp=unconfined \
  --device /dev/kfd --device /dev/dri \
  --group-add video \
  -v <path>/ThinkingCap-3.8-27B-PARO5:/app/models \
  -v <path>/cache/triton:/cache/triton \
  -v <path>/cache/vllm:/cache/vllm \
  -v <path>/cache/inductor:/cache/inductor \
  -e TRITON_CACHE_DIR=/cache/triton \
  -e VLLM_CACHE_ROOT=/cache/vllm \
  -e TORCHINDUCTOR_CACHE_DIR=/cache/inductor \
  -e OMP_NUM_THREADS=8 \
  -e VLLM_ROCM_USE_AITER=0 \
  -e GPU_MAX_HW_QUEUES=2 \
  -e HSA_ENABLE_INTERRUPT=1 \
  -e HSA_ENABLE_MWAITX=1 \
  -e ROCR_VISIBLE_DEVICES=0,1,2,3 \
  tcclaviger/vllm:latest \
  /app/models \
  --tensor-parallel-size 4 \
  --tool-call-parser qwen3_coder \
  --enable-auto-tool-choice \
  --max-num-seqs 8 \
  --enable-chunked-prefill \
  --max-num-batched-tokens 8192 \
  --gpu-memory-utilization 0.92 \
  --host 0.0.0.0 \
  --port 8077 \
  --kv-cache-dtype fp8 \
  --served-model-name ThinkingCap-3.8-27B-PARO5 \
  --max-model-len 262144 \
  --reasoning-parser qwen3 \
  --override-generation-config '{"max_tokens": 65536, "temperature": 0.6, "top_p": 0.95, "top_k": 20}' \
  --compilation-config '{"cudagraph_mode": "PIECEWISE", "cudagraph_capture_sizes": [1, 8, 16, 24, 32, 40, 48, 56, 64], "max_cudagraph_capture_size": 64}'

Speculative decoding with a Qwen3.8-27B DFlash draft works unchanged: add --speculative-config '{"method": "dflash", "model": "/app/draft", "num_speculative_tokens": 7}' with the draft mounted at /app/draft.

How this checkpoint was quantized

ParoQuant (arXiv 2511.10645) with the paper's two-stage activation-aware layer-wise optimization (Alg. A2), run inside the same tcclaviger/vllm image that serves the result, on 4 x R9700 from the BF16 source.

1. Attention first. Each layer's attention and GDN projections are quantized to FP8 inline before that layer trains, so every calibration signal flows through the attention that ships.
2. Calibration corpus. 850 sequences of 2048 tokens from a 2M-token, 26-domain corpus (code, math, reasoning, tools, dialogue, multilingual, long books, and more), 64 sequences held out. The corpus is disjoint from the KL evaluation set.
3. Data-free start. Every MLP projection begins from a weight-space ParoQuant fit: per 128-channel group, 8 pairwise Givens rotations and channel scales chosen to minimise the int5 rounding error.
4. Layer-serial two-stage training, 64 layers. For each layer in order, the BF16 layer's output on the calibration set is the target. Stage 1 trains the rotation angles and channel scales; stage 2 trains the weights and the quantizer scales and zero points with straight-through rounding. AdamW, 10 epochs per stage, SmoothL1 loss, cosine schedule, best epoch by held-out loss. The quantized layer's output is then propagated forward as the next layer's input, so later layers calibrate against the quantized path rather than BF16.
5. Export. Weights are packed as int5 bit-planes with fp16 group scales and zero points, plus the served rotation pairs, angles and channel scales; at inference the rotation is applied to the activations by a fused kernel ahead of the int5 GEMM.
6. Validate. End-of-prefill KL divergence and top-1 agreement against the BF16 source over a frozen 994-prompt, 27-domain set disjoint from the calibration corpus: 97.48% top-1 agreement, mean KLD 0.0115.

ThinkingCap: Qwen 3.8 27B

In the second installment of our ThinkingCap series, we focus on maintaining the performance of Qwen3.8-27B (Qwen Team, 2026) on challenging and agentic tasks while delivering a substantial reduction in thinking verbosity. ThinkingCap Qwen3.8-27B cuts reasoning tokens by 37% on average (11% to 66% depending on the benchmark) and holds an average accuracy of 85.8% against the base model's 86.6%. It shines in long-context retrieval, where it cuts reasoning by 39% with accuracy intact (+2.3pp). Check our blogpost for more details.

Token efficiency and benchmark performance

Benchmark	Accuracy		Thinking tokens
	Base	Ours	Base mean	Ours mean	Reduction
Knowledge & reasoning
GPQA-Diamond	89.93 ±0.70	88.04 ±1.09	12,772	7,267	↓ 43.1%
MMLU-Pro	85.54 ±0.63	84.67 ±0.64	3,725	1,591	↓ 57.3%
MMMLU	85.38 ±0.69	84.09 ±0.72	1,656	571	↓ 65.5%
Math & code
AIME 2026	98.13 ±0.74	94.27 ±1.47	15,663	10,934	↓ 30.2%
HMMT (Feb 2026)	95.83 ±1.16	94.70 ±1.50	23,211	18,099	↓ 22.0%
HMMT (Nov 2025)	97.08 ±1.43	96.04 ±2.17	14,443	10,037	↓ 30.5%
LiveCodeBench v6	91.14 ±1.11	91.21 ±1.28	28,395	22,645	↓ 20.3%
Long-context & multimodal
AA-LCR	81.75 ±1.07	84.00 ±0.77	2,550	1,565	↓ 38.6%
RealWorldQA	83.25 ±0.73	82.34 ±0.71	992	492	↓ 50.4%
Instruction following & agentic
IFBench	79.75 ±0.63	79.71 ±0.60	7,961	4,266	↓ 46.4%
τ²-bench	76.16 ±1.52	75.15 ±1.67	4,584	3,168	↓ 30.9%
Terminal-Bench 2.1	75.84 ±4.26	75.28 ±4.38	72,871	65,092	↓ 10.7%
Macro average	86.6	85.8	15,735	12,144	↓ 37.2%

Evaluation details

Models

Base Qwen/Qwen3.8-27B against bottlecapai/ThinkingCap-Qwen3.8-27B, shown as Ours.

Metrics

• Accuracy (Base / Ours) — fraction of correct answers. The τ²-bench score is an unweighted mean over its three domains (airline, retail, telecom), not pooled over tasks.

• Thinking tokens (Base / Ours) — mean length of the <think> trace, answer excluded. τ²-bench and Terminal-Bench are multi-turn agentic episodes, so for those two rows the count is the reasoning summed over every turn of the episode (about 15 and 40 turns on average), not a single trace.

• (Thinking token) Reduction — the relative change between the two mean columns beside it, (Ours − Base) / Base, where each mean is taken over every question and seed of the benchmark. A saving is shown as a green ↓ percentage.

• Macro average (bottom row) — equal-weight mean across benchmarks of each column, including the reduction: it is the mean of the per-benchmark reductions, not the ratio of the two token figures beside it.

We separately track two trace-quality failure modes. Truncation: on the single-turn benchmarks, the <think> trace never closes because the model hits the generation cap while still reasoning, so no answer is produced. On the multi-turn benchmarks it is an episode-level flag, so it compounds over turns — τ²-bench marks an episode if any of its turns hits the per-turn cap, and Terminal-Bench marks a trial that exhausted its three-hour agent budget, which scores 0. Looping: the model repeating the same reasoning until it never finishes, detected with a compression-ratio test on the single-turn benchmarks and by the agent harness's own stalled-turn rule on Terminal-Bench. Both stay below 1% overall and both improve: truncation 0.51% → 0.34% and looping 0.06% → 0.05%, equal-weight across benchmarks.

Serving

Hardware: NVIDIA H200. vLLM 0.29.0, with MTP speculative decoding (num_speculative_tokens=3).

Thinking is on at reasoning_effort=xhigh, the chat template's own default, with the base model's recommended sampling — temperature=1.0, top_p=0.95, top_k=20, min_p=0.0 — used unchanged for Ours.

The generation cap is 253,952 tokens for most benchmarks. AA-LCR uses 131,072 and τ²-bench 65,536, because their documents and multi-turn transcripts occupy the rest of the window.

Benchmarks

Eleven run the complete set: AIME 2026 (30 problems), HMMT Feb 2026 (33), HMMT Nov 2025 (30), GPQA-Diamond (198), IFBench (300), RealWorldQA (765), AA-LCR (100), τ²-bench (278 tasks across airline, retail and telecom), Terminal-Bench 2.1 (89 tasks, Terminus-2 agent under Harbor), LiveCodeBench v6 (175) and MMLU-Pro (12,032 — the whole test split).

MMMLU is the one subset: a 10,000-question random sample of the multilingual set, drawn with a fixed seed so every condition sees the same questions.

Seeds and intervals

Independent runs per benchmark: 32 seeds on AIME 2026; 16 seeds on GPQA-Diamond, HMMT (Feb 2026), HMMT (Nov 2025) and IFBench; 8 seeds on LiveCodeBench v6, AA-LCR, RealWorldQA and τ²-bench; 4 seeds on Terminal-Bench 2.1; a single seed on MMLU-Pro and MMMLU. The seed count decides what the accuracy interval means. Multi-seed rows show the 95% t-interval across seeds — how much the answer moves when only the sampling seed changes, at Qwen's recommended temperature of 1.0. MMLU-Pro and MMMLU run a single seed over 12,032 and 9,996 questions, so they instead show a 95% Wilson interval over the question outcomes — how much the answer would move on a different draw of questions. The two measure different sources of variance and should not be read against each other.

Thinking mode comparison

We recommend using this model at the xhigh thinking effort for the best balance between accuracy and reasoning token usage. At lower efforts, the ThinkingCap treatment amplifies the effect of the effort setting while keeping its original trade-off. We plan to focus on improving the individual thinking modes in a future release.

Usage

HuggingFace Transformers

from transformers import AutoModelForImageTextToText, AutoProcessor
model = AutoModelForImageTextToText.from_pretrained("bottlecapai/ThinkingCap-Qwen3.8-27B", dtype="bfloat16")
proc = AutoProcessor.from_pretrained("bottlecapai/ThinkingCap-Qwen3.8-27B")

Check https://huggingface.co/Qwen/Qwen3.8-27B for recommended usage, sampling params etc.

vLLM / SGLang

Serve the bf16 model with either engine. The reasoning parser returns the thinking in a separate reasoning / reasoning_content field instead of inline in content before </think>, and the tool-call parser turns the model's XML tool calls into structured tool_calls — the same flags the base model's serving recipes use. The model's own MTP (multi-token-prediction / NextN) head gives self-speculative decoding with no separate draft model:

# vLLM — standard
vllm serve bottlecapai/ThinkingCap-Qwen3.8-27B \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_xml
# vLLM — with MTP self-speculative decoding
vllm serve bottlecapai/ThinkingCap-Qwen3.8-27B \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_xml \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

# SGLang — standard
python -m sglang.launch_server --model-path bottlecapai/ThinkingCap-Qwen3.8-27B --trust-remote-code \
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder
# SGLang — with MTP self-speculative decoding
python -m sglang.launch_server --model-path bottlecapai/ThinkingCap-Qwen3.8-27B --trust-remote-code \
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder \
  --speculative-algorithm EAGLE --speculative-num-steps 3 \
  --speculative-eagle-topk 1 --speculative-num-draft-tokens 4

MTP speculative decoding is lossless — the output is identical to standard decoding. On these bf16 weights, vLLM 0.29.0 with num_speculative_tokens=3 accepted 53% of drafted tokens across our xhigh evaluation runs — about 2.6 tokens per decoding step, identical to the base model's 54% and 2.6 — ranging from 2.5 on LiveCodeBench to 2.9 on τ²-bench and AA-LCR; the shorter traces at medium/low lift this to 3.2–3.3.

Either server speaks the OpenAI Chat Completions API. One request covers text, images and the thinking-effort knob:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="-")   # SGLang: port 30000
r = client.chat.completions.create(
    model="bottlecapai/ThinkingCap-Qwen3.8-27B",
    messages=[{"role": "user", "content": [
        {"type": "image_url", "image_url": {"url": "https://example.com/photo.jpg"}},
        {"type": "text", "text": "What is happening in this picture?"},
    ]}],
    temperature=1.0, top_p=0.95,
    extra_body={"top_k": 20,
                "chat_template_kwargs": {"reasoning_effort": "xhigh"}},   # xhigh (default) | medium | low
)
print(r.choices[0].message.reasoning)               # the thinking (`reasoning_content` on SGLang)
print(r.choices[0].message.content)                 # the answer

A text-only request is the same call with a plain string as content.

Quantized versions

Same checkpoint, chat template and license, quantized for smaller footprints and faster serving:

• FP8 — bottlecapai/ThinkingCap-Qwen3.8-27B-FP8 — FP8 block-wise, vLLM, 31 GB; Hopper and Blackwell
• GGUF — bottlecapai/ThinkingCap-Qwen3.8-27B-GGUF — IQ4_XS to f16 (16–55 GB), llama.cpp / LM Studio / Ollama; CUDA, Apple Metal, Vulkan or CPU
• NVFP4 — bottlecapai/ThinkingCap-Qwen3.8-27B-NVFP4 — NVFP4 weight-only, vLLM, 21 GB; Hopper (Marlin kernel) and Blackwell
• NVFP4 W4A4 — bottlecapai/ThinkingCap-Qwen3.8-27B-NVFP4A4-AWQ — NVFP4 weights and activations (AWQ), vLLM, 23 GB; Blackwell only

All builds are gathered in this collection.

Where to find us

Need even more efficiency? The open release is production-ready. Our enterprise versions go further — fewer thinking tokens still, tuned to your workload, at matched accuracy on your own tasks. Built for AI labs, inference providers and enterprises running models at scale. Deployed on your infrastructure, or in the cloud and region you choose. Talk to our team

License

ThinkingCap: PolyForm Small Business 1.0.0 + BottleCap personal-use grant (see LICENSE).

Upstream Qwen materials: Apache-2.0 (see NOTICE).

Commercial license: contact BottleCap AI.

Citation

If you use this model, please cite:

@misc{ThinkingCap-Qwen3.8-27B,
  title     = {bottlecapai/ThinkingCap-Qwen3.8-27B},
  author    = {Osusky, Adam and Lindauer, Jan and Jirkovsky, Adam and Mihal, Filip and Platek, Ondrej and Herel, David and Ihnatchenko, Luka and Bartek, Vojtech and Jirak, Jiri and Kubista, Daniel and Krus, Frantisek and Mikolov, Tomas},
  year      = {2026},
}