mph/ThinkingCap-Qwen3.8-27B-MXFP8

ThinkingCap-Qwen3.8-27B-MXFP8

MXFP8 quantization of bottlecapai/ThinkingCap-Qwen3.8-27B, produced with NVIDIA TensorRT Model Optimizer (ModelOpt) v0.47.0.

What is quantized

MXFP8 stores weights and activations as FP8 (E4M3) values with a shared power-of-two (E8M0) scale for every block of 32 elements. This gives roughly half the weight memory of BF16 and keeps accuracy close to it, because the scaling is fine-grained.

The following modules are left in BF16, as listed in hf_quant_config.json:

• Language-model embeddings and lm_head
• Linear-attention convolution and projection modules
• All vision modules (visual.*)
• MTP modules

Quantized layers are therefore the main transformer linear layers (attention and MLP projections) of the language model.

Hardware requirements

• Native MXFP8 compute needs NVIDIA Blackwell (SM100/SM120) GPUs.
• Weights take about 30 GB, so one 40 GB+ GPU can hold the weights. Add headroom for KV cache and vision activations. Use more GPUs with tensor parallelism for long contexts or high concurrency.

Serving with vLLM

#!/usr/bin/env bash
set -euo pipefail

vllm serve mph/ThinkingCap-Qwen3.8-27B-MXFP8 \
    --port 8000 \
    --trust-remote-code \
    --enable-prefix-caching \
    --enable-chunked-prefill \
    --enable-auto-tool-choice \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_xml \
    --gpu-memory-utilization 0.95 \
    --attention-backend FLASHINFER \
    --mm-encoder-tp-mode data \
    --quantization modelopt_mxfp8 \
    --dtype bfloat16

Serving with SGLang

#!/usr/bin/env bash
set -euo pipefail

python -m sglang.launch_server \
    --model-path mph/ThinkingCap-Qwen3.8-27B-MXFP8 \
    --port 8000 \
    --trust-remote-code \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder \
    --mem-fraction-static 0.95 \
    --attention-backend flashinfer \
    --mm-enable-dp-encoder \
    --dtype bfloat16