mph/qwen3.8-27b-mxfp8

Qwen3.8-27B (MXFP8)

MXFP8-quantized weights for Qwen/Qwen3.8-27B, a multimodal vision–language model. The text decoder backbone is stored in MXFP8; vision components and the language-model head remain bfloat16.

Quantization details

This checkpoint was produced with TorchAO using MXDynamicActivationMXWeightConfig:

Component	Precision
model.language_model (text decoder Linear layers)	MXFP8 weights; activations quantized dynamically at inference
visual (ViT + merger), lm_head	bfloat16
Hybrid Gated DeltaNet Conv1d / A_log / dt_bias / in_proj_b / in_proj_a	bfloat16

• Format: torchao-flattened safetensors (MXTensor qdata/scale + metadata)
• Block size: 32
• Dtypes: float8_e4m3fn for weights and activations
• Scaling: RCEIL
• Base dtype: bfloat16

Weights were quantized once on GPU, exported to CPU, flattened with flatten_tensor_state_dict, and saved with a TorchAoConfig in config.json. Reload does not re-run weight quantization; the language model still applies dynamic activation quantization during forward passes.

Hardware requirements

MXFP8 inference requires a Blackwell-class NVIDIA GPU (compute capability SM100+, i.e. major version ≥ 10). Examples include B200, GB200, and RTX Pro 6000. Older architectures (Ampere, Hopper, etc.) are not supported for this checkpoint.

• CUDA GPU with SM100+
• Sufficient VRAM for a 27B multimodal model (peak usage depends on sequence length and vision inputs)

Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True if you hit fragmentation during load or generation.

Software requirements

pip install "transformers>=5.5.4" torch torchao safetensors

You need a recent torchao build with MXFP8 inference support. For serving, use a recent vllm with TorchAO MXFP8 support (nightly or a source build is typical).

Load the processor from the same directory as the weights (vLLM does this automatically; there is no --processor flag):

from transformers import AutoProcessor, Qwen3_5ForConditionalGeneration
import torch

QUANTIZED_MODEL = "YOUR_USERNAME/qwen3.8-27b-mxfp8"  # or local path

processor = AutoProcessor.from_pretrained(QUANTIZED_MODEL)
model = Qwen3_5ForConditionalGeneration.from_pretrained(
    QUANTIZED_MODEL,
    torch_dtype=torch.bfloat16,
)
model.to("cuda")
model.eval()

Qwen3.8 reuses the Qwen3.5 Transformers class (Qwen3_5ForConditionalGeneration).

Usage

Thinking mode is on by default. Pass enable_thinking=False to apply_chat_template for instruct (non-thinking) mode.

Text-only

messages = [
    {"role": "user", "content": "Explain MXFP8 in one sentence."},
]
inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    add_generation_prompt=True,
    enable_thinking=True,
)
inputs = inputs.to(model.device)

with torch.inference_mode():
    output_ids = model.generate(**inputs, max_new_tokens=128, do_sample=False)

response = processor.decode(
    output_ids[0, inputs["input_ids"].shape[-1]:],
    skip_special_tokens=False,
)
print(response)

Image + text

from PIL import Image

image = Image.open("example.png").convert("RGB")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": image},
            {"type": "text", "text": "What is shown in this image?"},
        ],
    }
]
# Same apply_chat_template → generate → decode flow as above.

Serving with vLLM

This checkpoint can be served with vllm serve. vLLM loads the Qwen3 VL image/video processor from the model directory, so preprocessor_config.json and video_preprocessor_config.json must sit next to the weights. --tokenizer is optional once those sidecars are present.

Save the following as serve.sh (or run it inline):

#!/usr/bin/env bash
set -euo pipefail

export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
# TorchAO MX kernels and vLLM compile cache currently compose poorly.
export VLLM_DISABLE_COMPILE_CACHE=1

MODEL=mph/qwen3.8-27b-mxfp8
TOKENIZER=Qwen/Qwen3.8-27B
SERVED_NAME=qwen3.8-27b-mxfp8
PORT=8000
MAX_MODEL_LEN=5000  # native context is 262144

vllm serve "$MODEL" \
  --tokenizer "$TOKENIZER" \
  --served-model-name "$SERVED_NAME" \
  --host 0.0.0.0 \
  --port "$PORT" \
  --max-model-len "$MAX_MODEL_LEN" \
  --max-num-seqs 115 \
  --gpu-memory-utilization 0.94 \
  --attention-backend FLASHINFER \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --mm-encoder-tp-mode data

chmod +x serve.sh
./serve.sh

The server exposes an OpenAI-compatible API at http://localhost:8000/v1.

Recommended sampling (from the base model card):

Mode	temperature	top_p	top_k	presence_penalty
Thinking (default)	1.0	0.95	20	0.0
Instruct / non-thinking	0.7	0.80	20	1.5

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.8-27b-mxfp8",
    "messages": [
      {"role": "user", "content": "Explain MXFP8 in one sentence."}
    ],
    "temperature": 1.0,
    "top_p": 0.95,
    "max_tokens": 1024,
    "chat_template_kwargs": {
      "enable_thinking": true,
      "preserve_thinking": true
    }
  }'

from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
resp = client.chat.completions.create(
    model="qwen3.8-27b-mxfp8",
    messages=[{"role": "user", "content": "Explain MXFP8 in one sentence."}],
    temperature=1.0,
    top_p=0.95,
    extra_body={
        "top_k": 20,
        "chat_template_kwargs": {"enable_thinking": True},
        "reasoning_effort": "low",  # xhigh (default), medium, or low
    },
)
print(resp.choices[0].message.content)

Optional flags:

• --language-model-only — skip the vision encoder (more KV cache for text-only serving)
• --tensor-parallel-size N — split across GPUs if one card is not enough
• --default-chat-template-kwargs '{"enable_thinking": false}' — disable thinking server-wide
• YaRN to 1M context (from the base model card):

VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 ./serve.sh
# then add to the vllm serve invocation:
#   --hf-overrides '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}'
#   --max-model-len 1000000

Files

File	Description
model.safetensors	Quantized weights
config.json	Model config + quantization_config (TorchAoConfig)
generation_config.json	Generation defaults from the base model
preprocessor_config.json	Image processor (required by vLLM)
video_preprocessor_config.json	Video processor (required by vLLM)
tokenizer.json / tokenizer_config.json / vocab.json / merges.txt / chat_template.jinja	Tokenizer + chat template
manifest.json	Provenance and quantization summary

Limitations

• Quantization quality has not been formally benchmarked against the full-precision base model; validate on your tasks before production use.
• MXFP8 kernels and TorchAO MX support are still evolving; pin compatible torch / torchao / vllm versions for reproducibility.
• Vision and lm_head paths run in bf16, so memory savings are concentrated in the text decoder.

License

Apache 2.0. Follow the license terms of Qwen/Qwen3.8-27B.