Qwen3.8-27B-FP8-dynamic

Quantized version of Qwen/Qwen3.8-27B, a Qwen3.5-family hybrid-attention VLM, produced with llm-compressor and served with vLLM.

Recipe

• Scheme: FP8 (W8A8), weights FP8 E4M3 per-channel RTN, activations FP8 per-token dynamic.
• Calibration: none (data-free).
• What is quantized: the text-decoder Linear layers only. The vision tower (re:.*visual.*), the hybrid linear-attention mixers (re:.*linear_attn.*), and lm_head stay in bf16. The full VLM (with vision_config) is saved in the compressed-tensors format, and the base MTP predictor is preserved for speculative decoding.
• Hardware: Native FP8 on Ada and Hopper (H100) and newer; runs elsewhere via the fp8-Marlin dequant path.

from transformers import AutoModelForImageTextToText, AutoTokenizer
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

MODEL_ID = "Qwen/Qwen3.8-27B"
model = AutoModelForImageTextToText.from_pretrained(MODEL_ID, dtype="bfloat16")
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)

recipe = QuantizationModifier(
    targets="Linear",
    scheme="FP8_DYNAMIC",
    ignore=["lm_head", "re:.*visual.*", "re:.*linear_attn.*", "re:.*mlp.gate$"],
)
oneshot(model=model, recipe=recipe)  # data-free
model.save_pretrained("Qwen3.8-27B-FP8-dynamic", save_compressed=True, save_original_format=False)
tokenizer.save_pretrained("Qwen3.8-27B-FP8-dynamic")

Serving (vLLM)

vllm serve soyrsoyr/Qwen3.8-27B-FP8-dynamic

from vllm import LLM, SamplingParams

llm = LLM(model="soyrsoyr/Qwen3.8-27B-FP8-dynamic")
out = llm.generate(["The capital of France is"], SamplingParams(max_tokens=32))
print(out[0].outputs[0].text)

Performance

Recovery vs. the bf16 base, evaluated through the vLLM backend with lm-evaluation-harness (OpenLLM v1) and lighteval (generative reasoning, pass@1 at temperature 0.6, top_p 0.95, up to 32k tokens).

OpenLLM Leaderboard v1

Benchmark	Qwen3.8-27B	FP8-dynamic	Recovery
ARC-Challenge (25-shot), acc_norm	50.68	49.91	98.5%
HellaSwag (10-shot), acc_norm	71.99	72.38	100.5%
TruthfulQA-mc2 (0-shot), acc	61.25	61.21	99.9%
Winogrande (5-shot), acc	76.87	76.01	98.9%
Average	65.20	64.88	99.5%

MMLU and GSM8K are omitted. Qwen3.8-27B is a reasoning model, so under the OpenLLM v1 protocol GSM8K has its chain of thought truncated and MMLU's loglikelihood is measured where the model wants to emit its think block, both of which collapse to a harness artifact rather than a real score. Math and knowledge are captured by the generative reasoning suite instead.

Reasoning suite (generative, pass@1)

Benchmarks are currently running (single H100). The recovery table will be filled in here as results land.