Tess-4-27B-FP8

FP8_DYNAMIC W8A8 quantisation of migtissera/Tess-4-27B — a dense 27B qwen3_5 reasoning fine-tune (multimodal vision tower + hybrid linear/full attention) with an MTP speculative-decoding head. Produced with llm-compressor.

Quantisation

• Scheme: FP8_DYNAMIC W8A8 — 8-bit FP8 (E4M3) weights with dynamic per-token FP8 activation quantisation (data-free; activation scales computed at inference).
• Tooling: llm-compressor 0.12.0 + compressed-tensors 0.17.1, transformers 5.8.1, on a Blackwell RTX PRO 6000 (SM120).
• Quantised: the self_attn and mlp (gate/up/down) projections of the 64 language-model layers.
• Kept in bf16 (quantization_config.ignore): lm_head, all model.visual.* (vision tower), the entire linear_attn Mamba/SSM block (in_proj_*, out_proj, conv1d), and mtp.*. Keeping the SSM block in bf16 is essential on this family — NVFP4-ing it inflates KLD by ~2×.
• Recipe: recipe.yaml.

Speculative decoding (MTP)

Tess-4-27B ships the 15 mtp.* tensors, and they are byte-identical to the Qwen/Qwen3.6-27B base MTP head — the fine-tune reused it rather than retraining. llm-compressor drops MTP during quantisation (transformers v5 doesn't instantiate the submodule), so it was regrafted in bf16 from the source checkpoint and the 15 tensors are present in this repo's safetensors index for vLLM/SGLang to pick up.

Measured with vLLM (method: qwen3_5_mtp, num_speculative_tokens: 1, greedy, 12 mixed chat/code/reasoning prompts): The fine-tune shifted the trunk so far that the inherited head is useless for speculation: 0.0% of drafted tokens were accepted, and speculative decoding is counterproductive (it slows generation, since every rejected draft is wasted compute). Serve without --speculative-config. The mtp.* tensors are still shipped (matching the parent's structure and available for anyone who retrains the head), but do not enable MTP speculative decoding as-is. Positive control: the base Qwen/Qwen3.6-27B MTP head accepts 92.7% on the identical vLLM stack, so the number below is a genuine property of Tess's fine-tuned trunk, not a serving artefact. End-to-end throughput was 35.3 tok/s with speculation on vs 40.6 tok/s off = 0.87×.

Quality

KLD is per-token KL divergence vs the bf16 Tess-4-27B parent over 8 neuralmagic/calibration samples (max_seq=1024) — it isolates quantisation loss (not the fine-tune delta). PPL is wikitext-2-raw (test), non-overlapping 2048-token chunks, identical tokenizer to the parent. Compressed weights were decompressed to bf16 for the eval (the quant→dequant round-trip is applied, so precision loss is captured).

Model	KLD vs bf16 Tess (nats)	PPL (wikitext-2-raw)	ΔPPL
migtissera/Tess-4-27B (bf16)	0	6.8180	—
this (FP8)	0.0132	6.8648	+0.69%

The FP8 KLD below is weight-only: the bf16-decompressed eval does not model FP8_DYNAMIC's per-token activation quant, so the served KLD is marginally higher.

Inference

This is a dense checkpoint, so it runs under both 🤗 transformers and vLLM.

transformers

from transformers import AutoModelForImageTextToText, AutoProcessor
import torch

model = AutoModelForImageTextToText.from_pretrained(
    "huginnfork/Tess-4-27B-FP8", dtype=torch.bfloat16, device_map="auto",
)
proc = AutoProcessor.from_pretrained("huginnfork/Tess-4-27B-FP8")

vLLM (with MTP speculative decoding)

vllm serve huginnfork/Tess-4-27B-FP8 \
    --trust-remote-code --quantization compressed-tensors --max-model-len 8192 \
    --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":1}'

On Blackwell (RTX PRO 6000 / B200, SM120/SM100): FlashInfer's JIT arch probe currently mis-detects SM120 and aborts engine startup. Disable it:

TORCH_CUDA_ARCH_LIST=12.0+PTX VLLM_USE_FLASHINFER_SAMPLER=0 VLLM_USE_FLASHINFER=0 \
    vllm serve huginnfork/Tess-4-27B-FP8 --trust-remote-code --quantization compressed-tensors --max-model-len 8192