tacodevs/Wicked-Stuff-31B-Final-FP8-Dynamic

Wicked-Stuff-31B-Final FP8-Dynamic (W8A8)

FP8 quantization of ReadyArt/Wicked-Stuff-31B-Final (Gemma 4 31B).

• Scheme: FP8_DYNAMIC W8A8: per-channel FP8 weights, dynamic per-token FP8 activations, on all Linear layers (llm-compressor). Vision, audio, embeddings and lm_head stay BF16. No calibration data needed.
• Hardware: Hopper (H100/H200) and Blackwell; Ada (RTX 4090/L40S) also has FP8.

vllm serve tacodevs/Wicked-Stuff-31B-Final-FP8-Dynamic --max-model-len 32768 --kv-cache-dtype fp8 \
  --reasoning-parser gemma4 --enable-auto-tool-choice --tool-call-parser gemma4

Thinking: pass chat_template_kwargs: {"enable_thinking": true}.