huseinzolkepliscicom/gemma-4-12B-it-FP8-Dynamic

gemma-4-12B-it — FP8 Dynamic

FP8-dynamic (compressed-tensors) quantization of google/gemma-4-12B-it, produced with llm-compressor.

The language model is quantized to FP8 (E4M3, per-channel weights / per-token dynamic activations). The vision + audio embedders/towers are kept in bf16 — a quantized multimodal tower is not vLLM-servable (an FP8 gemma-4 vision tower dies with RuntimeError: Not yet supported ScalarType 46, and a quantized embed_audio/embed_vision fails vLLM weight-loading).

Verified serving on vLLM 0.23.0 (TP1), including tool-calling and reasoning:

vllm serve huseinzolkepliscicom/gemma-4-12B-it-FP8-Dynamic \
  --tensor-parallel-size 1 \
  --tool-call-parser gemma4 --enable-auto-tool-choice \
  --reasoning-parser gemma4 \
  --gpu-memory-utilization 0.9