mattbucci/gemma-4-12B-AWQ

gemma-4-12B-it AWQ (int4, W4A16)

In-house AWQ-Marlin (4-bit) quantization of google/gemma-4-12B-it — the encoder-free omni Gemma 4 (Gemma4UnifiedForConditionalGeneration): text + reasoning + tool-use + vision + audio, global-MQA attention, 5:1 sliding:full, ~262K native context.

How it was built

Packed data-free from the QAT base google/gemma-4-12B-it-qat-q4_0-unquantized (Gemma's quantization-aware-trained release). Because QAT already conditions the weights onto a 4-bit grid, a plain RTN re-quantization to AWQ group-128 is near-lossless — no calibration corpus needed:

• Text-decoder linears (q/k/v/o_proj, mlp gate/up/down_proj) → AWQ int4, group_size 128.
• Vision/audio embedders + all norms + tied lm_head kept BF16 (modality preservation).
• Quantization fidelity vs the QAT base: MLP layers cosine 1.0000, attention projections 0.993–0.996.

Quality (2x RTX 3090, SGLang v0.5.12, TP=2)

MMLU	HumanEval	Needle@long-ctx	256K tool-use	Vision	Thinking
80%	95%	100%	100% (→95K)	✓	clean

Weights are 5.4 GB/rank at int4. Identical text quality to the BF16 base at matched context length.

Serving (SGLang)

This is the encoder-free unified Gemma 4 arch (gemma4_unified, a transformers-5.10.dev architecture). Serving on SGLang needs the back-port patches 042–048 from 2x-3090-GA102-300-A1-sglang-inference (loader + vendored config/processor + the hybrid-SWA TP>1 KV-pool routing fix + the processor __call__ image-token expansion). On Ampere (sm_86), use --attention-backend triton (FlashInfer rejects the 512-dim global head) and --disable-cuda-graph.

License

Governed by the Gemma Terms of Use. Derived from google/gemma-4-12B-it.