keptn/gemma4-31b-a4

Gemma 4 31B-it · a4 · for radiance

google/gemma-4-31B-it in one 20 GiB file for the radiance inference engine on AMD RDNA4. The file reads text, images and video and includes Google's assistant head for speculative decoding.

Version 2.0 uses the a4 format. The weights are int4 with a zero point and one f16 scale per 128 weights. The activations are int8 with one scale per 128 values. It is calibrated with keptn/quant-calibration-v1. Its KL divergence is about half that of version 1.0. It decodes 1.6 % slower than version 1.0 on one card and 7.5 % slower on two cards.

Version 1.0 was called w4a8: int4 weights without a zero point, one f16 scale per 128 weights, and int8 activations with one scale per row. The repository was named keptn/gemma4-31b-w4a8 then, and that name redirects here. Version 1.0 is still available at revision v1.0.

File	gemma4-31b-a4.rad, 19.95 GiB
Needs	radiance 1.16.0 or newer; 1, 2 or 4 cards with 32 GB (tested on one, two and four Radeon AI PRO R9700)
Format	weights int4 with a zero point, one f16 scale per 128 weights (all linear layers of the trunk, GPTQ); activations int8, one scale per token and 128 values; at decode an f16 copy of the int4 layers; lm_head int8, one bf16 scale per 128 weights; the assistant head's lm_head also int8
Speculation	Google's Gemma 4 assistant head, bf16, 3 draft tokens
Inputs	text · image · video
Context	262,144 tokens
Version	2.0 (2026-11-04). Version 1.0 (2026-10-13, w4a8) is still available at revision v1.0.

Speed

Radeon AI PRO R9700 at 215 W, radiance 1.16.0, fp8 KV, 3 draft tokens, thinking off. BetterBench 0.6.1, full run, temperature 0.7. Tokens per second (median):

	1 card	2 cards	4 cards
decode, one stream	78.4	132.3	202.0
prefill at 24k tokens	1,463	2,657	4,318
4 streams, total	130	349	516

Quality

Distance to the bf16 original on the 80 MT-Bench questions, both turns. The text is the GPT-4 answers from lmsys/mt_bench_human_judgments, with the model's chat template and thinking off. The reference is the bf16 original in HF transformers, with fp32 arithmetic and the full vocabulary. radiance scores this file with --kld-ref, on the answer tokens only (42,283 tokens).

	mean KL	99th percentile KL	top token the same	perplexity ratio
this file	0.0251	0.331	96.1%	×0.993
keptn/gemma4-31b-a4 version 1.0 (w4a8)	0.0517	0.721	94.6%	×1.056
keptn/gemma4-31b-i8g	0.00236	0.0311	98.8%	×1.004
keptn/gemma4-31b-w5	0.00803	0.107	97.8%	×1.010

Quick start

Start the model on two cards:

hf download keptn/gemma4-31b-a4 gemma4-31b-a4.rad --local-dir ~/models
docker pull stilldeadcode/radiance:1.16.0
docker run -d --init --name gemma4 --network host --group-add video --group-add render \
    --device /dev/kfd --device /dev/dri --security-opt seccomp=unconfined --ulimit memlock=-1:-1 \
    -v ~/models:/models:ro -e HSA_ENABLE_IPC_MODE_LEGACY=1 -e ROCR_VISIBLE_DEVICES=0,1 \
    stilldeadcode/radiance:1.16.0 --model /models/gemma4-31b-a4.rad --host 0.0.0.0 --port 8000 \
    --tp 2 --p2p on --kv-cache-dtype fp8 --max-model-len 65536 --max-num-seqs 8 \
    --max-num-batched-tokens 2048 --num-speculative-tokens 3 --vision-tower-placement ram \
    --temp 1.0 --top-p 0.95 --top-k 64

	change
one card	ROCR_VISIBLE_DEVICES=0, drop --tp 2 --p2p on, --max-num-seqs 2
four cards	ROCR_VISIBLE_DEVICES=0,1,2,3, --tp 4
full context	--max-model-len 262144 --max-num-seqs 2
thinking	per request: "chat_template_kwargs": {"enable_thinking": true}

ROCR_VISIBLE_DEVICES selects the cards. Use amd-smi list to find the card numbers. Three cards do not work. Gemma has 32 attention heads, and they do not split into three. The sampling values are Google's: temperature 1.0, top_p 0.95 and top_k 64.

Memory and context

KV pool at --max-model-len 262144, fp8 KV, radiance 1.16.0, from radiance's startup report. Tokens of context in total, shared by all sequences:

	2 sequences	4 sequences	8 sequences
1 card	298,848	277,648	234,672
2 cards	524,304	1,048,624	1,017,136
4 cards	524,352	1,048,624	2,097,248

On one card each sequence also keeps its own sliding-window state. So more sequences leave less for the shared pool.

Images, audio and video

The same inputs go to radiance and to the bf16 original in HF transformers. Greedy, thinking off, 2 cards, radiance 1.16.0. There is one measurement for each vision or audio encoder, made with gemma4-31b-a4.rad.

input	cases	same answer	mean KL	top token the same
images (DocVQA and AI2D, fixed selection)	20	16/20	0.268	66/70
video (MVBench, 5 clips, 4 frames each)	5	4/5	0.0556	31/32

How it was made

1. Started from the bf16 checkpoint google/gemma-4-31B-it (revision 842da379).
2. Added Google's assistant head google/gemma-4-31B-it-assistant (revision 627c5ec1).
3. Measured the GPTQ statistics with the bf16 model on the calibration set named below (keptn/quant-calibration-v1, revision 3969b2f9). Rows were picked by category with a fixed seed, and each row was cut at 4,096 tokens. No SmoothQuant.
4. Quantised every linear layer of the trunk with GPTQ to int4: a u8 zero point and one f16 scale per 128 weights. q, k and v share one GPTQ pass.
5. Rounded both lm_heads to int8, one bf16 scale per 128 weights. The other layers of the assistant head and the vision tower stay bf16.

rad-convert gemma-4-31B-it --draft-model gemma-4-31B-it-assistant \
    --recipe gemma4-31b-a4.recipe --set quantization_config.quant_method=compressed-tensors \
    -o gemma4-31b-a4.rad

GPTQ needs statistics from the calibration set (calib= in the recipe).

Calibration: 662 rows and 708,764 tokens of keptn/quant-calibration-v1, rendered with the model's chat template.

Model card with: radiance-model-cards-framework 0.3 (codeberg.org/keptn42/radiance-model-cards-framework).

Limits

• radiance runs on AMD RDNA4 cards.
• The file needs radiance 1.16.0 or newer.
• Version 1.0 (gemma4-31b-w4a8.rad) needs radiance 1.5.0 or newer. Download it with hf download keptn/gemma4-31b-a4 gemma4-31b-w4a8.rad --revision v1.0.

Credits

• Gemma 4 support in radiance: based on the initial work of sfjes (sfjes@vmdaemon.org) in PR #75.
• radiance: StillDeadcode (codeberg.org/StillDeadcode/radiance). The int4 kernels and GPTQ come from radiance and its libquant.
• Base model, assistant head and vision tower: Google.
• Test and calibration sets: MT-Bench, DocVQA, AI2D, keptn/quant-calibration-v1.

License

Apache 2.0, the license of the base model. See LICENSE.