keptn/gemma4-31b-i8g

Gemma 4 31B-it · int8 · for radiance

google/gemma-4-31B-it in one 33.4 GiB file for the radiance inference engine on AMD RDNA4. All linear layers of the trunk are int8, rounded to nearest.

File	gemma4-31b-i8g.rad, 33.37 GiB
Needs	radiance 1.13.0 or newer; 2 or 4 cards with 32 GB (tested on two and four Radeon AI PRO R9700). It does not fit one card
Format	weights int8, one bf16 scale per 128 weights (all linear layers of the trunk, round to nearest); activations int8, one scale per token and 128 values; lm_head int8, one bf16 scale per 128 weights; the assistant head's lm_head also int8
Speculation	Google's Gemma 4 assistant head, bf16, 3 draft tokens
Inputs	text · image · video
Context	262,144 tokens
Version	1.0 (2026-10-24)

Speed

Radeon AI PRO R9700 at 215 W, radiance main 248ecf5, fp8 KV, 3 draft tokens, thinking off. BetterBench 0.6.1, full run, temperature 0.7. Tokens per second (median):

	1 card	2 cards	4 cards
decode, one stream	does not fit	89.6	142.6
prefill at 24k tokens	does not fit	2,534	3,451
4 streams, total	does not fit	264	404

Quality

Distance to the bf16 original on the 80 MT-Bench questions, both turns. The text is the GPT-4 answers from lmsys/mt_bench_human_judgments, with the model's chat template and thinking off. The reference is the bf16 original in HF transformers, with fp32 arithmetic and the full vocabulary. radiance scores this file with --kld-ref, on the answer tokens only (42,283 tokens).

	mean KL	99th percentile KL	top token the same	perplexity ratio
this file	0.00236	0.0311	98.8%	×1.004
keptn/gemma4-31b-w4a8	0.0517	0.721	94.6%	×1.056
keptn/gemma4-31b-w5	0.00803	0.107	97.8%	×1.010

Quick start

Start the model on two cards:

hf download keptn/gemma4-31b-i8g gemma4-31b-i8g.rad --local-dir ~/models
docker pull stilldeadcode/radiance:1.13.0
docker run -d --init --name gemma4-i8g --network host --group-add video --group-add render \
    --device /dev/kfd --device /dev/dri --security-opt seccomp=unconfined --ulimit memlock=-1:-1 \
    -v ~/models:/models:ro -e HSA_ENABLE_IPC_MODE_LEGACY=1 -e ROCR_VISIBLE_DEVICES=0,1 \
    stilldeadcode/radiance:1.13.0 --model /models/gemma4-31b-i8g.rad --host 0.0.0.0 --port 8000 \
    --tp 2 --kv-cache-dtype fp8 --max-model-len 65536 --max-num-seqs 8 \
    --max-num-batched-tokens 2048 --num-speculative-tokens 3 --vision-tower-placement ram \
    --temp 1.0 --top-p 0.95 --top-k 64

	change
four cards	ROCR_VISIBLE_DEVICES=0,1,2,3, --tp 4
full context	--max-model-len 262144 --max-num-seqs 2
thinking	per request: "chat_template_kwargs": {"enable_thinking": true}

ROCR_VISIBLE_DEVICES selects the cards. Use amd-smi list to find the card numbers. Three cards do not work. Gemma has 32 attention heads, and they do not split into three. The sampling values are Google's: temperature 1.0, top_p 0.95 and top_k 64.

Memory and context

KV pool at --max-model-len 262144, fp8 KV, radiance main 248ecf5, from radiance's startup report. Tokens of context in total, shared by all sequences:

	2 sequences	4 sequences	8 sequences
1 card	does not fit	does not fit	does not fit
2 cards	524,304	765,328	721,968
4 cards	524,352	1,048,624	2,097,248

Images, audio and video

The same inputs go to radiance and to the bf16 original in HF transformers. Greedy, thinking off, 2 cards, radiance w5pr-3a701182-b115. There is one measurement for each vision or audio encoder, made with gemma4-31b-w5.rad.

input	cases	same answer	mean KL	top token the same
images (DocVQA and AI2D, fixed selection)	20	16/20	0.471	66/70
video (MVBench, 5 clips, 4 frames each)	5	4/5	0.0272	31/32

How it was made

1. Started from the bf16 checkpoint google/gemma-4-31B-it (revision 842da379).
2. Added Google's assistant head google/gemma-4-31B-it-assistant (revision 627c5ec1).
3. Rounded every linear layer of the trunk to int8, one bf16 scale per 128 weights. No calibration data, no smoothing, no GPTQ.
4. Rounded both lm_heads to int8 the same way. The other layers of the assistant head and the vision tower stay bf16.

rad-convert gemma-4-31B-it --draft-model gemma-4-31B-it-assistant \
    --recipe gemma4-31b-i8g.recipe --set quantization_config.quant_method=compressed-tensors \
    -o gemma4-31b-i8g.rad

A rebuild with rad-convert gives the same weights. Only the converter's build date in the file header differs (one byte), so the sha256 differs.

Calibration: no calibration data.

Model card with: radiance-model-cards-framework 0.3 (codeberg.org/keptn42/radiance-model-cards-framework).

Limits

• radiance runs on AMD RDNA4 cards.
• The file needs radiance 1.13.0 or newer. radiance 1.12.0 does not contain the i8a8 format.
• One card is not possible. The weights need 33.4 GiB.
• Images on two or four cards need radiance 1.13.0 or newer (PR #238).
• The compose file is in PR #368, not in a release yet. Until then, use the docker run command in Quick start.

Credits

• Gemma 4 support in radiance: based on the initial work of sfjes (sfjes@vmdaemon.org) in PR #75.
• radiance: StillDeadcode (codeberg.org/StillDeadcode/radiance). The i8a8 format comes from radiance's Gemma 3 support.
• Base model, assistant head and vision tower: Google.
• Test sets: MT-Bench, DocVQA, AI2D.

License

Apache 2.0, the license of the base model. See LICENSE.