Instructions to use kingjones777/Gemma-4-31B-it-ROCmFP4-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use kingjones777/Gemma-4-31B-it-ROCmFP4-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf kingjones777/Gemma-4-31B-it-ROCmFP4-GGUF:Q4_0_ROCMFP
# Run inference directly in the terminal:
llama cli -hf kingjones777/Gemma-4-31B-it-ROCmFP4-GGUF:Q4_0_ROCMFP

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf kingjones777/Gemma-4-31B-it-ROCmFP4-GGUF:Q4_0_ROCMFP
# Run inference directly in the terminal:
llama cli -hf kingjones777/Gemma-4-31B-it-ROCmFP4-GGUF:Q4_0_ROCMFP

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf kingjones777/Gemma-4-31B-it-ROCmFP4-GGUF:Q4_0_ROCMFP
# Run inference directly in the terminal:
./llama-cli -hf kingjones777/Gemma-4-31B-it-ROCmFP4-GGUF:Q4_0_ROCMFP

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf kingjones777/Gemma-4-31B-it-ROCmFP4-GGUF:Q4_0_ROCMFP
# Run inference directly in the terminal:
./build/bin/llama-cli -hf kingjones777/Gemma-4-31B-it-ROCmFP4-GGUF:Q4_0_ROCMFP

Use Docker

docker model run hf.co/kingjones777/Gemma-4-31B-it-ROCmFP4-GGUF:Q4_0_ROCMFP

LM Studio
Jan

vLLM

How to use kingjones777/Gemma-4-31B-it-ROCmFP4-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "kingjones777/Gemma-4-31B-it-ROCmFP4-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kingjones777/Gemma-4-31B-it-ROCmFP4-GGUF",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/kingjones777/Gemma-4-31B-it-ROCmFP4-GGUF:Q4_0_ROCMFP

Ollama
How to use kingjones777/Gemma-4-31B-it-ROCmFP4-GGUF with Ollama:
```
ollama run hf.co/kingjones777/Gemma-4-31B-it-ROCmFP4-GGUF:Q4_0_ROCMFP
```

Unsloth Studio

How to use kingjones777/Gemma-4-31B-it-ROCmFP4-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for kingjones777/Gemma-4-31B-it-ROCmFP4-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for kingjones777/Gemma-4-31B-it-ROCmFP4-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for kingjones777/Gemma-4-31B-it-ROCmFP4-GGUF to start chatting

How to use kingjones777/Gemma-4-31B-it-ROCmFP4-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf kingjones777/Gemma-4-31B-it-ROCmFP4-GGUF:Q4_0_ROCMFP

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "kingjones777/Gemma-4-31B-it-ROCmFP4-GGUF:Q4_0_ROCMFP"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use kingjones777/Gemma-4-31B-it-ROCmFP4-GGUF with Docker Model Runner:
```
docker model run hf.co/kingjones777/Gemma-4-31B-it-ROCmFP4-GGUF:Q4_0_ROCMFP
```

Lemonade

How to use kingjones777/Gemma-4-31B-it-ROCmFP4-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull kingjones777/Gemma-4-31B-it-ROCmFP4-GGUF:Q4_0_ROCMFP

Run and chat with the model

lemonade run user.Gemma-4-31B-it-ROCmFP4-GGUF-Q4_0_ROCMFP

List all available models

lemonade list

Hermes Agent

How to use kingjones777/Gemma-4-31B-it-ROCmFP4-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf kingjones777/Gemma-4-31B-it-ROCmFP4-GGUF:Q4_0_ROCMFP

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default kingjones777/Gemma-4-31B-it-ROCmFP4-GGUF:Q4_0_ROCMFP

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use kingjones777/Gemma-4-31B-it-ROCmFP4-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf kingjones777/Gemma-4-31B-it-ROCmFP4-GGUF:Q4_0_ROCMFP

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "kingjones777/Gemma-4-31B-it-ROCmFP4-GGUF:Q4_0_ROCMFP" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Gemma-4-31B-it — ROCmFP4 / ROCmFPX GGUF

First ROCmFP4/ROCmFPX quantization of google/gemma-4-31B-it. Four quants, built from the Hub BF16 GGUF (no re-conversion from safetensors), each individually verified — text and vision — on real hardware.

Built for AMD Strix Halo (gfx1151) — Ryzen AI MAX+ 395, 128 GB unified memory, ROCm 7.2.4 — using the ROCmFPX llama.cpp fork, which adds AMD-native FP4/FP8 tensor types that mainline llama.cpp does not have.

⚠️ These files need a ROCmFPX-capable llama.cpp build. They will not load in stock llama.cpp / Ollama / LM Studio — Q4_0_ROCMFP4_* and Q*_0_ROCMFPX* are not mainline types.

Variants — all four in this repo

file	ftype	size	BPW	`token_embd`	decode + MTP	decode alone	speedup
`gemma-4-31B-it-Q4_0_ROCMFP4_COHERENT.gguf`	102	16.44 GiB	4.60	q6_K	21.67 t/s	11.77	1.84×
`gemma-4-31B-it-Q6_0_ROCMFPX_AGENT.gguf`	114	26.63 GiB	7.45	q8_0	14.82 t/s	7.33	2.02×
`gemma-4-31B-it-Q8_0_ROCMFPX.gguf`	111	29.54 GiB	8.26	q8_0	14.86 t/s	6.71	2.21×
`gemma-4-31B-it-Q8_0_ROCMFPX_AGENT.gguf`	115	30.04 GiB	8.40	q8_0	15.34 t/s	6.57	2.33×

mmproj-BF16.gguf (vision projector) is included — required for image input.

Which to pick: the 4-bit (102) — 21.67 t/s with the drafter, 1.46× the fastest 8-bit at 55% of the size, with identical correctness and vision results. If you want an 8-bit, take the AGENT variant Q8_0_ROCMFPX_AGENT (115), not the plain one: with speculation on it is both faster (15.34 vs 14.86) and accepts more drafts (0.619 vs 0.603), because the AGENT recipe holds more attention-output tensors at higher precision — which is exactly what raises draft acceptance. (Without a drafter the ordering reverses and plain Q8 wins; the AGENT recipe only pays when you are actually speculating.)

Measured — not estimated

Hardware: AMD Ryzen AI MAX+ 395 (Strix Halo, gfx1151), 128 GB unified, ROCm 7.2.4. Load: -ngl 999 -c 4096 -fa on -fit off. Decode = 300 tokens, warm-up discarded, median of 3.

With the drafter (n-max 5), correctness re-checked under the shipping config:

quant	run 1 / 2 / 3	median	acceptance	correctness
102	21.69 / 21.68 / 21.66	21.68	0.735	`391` · Tokyo · `366` ✅
114	14.82 / 14.79 / 14.86	14.82	0.629	✅
111	14.80 / 14.86 / 14.97	14.86	0.603	✅
115	15.25 / 15.34 / 15.39	15.34	0.619	✅

Without the drafter (vision path, -fa off):

quant	run 1 / 2 / 3	median	spread	vision (4-colour quadrants)
102	11.81 / 11.81 / 11.81	11.81	1.000	✅ TL red · TR blue · BL yellow · BR green
114	7.33 / 7.33 / 7.33	7.33	1.000	✅ same
111	6.72 / 6.71 / 6.71	6.71	1.001	✅ same
115	6.59 / 6.57 / 6.57	6.57	1.003	✅ same

Spreads ≤1.003 with nothing else running — clean numbers, not contended ones. Speculative decoding is output-preserving (the target verifies every drafted token) and the correctness answers are identical with and without it.

Why you can trust the no-drafter numbers

This model is conventionally dense, so weights stream once per token and t/s × file_GB gives effective memory bandwidth. Against Strix Halo's ~256 GB/s peak, all four land at 81–83% — the hardware ceiling, so nothing is spilling to CPU or silently misconfigured. (With MTP the model verifies several tokens per weight read, which is exactly how it exceeds that ceiling.)

Vision was tested with a 512² four-colour quadrant image, asking which colour is where. A solid-colour square would pass trivially and proves nothing. Use -fa off for image input. ⚠️ Give vision prompts max_tokens: 1024; at 256 the model spends the budget thinking and returns finish_reason: length with empty content, which looks like a failure and is not.

⚡ Speculative decoding (MTP) — supported, and worth 1.84×

This model has an official drafter, google/gemma-4-31B-it-assistant. It is included in this repo as mtp-gemma-4-31B-it-Q8_0.gguf (491 MB) so the numbers above are reproducible.

Recommended text command — this is what produced 21.67 t/s:

llama-server -m gemma-4-31B-it-Q4_0_ROCMFP4_COHERENT.gguf \
  --spec-type draft-mtp --model-draft mtp-gemma-4-31B-it-Q8_0.gguf \
  --spec-draft-ngl 999 --spec-draft-n-max 5 \
  -ngl 999 -c 4096 -fa on -fit off

Measured tuning (median of 3 each, code prompt, idle box) — don't guess these:

setting	result
`n-max 5`	21.67 t/s ← best
`n-max 3`	21.19
`n-max 4`	20.81
`n-max 6`	20.65
`n-max 8`	12.83 — collapses
Q8 drafter	20.81 @ n4 ← use this
BF16 drafter	18.84 @ n4 — slower despite higher acceptance (0.704 vs 0.688); the bigger drafter costs more than it returns
`p_min` 0.5 / 0.0	no reliable gain — apparent +0.9% carries 7× the run-to-run spread

Draft acceptance at the recommended config is 0.649, mean accepted length 3.18. Acceptance is workload-dependent: code drafts accept far better than prose, so treat 21.67 as a code-workload figure and expect less on free-form text.

⛔ MTP and vision cannot be used together

Enabling the drafter and passing an image crashes the server:

process: missing MTP boundary for seq_id=2 pos=132
srv update_slots: failed to process speculative batch
server-context.cpp:3202: fatal error

Image embeddings are injected outside the normal token path and the speculative batch loses its boundary (upstream llama.cpp PR #20277). For vision, drop the --spec-type flags and use -fa off — that path is fully verified (see below) and runs at 11.81 t/s.

Verification

Each artifact was checked individually:

Size vs --dry-run projection — constant ~15.3 MiB GGUF-header delta across all four
token_embd audited by exact tensor name (not substring — output.weight false-matches blk.N.attn_output.weight)
Loaded, 3/3 correctness, real-content vision test, decode median of 3 with spread reported

  514687104  mtp-gemma-4-31B-it-Q8_0.gguf   (drafter — needed for the headline speed)
17651002048  gemma-4-31B-it-Q4_0_ROCMFP4_COHERENT.gguf
28594301632  gemma-4-31B-it-Q6_0_ROCMFPX_AGENT.gguf
31720467136  gemma-4-31B-it-Q8_0_ROCMFPX.gguf
32252046016  gemma-4-31B-it-Q8_0_ROCMFPX_AGENT.gguf

⚠️ Note for anyone quantizing this model themselves

gemma-4-31B-it has tie_word_embeddings = true — there is no output.weight tensor. --output-tensor-type is a silent no-op; --token-embedding-type is the only flag that actually protects the head:

llama-quantize --output-tensor-type q6_K --token-embedding-type q6_K \
  gemma-4-31B-it-BF16.gguf out.gguf Q4_0_ROCMFP4_COHERENT 16

Verify by reading the finished file's token_embd.weight type — do not assume it worked because the flag was accepted.

Credits

Base model: google/gemma-4-31B-it — Gemma Terms of Use
Quantization stack: ROCmFPX

Downloads last month: 192

GGUF

Model size

31B params

Architecture

gemma4

Hardware compatibility

4-bit

8-bit

View +1 variant

Model tree for kingjones777/Gemma-4-31B-it-ROCmFP4-GGUF

Base model

google/gemma-4-31B

Finetuned

google/gemma-4-31B-it

Quantized

(301)

this model