Instructions to use maczzinatui/Grug-12B-ROCmFP4_FAST-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use maczzinatui/Grug-12B-ROCmFP4_FAST-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="maczzinatui/Grug-12B-ROCmFP4_FAST-GGUF",
	filename="Grug-12B-ROCmFP4_FAST.gguf",
)

llm.create_chat_completion(
	messages = "No input example has been defined for this model task."
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use maczzinatui/Grug-12B-ROCmFP4_FAST-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf maczzinatui/Grug-12B-ROCmFP4_FAST-GGUF
# Run inference directly in the terminal:
llama cli -hf maczzinatui/Grug-12B-ROCmFP4_FAST-GGUF

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf maczzinatui/Grug-12B-ROCmFP4_FAST-GGUF
# Run inference directly in the terminal:
llama cli -hf maczzinatui/Grug-12B-ROCmFP4_FAST-GGUF

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf maczzinatui/Grug-12B-ROCmFP4_FAST-GGUF
# Run inference directly in the terminal:
./llama-cli -hf maczzinatui/Grug-12B-ROCmFP4_FAST-GGUF

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf maczzinatui/Grug-12B-ROCmFP4_FAST-GGUF
# Run inference directly in the terminal:
./build/bin/llama-cli -hf maczzinatui/Grug-12B-ROCmFP4_FAST-GGUF

Use Docker

docker model run hf.co/maczzinatui/Grug-12B-ROCmFP4_FAST-GGUF

LM Studio
Jan
Ollama
How to use maczzinatui/Grug-12B-ROCmFP4_FAST-GGUF with Ollama:
```
ollama run hf.co/maczzinatui/Grug-12B-ROCmFP4_FAST-GGUF
```

Unsloth Studio

How to use maczzinatui/Grug-12B-ROCmFP4_FAST-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for maczzinatui/Grug-12B-ROCmFP4_FAST-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for maczzinatui/Grug-12B-ROCmFP4_FAST-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for maczzinatui/Grug-12B-ROCmFP4_FAST-GGUF to start chatting

How to use maczzinatui/Grug-12B-ROCmFP4_FAST-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf maczzinatui/Grug-12B-ROCmFP4_FAST-GGUF

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "maczzinatui/Grug-12B-ROCmFP4_FAST-GGUF"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use maczzinatui/Grug-12B-ROCmFP4_FAST-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf maczzinatui/Grug-12B-ROCmFP4_FAST-GGUF

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default maczzinatui/Grug-12B-ROCmFP4_FAST-GGUF

Run Hermes

hermes

Atomic Chat new

OpenClaw new

How to use maczzinatui/Grug-12B-ROCmFP4_FAST-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf maczzinatui/Grug-12B-ROCmFP4_FAST-GGUF

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "maczzinatui/Grug-12B-ROCmFP4_FAST-GGUF" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use maczzinatui/Grug-12B-ROCmFP4_FAST-GGUF with Docker Model Runner:
```
docker model run hf.co/maczzinatui/Grug-12B-ROCmFP4_FAST-GGUF
```

Lemonade

How to use maczzinatui/Grug-12B-ROCmFP4_FAST-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull maczzinatui/Grug-12B-ROCmFP4_FAST-GGUF

Run and chat with the model

lemonade run user.Grug-12B-ROCmFP4_FAST-GGUF-{{QUANT_TAG}}

List all available models

lemonade list

Grug-12B ROCmFP4_FAST — GGUF

ROCmFP4_FAST quant of kai-os/Grug-12B — a compact-reasoning fine-tune of google/gemma-4-12B-it, produced via charlie12345/ROCmFPX. Benchmarked on an RDNA4 RX 9060 XT (16 GB). Mean AEON score: 0.558 (120 scored / 150 cases).

Grug-12B is a specialized reasoning model trained to compress chain-of-thought while maintaining reasoning quality. At 6.0 GB, it leaves >7 GB VRAM headroom even at 152k context — by far the most comfortable quant on 16 GB cards.

File

File	Size	Quant	BPW
`Grug-12B-ROCmFP4_FAST.gguf`	6.0 GB	Q4_0_ROCMFP4_FAST	~4.25 bpw

NOT a stock llama.cpp quant

ROCmFP4_FAST is a custom weight format from charlie12345/ROCmFPX. Stock llama.cpp will exit with unknown quantization at load time. The system_fingerprint of a correctly-served ROCmFPX GGUF is b1-11d76c2.

Scope of these benchmarks — read this first

These numbers are a light baseline, not a thorough ROCmFPX evaluation. The mesh's bench framework is built for production agent workload regression-detection on the local stack, not for the kind of multi-axis sweep that upstream quant maintainers typically publish. Specifically:

Harness scope is bounded. The numbers below come from the mesh's Aeon-Bench-Pod (self-reported mode, 150-case full suite, text-only). That's a regression suite, not a quality benchmark.
Sample sizes are small. 150 cases on a single GPU, single rep. None are powered for statistical significance.
No perplexity / wikitext / MMLU / GSM8K. The mesh's stack isn't a quality benchmark — those are upstream's territory.
Single GPU class (RDNA4 16 GB). All measurements are on an AMD RX 9060 XT 16 GB (gfx1200, ROCm 7.x). No Blackwell, no CDNA, no multi-GPU. Cross-hardware generalization is NOT implied.
No human eval. "0.558 mean on the AEON suite" is not a quality verdict on this specific quant.
30 prose cases unscored — no frontier judge endpoint configured, so the prose category is excluded from the mean.

What this IS good for: a quick signal that the quant (a) loads, (b) runs at good throughput on RDNA4, (c) has ample VRAM headroom, (d) produces coherent output across math, instruction, reasoning, and coding. What this is NOT good for: claiming "this is the best quant of this model," reproducing academic benchmark results, or substituting for upstream's validation work.

For a rigorous view, see kai-os/Grug-12B (parent model) and charlie12345/ROCmFPX (quantizer). Raw bench reports are attached as BENCH-*.md files in this repo.

What we measured

AEON v3 — 150 cases (120 scored)

Benchmarked on AMD RDNA4 RX 9060 XT 16 GB, full suite.

Category	Mean	N
Overall	0.558	120
reasoning	0.767	30
coding	0.800	30
instruction	0.433	30
math	0.233	30

Difficulty	Mean	N
easy	1.000	8
medium	0.833	12
hard	0.650	20
expert	0.531	32
frontier	0.396	48

Profile: Grug excels at reasoning (0.767) and coding (0.800), consistent with its compact-reasoning training. Weak on math (0.233) — the compression training may penalize arithmetic precision. Strong difficulty scaling: holds up well at frontier (0.396) compared to other small quants.

Quick start

# Build charlie12345/ROCmFPX
git clone https://github.com/charlie12345/ROCmFPX
cd ROCmFPX
mkdir build && cd build
cmake .. -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1200
make -j$(nproc)

# Serve
llama-server \
  -m Grug-12B-ROCmFP4_FAST.gguf \
  --host 0.0.0.0 --port 8081 \
  -ngl 99 -c 32768 \
  -ctk q4_0 -ctv q4_0 \
  -fa on --cache-ram 0 \
  -np 1 --batch-size 512 --ubatch-size 128 \
  --jinja --metrics -rea off

Do NOT set GGML_HIP_ENABLE_UNIFIED_MEMORY=1 on discrete AMD cards — it moves the entire model to system RAM (30× regression).

Reproduce the quant

# From kai-os/Grug-12B safetensors, converted to BF16 GGUF first:
python3 convert_hf_to_gguf.py /path/to/kai-os/Grug-12B --outfile Grug-12B-BF16.gguf

# Then quantize:
llama-quantize --allow-requantize Grug-12B-BF16.gguf \
  Grug-12B-ROCmFP4_FAST.gguf Q4_0_ROCMFP4_FAST

Files in this repo

File	Description
`Grug-12B-ROCmFP4_FAST.gguf`	The quantized model (LFS-tracked, 6.0 GB)
`README.md`	This model card
`BENCH-aeon-full-suite.md`	AEON bench results (150 cases, 120 scored)

What's NOT in this repo (caveats)

Stock llama.cpp will not load this file. ROCmFP4_FAST is a custom weight format from charlie12345/ROCmFPX.
Grug is NOT a general-purpose instruct model. It's a specialized compact-reasoning model. Use for math, code reasoning, and agentic pipelines where reasoning-token compression matters. Do NOT use for instruction-following tasks where clean output is expected — the reasoning bloat will hurt quality.
No CUDA / non-AMD GPU bench. All measurements are RDNA4 (gfx1200).
No quality benchmark (perplexity, MMLU, GSM8K). Compact-reasoning fine-tunes may not reflect academic benchmarks.
No MTP / speculative-decode bench. Not applicable on this architecture.

Provenance

Source model: kai-os/Grug-12B (license: other — verify commercial use terms)
Base model: google/gemma-4-12B-it (Gemma 4 license)
Conversion: convert_hf_to_gguf.py (BF16 GGUF from 7 sharded safetensors, 23.8 GB → ROCmFP4_FAST 6.0 GB)
Quantizer: charlie12345/ROCmFPX commit 5b39566, preset Q4_0_ROCMFP4_FAST
Build hardware: Node B — RDNA4 RX 9060 XT 16 GB (gfx1201), ROCm 7.2.3, NixOS
Bench harness: Aeon-Bench-Pod v1 (self-reported mode)

License

This quant is derived from kai-os/Grug-12B (license: other — verify commercial use is permitted before deployment), which is a fine-tune of google/gemma-4-12B-it (Gemma 4 license). The ROCmFP4 quant format is provided by charlie12345/ROCmFPX (MIT).