Instructions to use kingjones777/Arete-27B-ROCmFP4-STRIX-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use kingjones777/Arete-27B-ROCmFP4-STRIX-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf kingjones777/Arete-27B-ROCmFP4-STRIX-GGUF:Q4_0_ROCMFP
# Run inference directly in the terminal:
llama cli -hf kingjones777/Arete-27B-ROCmFP4-STRIX-GGUF:Q4_0_ROCMFP

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf kingjones777/Arete-27B-ROCmFP4-STRIX-GGUF:Q4_0_ROCMFP
# Run inference directly in the terminal:
llama cli -hf kingjones777/Arete-27B-ROCmFP4-STRIX-GGUF:Q4_0_ROCMFP

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf kingjones777/Arete-27B-ROCmFP4-STRIX-GGUF:Q4_0_ROCMFP
# Run inference directly in the terminal:
./build/bin/llama cli -hf kingjones777/Arete-27B-ROCmFP4-STRIX-GGUF:Q4_0_ROCMFP

Use Docker

docker model run hf.co/kingjones777/Arete-27B-ROCmFP4-STRIX-GGUF:Q4_0_ROCMFP

Llama new
LM Studio
Jan

vLLM

How to use kingjones777/Arete-27B-ROCmFP4-STRIX-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "kingjones777/Arete-27B-ROCmFP4-STRIX-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kingjones777/Arete-27B-ROCmFP4-STRIX-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/kingjones777/Arete-27B-ROCmFP4-STRIX-GGUF:Q4_0_ROCMFP

Ollama
How to use kingjones777/Arete-27B-ROCmFP4-STRIX-GGUF with Ollama:
```
ollama run hf.co/kingjones777/Arete-27B-ROCmFP4-STRIX-GGUF:Q4_0_ROCMFP
```
Unsloth Desktop

How to use kingjones777/Arete-27B-ROCmFP4-STRIX-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf kingjones777/Arete-27B-ROCmFP4-STRIX-GGUF:Q4_0_ROCMFP

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "kingjones777/Arete-27B-ROCmFP4-STRIX-GGUF:Q4_0_ROCMFP"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use kingjones777/Arete-27B-ROCmFP4-STRIX-GGUF with Docker Model Runner:
```
docker model run hf.co/kingjones777/Arete-27B-ROCmFP4-STRIX-GGUF:Q4_0_ROCMFP
```

Lemonade

How to use kingjones777/Arete-27B-ROCmFP4-STRIX-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull kingjones777/Arete-27B-ROCmFP4-STRIX-GGUF:Q4_0_ROCMFP

Run and chat with the model

lemonade run user.Arete-27B-ROCmFP4-STRIX-GGUF-Q4_0_ROCMFP

List all available models

lemonade list

Hermes Agent

How to use kingjones777/Arete-27B-ROCmFP4-STRIX-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf kingjones777/Arete-27B-ROCmFP4-STRIX-GGUF:Q4_0_ROCMFP

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default kingjones777/Arete-27B-ROCmFP4-STRIX-GGUF:Q4_0_ROCMFP

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use kingjones777/Arete-27B-ROCmFP4-STRIX-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf kingjones777/Arete-27B-ROCmFP4-STRIX-GGUF:Q4_0_ROCMFP

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "kingjones777/Arete-27B-ROCmFP4-STRIX-GGUF:Q4_0_ROCMFP" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Arete-27B — ROCmFP4 for AMD Strix Halo (gfx1151)

A ROCmFP4 STRIX quant of NousResearch/Arete-27B (Apache-2.0; a fine-tune of Qwen3.8-27B). Importance-matrix weighted; output head and token embedding at Q6_K. Two Super ROC 4-bit prefill route files, rounded from this model's own BF16 weights, ship next to the GGUF.

This fine-tune has 64 layers and no in-file nextn / MTP tensors. Serve it with the base model's DFlash2 drafter.

Files

File	Bytes	sha256	Role
`Arete-27B-Q4_0_ROCMFP4_STRIX-imatrix-headQ6K.gguf`	15127583872	`4e674e163915d94f782c50b981e2ffe5d877d91508f5f91d988fd499c96a1f62`	model, recipe STRIX (ggml type 105), 4.50 BPW
`Arete-27B-Q4_0_ROCMFP4_STRIX-imatrix-headQ6K.route.a4w`	8638237208	`328ffd989206565c0693860a31814a037b9d69ed5bead7ee96fe58500fa29f90`	4-bit prefill route (`ffn_gate,ffn_up,attn_gate,attn_q,attn_qkv`), clip 3, fine 64
`Arete-27B-Q4_0_ROCMFP4_STRIX-imatrix-headQ6K.down-rot256.a4r`	2878866208	`ec3964ce24fc32db46d2cedab1474089a810e2861daf701c30b01adeaa32f9e1`	rotated `ffn_down` route, block 256, clip 3, fine 64

The route files belong to this GGUF only. The engine checks layer count and width, so using them with any other 27B file still loads and gives wrong answers.

Drafter (not in this repository): Qwen3.8-27B-DFlash2-Q4_K_M.gguf from z-lab/Qwen3.8-27B-DFlash2-GGUF.

How it was built

Source: NousResearch/Arete-27B BF16 safetensors, converted to a single BF16 GGUF (qwen35, 26,895,998,464 parameters, 851 tensors, 64 blocks, vocab 248320).
Importance matrix: 531 chunks × 512 tokens of the same agentic calibration text used for the Qwen3.8-27B STRIX file, computed on a Q8_0 of Arete (497 imatrix entries).
Quantize: llama-quantize --imatrix <that file> --output-tensor-type q6_K --token-embedding-type q6_K <BF16> <out> Q4_0_ROCMFP4_STRIX 16. Census: 851 tensors; output.weight and token_embd.weight converted to Q6_K; model 51305.09 MiB (16.00 BPW) → 14416.30 MiB (4.50 BPW). Wall 191.9 s.
Prefill route: Super ROC int4_weights from the Arete BF16 GGUF. Column scales dumped on this STRIX file (8K-token prompt, context 4096). Rotated-down scales dumped on the held-out quality text with the five-tensor route on and SUPERROC_INT4_DOWN_ROT=256.

Requires a runtime that loads ROCmFP4 (ggml types 100–106). Stock llama.cpp rejects these tensor types.

Run it

SUPERROC_ATTN_FOLDED=1 SUPERROC_ATTN_VW=1 \
SUPERROC_INT4_WEIGHTS=/path/to/Arete-27B-Q4_0_ROCMFP4_STRIX-imatrix-headQ6K.route.a4w \
SUPERROC_INT4_TENSORS=ffn_gate,ffn_up,attn_gate,attn_q,attn_qkv \
SUPERROC_INT4_DOWN_WEIGHTS=/path/to/Arete-27B-Q4_0_ROCMFP4_STRIX-imatrix-headQ6K.down-rot256.a4r \
SUPERROC_INT4_DOWN_ROT=256 \
superroc serve llm \
  -m /path/to/Arete-27B-Q4_0_ROCMFP4_STRIX-imatrix-headQ6K.gguf \
  --served-model-name arete-27b \
  --context 131072 --think off -t 0 --presence-penalty 0 \
  --speculative dflash2 --dflash-model /path/to/Qwen3.8-27B-DFlash2-Q4_K_M.gguf \
  --prompt-lookup

ROCm 7.2.4, LD_LIBRARY_PATH pointing at that ROCm lib. Thinking off. Presence penalty 0.

Quality

Perplexity with Super ROC's qwen35_ppl on a held-out text, context 4096, 5 chunks, 10,235 scored tokens (second half of each chunk). Corpus sha256 167d6b72bcaf3e0139fa407889b7b1aeee18e304ec5b5ce6135efb669b9647f8. Lower is better.

File / route	PPL	vs Q8_0
Q8_0 reference (not shipped)	4.79726	1.000
STRIX, route off	4.87744	1.0167
STRIX, five-tensor route on	4.88189	1.0176
STRIX, five-tensor route + rotated down + folded attention VW	4.90393	1.0222

The 1.02 × Q8_0 bar is 4.89321. Route off and the five-tensor route sit under it. The full stack (the serve command above) is 1.0222 × Q8_0.

Chat smoke (temperature 0): capital of Australia → Canberra; a weather-tool request returned get_weather({"city":"Paris"}) with finish_reason=tool_calls. Second turn of a 1,513-token prefix reused 1,513 / 1,513 cached tokens; the follow-up turn reused 1,572 / 1,592.

Speed

AMD Ryzen AI Max+ 395 (Radeon 8060S, gfx1151), 128 GB unified, ROCm 7.2.4. Super ROC, the serve command above, temperature 0, cold prompt cache (cached_tokens=0 on every timed request), means of two runs.

Measurement	Result
Prompt processing, 1,566-token prompt	457.0 tok/s
Generation after that prompt (up to 256 tokens)	35.0 tok/s
Draft tokens accepted on that prompt	286 / 646
Prompt processing, 13,384-token prompt	840.05 tok/s
Generation after that prompt	26.1 tok/s
Draft tokens accepted on that prompt	230 / 638
Prompt processing, 26,821-token prompt	775.8 tok/s
Generation after that prompt	21.4 tok/s
Draft tokens accepted on that prompt	220 / 676
Code generation, 6 prompts, up to 256 tokens, mean of two runs (range)	56.87 tok/s (38.9–64.9)
Draft tokens accepted on those code prompts	2386 / 3646
Tool calls valid, `tool_choice` required / auto	8/8 / 7/8

The DFlash2 drafter is the base Qwen3.8-27B file. A fine-tune can accept fewer drafts than the base; the table is the measured acceptance on this file.

Architecture

Arete-27B is dense hybrid-attention qwen35: 64 layers, hidden 5120, FFN 17408, SSM inner 6144, vocab 248320. Native context in the GGUF is 262144; the serve command above uses 131072.

License

Base model NousResearch/Arete-27B, Apache-2.0. This repository is the quantised weights, the two route files, and the measurements above.

Downloads last month: -

GGUF

Model size

27B params

Architecture

qwen35

Hardware compatibility

4-bit

Model tree for kingjones777/Arete-27B-ROCmFP4-STRIX-GGUF

Base model

Qwen/Qwen3.8-27B

Finetuned

NousResearch/Arete-27B

Quantized

(41)

this model