Instructions to use vmlinux/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use vmlinux/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="vmlinux/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF",
	filename="Qwen3.5-122B-A10B-Heretic-ROCmFP4-MTP.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use vmlinux/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf vmlinux/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF
# Run inference directly in the terminal:
llama cli -hf vmlinux/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf vmlinux/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF
# Run inference directly in the terminal:
llama cli -hf vmlinux/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf vmlinux/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF
# Run inference directly in the terminal:
./llama-cli -hf vmlinux/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf vmlinux/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF
# Run inference directly in the terminal:
./build/bin/llama-cli -hf vmlinux/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF

Use Docker

docker model run hf.co/vmlinux/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF

LM Studio
Jan

vLLM

How to use vmlinux/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "vmlinux/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "vmlinux/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/vmlinux/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF

Ollama
How to use vmlinux/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF with Ollama:
```
ollama run hf.co/vmlinux/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF
```

Unsloth Studio

How to use vmlinux/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for vmlinux/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for vmlinux/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for vmlinux/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF to start chatting

How to use vmlinux/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf vmlinux/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "vmlinux/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use vmlinux/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf vmlinux/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default vmlinux/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF

Run Hermes

hermes

Atomic Chat new

OpenClaw new

How to use vmlinux/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf vmlinux/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "vmlinux/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use vmlinux/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF with Docker Model Runner:
```
docker model run hf.co/vmlinux/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF
```

Lemonade

How to use vmlinux/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull vmlinux/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF

Run and chat with the model

lemonade run user.Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF-{{QUANT_TAG}}

List all available models

lemonade list

Qwen3.5 122B-A10B Heretic · ROCmFP4 iMatrix

A compact, importance-calibrated ROCmFP4 build for high-memory AMD systems

122B total · 10B active · 60.70 GiB · 28.45 tok/s native · 33.68 tok/s MTP · BF16 KLD 0.100716 · PP 353.3 W/MTP 328.6 t/s

This GGUF uses custom ROCmFP4 tensor types. It requires ROCmFPX or a runtime with equivalent support; stock llama.cpp, Ollama, LM Studio, and similar stock runtimes cannot load it.

Downloads

File	Purpose	Size	SHA-256
`Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix.gguf`	Main text model	65,184,265,248 bytes	`f563fba693c65a05431d4beecd11736edf7e54208902cae70d7b725057a7556e`
`Qwen3.5-122B-A10B-Heretic-ROCmFP4-MTP.gguf`	Optional external MTP companion	2,294,290,880 bytes	`5554538448efb668099be898ce7c8da5920e86b321f45e8ac1f7dca3518cc0a8`
`Qwen3.5-122B-A10B-Heretic-code-technical.imatrix`	Calibration matrix	358,906,432 bytes	`3a2a0e0bea5fe380a27cc938ad02e727dcbd840744ab3ddb3a8e8402a67deb05`
`chat_template.jinja`	Pinned Qwen3.5 chat template	7,756 bytes	`a4aee8afcf2e0711942cf848899be66016f8d14a889ff9ede07bca099c28f715`

Browse every repository file →

What it is

This is an iMatrix-calibrated quantization of the full-weight Qwen3.5 122B-A10B Heretic checkpoint. It was built with ROCmFPX's compact Q4_0_ROCMFP4_STRIX_LEAN recipe, but the hardware-oriented preset name has been left out of the public filename.

	Result
BF16 mean KLD	0.100716 ± 0.003407
Two-sigma KLD upper bound	0.107530
Improvement from iMatrix	14.01% lower KLD
Greedy decode	28.448 tok/s
Greedy decode with MTP	33.677 tok/s
4,277-token prefill	350.248 tok/s

The main model is complete and runs independently. The 23-tensor, one-layer MTP file is an optional external companion; no MTP layer is embedded in the main GGUF. Runtimes call this role a draft model, but Draft is omitted from the public filename because it does not mean unfinished.

Quality

The calibrated model and its uncalibrated control were replayed against the same saved distributions from the exact BF16 parent on ROCm0.

Build	Mean KLD	PPL ratio	Same top	RMS Δp
iMatrix	0.100716 ± 0.003407	1.080217 ± 0.010198	87.629% ± 0.622%	10.881%
Uncalibrated	0.117119 ± 0.004183	1.109251 ± 0.011299	86.774% ± 0.640%	12.077%

iMatrix calibration reduced mean KLD by 14.01%, reduced the PPL ratio by 0.029034, and improved same-top agreement by 0.855 percentage points. The two-sigma KLD upper bound is 0.107530, clearing the selected 0.126 performance-oriented target.

The paired replay covers 11 complete 512-token chunks and 2,805 evaluated next-token distributions. These were recovered at an exact probability-block boundary from an interrupted 32-chunk BF16 run. This is useful preliminary evidence, not a completed 32-chunk acceptance gate.

Performance

Measured on a 128 GB AMD Strix Halo system with Vulkan/RADV, 131,072 context, parallel 1, batch/ubatch 2048/1024, Q8_0 KV cache, flash attention, and MTP off. Values are medians from repeated runs.

Workload	Repeats	Decode	Prompt processing
Greedy, 256 generated tokens	5	28.448 tok/s	71.929 tok/s
Sampled, 256 generated tokens	3	28.418 tok/s	71.294 tok/s
4,277-token prefill + 128 generated	3	28.041 tok/s	350.248 tok/s

Relative to the otherwise matched uncalibrated build, decode improved by 2.43%, 5.23%, and 2.84% respectively. Long-prompt prefill changed by -1.03%. These are single-system measurements, not general performance guarantees.

Optional MTP acceleration

The external MTP companion was tested against this exact iMatrix trunk in a matched MTP-off/on A/B. The model, runtime, context, cache, batch geometry, prompts, and companion identity were held constant; only MTP execution changed.

Workload	Native decode	MTP decode	Decode gain	Acceptance	End-to-end gain
Greedy, 256 tokens	28.328 tok/s	33.677 tok/s	+18.89%	85.03%	+16.44%
Sampled, 256 tokens	28.035 tok/s	30.953 tok/s	+10.41%	80.00%	+8.76%
4,277-token prefill + 128	27.585 tok/s	33.752 tok/s	+22.36%	90.73%	-0.45%

MTP reduced prompt-processing throughput by about 7% because the companion must catch up on the prompt. It therefore helps most when generation is long enough to repay that fixed cost. The exact identity probe was unchanged, and both the MTP-off and MTP-on lanes passed technical prose, executable code semantics, strict JSON, tool-calling, balanced-analysis, and defensive-safety smokes.

Quantization

The model was quantized once from the validated BF16 GGUF using a 339-chunk code/technical importance matrix and the Q4_0_ROCMFP4_STRIX_LEAN preset.

Tensor type	Count
`Q4_0_ROCMFP4_FAST`	457
`Q4_0_ROCMFP4`	60
`F32`	361
`Q5_K`	1

In this preset, attention K/V tensors retain dual-scale ROCmFP4 protection, most transformer weights use the compact FAST layout, and token embeddings/output use Q5_K. The artifact contains 879 tensors across 48 blocks and no embedded MTP layer.

Run

llama-server \
  --model Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix.gguf \
  --host 127.0.0.1 --port 8080 \
  -dev Vulkan0 --n-gpu-layers 999 \
  --ctx-size 131072 --parallel 1 \
  --flash-attn on --batch-size 2048 --ubatch-size 1024 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --jinja --reasoning-format deepseek

Adjust context and cache settings for your memory budget. The tested runtime was charlie12345/ROCmFPX commit a6a93765f7ce9779c13f9881164a65f7a9f31198, built in Release mode for gfx1151 with Vulkan and HIP enabled, plus a local Qwen3.5 MoE graph correction. Results with newer ROCmFPX revisions have not yet been established.

To enable the optional companion, append:

  --spec-type draft-mtp \
  --spec-draft-model Qwen3.5-122B-A10B-Heretic-ROCmFP4-MTP.gguf \
  --spec-draft-device Vulkan0 --spec-draft-ngl 999 \
  --spec-draft-type-k f16 --spec-draft-type-v f16 \
  --spec-draft-n-max 2 --spec-draft-p-min 0.6 \
  --spec-draft-p-split 0.10 --spec-draft-backend-sampling

Tested system


Platform	AMD Strix Halo, 128 GB unified memory
GPU target	Radeon 8060S / `gfx1151`
Backend	Vulkan / RADV
Kernel	Linux 6.17.0-1028-oem
Mesa	25.2.8

No vision projector is included; treat this release as text-only.

Lineage and credits

This repository exists because several people published substantial work openly. Their roles are kept distinct so downstream users can find the original sources.

Official base model: Qwen/Qwen3.5-122B-A10B, Apache 2.0.
Full-weight Heretic source: trohrbaugh/Qwen3.5-122B-A10B-heretic, accessed through the pinned CCSSNE mirror at fd6d6bc89e9be47577736c952dd4d1d7c830f09f.
MTP tensor source: OptimizeLLM/Qwen3.5-122B-A10B-heretic-MTP-NVFP4 at 07b7c2101c33f23a84fd3c618d00f17ea12269c3; its model card reports that the MTP tensors were restored from the official Qwen checkpoint.
Heretic software and method: p-e-w/heretic, created by Philipp Emanuel Weidmann. The source checkpoint reports Heretic v1.2.0.
Related full-weight Heretic reference: scrunter/Qwen3-VL-235B-A22B-Thinking-heretic. This related 235B model is not the tensor source for this release.
ROCmFP4 implementation, quantizer, and runtime: charlie12345/ROCmFPX.
Strix Halo deployment stack: hec-ovi/llama-vulkan-strix.
Release inspiration: plunderstruck/Qwen3.6-27B-MTP-ROCmFP4-GGUF.
Conversion, validation, benchmarking, and packaging: vmlinux.

Please preserve this lineage, the Apache 2.0 license, and a description of your changes when redistributing derivatives.

License

The distributed model derivative is provided under the upstream Apache License 2.0. The included LICENSE is copied from the pinned full-weight source. Runtime and tooling repositories retain their own licenses: Heretic software is AGPL-3.0, while ROCmFPX and the deployment tooling are separately licensed. No runtime source code is bundled in this model repository.

Downloads last month: -

GGUF

Model size

122B params

Architecture

qwen35moe

Hardware compatibility

We're not able to determine the quantization variants.

View all variants

Model tree for vmlinux/Qwen3.5-122B-A10B-Heretic-ROCmFP4-iMatrix-GGUF

Base model

CCSSNE/trohrbaugh-Qwen3.5-122B-A10B-heretic

Quantized

(3)

this model