Instructions to use jakeatx/Agnes-3.0-Flash-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use jakeatx/Agnes-3.0-Flash-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf jakeatx/Agnes-3.0-Flash-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf jakeatx/Agnes-3.0-Flash-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf jakeatx/Agnes-3.0-Flash-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf jakeatx/Agnes-3.0-Flash-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf jakeatx/Agnes-3.0-Flash-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf jakeatx/Agnes-3.0-Flash-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf jakeatx/Agnes-3.0-Flash-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf jakeatx/Agnes-3.0-Flash-GGUF:Q4_K_M

Use Docker

docker model run hf.co/jakeatx/Agnes-3.0-Flash-GGUF:Q4_K_M

LM Studio
Jan
Ollama
How to use jakeatx/Agnes-3.0-Flash-GGUF with Ollama:
```
ollama run hf.co/jakeatx/Agnes-3.0-Flash-GGUF:Q4_K_M
```
Unsloth Desktop

How to use jakeatx/Agnes-3.0-Flash-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf jakeatx/Agnes-3.0-Flash-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "jakeatx/Agnes-3.0-Flash-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use jakeatx/Agnes-3.0-Flash-GGUF with Docker Model Runner:
```
docker model run hf.co/jakeatx/Agnes-3.0-Flash-GGUF:Q4_K_M
```

Lemonade

How to use jakeatx/Agnes-3.0-Flash-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull jakeatx/Agnes-3.0-Flash-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Agnes-3.0-Flash-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use jakeatx/Agnes-3.0-Flash-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf jakeatx/Agnes-3.0-Flash-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default jakeatx/Agnes-3.0-Flash-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use jakeatx/Agnes-3.0-Flash-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf jakeatx/Agnes-3.0-Flash-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "jakeatx/Agnes-3.0-Flash-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Agnes 3.0 Flash GGUF

These GGUFs do not load on stock llama.cpp yet. Agnes runs a second, narrower SwiGLU in parallel with the main FFN on every layer, and upstream has no support for it, so a stock build stops with:
error loading model: done_getting_tensors: wrong number of tensors; expected 1188, got 972
(the 216 missing tensors are the parallel branch; the 15 MTP tensors are dropped too). Until upstream lands support, use our fork:
git clone -b agnes-3.0-flash https://github.com/JakeATX/llamAmpere.git
cd llamAmpere && cmake -B build -DGGML_CUDA=ON && cmake --build build -j --target llama-server
Details: docs/agnes-3.0-flash.md

Quantization build for the Preview release of Agnes-AI/Agnes-3.0-Flash, pinned to source revision 24f712ce59379b54c4a141d2708c35daf5ff613b.

Build status

The source safetensors are downloaded and verified locally. Conversion and uploads are in progress; quant folders will be published independently as they finish:

ATX-Q3_K_XL/
ATX-Q4_K_XL/
ATX-Q4_K_M/
ATX-Q5_K_XL/
ATX-Q6_K_XL/
ATX-Q8_K_XL/
ATX-IQ4_XS-M/

Every target GGUF includes Agnes' available one-layer MTP/NextN head. The GGUFs also retain the checkpoint's second 2,048-wide parallel FFN branch.

All Agnes model conversion, model-specific recipe adaptation, quantization, validation, and publication in this repository were performed by ATX. Unsloth did not produce these Agnes quants. Where an Unsloth Qwen3.8 calibration asset is named in GGUF metadata, that records an input to ATX's layer-role transfer; it is not an authorship or publisher claim for these artifacts.

Compatibility

Agnes' parallel FFN is not supported by stock llama.cpp at publication time. The artifacts are built from official ggml-org/llama.cpp commit 3057bb66c86c46d5781e50e85462a760ba7d1feb plus the narrowly scoped patch in build-info/. The patch commit 5bb7de853836418731d655a287af645bdf8257a5 has that official commit as its direct parent. These artifacts contain the text model and native MTP head. The vision projector is not included because the current converter does not support an Agnes mmproj export.

The model is dense rather than MoE. The dynamic quant recipes therefore apply role-aware bit allocation across recurrent attention, global attention, main FFN, and parallel FFN tensors instead of routed-expert formulas. The parallel FFN follows the corresponding main FFN tensor type in each layer.

Downloads last month: 321

GGUF

Model size

33B params

Architecture

qwen35

Hardware compatibility

3-bit

4-bit

5-bit

6-bit

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for jakeatx/Agnes-3.0-Flash-GGUF

Base model

Agnes-AI/Agnes-3.0-Flash

Quantized

(8)

this model