Instructions to use unsloth/Qwen3-30B-A3B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use unsloth/Qwen3-30B-A3B-GGUF with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="unsloth/Qwen3-30B-A3B-GGUF")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("unsloth/Qwen3-30B-A3B-GGUF")
model = AutoModelForCausalLM.from_pretrained("unsloth/Qwen3-30B-A3B-GGUF", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use unsloth/Qwen3-30B-A3B-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf unsloth/Qwen3-30B-A3B-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
llama cli -hf unsloth/Qwen3-30B-A3B-GGUF:UD-Q4_K_XL

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf unsloth/Qwen3-30B-A3B-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
llama cli -hf unsloth/Qwen3-30B-A3B-GGUF:UD-Q4_K_XL

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf unsloth/Qwen3-30B-A3B-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
./llama-cli -hf unsloth/Qwen3-30B-A3B-GGUF:UD-Q4_K_XL

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf unsloth/Qwen3-30B-A3B-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
./build/bin/llama-cli -hf unsloth/Qwen3-30B-A3B-GGUF:UD-Q4_K_XL

Use Docker

docker model run hf.co/unsloth/Qwen3-30B-A3B-GGUF:UD-Q4_K_XL

LM Studio
Jan

vLLM

How to use unsloth/Qwen3-30B-A3B-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "unsloth/Qwen3-30B-A3B-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "unsloth/Qwen3-30B-A3B-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/unsloth/Qwen3-30B-A3B-GGUF:UD-Q4_K_XL

SGLang

How to use unsloth/Qwen3-30B-A3B-GGUF with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "unsloth/Qwen3-30B-A3B-GGUF" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "unsloth/Qwen3-30B-A3B-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "unsloth/Qwen3-30B-A3B-GGUF" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "unsloth/Qwen3-30B-A3B-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Ollama
How to use unsloth/Qwen3-30B-A3B-GGUF with Ollama:
```
ollama run hf.co/unsloth/Qwen3-30B-A3B-GGUF:UD-Q4_K_XL
```

Unsloth Studio

How to use unsloth/Qwen3-30B-A3B-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for unsloth/Qwen3-30B-A3B-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for unsloth/Qwen3-30B-A3B-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for unsloth/Qwen3-30B-A3B-GGUF to start chatting

How to use unsloth/Qwen3-30B-A3B-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf unsloth/Qwen3-30B-A3B-GGUF:UD-Q4_K_XL

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "unsloth/Qwen3-30B-A3B-GGUF:UD-Q4_K_XL"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use unsloth/Qwen3-30B-A3B-GGUF with Docker Model Runner:
```
docker model run hf.co/unsloth/Qwen3-30B-A3B-GGUF:UD-Q4_K_XL
```

Lemonade

How to use unsloth/Qwen3-30B-A3B-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull unsloth/Qwen3-30B-A3B-GGUF:UD-Q4_K_XL

Run and chat with the model

lemonade run user.Qwen3-30B-A3B-GGUF-UD-Q4_K_XL

List all available models

lemonade list

Hermes Agent

How to use unsloth/Qwen3-30B-A3B-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf unsloth/Qwen3-30B-A3B-GGUF:UD-Q4_K_XL

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default unsloth/Qwen3-30B-A3B-GGUF:UD-Q4_K_XL

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use unsloth/Qwen3-30B-A3B-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf unsloth/Qwen3-30B-A3B-GGUF:UD-Q4_K_XL

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "unsloth/Qwen3-30B-A3B-GGUF:UD-Q4_K_XL" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

`UD-Q4_K_XL` or `Q4_K_M`?

by pootow - opened Apr 29, 2025

Discussion

pootow

Apr 29, 2025

what does UD mean in the name?
why Q4_K_XL is smaller than Q4_K_M? I think XL is supposed to be larger than M.
what to choose?

CyberTimon

Apr 29, 2025

wondering the same

fieryTransition

Apr 29, 2025

Pretty sure UD means unsloth dynamic, so based on their blog posts, you'll want that.

Dampfinchen

Apr 29, 2025

Why would they continue to upload the old quants when their UD quants are significantly better?

TobDeBer

Apr 29, 2025

Q4_K_XL decides to use Q5_K on important matrices if it considers it safe. Q4_K_M uses Q6_K there mostly.
Most matrices don't differ and use Q4_K.
I'd always go with the XL variant.

shimmyshimmer

Unsloth AI org Apr 29, 2025

•

edited Apr 30, 2025

For better results, Always use the XL quants.

yes, UD means Unsloth Dynamic

All of them use our calibration dataset

NightwingNg

Apr 29, 2025

•

edited Apr 29, 2025

For better results, Always use the XL quants.

yes, UD means Unsloth Dynamic

So no UD means not using it? But you said all Qwen3 models are using it, right?

urtuuuu

Apr 29, 2025

•

edited Apr 29, 2025

If the highest quant/size i can run is Q3_K_M.gguf, then does it make sense to download UD-Q3_K_XL.gguf instead? That's all i want to know :)
(because its also smaller in size, but performance?)

shimmyshimmer

Unsloth AI org Apr 30, 2025

For better results, Always use the XL quants.

yes, UD means Unsloth Dynamic

So no UD means not using it? But you said all Qwen3 models are using it, right?

Correct, all Qwen3 models are using it but some quants of the Qwen3 model do not use it. But all quants do use our calibration dataset

shimmyshimmer

Unsloth AI org Apr 30, 2025

If the highest quant/size i can run is Q3_K_M.gguf, then does it make sense to download UD-Q3_K_XL.gguf instead? That's all i want to know :)
(because its also smaller in size, but performance?)

I would recommend you to try both and see which you like better

LagOps

May 13, 2025

Where did the UD-Q4_K_XL go? was something wrong with it?

shimmyshimmer

Unsloth AI org May 13, 2025

Where did the UD-Q4_K_XL go? was something wrong with it?

Whoops because of the amount of quants hugging face hides it cause it's overwhelming

LagOps

May 13, 2025

•

edited May 13, 2025

Oh i see, it is not grouped with the regular quants in the files section either, so i overlooked it. i see there has been an update 4 days ago - what changed?

Nevermind, there has been a new version right now! Still, I wonder what changed since it's not apparent from the commit history.

shimmyshimmer

Unsloth AI org May 14, 2025

Oh i see, it is not grouped with the regular quants in the files section either, so i overlooked it. i see there has been an update 4 days ago - what changed?

Nevermind, there has been a new version right now! Still, I wonder what changed since it's not apparent from the commit history.

We're working with HF to fix this btw, see:
https://github.com/huggingface/huggingface.js/pull/1452/files

And we are updating the models with the latest chat template fixes, see: https://www.reddit.com/r/LocalLLaMA/comments/1klltt4/the_qwen3_chat_template_is_still_bugged/

CHNtentes

May 14, 2025

Oh i see, it is not grouped with the regular quants in the files section either, so i overlooked it. i see there has been an update 4 days ago - what changed?

Nevermind, there has been a new version right now! Still, I wonder what changed since it's not apparent from the commit history.

We're working with HF to fix this btw, see:
https://github.com/huggingface/huggingface.js/pull/1452/files

And we are updating the models with the latest chat template fixes, see: https://www.reddit.com/r/LocalLLaMA/comments/1klltt4/the_qwen3_chat_template_is_still_bugged/

Maybe get rid of Q4_1? Q4_0 seems still useful in some cases.

shimmyshimmer

Unsloth AI org May 14, 2025

@LagOps @CHNtentes it should be fixed now guys :)

imoc

Jun 23, 2025

Q4_K_XL decides to use Q5_K on important matrices if it considers it safe. Q4_K_M uses Q6_K there mostly.
Most matrices don't differ and use Q4_K.
I'd always go with the XL variant.

Hmmm... Why XL? Smaller?

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment