Instructions to use unsloth/Qwen3.8-27B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use unsloth/Qwen3.8-27B-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M
# Run inference directly in the terminal:
llama cli -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M
# Run inference directly in the terminal:
llama cli -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M

Use Docker

docker model run hf.co/unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M

LM Studio
Jan
Ollama
How to use unsloth/Qwen3.8-27B-GGUF with Ollama:
```
ollama run hf.co/unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M
```

Unsloth Studio

How to use unsloth/Qwen3.8-27B-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for unsloth/Qwen3.8-27B-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for unsloth/Qwen3.8-27B-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for unsloth/Qwen3.8-27B-GGUF to start chatting

How to use unsloth/Qwen3.8-27B-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use unsloth/Qwen3.8-27B-GGUF with Docker Model Runner:
```
docker model run hf.co/unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M
```

Lemonade

How to use unsloth/Qwen3.8-27B-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M

Run and chat with the model

lemonade run user.Qwen3.8-27B-GGUF-UD-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use unsloth/Qwen3.8-27B-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use unsloth/Qwen3.8-27B-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

New discussion

Resources

View closed (24)

Introducing Unsloth Dynamic v3 Qwen3.8

pinned

👍🔥 45

#74 opened 2 days ago by

danielhanchen

What's the use of mtp-Qwen3.8-27B-Q4_0.gguf?

#97 opened about 13 hours ago by

cnayan01

Could not download MTP drafter: RemoteEntryNotFoundError: 404 Client Error

#95 opened about 14 hours ago by

CyberTod

Qwen in the UD-IQ3_XXS quant is honestly pretty impressive on my RTX 5070 Ti with 32 GB of 6400 MT/s RAM.

🚀 1

#94 opened about 14 hours ago by

Nevermorye

Why Qwen3.8-27B-UD-Q8_K_XL.gguf is not updated like other quants?

#91 opened 1 day ago by

akierum

KV-cache KLD scales with model fidelity, not quant family — three null results and one metric trap

#89 opened 1 day ago by

Knappy

Imatrix now available!

🔥🚀 13

#88 opened 1 day ago by

danielhanchen

# VRAM is linear in context: a formula that predicts any Qwen3.8-27B GGUF config (and settles the 150K debate)

#87 opened 1 day ago by

Knappy

Why delete Qwen3.8-27B-IQ4_NL.gguf with huggingface_hub?

👍 5

#86 opened 1 day ago by

puchuu

Even at low Q6 quant it's very good!

🤗 2

#84 opened 1 day ago by

auf1r2

The model loads successfully via llama.cpp, but it crashes with an abnormal exit as soon as inference begins. Why is this happening?

🤗 1

#83 opened 1 day ago by

Matheartcis

Can i ask the sweet point for 12gb cards?

#81 opened 2 days ago by

AsThirtyThree

Quality enhance

#80 opened 2 days ago by

r1yyxn

DFlash2?

👀 5

#79 opened 2 days ago by

guarism0

What is mtp-Qwen3.8-27B-Q4_0.gguf used for?

#78 opened 2 days ago by

artden111

Qwen3.8-27B on Dual RTX 3060 — llama.cpp Benchmark

#77 opened 2 days ago by

marjimgu

`Qwen3.8-27B-UD-IQ1_S.gguf`Has anyone tried this?

#76 opened 2 days ago by

artden111

NVFP4/MXFP4 GGUF Version?

👍 3

#75 opened 2 days ago by

CYISNOTHERE

Qwen3.8-27B: Q5_K_M, RTX 5060ti 16gb x 2, total 32gb, 30t/s, 215k context

🤗 1

#72 opened 3 days ago by

Artem7799

Squeezing Qwen 3.8 27B into a Single 16 GB GPU — Almost 42 tok/s, 64K Context

🚀 1

#70 opened 4 days ago by

Ataa

Crash when used with llama.cpp on image loading.

#69 opened 4 days ago by

MalcolmMielle

Running Qwen3.8-27B on a Single RTX 3090 (24 GB) 45-70t/s with 150k context

🔥 1

#68 opened 4 days ago by

SergeySS8

Qwen3.8-27B only 18 tok/s vs Qwen3.6-27B 62 tok/s on A100 80GB + vLLM — expected?

👍 4

#66 opened 5 days ago by

Alecone

Qwen3.8-27B on an RTX 4070 Ti 12GB: IQ2 vs Q2, 16K context, and MTP

👍 2

#65 opened 5 days ago by

Hugosmr

Enable Reasoning Level with LM Studio and VS Code GitHub Copilot Chat

❤️ 2

#63 opened 5 days ago by

asage-me

Running Qwen3.8-27B Q4_K_S on an RTX 3060 12GB with 96K Context + MTP (~10 t/s)

#61 opened 5 days ago by

Hjx2

UD_IQ1_XXXS possible like you did with the big one?

🔥 1

#60 opened 5 days ago by

TheWegemann

Qwen3.8-27B UD-Q2_K_XL: unexpectedly slow ROCm prefill; ssm_alpha/ssm_beta are IQ1_M

🔥 1

#59 opened 5 days ago by

djtrondheim

Q3 K M (13.8 GB) is larger than UD Q3 K XL(13.4GB)

👍 1

#58 opened 6 days ago by

whatitall

llama.cpp Strix Halo 128GB config for coding tasks

#57 opened 6 days ago by

lukasz-lobocki

My issues and what I had to do to get Qwen3.8-27B /Qwen3.8-27B-MTP-ONLY-Q6_K.gguf /Qwen3.8-27B-MTP-ONLY-Q4_K_M.gguf to work with Llama.cpp

👍 2

#56 opened 6 days ago by

Polyj

Tested the model (locally) on coding and agentic work

#54 opened 6 days ago by

curiousily

llama server settings on Hermes agent 16GB+8GB GPU - MANY THANKS UNSLOTH!

#53 opened 6 days ago by

Seraph-Northforge

Qwen3.8-27B有大量的无效思考量，导致任务完成时间非常长。

#51 opened 6 days ago by

kelei999999

KL divergence on code Vs long prose/documents

#50 opened 6 days ago by

Luke2642

Independent KLD benchmark: UD-IQ2_XXS beats bartowski IQ2_XXS on both size and quality

👍 9

#49 opened 6 days ago by

x0me

No valid custom reasoning fields found in model 'unsloth/Qwen3.8-27B-GGUF/Qwen3.8-27B-xxx.gguf' ?

#48 opened 6 days ago by

Tropicalbreeze

Qwen3.8-27B Q8_0 on 3x Radeon Instinct MI50 16Gb

🚀🤗 3

#47 opened 6 days ago by

NunoS

Single rtx rog 3090

#46 opened 6 days ago by

Utku92

Qwen 3.8-27b context usage is about 10x of 3.6-27b

👍 4

#45 opened 7 days ago by

WhiteDan64

Dual 3080 20G, Q6-K_XL, avg 40 - 50 t/s

👍 1

#44 opened 7 days ago by

xudongCNCS

Chat template issue and ways to solve it

❤️👍 7

#42 opened 7 days ago by

Snol-std

Ryzen AI Max+ 395 results: 16.68 tok/s with UD-Q5_K_XL, Vulkan and MTP 4

❤️ 3

#41 opened 7 days ago by

erstmalreden

I think this model is not ready for use

👀👍 7

#40 opened 7 days ago by

sa13ma

anyone feels the same? the ocr and vlm ability is inferior to 3.6 27B

#39 opened 7 days ago by

Albert-Liu-Hongkang

Why UD-Q2_K_XL has IQ1_M SSM alpha/beta and only token_embd use 2bit quant? It's a Q2 GGUF!

➕ 2

#36 opened 7 days ago by

Gavin-chen

UD quants for 16GB VRAM?

#35 opened 7 days ago by

Witherhoard

something weird. ggml version of gguf doesn't have this thing

#34 opened 7 days ago by

rosspanda0

codeneedle benchmark results

#33 opened 7 days ago by

akierum

The model is really slow due to unclear reason

🚀 2

#32 opened 7 days ago by

SNMetamorph