Instructions to use unsloth/DeepSeek-V3.1-Terminus-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use unsloth/DeepSeek-V3.1-Terminus-GGUF with Transformers:

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("unsloth/DeepSeek-V3.1-Terminus-GGUF", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use unsloth/DeepSeek-V3.1-Terminus-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf unsloth/DeepSeek-V3.1-Terminus-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
llama cli -hf unsloth/DeepSeek-V3.1-Terminus-GGUF:UD-Q4_K_XL

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf unsloth/DeepSeek-V3.1-Terminus-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
llama cli -hf unsloth/DeepSeek-V3.1-Terminus-GGUF:UD-Q4_K_XL

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf unsloth/DeepSeek-V3.1-Terminus-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
./llama-cli -hf unsloth/DeepSeek-V3.1-Terminus-GGUF:UD-Q4_K_XL

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf unsloth/DeepSeek-V3.1-Terminus-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
./build/bin/llama-cli -hf unsloth/DeepSeek-V3.1-Terminus-GGUF:UD-Q4_K_XL

Use Docker

docker model run hf.co/unsloth/DeepSeek-V3.1-Terminus-GGUF:UD-Q4_K_XL

LM Studio
Jan
Ollama
How to use unsloth/DeepSeek-V3.1-Terminus-GGUF with Ollama:
```
ollama run hf.co/unsloth/DeepSeek-V3.1-Terminus-GGUF:UD-Q4_K_XL
```

Unsloth Studio

How to use unsloth/DeepSeek-V3.1-Terminus-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for unsloth/DeepSeek-V3.1-Terminus-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for unsloth/DeepSeek-V3.1-Terminus-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for unsloth/DeepSeek-V3.1-Terminus-GGUF to start chatting

Docker Model Runner
How to use unsloth/DeepSeek-V3.1-Terminus-GGUF with Docker Model Runner:
```
docker model run hf.co/unsloth/DeepSeek-V3.1-Terminus-GGUF:UD-Q4_K_XL
```

Lemonade

How to use unsloth/DeepSeek-V3.1-Terminus-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull unsloth/DeepSeek-V3.1-Terminus-GGUF:UD-Q4_K_XL

Run and chat with the model

lemonade run user.DeepSeek-V3.1-Terminus-GGUF-UD-Q4_K_XL

List all available models

lemonade list

Atomic Chat

why Q4_K_M > Q4_K_XL

by bobchenyx - opened Oct 2, 2025

Discussion

bobchenyx

Oct 2, 2025

•

edited Oct 2, 2025

interesting thing that the size of Q4_K_M is larger than Q4_K_XL

perhaps it's with this part of ffn_down pattern matching which bump all ffn_down_exps and shexps as well ?
llama-quant.cpp#L336

is it designed on purpose to be like this?

[  53/1086]           blk.3.ffn_down_exps.weight - [ 2048,  7168,   256,     1], type =   bf16, converting to q6_K .. size =  7168.00 MiB ->  2940.00 MiB
[  54/1086]          blk.3.ffn_down_shexp.weight - [ 2048,  7168,     1,     1], type =   bf16, converting to q6_K .. size =    28.00 MiB ->    11.48 MiB

bobchenyx changed discussion title from why Q4_K_M > Q4_K_XL (lol) to why Q4_K_M > Q4_K_XL Oct 2, 2025

Cosmo77

Nov 20, 2025

Q4_K_XL is actually a dynamic quant version of the model with long name as UD-Q4_K_XL, so given that my assumption would be that the base model of both is the Q4_K_M and the dynamic quant version has some portions that have been degraded to 1 bit while other have been left at native or varying degrees between. So it was a value judgement by usloth on what size to the dynamic quant version.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment