Instructions to use quevirus/Xevn-4B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use quevirus/Xevn-4B-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf quevirus/Xevn-4B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf quevirus/Xevn-4B-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf quevirus/Xevn-4B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf quevirus/Xevn-4B-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf quevirus/Xevn-4B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf quevirus/Xevn-4B-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf quevirus/Xevn-4B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf quevirus/Xevn-4B-GGUF:Q4_K_M

Use Docker

docker model run hf.co/quevirus/Xevn-4B-GGUF:Q4_K_M

LM Studio
Jan
Ollama
How to use quevirus/Xevn-4B-GGUF with Ollama:
```
ollama run hf.co/quevirus/Xevn-4B-GGUF:Q4_K_M
```
Unsloth Desktop

How to use quevirus/Xevn-4B-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf quevirus/Xevn-4B-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "quevirus/Xevn-4B-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use quevirus/Xevn-4B-GGUF with Docker Model Runner:
```
docker model run hf.co/quevirus/Xevn-4B-GGUF:Q4_K_M
```

Lemonade

How to use quevirus/Xevn-4B-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull quevirus/Xevn-4B-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Xevn-4B-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use quevirus/Xevn-4B-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf quevirus/Xevn-4B-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default quevirus/Xevn-4B-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use quevirus/Xevn-4B-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf quevirus/Xevn-4B-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "quevirus/Xevn-4B-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Xevn-4B-GGUF

Transformers | MLX | GitHub | Website

GGUF files of Xevn-4B for llama.cpp. Xevn is a 4B decision model built for Korean, trained for Korean understanding, knowledge, and judgment.

Xevn follows the Jev decision format. It reads a question and its options once and returns the answer together with a probability for every option. There is no text generation, so a request costs one forward pass.

Model overview

Type: Decision model with one forward pass per question and no text generation
Parameters: 4B
Architecture: Hybrid Gated DeltaNet and attention, 32 layers (24 Gated DeltaNet, 8 full attention)
Hidden size / vocabulary: 2,560 / 248,320
Question types: choice, noul, score
Languages: Korean (primary), English
Format: GGUF (llama.cpp)
Files: BF16, Q8_0, Q6_K, Q5_K_M, Q4_K_M

Files

File	Quant	Size	Same answer as 16-bit
Xevn-4B-BF16.gguf	BF16	8.67 GB	39/40
Xevn-4B-Q8_0.gguf	Q8_0	4.61 GB	38/40
Xevn-4B-Q6_K.gguf	Q6_K	3.56 GB	37/40
Xevn-4B-Q5_K_M.gguf	Q5_K_M	3.16 GB	36/40
Xevn-4B-Q4_K_M.gguf	Q4_K_M	2.78 GB	38/40

Xevn-4B-Q8_0.gguf is the recommended default. "Same answer as 16-bit" counts held-out questions where the file picks the same top answer as the 16-bit Transformers model. With 40 questions, differences of one or two answers between files are within noise.

Quickstart

pip install -U "llama-cpp-python>=0.3.36" huggingface_hub
# NVIDIA: CMAKE_ARGS="-DGGML_CUDA=on" pip install -U "llama-cpp-python>=0.3.36"
hf download quevirus/Xevn-4B-GGUF Xevn-4B-Q8_0.gguf xevn_gguf.py xevn_infer.py temps.json --local-dir Xevn-4B-GGUF

Older llama-cpp-python releases do not support this architecture. On macOS the default build uses Metal.

import sys
sys.path.insert(0, "Xevn-4B-GGUF")  # the repository ships the xevn_gguf.py helper
from xevn_gguf import XevnGGUF

model = XevnGGUF("Xevn-4B-GGUF/Xevn-4B-Q8_0.gguf")  # n_ctx=8192, n_gpu_layers=-1 (all layers on the GPU)

state = "고객: 어제 주문한 이어폰 값이 카드에서 두 번 결제됐어요. 오늘 안에 한 건 취소해 주세요."
# "Customer: the earphones I ordered yesterday were charged to my card twice. Please cancel one today."
questions = {
    "team": {"type": "choice", "instructions": "Which team should handle this ticket?",
             "criteria": {"billing": "Payments and refunds", "shipping": "Delivery", "account": "Login and account"}},
    "charged_twice": {"type": "noul", "instructions": "Was the customer charged more than once?"},
    "urgency": {"type": "score", "instructions": "How urgent is this ticket?",
                "criteria": ["Low", "Medium", "High"]},
}
result = {key: model.predict(state, q) for key, q in questions.items()}

Output (rounded):

{
  "team": {"answer": "billing", "probs": {"billing": 0.998, "shipping": 0.001, "account": 0.001}},
  "charged_twice": {"answer": true, "probs": {"yes": 0.993, "no": 0.007}},
  "urgency": {"answer": 1.77, "probs": {"0": 0.041, "1": 0.150, "2": 0.809}}
}

Values differ slightly between quant types.

Request format

A request is a state (text or any JSON value) and one or more named questions. Each question has a type, instructions, and criteria.

Type	Use	`criteria`	`answer` in the result
`choice`	Pick one of up to 26 options	`{"key": "description", ...}`, 2 to 26 options	The key of the most probable option
`noul`	Yes or no	Optional `{"true": "...", "false": "..."}`	`true` or `false`
`score`	Rate on a graded scale	`["lowest level", ..., "highest level"]`, 2 to 10 levels	The expected level (0 is the first level)

probs holds a probability for every option, yes/no, or level. For noul, a yes probability strictly between 0.2 and 0.8 is moved to 0.2 or 0.8 on the same side, so the model always commits to an answer. Readout temperatures per type are in temps.json (choice 1.1, noul 1.2, score 1.2).

Benchmark results are in the Xevn-4B evaluation; they were measured with the 16-bit weights.

Model files

Format	Precision	Size	Speed (p50)	Repository
Transformers	16-bit	9.3 GB		Xevn-4B
MLX	16-bit	8.4 GB	88 ms	Xevn-4B-MLX-bf16
MLX	8-bit	4.5 GB	83 ms	Xevn-4B-MLX-8bit
MLX	6-bit	3.4 GB	82 ms	Xevn-4B-MLX-6bit
MLX	5-bit	2.9 GB	80 ms	Xevn-4B-MLX-5bit
MLX	4-bit	2.4 GB	78 ms	Xevn-4B-MLX-4bit
MLX	3-bit	1.8 GB	77 ms	Xevn-4B-MLX-3bit
GGUF	BF16	8.67 GB		Xevn-4B-GGUF
GGUF	Q8_0	4.61 GB		Xevn-4B-GGUF
GGUF	Q6_K	3.56 GB		Xevn-4B-GGUF
GGUF	Q5_K_M	3.16 GB		Xevn-4B-GGUF
GGUF	Q4_K_M	2.78 GB		Xevn-4B-GGUF

Speed is the MLX p50 over an equal mix of the three input lengths in the Speed table, measured in one session on an Apple M4 Max. On every benchmark, MLX 8-bit to 5-bit stay within 2 points of 16-bit accuracy and 4-bit within 3 points. 3-bit loses 7 to 21 points and is provided for experimentation. The Transformers weights keep the original multimodal layout (9.3 GB); the helper uses the text model only.

Intended use and limitations

Xevn is built for decisions: classification, multiple choice, yes/no judgments, and rating. It is not a chat or text-generation model. It is tuned for Korean; English input works but is not the focus.

Xevn returns probabilities over the options you give it. It does not generate text and cannot answer open questions.
The bundled helpers run one forward pass per question and accept 2 to 26 options and 2 to 10 score levels.
Speed figures are for MLX on an Apple M4 Max with inputs up to 256 tokens.
Ollama is not supported, because this readout needs the prompt logits.

Citation

@misc{xevn2026,
  title  = {Xevn-4B: a Korean decision model},
  author = {Nhahan},
  year   = {2026},
  url    = {https://huggingface.co/quevirus/Xevn-4B}
}