Instructions to use yeaay/Intern-Decision-4B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use yeaay/Intern-Decision-4B-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf yeaay/Intern-Decision-4B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf yeaay/Intern-Decision-4B-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf yeaay/Intern-Decision-4B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf yeaay/Intern-Decision-4B-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf yeaay/Intern-Decision-4B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama cli -hf yeaay/Intern-Decision-4B-GGUF:Q4_K_M

Use Docker

docker model run hf.co/yeaay/Intern-Decision-4B-GGUF:Q4_K_M

Llama new
LM Studio
Jan

vLLM

How to use yeaay/Intern-Decision-4B-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "yeaay/Intern-Decision-4B-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "yeaay/Intern-Decision-4B-GGUF",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/yeaay/Intern-Decision-4B-GGUF:Q4_K_M

Ollama
How to use yeaay/Intern-Decision-4B-GGUF with Ollama:
```
ollama run hf.co/yeaay/Intern-Decision-4B-GGUF:Q4_K_M
```
Unsloth Desktop

How to use yeaay/Intern-Decision-4B-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf yeaay/Intern-Decision-4B-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "yeaay/Intern-Decision-4B-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use yeaay/Intern-Decision-4B-GGUF with Docker Model Runner:
```
docker model run hf.co/yeaay/Intern-Decision-4B-GGUF:Q4_K_M
```

Lemonade

How to use yeaay/Intern-Decision-4B-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull yeaay/Intern-Decision-4B-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Intern-Decision-4B-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use yeaay/Intern-Decision-4B-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf yeaay/Intern-Decision-4B-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default yeaay/Intern-Decision-4B-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use yeaay/Intern-Decision-4B-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf yeaay/Intern-Decision-4B-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "yeaay/Intern-Decision-4B-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Intern-Decision-4B GGUF (F16 / Q8_0 / Q5_K_M / Q4_K_M)

GGUF conversions of internlm/Intern-Decision-4B with a decision-level fidelity evaluation on the official Intern-Decision benchmarks: every quantization is compared, decision by decision, against the original BF16 checkpoint run in the same environment.

This is an unofficial community conversion; it is not affiliated with InternLM / Shanghai AI Lab.

Recommended file: Intern-Decision-4B-Q5_K_M.gguf — 3.07 GB, ~3.9 GB VRAM at 8K context, 98.5% of decisions identical to BF16, unchanged average accuracy.

Files

File	Size	SHA-256
`Intern-Decision-4B-F16.gguf`	8.42 GB	`97525f7157cda0b44b73b714ee2fa7297c94e3cb56b5981fd80de365f496bcca`
`Intern-Decision-4B-Q8_0.gguf`	4.48 GB	`86198e50f22d6a43202b3e73fdb74a0bd7caece1136975ac3ecf530787a50813`
`Intern-Decision-4B-Q5_K_M.gguf`	3.07 GB	`a322e305daf5f78c3865d9dd812a48ccd276035d3882af6f09f5b1e6e002b359`
`Intern-Decision-4B-Q4_K_M.gguf`	2.71 GB	`3c3be9bd66008743fc24a763f28e2de51fb2417d2e4406ae59a3015ec7eaeaab`
`mmproj-Intern-Decision-4B-F16.gguf`	0.67 GB	`beaeba2ae864a437ab4ef681a9e58b5ab7e1882e23eb3c66f0d30911178715a9`
`decision_gguf.py`		Reference scorer (see below)
`eval/`		Evaluation runner, comparison script and summary results

The vision projector is provided for completeness; all evaluations below are text-only.

This is a scoring model, not a chat model

Intern-Decision does not generate answers. A request (state + named questions with options) is compiled into a chat transcript whose assistant turn is a JSON skeleton with one <decision> placeholder per field; the model's logits at the position immediately before each placeholder are restricted to that field's option symbols (A, B, …), softmaxed, and calibrated with temperature scaling (T = 1.99241824, fitted upstream on BF16 and left unchanged here).

Using these files through a chat UI or llama-server completion endpoint will not reproduce the model's behaviour. Use decision_gguf.py, which takes the prompt compiler, chat template, symbol mapping and calibration directly from the official inference.py and only runs the forward pass on llama.cpp (no PyTorch needed):

pip install llama-cpp-python huggingface_hub transformers numpy
python decision_gguf.py --gguf Intern-Decision-4B-Q5_K_M.gguf --request request.json

request.json uses the upstream format, e.g.

{"state": "The customer was charged twice and asks for the extra payment back.",
 "questions": {"team": {"type": "choice", "instructions": "Which team should handle this request?",
                        "criteria": {"billing": "Payments and refunds", "delivery": "Shipping and delivery"}},
               "refund_requested": {"type": "noul", "instructions": "Is the customer asking for a refund?"}}}

Each <decision> position is scored with one batched forward over its full prefix. Feeding the prompt incrementally (continuing the KV/recurrent state between decision positions) is mathematically equivalent but, for K-quants, changed logits by up to ~0.6 because llama.cpp picks different quantized kernels for small batches — enough to flip near-tie decisions.

Conversion

Source: internlm/Intern-Decision-4B revision 0e5e6aa7d6d750e2b1504ba11a8136cb58aeb3cd. All 14 checkpoint files match the SHA-256 hashes recorded for the 4B model in the official benchmarks/temperature-presets.json, i.e. this is the same checkpoint the published results used.
llama.cpp d81235049384534c167caea52b85a694f6103d14 (CPU build), architecture qwen35.
python convert_hf_to_gguf.py <hf_dir> --no-nextn --outtype f16, then llama-quantize to each type (no imatrix). --no-nextn is required: config.json declares mtp_num_hidden_layers: 1 but the checkpoint contains no MTP weights, and without the flag the GGUF declares a block with no tensors and fails to load.
mmproj: convert_hf_to_gguf.py <hf_dir> --mmproj --outtype f16.
Tokenizer: <decision> is preserved as a single special token (id 248077); in every spot check (benchmark self-checks and test requests) the GGUF tokenizer reproduced the HF token ids exactly.

Evaluation

Official bundle from InternLM/Intern-Decision at commit 3572c8a68b5df5dafe02d0e093989ba8ec0183bc: the seven accuracy-v1 suites (10,751 rows / 12,351 decisions) and the 96-case known-distribution calibration pilot, scored with the repository's own src.eval.jev.evaluate, upstream Jevbench scoring, run_pilot.predict_inputs and score_known_distribution.score_benchmark.

Baseline: original BF16 weights through the official HF backend (--backend hf, transformers 5.14.1, SDPA).
GGUF: a backend subclassing the official HFBackend (identical encode()), forward pass via llama-cpp-python 0.3.36 with CUDA, all layers on one GPU.
Hardware: NVIDIA Tesla T4 16 GB (Kaggle), one model per GPU.

Accuracy (%)

Suite	n	BF16 (HF)	F16	Q8_0	Q5_K_M	Q4_K_M
Jevbench-Easy	48	100.00	100.00	100.00	100.00	100.00
Jevbench-Original	72	98.61	98.61	98.61	100.00	100.00
Jevbench-Hard	111	72.97	72.97	72.97	72.07	70.27
Typed Decision	2,000	80.50	80.75	80.50	81.35	79.80
ToolACE	310	96.45	96.45	96.45	96.45	96.45
AG News	7,600	90.80	90.82	90.80	90.68	90.39
WildJailBreak	2,210	89.82	89.95	89.86	89.14	89.00
Average (7 suites, unweighted)		89.88	89.94	89.88	89.96	89.42

For reference, the upstream model card reports 90.02 for Intern-Decision-4B using the XTuner backend; upstream documents HF-backend results as a different execution setting.

Decision-level agreement with BF16 (12,351 decisions)

	Same argmax as BF16	BF16 right → quant wrong	BF16 wrong → quant right	wrong → different wrong	Mean / p99 \|Δ top-prob\|
F16	99.69%	13	22	3	0.003 / 0.027
Q8_0	99.62%	22	23	2	0.004 / 0.045
Q5_K_M	98.51%	91	84	9	0.014 / 0.182
Q4_K_M	97.13%	201	136	18	0.026 / 0.297

F16 involves no quantization, so its 0.31% disagreement is the floor set by the backend change (llama.cpp vs PyTorch kernels). Q8_0 is indistinguishable from that floor. Q5_K_M changes ~1.2% more decisions, with gains and losses nearly balanced. Q4_K_M loses net accuracy, most visibly on Jevbench-Hard and AG News.

Calibration (fixed T = 1.99241824)

	Jevbench-Hard Brier ↓	Jevbench-Hard ECE ↓	Pilot TVD (raw → calibrated) ↓	Pilot excess Brier (calibrated) ↓
BF16 (HF)	0.347	0.080	0.269 → 0.216	0.105
F16	0.349	0.074	0.268 → 0.214	0.104
Q8_0	0.350	0.073	0.271 → 0.216	0.104
Q5_K_M	0.347	0.075	0.278 → 0.218	0.109
Q4_K_M	0.372	0.065	0.257 → 0.209	0.096

Jevbench-Hard has 111 items, so its Brier/ECE are noisy. Q8_0 and Q5_K_M stay close to BF16 on all calibration metrics with the upstream temperature unchanged.

Memory (single T4, n_ctx = 8,256, all layers offloaded)

	Load	Peak during suites
BF16 (HF, SDPA)	8.9 GB	12.4 GB
F16	9.0 GB	9.3 GB
Q8_0	5.1 GB	5.4 GB
Q5_K_M	3.9 GB	4.2 GB
Q4_K_M	3.4 GB	3.7 GB

Smaller contexts need less. Latencies measured on the T4 are not reported as representative (BF16 has no native support on Turing and the HF path used reference kernels for the linear-attention layers).

Reproducing

eval/worker.py (runs inside a checkout of the official repository) and eval/analyze.py reproduce the tables; eval/comparison.json holds the summary numbers and sample flips.

License

Apache-2.0, inherited from Intern-Decision-4B, which is derived from Qwen3.5; LICENSE and LICENSE-QWEN are retained. The weights are unmodified apart from format conversion and quantization.

Downloads last month: 448

GGUF

Model size

4B params

Architecture

qwen35

Hardware compatibility

4-bit

5-bit

8-bit

16-bit

Model tree for yeaay/Intern-Decision-4B-GGUF

Base model

Qwen/Qwen3.5-4B-Base

Finetuned

Qwen/Qwen3.5-4B

Finetuned

internlm/Intern-Decision-4B

Quantized

(2)

this model