tinyopsec/Nanbeige4.2-3B-GGUF

Nanbeige4.2-3B — GGUF Quantizations

Community-provided GGUF quantizations of Nanbeige/Nanbeige4.2-3B, a compact agentic LLM developed by Nanbeige LLM Lab.

All weight files were converted from the original BF16 SafeTensors checkpoint using the Nanbeige fork of llama.cpp (-b nanbeige42). No fine-tuning or architectural changes were applied.

---

Model Overview

Property	Value
Base model	Nanbeige/Nanbeige4.2-3B
Architecture	Looped Transformer (nanbeige)
Total parameters	4B
Non-embedding parameters	3B
Context length	262,144 tokens (256K)
Languages	English, Chinese
Format	GGUF
Original license	Apache 2.0
Technical report	arXiv:2607.22083

Nanbeige4.2-3B is a supervised fine-tuned + RL-trained agentic model built on Nanbeige4.2-3B-Base. Its Looped Transformer architecture reuses transformer layers to increase effective model capacity without increasing parameter count. At 3B non-embedding parameters it outperforms models such as Qwen3.5-9B and Gemma4-12B on several agentic and reasoning benchmarks.

---

Available Quantizations

Quantization	File size	Notes
Q2_K	1.76 GB	Minimum footprint; noticeable quality loss
Q3_K_S	2.00 GB	Small 3-bit; good for very constrained memory
Q3_K_M	2.17 GB	Medium 3-bit; better quality than S
Q4_0	2.48 GB	Legacy 4-bit; lower quality than K-quants
Q4_K_S	2.50 GB	Small 4-bit K-quant
Q4_K_M	2.57 GB	Recommended default — best quality/size ratio
Q5_K_S	2.94 GB	Small 5-bit K-quant
Q5_K_M	2.99 GB	Medium 5-bit; high quality with modest size
Q6_K	3.42 GB	Near-lossless; use when RAM allows
Q8_0	4.43 GB	Minimal precision loss; reference quality
F16	8.34 GB	Full half-precision; for benchmarking only

Recommendation: Q4_K_M is the best starting point for most deployments. Use Q6_K or Q8_0 if you have sufficient RAM and need maximum fidelity.

---

Quickstart

llama.cpp

Install (macOS / Linux)

curl -LsSf https://llama.app/install.sh | sh

Install (Windows via WinGet)

winget install llama.cpp

Run a local OpenAI-compatible server

llama serve -hf tinyopsec/Nanbeige4.2-3B-GGUF:Q4_K_M

Run inference in the terminal

llama cli -hf tinyopsec/Nanbeige4.2-3B-GGUF:Q4_K_M

---

Ollama

ollama run hf.co/tinyopsec/Nanbeige4.2-3B-GGUF:Q4_K_M

---

Docker

docker model run hf.co/tinyopsec/Nanbeige4.2-3B-GGUF:Q4_K_M

---

LM Studio

Download any GGUF file from the Files tab and open it directly in LM Studio.

Note: The bundled llama.cpp inside LM Studio does not yet support the nanbeige architecture. You must replace the LM Studio backend binaries with the Nanbeige fork:

1. Build llama.cpp from the nanbeige42 branch
2. Copy build/bin/* into your LM Studio backend directory (e.g. ~/.lmstudio/extensions/backends/llama.cpp-<version>)

---

Inference Parameters

The following settings are recommended by the original authors:

Scenario	Temperature	Max new tokens
Agentic / tool-use tasks	1.0	65,536
Reasoning / chat tasks	0.6	131,072

The chat template exposes two configuration flags:

• enable_thinking — enables chain-of-thought reasoning (on by default; set to false for non-thinking mode)
• preserve_thinking — controls whether reasoning tokens from previous turns are retained in multi-turn conversations (recommended true for tool-use and code-agent workflows; false for general chat)

For tool-calling, tool_call_format="xml" yields the best performance; json is supported for compatibility.

---

Benchmark Summary

Results reported by the original authors (thinking mode, preserve_thinking=true):

Benchmark	Nanbeige4.2-3B	Qwen3.5-9B	Qwen3.5-4B	Gemma4-12B
SWE-Bench Verified	63.6	53.1	38.8	44.2
SWE-Bench Pro	46.9	33.8	29.4	21.9
GPQA-Diamond	87.4	81.7	78.2	78.8
HMMT-Feb-2026	82.8	69.6	60.6	51.5
LiveCodeBench-V6	72.5	65.6	55.8	72.0
HLE (w/o Search)	17.8	12.5	6.8	14.8
Claw-Evalpass	52.2	47.1	36.9	25.5
GDPval rubrics	74.3	61.9	46.7	68.5

Full benchmark tables and methodology are available in the original model card and technical report.

---

Conversion Notes

GGUF files were produced using the Nanbeige fork of llama.cpp. The standard upstream llama.cpp (ggerganov/llama.cpp) does not support the nanbeige architecture at the time of publication; use the forked build for any operations that require re-quantizing or converting these files.

# Clone the Nanbeige fork
git clone -b nanbeige42 https://github.com/Nanbeige/llama.cpp.git
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

# BF16 → GGUF (reference step)
python3 convert_hf_to_gguf.py /path/to/Nanbeige4.2-3B \
  --outfile Nanbeige4.2-3B-BF16.gguf \
  --outtype bf16

# Re-quantize to Q4_K_M
./build/bin/llama-quantize \
  Nanbeige4.2-3B-BF16.gguf \
  Nanbeige4.2-3B-Q4_K_M.gguf \
  Q4_K_M

---

Limitations

As noted by the original authors: the model may produce inaccurate, biased, or otherwise inappropriate outputs due to its probabilistic nature. These GGUF files are unmodified quantizations of the original checkpoint and inherit all limitations documented in the original model card.

---

License

The original model weights are released under the Apache 2.0 license by Nanbeige LLM Lab. This repository provides only format-converted quantizations. The Apache 2.0 license terms apply; see LICENSE in the original repository.

---

Citation

If you use this model, please cite the original work:

@article{lab2026nanbeige4,
  title={Nanbeige4.2-3B: Unlocking Agentic Capabilities in a Compact Model},
  author={Lab, Nanbeige and Yang, Chen and Huang, Chengrui and Lan, Fufeng and others},
  journal={arXiv preprint arXiv:2607.22083},
  year={2026}
}

---

Links

• Original model — Nanbeige/Nanbeige4.2-3B
• Base model — Nanbeige/Nanbeige4.2-3B-Base
• Technical report — arXiv:2607.22083
• Nanbeige llama.cpp fork