Instructions to use owao/Nanbeige4.2-3B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use owao/Nanbeige4.2-3B-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="owao/Nanbeige4.2-3B-GGUF",
	filename="nanbeige4.2-3b-BF16.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

llama-cpp-python

How to use owao/Nanbeige4.2-3B-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="owao/Nanbeige4.2-3B-GGUF",
	filename="nanbeige4.2-3b-BF16.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use owao/Nanbeige4.2-3B-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf owao/Nanbeige4.2-3B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf owao/Nanbeige4.2-3B-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf owao/Nanbeige4.2-3B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf owao/Nanbeige4.2-3B-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf owao/Nanbeige4.2-3B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf owao/Nanbeige4.2-3B-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf owao/Nanbeige4.2-3B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf owao/Nanbeige4.2-3B-GGUF:Q4_K_M

Use Docker

docker model run hf.co/owao/Nanbeige4.2-3B-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use owao/Nanbeige4.2-3B-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "owao/Nanbeige4.2-3B-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "owao/Nanbeige4.2-3B-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/owao/Nanbeige4.2-3B-GGUF:Q4_K_M

Ollama
How to use owao/Nanbeige4.2-3B-GGUF with Ollama:
```
ollama run hf.co/owao/Nanbeige4.2-3B-GGUF:Q4_K_M
```

Unsloth Studio

How to use owao/Nanbeige4.2-3B-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for owao/Nanbeige4.2-3B-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for owao/Nanbeige4.2-3B-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for owao/Nanbeige4.2-3B-GGUF to start chatting

How to use owao/Nanbeige4.2-3B-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf owao/Nanbeige4.2-3B-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "owao/Nanbeige4.2-3B-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use owao/Nanbeige4.2-3B-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf owao/Nanbeige4.2-3B-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default owao/Nanbeige4.2-3B-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat new

OpenClaw new

How to use owao/Nanbeige4.2-3B-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf owao/Nanbeige4.2-3B-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "owao/Nanbeige4.2-3B-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use owao/Nanbeige4.2-3B-GGUF with Docker Model Runner:
```
docker model run hf.co/owao/Nanbeige4.2-3B-GGUF:Q4_K_M
```

Lemonade

How to use owao/Nanbeige4.2-3B-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull owao/Nanbeige4.2-3B-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Nanbeige4.2-3B-GGUF-Q4_K_M

List all available models

lemonade list

Nanbeige4.2-3B GGUF Quantizations

GGUF quantized versions of Nanbeige4.2-3B.

Original Model Card

See the original model card for details on how to use with llama.cpp and ollama (you'll need to build from their fork cause their changes didn't land in llama.cpp main yet).

Why Ollama is Bad 👎 (needs to be put down somewhere)

@ngxson - https://github.com/ollama/ollama/issues/11714#issuecomment-3174632621

"Some of llama.cpp maintainers even have to work during their vacations just to have someone else copy their work without giving any credits."
@mudler - https://github.com/ollama/ollama/issues/11714#issuecomment-3175288625

"it would have been much better if all projects that depend on @ggerganov's and the ggml team work would have upstreamed the contributions directly so anyone in the ecosystem could benefit, and avoid vendor lock-in and the duplicated efforts everywhere... consuming llama.cpp and reporting issues and upstream any change directly there. It is quite frustrating to see that the Open source scene is really getting derailed lately by this kind of bad attitude."
@pwilkin - https://github.com/ollama/ollama/issues/11714#issuecomment-3175999505

"If you build upon a technology, in the OSS world it's a good habit to actually contribute back to the technology you use if you build something new... But Ollama has, again and again, done the opposite of that - made hacky solutions of their own on top of existing llama.cpp / ggml code instead of contributing to the baseline, then taken the fixes that the ggml team has done as 'new features' or 'bugfixes' of their own platform."
@Teravus - https://github.com/ollama/ollama/issues/11714#issuecomment-3176339445

"Ollama, for sure, needs to provide something to the user that says that they're using code from llama.cpp. Usually, this is in an about box. I don't even see an about box. Therefore, doesn't look like they're complying with that... The only mention that I see of llama.cpp isn't really a 'we use this software' reference. It's just: Supported backends llama.cpp project founded by Georgi Gerganov. This doesn't seem like enough. Skirting the issue, by treating llama.cpp like a back-end."
@Ggerganov - https://github.com/ggml-org/llama.cpp/pull/19324#issuecomment-3847213274

"it's quite funny watching the ollama bros copy-pasting our bugs into their "new engine" 🤣. Let's see how long it will take them to realize."
@Ggerganov - https://github.com/ollama/ollama/issues/11714#issuecomment-3172893576

Before the model was released, the ollama devs decided to fork the ggml inference engine in order to implement gpt-oss support (#11672). In the process, they did not coordinate the changes with the upstream maintainers of ggml. As a result, the ollama implementation is not only incompatible with the vast majority of gpt-oss GGUFs that everyone else uses, but is also significantly slower and unoptimized. On the bright side, they were able to announce day-1 support for gpt-oss and get featured in the major announcements on the release day.

Now after the model has been released, the blogs and marketing posts have circled the internet and the dust has settled, it's time for ollama to throw out their ggml fork and copy the upstream implementation (#11823). For a few days, you will struggle and wonder why none of the GGUFs work, wasting your time to figure out what is going on, without any help or even with some wrong information. But none of this matters, because soon the upstream version of ggml will be merged and ollama will once again be fast and compatible."

But hopefully here is an alternative 🤗 https://github.com/mostlygeek/llama-swap

with a happy user example: https://huggingface.co/unsloth/gpt-oss-20b-GGUF/discussions/17#68aa6eb05372ae5a8eeac9ed

Citation

@misc{nanbeige2026,
  title        = {Nanbeige4.2-3B},
  author       = {Nanbeige Team},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/Nanbeige/Nanbeige4.2-3B}}
}

Downloads last month: 4,775

GGUF

Model size

4B params

Architecture

nanbeige

Hardware compatibility

3-bit

4-bit

5-bit

6-bit

8-bit

16-bit

Model tree for owao/Nanbeige4.2-3B-GGUF

Base model

Nanbeige/Nanbeige4.2-3B-Base

Finetuned

Nanbeige/Nanbeige4.2-3B

Quantized

(21)

this model