mosesman/Nanbeige4.1-3B-openvino-int4

Nanbeige4.1-3B - OpenVINO INT4 (runs on NPU / iGPU / CPU)

OpenVINO IR export of Nanbeige/Nanbeige4.1-3B quantized with the NPU-correct recipe used across the mosesman/LFM2.5-2.6B-openvino-int4-npu family:

--weight-format int4 --sym --group-size 128 --backup-precision int8_sym

One OpenVINO IR file runs on NPU, GPU, or CPU - the device is a string at pipeline init (device="NPU" / "GPU" / "CPU").

Role in the Lex NPU Runtime dual-device suite:

• LFM2.5-2.6B -> NPU (base agent, pinned)
• Nanbeige4.1-3B -> iGPU (coding/reasoning specialist, swap pool)

So despite the neutral repo name: this model is the iGPU resident in the suite, while LFM2.5 owns the NPU.

Why 4.1 and not 4.2

Nanbeige4.2-3B uses a custom nanbeige architecture that has no OpenVINO export config in optimum-intel (and needs a fork of llama.cpp). The 4.1 line is standard LlamaForCausalLM, so it exports cleanly and stays compatible with every OpenVINO release. For an edge runtime that beginners install, standard-arch compatibility beats the newest checkpoint.

Export	Arch	OpenVINO
Nanbeige4.2-3B	custom nanbeige	unsupported (optimum-intel error)
This repo (4.1-3B)	LlamaForCausalLM	exportable + validated

Quantization stats

Weight compression mode	% all parameters
int4_sym, group size 128	78% (224/226 layers, 100% of ratio-defining)
int8_sym, per-channel (backup)	22% (2/226 layers: embeddings + LM head)

Quick start (OpenVINO GenAI)

pip install openvino openvino-genai transformers

import openvino_genai as ov_genai
from transformers import AutoTokenizer

model_id = "mosesman/Nanbeige4.1-3B-openvino-int4"
device = "GPU"  # suite role: iGPU specialist. Also works on "NPU" / "CPU"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Write a Python function that returns the sum of a list."}],
    tokenize=False,
    add_generation_prompt=True,
)

pipe = ov_genai.LLMPipeline(model_id, device)
config = ov_genai.GenerationConfig()
config.max_new_tokens = 128
config.do_sample = False
print(pipe.generate(prompt, config))

Note: Nanbeige4.1 is a reasoning model - it emits a <think> block before answering. Keep add_generation_prompt=True so the model knows when to reason.

Reproduce the export

optimum-cli export openvino \
  --model Nanbeige/Nanbeige4.1-3B \
  --trust-remote-code \
  --weight-format int4 \
  --sym \
  --group-size 128 \
  --backup-precision int8_sym \
  ./Nanbeige4.1-3B-openvino-int4-npu

Tooling: optimum-intel 2.1.0 / optimum 2.3.0, nncf 3.3.0, openvino 2026.3.0, transformers 5.4.0.

Files

OpenVINO GenAI layout:

• openvino_model.xml / .bin (2.4GB)
• openvino_tokenizer.xml / .bin, openvino_detokenizer.xml / .bin
• tokenizer + chat template configs

License / attribution

Quantized derivative of Nanbeige/Nanbeige4.1-3B under Apache-2.0. See LICENSE and NOTICE.

Disclaimer

Provided as-is for interoperability with Intel NPU / OpenVINO. Not affiliated with Nanbeige or Intel. Validate quality for your own workload before production use.