srv-sngh/Nanbeige4.2-3B-mlx-nvfp4

Nanbeige4.2-3B — MLX NVFP4

Nanbeige4.2-3B quantized to NVFP4 (4-bit float, group size 16) in MLX format, for Krill on Apple Silicon.

Base	Nanbeige/Nanbeige4.2-3B (bf16, 8.3 GB)
This build	NVFP4, 4-bit / group 16 (2.26 GB)
Quantized tensors	156 (22 layers x 7, plus embed_tokens and lm_head)
Context	262,144

Quantized with Krill's native Swift+MLX checkpoint quantizer (krill quantize --mode nvfp4) — no Python, no mlx_lm.convert.

⚠️ Runtime support: Krill only

This build runs in Krill and nowhere else. That is not a packaging choice — as of this writing no other local inference engine implements the Nanbeige looped architecture:

Runtime	Status
Krill ≥ 0.17.0	✅ native Swift+MLX runtime
llama.cpp / Ollama / LM Studio	❌ unknown model architecture: 'nanbeige'
mlx-lm (0.31.3)	❌ no nanbeige model module
vLLM / SGLang	❌ upstream ships forks, not mainline support
transformers	✅ but only via trust_remote_code on the original bf16 repo — it cannot read these nvfp4-packed tensors

GGUF conversions of Nanbeige 4.2 exist on the Hub, but they do not load: llama.cpp rejects the architecture before it reads any weights. If you are not using Krill, use the original Nanbeige/Nanbeige4.2-3B with transformers.

Performance

Measured on Apple Silicon, 512-token prompt, same machine. The baseline is the upstream PyTorch reference (modeling_nanbeige.py, fp16 on MPS) because it is the only other runtime that can serve this model at all — an Ollama comparison is impossible, not merely absent.

	Prefill	Decode	Load	Peak memory
Krill 0.17.0, this nvfp4 build	294.6 tok/s	37.5 tok/s	0.15 s	1.7 GB
PyTorch reference, fp16 (MPS)	204.0 tok/s	9.7 tok/s	10.2 s	~8 GB
speed-up	1.4×	3.9×	68×	~4.7× smaller

For scale, a dense qwen2.5-3b 4-bit in the same Krill build does 508.5 tok/s prefill / 49.8 tok/s decode at 1.5 GB. Nanbeige decodes ~25% slower at a comparable footprint — that is the loop being paid for: 44 effective layers out of 22 blocks' worth of weights.

Usage

krill pull srv-sngh/Nanbeige4.2-3B-mlx-nvfp4
krill run Nanbeige4.2-3B-mlx-nvfp4 "Which number is bigger, 9.11 or 9.8?"

Nanbeige 4.2 is a reasoning model: its chat template opens an assistant turn inside a <think> block, so the model spends tokens reasoning before it emits any visible answer. Give it room — a few hundred tokens is often not enough to close the block, and you will see an empty reply. --max-tokens 4000 or more is a sensible floor for anything non-trivial.

Upstream's recommended sampling settings (from generation_config.json): temperature 0.6, top_p 0.95, top_k 20 for reasoning and chat; temperature 1.0 for agentic and tool-use work.

About the architecture

This is not a plain Llama clone, and the quantized layout reflects that:

• Looped transformer. num_loops: 2 runs the whole 22-block stack twice over the same weights — 44 effective layer executions from 22 blocks' worth of parameters. Each execution keeps its own KV cache slot, so a runtime must allocate num_loops * num_hidden_layers = 44 caches, not 22. The final model.norm is applied at the end of every loop.
• head_dim decoupled from hidden_size. 48 heads x 128 dims against a 3072-wide residual, so q_proj is 3072 -> 6144 and o_proj 6144 -> 3072. A loader that derives hidden_size / num_attention_heads gets 64 and mis-shapes every attention projection.

The optional research features described in upstream's configuration_nanbeige.py (multi-head hyper-connections with depth attention, concatenated n-gram embeddings, LoopSplit, loop-shared KV) are off in this checkpoint and its weight map contains none of their tensors.

Verification

The native Krill runtime backing this build is logit-parity gated against upstream's own modeling_nanbeige.py (mlx-lm has no nanbeige port), on both a synthetic fp32 fixture and these real weights — prefill and incremental cached decode, matching argmax and cosine > 0.9999. A live smoke gate additionally drives this checkpoint through the whole serving path (prompt → looped prefill → 44-cache decode → reasoning filter → detokenizer), and tool calling is verified end to end.

Reproducing the reference? NanbeigeRotaryEmbedding registers inv_freq with persistent=False, so it is absent from the checkpoint. Under transformers 5.x, from_pretrained materializes the model from checkpoint tensors and that buffer is left zero-initialized on every layer — RoPE then applies no rotation at all and the reference emits positionless gibberish. Recompute inv_freq after loading before trusting any comparison. The shipped config.json also has rope_scaling: null, which transformers 5.x repopulates, tripping upstream's _init_rope on a missing "type" key.

License

Apache 2.0, inherited from the base model. All credit for the model itself goes to the Nanbeige team at BOSS Zhipin.