leonsarmiento/Nex-N2-mini-5bit-XL-mlx

This model was converted to MLX format from nex-agi/Nex-N2-mini using BaseQuant_XL 5/8-bit mixed quantization optimized for Apple Silicon. The vision encoder is preserved and quantized at 5-bit, making this a full multimodal model.

BaseQuant_XL keeps the most routing-critical layers in full bf16 precision — the MoE router gate, shared expert gate, shared expert, and lm_head — while applying aggressive quantization to the bulk parameters. This preserves routing accuracy and output quality where it matters most.

Nex-N2-mini is a 35B-parameter MoE (Mixture of Experts) model fine-tuned from Qwen3.5-35B-A3B-Base by Nex-AGI, featuring 256 experts (8 active per token + 1 shared expert), hybrid full + linear (Gated DeltaNet) attention, an "Agentic Thinking" framework (Adaptive Thinking + Coherent Thinking), and a vision encoder for multimodal input. Despite 35B total parameters, only ~3B are activated per token for efficient inference.

Intelligence Benchmarks (n=30 samples)

Benchmark	Nex-N2-mini XL (5-bit)	Nex-N2-mini (5-bit)	Delta
MMLU	60.0%	60.0%	—
MMLU_PRO	63.3%	56.7%	+6.6
HellaSwag	86.7%	83.3%	+3.4
TruthfulQA	96.7%	96.7%	—
ARC Challenge	80.0%	83.3%	-3.3
Winogrande	76.7%	73.3%	+3.4
MathQA	53.3%	53.3%	—
HumanEval	86.7%	90.0%	-3.3
MBPP	73.3%	76.7%	-3.4

XL shows clear gains on MMLU_PRO, HellaSwag, and Winogrande. Minor regressions on coding benchmarks (HumanEval, MBPP) are within sampling noise at n=30.

Use with mlx

pip install -U mlx-vlm

python -m mlx_vlm.generate --model leonsarmiento/Nex-N2-mini-5bit-XL-mlx --max-tokens 256 --temperature 0.7 --top-p 0.95 --prompt "Hello"

BaseQuant_XL Quantization Strategy

Bit Depth	Layers	Rationale
bf16 (unquantized)	mlp.gate (router), shared_expert_gate, lm_head, shared_expert	Routing decisions and shared computation path — errors here are qualitatively different from precision loss
8-bit	embed_tokens, self_attn (full attention), linear_attn (DeltaNet)	Every-token layers with moderate sensitivity — 8-bit is near-lossless
5-bit	vision_tower, switch_mlp (routed experts)	Bulk of parameters, only 8 of 256 experts active per token — natural redundancy tolerates lower precision

Quantization Details

Layer	Bits	Group Size
mlp.gate (router)	bf16	—
shared_expert_gate	bf16	—
lm_head	bf16	—
shared_expert	bf16	—
embed_tokens	8	64
self_attn (full attention)	8	64
linear_attn (DeltaNet)	8	64
vision_tower	5	64
switch_mlp (routed experts)	5	64
Default fallback	8	64

• Quantization type: BaseQuant_XL mixed (multimodal, vision preserved)
• Bits per weight: 5.881
• Total size: ~24 GB (5 shards)
• Group size: 64
• Method: Custom quant_predicate via mlx_vlm

Recommended Inference Parameters

Parameter	Value
temperature	0.7
top_p	0.95
top_k	40
min_p	0.01
repeat_penalty	1.05

Note: This is a Qwen3.5-based model — preserve_thinking is not applicable.