majentik/Nemotron-3-Nano-30B-A3B-RotorQuant-MLX-2bit

KV-cache quantization without any fork (recommended, 2026): upstream llama.cpp/Ollama now cover this natively — use -ctk q8_0 -ctv q8_0 (half KV memory, negligible quality loss: perplexity +0.002–0.05) or -ctk q4_0 -ctv q4_0 (quarter memory, ≈7.6% perplexity increase). In Ollama: OLLAMA_KV_CACHE_TYPE=q8_0 with OLLAMA_FLASH_ATTENTION=1. Keep K and V types symmetric to stay on the fast fused Flash-Attention path. Since April 2026, mainline llama.cpp also applies Hadamard rotation to KV activations (PR #21038), which greatly improves low-bit KV quality (opt-out: LLAMA_ATTN_ROT_DISABLE=1).

The RotorQuant/TurboQuant fork flow below is experimental/legacy: the TurboQuant llama.cpp PR was closed without merging (June 2026) and the fork is unmaintained relative to mainline. It is NOT required to use this model.

Nemotron-3-Nano-30B-A3B - RotorQuant MLX 2-bit

2-bit weight-quantized MLX version of nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 with RotorQuant KV-cache quantization. Optimized for Apple Silicon inference via the MLX framework. RotorQuant delivers 5.3x faster prefill and 28% faster decode compared to TurboQuant. Maximum compression for running on memory-constrained devices. Only 3.2B parameters are active per token despite 30.7B total, making this model significantly more efficient at inference time than its parameter count suggests. The hybrid Mamba-2 + Transformer MoE architecture supports up to 1M context length.

Approximate model size: ~9 GB

Model Specifications

Property	Value
Base Model	nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16
Parameters	30.7 billion total (3.2 billion active per token)
Architecture	Hybrid Mamba-2 + Transformer MoE (3.2B active per token)
Context Length	1,048,576 tokens (1M)
License	NVIDIA Open Model License (commercial use OK)
Weight Quantization	2-bit (~9 GB)
KV-Cache Quantization	RotorQuant
Framework	MLX (Apple Silicon)

Quickstart

from mlx_lm import load, generate
from rotorquant import IsoQuantCache

model, tokenizer = load("majentik/Nemotron-3-Nano-30B-A3B-RotorQuant-MLX-2bit")

prompt = "Explain the theory of relativity."
response = generate(model, tokenizer, prompt=prompt, max_tokens=512)
print(response)

What is RotorQuant?

RotorQuant applies block-diagonal rotations (Clifford algebra) for KV cache compression. Combined with 2-bit weight quantization in MLX, this provides a dual compression strategy with superior KV-cache performance: smaller model weights plus faster compressed KV cache for efficient long-context generation.

Key advantages over TurboQuant:

• 5.3x faster prefill
• 28% faster decode
• Equivalent memory savings

KV-Cache Quantization Comparison

Method	Prefill Speed	Decode Speed	Memory Savings	Reference
TurboQuant	1x (baseline)	1x (baseline)	High	arXiv: 2504.19874
RotorQuant	5.3x faster	28% faster	High	GitHub

Memory Estimates (Nemotron-3-Nano-30B-A3B)

Precision	Approximate Size	MLX Variant
BF16 (original)	~60 GB	--
8-bit quantized	~30 GB	RotorQuant-MLX-8bit
4-bit quantized	~17 GB	RotorQuant-MLX-4bit
2-bit quantized	~9 GB	This model

Hardware Requirements

This model requires approximately 9 GB of unified memory. Recommended hardware:

• Apple M1 (16 GB+)
• Apple M2 (16 GB+)
• Apple M3 (16 GB+)
• Apple M4 (16 GB+)
• Any Apple Silicon Mac with 16 GB+ unified memory

See Also

• nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 -- Base model
• majentik/Nemotron-3-Nano-30B-A3B-RotorQuant -- RotorQuant KV-cache only (transformers)
• majentik/Nemotron-3-Nano-30B-A3B-RotorQuant-MLX-8bit -- MLX 8-bit variant
• majentik/Nemotron-3-Nano-30B-A3B-RotorQuant-MLX-4bit -- MLX 4-bit variant
• majentik/Nemotron-3-Nano-30B-A3B-TurboQuant-MLX-2bit -- TurboQuant MLX 2-bit variant
• RotorQuant GitHub
• MLX Framework