mlx-community/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-OptiQ-4bit

Built with mlx-optiq, the MLX-native toolkit to quantize, fine-tune, and serve LLMs locally on Apple Silicon, no PyTorch and no cloud. All OptiQ quants · Docs

NVIDIA's Mamba-attention hybrid MoE, running locally on a Mac. This is an OptiQ mixed-precision quant of nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16. 22.8 GB on disk, down from 65.8 GB.

Lightning is a nemotron_h hybrid: Mamba2 state-space blocks with a handful of attention layers, and a 128-expert MoE with 6 experts active per token. The routed experts carry most of the weight, so that is where the bits come from.

What it is

Property	Value
Base	nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 (~30B total, ~3B active)
Method	OptiQ mixed-precision, sensitivity-driven (uniform-4-bit reference)
Precision	per-layer 4/8-bit: 36 layers at 4-bit, 127 at 8-bit
Experts	46 fused expert tensors: 36 at 4-bit, 10 at 8-bit
On disk	22.8 GB
Speculative decoding	MTP head preserved in optiq/mtp.safetensors

All 164 quantizable projections were measured. Every 4-bit assignment in the model is an expert tensor: the 128-expert stack is where the size is, and the sweep found it is also where the precision is least needed.

Following llama.cpp's naming for mixed quants, the "4bit" label denotes the family, not the weighted average.

Capability Score

Six-metric mean (the standard OptiQ text eval). Strong code generation; long-context multi-hop retrieval is the weak spot, which is what a hybrid with few attention layers would predict.

Metric	Score
MMLU (5-shot, 969 samples)	78.9%
GSM8K (1000 samples)	85.5%
IFEval (full set, strict)	68.4%
BFCL-V3 simple (200 calls)	83.0%
HumanEval (164 problems, pass@1)	87.8%
HashHop (long-context retrieval)	40.0%
Capability Score (mean of 6)	73.94

Run it

pip install "mlx-optiq>=0.4.20"

import optiq  # registers the nemotron_h MoE + MTP sidecar
from mlx_lm import load, generate

model, tok = load("mlx-community/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-OptiQ-4bit")
msgs = [{"role": "user", "content": "Explain why the sky is blue."}]
prompt = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
print(generate(model, tok, prompt=prompt, max_tokens=600))

It is a reasoning model and emits a <think> block before its answer, so give it room and read the text after </think>.

Serve it on a smaller Mac

At 22.8 GB this does not fit resident on a 24 GB machine. It is a 128-expert MoE, so stream the experts from SSD instead:

optiq serve --model mlx-community/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-OptiQ-4bit --stream-experts

Measured on a 26 GB M4 Pro: 4.2 GB resident, 6.0 GB load peak, against a resident load that runs out of memory. The same policy is available from the library API via OptiqEngine(path, stream_experts="auto"), and from optiq eval --stream-experts for benchmarking.

Links

• Project website: mlx-optiq.com
• Family guide: mlx-optiq.com/docs/nemotron3
• All OptiQ quants: mlx-optiq.com/models
• Base model: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16