scratchery/gpt-oss-120b-3bit

gpt-oss-120b — MLX 3-bit (mixed-width affine)

openai/gpt-oss-120b converted to MLX with a mixed-width affine recipe: low bits only where the parameters actually live, wide bits where gpt-oss is fragile.

module	quant	group size
mlp.experts gate/up/down (98% of parameters)	3-bit affine	64
self_attn q/k/v/o	4-bit affine	64
mlp.router	8-bit affine	64
embed_tokens, lm_head	4-bit affine	64
norms, attention sinks, all linear biases	bf16 (unquantized)	—

~48 GiB on disk / in memory. 10 safetensors shards; config.json carries the full per-module quantization map.

Why mixed widths

Uniform low-bit gpt-oss releases degrade badly. Two controlled ablations (same converter, same source, one variable at a time):

• Attention is the fragile part. Attention projections at 2-bit alone collapse output into repetition, even with everything else at 4-bit.
• At 120b, 3 is the floor for the experts. A 2-bit-experts leg of this exact recipe produces incoherent token soup at 120b (128 experts), while the same recipe is coherent at 20b (32 experts) — the 128-expert pool is four times more diluted per expert. The 3-bit leg produces clean harmony reasoning out of the box.
• The router stays at 8-bit: top-4 routing over 128 experts is knife-edge-sensitive (the measured 4th-vs-5th logit gap at 120b has median 0.039), so narrowing it flips expert selection.

This is the same conclusion mlx_lm's own --quant-predicate mixed_* modes reach: squeeze the experts, keep attention and router wide.

Usage

from mlx_lm import load, generate

model, tokenizer = load("scratchery/gpt-oss-120b-3bit")

The harmony chat template is included (chat_template.jinja); use the tokenizer's apply_chat_template and the analysis/final channels render correctly. Needs ~48 GB of unified memory.

Provenance

Converted from the official openai/gpt-oss-120b release (MXFP4 experts at group 32, bf16 elsewhere) with mlx_lm + per-tensor mlx.core.quantize for the expert stacks (QuantizedSwitchLinear has no to_quantized, so the stock nn.quantize never touches them). Validated with harmony-templated greedy probes: coherent analysis-channel reasoning on short and long prompts.

License: Apache-2.0, inherited from the base model.