Laguna-S-2.1-oQ3e-fast

Speed-tuned variant of Laguna-S-2.1-oQ3e, the calibrated 3-bit MLX quantization of poolside/Laguna-S-2.1 (118B total, 8B activated per token). Same oQ level 3 enhanced, same 3-bit experts — the one thing I changed is the budget: instead of letting oQ spend freely on the dense spine, I capped it at target_bpw=3.50, and oQ dropped most of the attention to the 3-bit base on its own. 3.56 bits/weight effective, 49 GB on disk. For Apple Silicon.

• 49 GB on disk — essentially the same as the oQ3e it derives from; the fast trick trades bandwidth, not size, and its disk saving shrinks as the base bit-width rises
• 48 layers, 47 of them MoE with 256 routed experts + 1 shared, top-10 (L0 is a dense MLP); interleaved attention (12 global with YaRN to 1M context, 36 sliding-window 512)
• Peak memory in my tests: 46.3 GB at 1k context, 49.3 GB at 64k — fits a 64 GB Mac
• 77.2 tok/s at 1k context against 67.5 for the oQ3e, at the same accuracy within noise
• Converted and tested on a Macbook Pro M5 Max 128GB 40 GPU

What changed against the oQ3e

Only the bit budget. oQ derives bit-width per tensor from an imatrix-calibrated sensitivity pass; at level 3 enhanced it lands on 3-bit experts and spreads the attention across 4/5/6/8 bits. Attention is ~4% of the file but ~37% of the bytes read per token — the dense spine is read in full on every token while only 10 of 256 experts are touched — so it's the cheapest place on disk to buy back bandwidth.

Passing target_bpw=3.50 (with hard_cap_bpw=3.56) tells the allocator to hit a lower budget, and it picks where to take the loss. It kept the shared expert, embeddings and lm_head at 8 bits and dropped most of the attention to the 3-bit base, holding q/k/v of the global-attention layers at 6–8. Those are the layers that carry the long-context path; the 36 sliding-window layers only see 512 tokens.

	oQ3e	this build
routed experts	3b gs128	3b gs128 (re-calibrated)
shared expert, embeddings, lm_head	8b	8b
attention q/k/v	4b ×81, 5b ×27, 6b ×14, 8b ×22	3b ×108, 5b ×6, 6b ×12, 8b ×18
attention o/g	4b ×54, 5b ×32, 8b ×10	3b ×75, 5b ×11, 8b ×10

Requirements

mlx-lm doesn't support the laguna architecture yet — there's an open PR: mlx-lm#1223. Until it lands, use mlx-vlm (0.6.3+), which implements laguna as a text-only model:

uvx --from mlx-vlm mlx_vlm.generate --model mlx-community/Laguna-S-2.1-oQ3e-fast --prompt "..."

oMLX serves it directly from 0.5.3 on — it vendors the PR and patches it into mlx-lm at import, so no model setting is needed. On older builds that predate the patch, discovery lands on mlx-lm and fails with Model type laguna not supported; set model_type_override: "vlm" and refresh discovery (omlx restart).

How it was built

omlx.oq.quantize_oq_streaming at oq_level=3, enhanced=True, group_size=128, dtype bfloat16, with target_bpw=3.50 and hard_cap_bpw=3.56. The ladder's cached imatrix was no longer on disk, so I re-collected one (128 samples, same oqe_code_multilingual calibration); the experts are the same 3-bit as the oQ3e, re-calibrated, and the accuracy landed on top of it regardless (table below). Sensitivity was measured against a uniform 4-bit proxy on disk — the 235 GB FP model doesn't fit in 128 GB of RAM. Output is standard MLX affine quantization: no custom kernels or runtime required.

Unlike the upstream config, generation_config.json here ships repetition_penalty: 1.05: at this bit-width the model can fall into verbatim repetition loops in long-form generation, and this is the mildest setting that reliably broke them in my tests.

Why it's faster

Generation is memory-bandwidth bound, and what matters is bytes read per token, not bytes on disk. The oQ3e reads 4.65 GB per token, this build reads 4.22 — the difference is the attention dropping from a 4/5/6/8 mix to the 3-bit base. The byte count alone predicts ~10% at 1k; measured is 14%, rising to 21% at 64k as attention's share of per-token work grows with context. Disk and peak memory barely move — this is a strictly-faster oQ3e at the same size and accuracy.

Conversion check

Smoke-tested after conversion with mlx_vlm.generate: coherent — solved 17 * 24 = 408, broke it down by the distributive property and verified the result a second way, no repetition loop, at 73 tok/s.

Performance

Measured with oMLX's benchmark harness on a Macbook Pro M5 Max 128GB 40 GPU, single request, 128 generated tokens:

prompt	gen tok/s	prefill tok/s	TTFT ms	peak GB
1k	77.2	981.7	1044	46.27
4k	75.6	1114.5	3676	46.41
8k	72.8	998.5	8205	46.58
16k	66.5	914.7	17913	46.97
32k	59.7	842.6	38892	47.72
64k	48.4	753.3	87005	49.33

Continuous batching at 1k prompt / 128 generated:

batch	tg tok/s	speedup	TTFT ms	E2E s
1	77.2	1.00x	1044	2.71
2	101.9	1.32x	1868	4.38
4	133.2	1.73x	3174	7.10
8	158.8	2.06x	4628	12.37

Benchmarks & Variants

mmlu_pro, mathqa and winogrande, n=300 seeded samples each, thinking off. This build was measured on the oMLX LM engine directly (greedy, mathqa/winogrande at 1024 max tokens so reasoning isn't truncated). Standard error at this n is around 2.5 points, so the oQ3e-fast and the oQ3e are indistinguishable here.

Variant	Size	bpw	gen tok/s (1k → 64k)	mmlu_pro	mathqa	winogrande
Laguna-S-2.1-oQ2e-fast	35 GB	2.60	78.8 → 48.6	0.700	0.850	0.713
Laguna-S-2.1-oQ2e	36 GB	2.70	61.5 → 38.8	0.703	0.840	0.707
Laguna-S-2.1-oQ3e-fast (this repo)	49 GB	3.56	77.2 → 48.4	0.750	0.887	0.760
Laguna-S-2.1-oQ3e	49 GB	3.59	67.5 → 40.1	0.750	0.880	0.760
Laguna-S-2.1-oQ4e-fast	63 GB	4.54	69.3 → 45.5	0.787	0.873	0.777
Laguna-S-2.1-oQ4e	64 GB	4.60	55.9 → 39.8	0.757	0.887	0.777
Laguna-S-2.1-oQ5e	78 GB	5.30	57.5 → 38.1	0.773	0.883	0.797
Laguna-S-2.1-oQ6e	92 GB	6.27	53.0 → 32.9	0.763	0.873	0.777
Laguna S 2.1 (API, bf16)	—	16	—	0.773	0.880	0.810

Treat this as a rough sighting, not a verdict. Three benchmarks at n=300 cover a narrow slice of what the model does — no long-context work, no agentic loops, no real code — and at this sample size most of the ladder above 3.6 bpw sits inside the error bars. I ran them to size the drop between levels, not to rank the variants against each other. Test the one you're considering on your own workload before trusting any of it.

Pick this one over the oQ3e whenever you'd have reached for the oQ3e — it's the same size, the same accuracy, and 14–21% faster. Pick oQ4e or above if accuracy matters more than throughput.

Usage

# mlx-vlm — plain mlx-lm doesn't support the laguna architecture
uvx --from mlx-vlm mlx_vlm.generate --model mlx-community/Laguna-S-2.1-oQ3e-fast \
  --prompt "Explain Bayes' theorem in two sentences." --max-tokens 300

# oMLX — discovers the model from the HF cache
omlx serve

License

OpenMDW-1.1, inherited from the base model. Refer to the original model card for architecture, benchmarks, and intended use.