Muse-Glimmer-30B-MLX-4bit

MLX (Apple Silicon) build of Muse-Glimmer-30B, quantized to 4-bit (group size 64).

These files are modified — the weights have been converted to MLX and quantized. The architecture is unchanged.

You need the runtime

muse_glimmer is carried by neither mlx-lm nor mlx-vlm, so mlx_vlm.load() cannot read this repository (or any other MLX build of this model — there are several, and none of them load with stock tooling):

>>> importlib.import_module("mlx_vlm.models.muse_glimmer")
ModuleNotFoundError

Use the port:

git clone https://github.com/PipeNetwork/muse-glimmer-mlx
cd muse-glimmer-mlx && pip install -r requirements.txt

from muse_glimmer_mlx.load import load
model = load("/path/to/Muse-Glimmer-30B-MLX-4bit")

That port is validated against transformers 5.15 — text stack to 2.0e-05, vision tower to 5.5e-06, vision geometry exactly — and it also loads every other published MLX conversion of this model unmodified.

Size

this build	21.4 GB
bfloat16	59.6 GB

Quantization covers the 52 text layers, lm_head, and the vision adapter and projection — 420 layers. The 50-layer vision tower (3.7 GB) and the embedding table (2.7 GB) stay bfloat16, as in every published build, so 6.4 GB of this file is unquantized and the bit width understates the download.

Quality

Measured against bfloat16 on ~98k tokens of wikitext-2, scored over identical windows with differences taken per window and a bootstrap over one shared index set. Perplexity varies far more between windows than between quants, so pairing is what makes the comparison readable at all.

build	size	perplexity	ΔNLL vs bf16 [95% CI]	windows worse
bfloat16	59.6 GB	6.3224	—	—
8-bit	34.6 GB	6.3224	−0.0000 [−0.0003, +0.0003]	25/48
4-bit	21.3 GB	6.5206	+0.0309 [+0.0285, +0.0334]	48/48
3-bit	18.1 GB	7.4400	+0.1628 [+0.1547, +0.1718]	48/48
2-bit (withheld)	14.7 GB	78.1019	+2.5139 [+2.4730, +2.5552]	48/48

Read the last column with the interval. 8-bit is worse on only 25 of 48 windows — a coin flip — and its interval straddles zero: it is statistically indistinguishable from bfloat16 here. Every other width is worse on every window, because quantization damage is systematic, so once it exists the paired test finds it everywhere. That also means 4-bit's +3.1% is unambiguous and small: significant is not the same as large.

The knee is 4 → 3 bits (5.3x the delta) and the cliff is 3 → 2 (15.4x).

What that looks like

Greedy decoding, same prompts, against bfloat16:

prompt	bfloat16	3-bit
The capital of France is	"Paris. It is the most populous city in France…"	fluent and on-topic, but never directly completes the statement — "a city of contrasts…"
Explain why the sky appears blue, in two sentences	Rayleigh scattering, two distinct sentences	correct physics, but repeats the same sentence as (1)(2)(3)
Write a Python function that reverses a linked list	def reverse_linked_list(head): then explains	restates the problem first, reaches class ListNode:

So: grammatical, factually largely intact, but weaker at following an instruction directly and prone to repetition loops that bfloat16 does not exhibit on the same prompts. Choose it to fit the model in ~18 GB, not for fidelity.

2-bit is not published. It was built and measured: perplexity 78.1, and it emits "Paris, not Paris. The rest of the world, the rest of the world…". It would also have saved only 3.3 GB over this build, because the unquantized tower and embeddings dominate either way.

License

Apache-2.0, as is the upstream model. Port code: https://github.com/PipeNetwork/muse-glimmer-mlx.