dudeman2512/Muse-Glimmer-30B-NVFP4

Muse-Glimmer-30B — NVFP4 (compressed-tensors)

A 4-bit NVFP4 weight-only quantization of meta-models/Muse-Glimmer-30B, in compressed-tensors / nvfp4-pack-quantized format. 59.6 GB → 22 GB, which fits a single 32 GB consumer card (RTX 5090, sm_120) with room for full 131K context.

What is quantized

Component	Precision
LM attention q/k/v/o/gate_proj (52 layers)	NVFP4 (E2M1, 16-element blocks, fp8_e4m3 block scales + fp32 global scale)
LM MLP gate/up/down_proj (52 layers)	NVFP4
Vision tower / adapter / projection	bf16
embed_tokens, lm_head, all norms, all biases	bf16

416 language-model Linears quantized (52 layers × 8 modules, including the unusual self_attn.gate_proj); 1020 tensors passed through untouched. The perception encoder stays bf16, so image input is unaffected.

Scheme is exactly compressed_tensors' NVFP4A16 preset: num_bits=4, type=float, symmetric=True, group_size=16, strategy=tensor_group, dynamic=False.

Quality

Mean relative weight reconstruction error vs the original BF16, measured over sampled layers:

quant	rel. error	size	% of bf16
NVFP4 (this repo)	0.0947	22 GB	28.1%
INT4 W4A16 g128	0.1156	21 GB	25.8%

NVFP4 is both more accurate and only marginally larger than int4-g128 — the finer group size (16 vs 128) and E2M1's non-uniform spacing both help on Gaussian-ish weights. An INT4 build is published separately for kernel/back-end comparison.

Fit

GQA is 32:2 with head_dim=128, and only 13 of 52 layers are full-attention (the rest are sliding-window 2048). Full 131,072-token context costs roughly 1.8 GB of KV in bf16, so on a 32 GB card the ~8 GB left after weights supports full context plus concurrency.

How it was made

Produced by streaming the raw safetensors tensor-by-tensor and never instantiating the model, so peak RAM is about one tensor rather than the full 59.6 GB checkpoint. All quantization math comes from compressed_tensors itself (generate_gparam, calculate_qparams, NVFP4PackedCompressor).

A note on llm-compressor, since this is easy to get wrong: the PyPI wheel pins transformers<=5.14.1, which collides with the >=5.15.0 that muse_glimmer requires. That cap only applies when BUILD_TYPE=release, so installing llm-compressor from source leaves transformers unpinned and it works fine with 5.15.0. The reason a streaming script was used here is memory, not the pin: oneshot() instantiates the whole model, and this 59.6 GB checkpoint does not fit the host it was built on.

Implementation note worth knowing

weight_global_scale is stored 0-dim, not shape (1,). compressed_tensors' generate_gparam returns (1,), but vLLM's PerTensorScaleParameter fused path does param[shard_id].copy_(loaded), which raises

RuntimeError: output with shape [] doesn't match the broadcast shape [1]

on qkv_proj / gate_up_proj. A (1,)-shaped checkpoint crashes on load. If you build your own NVFP4 artifact, squeeze that scalar.

Verification

• 416/416 Linears present; 0 missing tensors, 0 stray zero-point/g_idx tensors.
• 0 shape/dtype conformance failures against what vLLM's loaders actually allocate.
• Byte-level structural match against an independently-produced third-party NVFP4 build of the same base model (2268 tensors, identical name sets, 0 dtype/shape mismatches).

Verified: fits a single 32 GB GPU

Loaded with transformers 5.15.0 on an RTX 5090 (sm_120):

[load] 3s   [placement] cuda:0=23.4GB

The whole checkpoint sits on one 32 GB card with roughly 8 GB left over, which is what makes full 131K context viable (KV is only ~1.8 GB, see above).

Gotcha if you try to run it in transformers

compressed-tensors installs a forward pre-hook that decompresses the entire model to bf16 on the first forward pass -- transformers has no NVFP4/INT4 compute kernels, those live in vLLM. So the model loads at 23.4 GB but needs ~60 GB the moment you run it, and you get:

torch.OutOfMemoryError: CUDA out of memory

Splitting across GPU+CPU with device_map="auto" does not rescue it: the decompress hook looks up the plain weight key, which offloaded modules do not have, and fails with KeyError: 'model.language_model.layers.0.self_attn.q_proj.weight'.

Practical consequence: use vLLM for inference (once #51655 lands). transformers is fine for loading and inspecting the checkpoint, but running it there dequantizes to bf16 and therefore tells you nothing about 4-bit kernel behaviour anyway.

Serving

⚠️ vLLM does not yet support muse_glimmer upstream. vllm-project/vllm#51655 is open, not merged. Until it lands you need to build from that branch:

git clone -b tiezhen/new-model-support https://github.com/xianbaoqian/vllm && cd vllm
VLLM_USE_PRECOMPILED=1 pip install -e .

vllm serve dudeman2512/Muse-Glimmer-30B-NVFP4 \
  --served-model-name muse --max-model-len 131072 \
  --reasoning-parser muse_glimmer --tool-call-parser muse_glimmer

The reasoning parser is required — the model uses channel-scoped output framing rather than <think> tags, and without it the channels collapse and output comes back empty.

End-to-end generation has not been validated, because that needs vLLM. What is verified: the checkpoint loads on a single 32 GB GPU (above), is structurally conformant to vLLM's loaders, and its weight reconstruction error is measured against the original BF16.

Caveats

• Built on sm_120 (RTX 5090). FP4 kernel behaviour differs on GB10 / sm_121 — test before committing on DGX Spark.
• Weight-only (W4A16). Activations stay in bf16.

License

Apache 2.0, inherited from the base model. LICENSE and USAGE_POLICY.md are included unmodified.