Muse-Glimmer-30B-int4-AutoRound

int4 · W4A16 — 4-bit weights, 16-bit activations. Quantization of meta-models/Muse-Glimmer-30B produced with Intel AutoRound, packaged in compressed-tensors for vLLM. The W4A16 scheme (weights-only) keeps activations at BF16 → highest accuracy of the 4-bit variants and runs on any modern NVIDIA GPU (Ampere → Hopper → Blackwell, incl. DGX Spark). Text decoder is int4; vision tower + lm_head stay BF16. ~20.7 GB (vs ~60 GB BF16).

→ On a Blackwell card and want FP4 activation speed? Use the NVFP4 / W4A4 sibling.

Model overview

• Base model: meta-models/Muse-Glimmer-30B — a dense (Gemma2-derived) multimodal decoder + vision tower.
• Quantization: W4A16 (int4 weight-only, group size 128) on the text-decoder Linear layers; vision tower / adapter / projection / patch-embedder / lm_head kept BF16.
• Format: compressed-tensors (pack-quantized), auto-detected by vLLM.
• Quantizer: Intel AutoRound (arXiv:2309.05516).
• Intended use: efficient inference on any modern NVIDIA GPU (RTX 3090 / 4090 / 5090, A100, H100, RTX PRO 6000, DGX Spark). Weight-only int4 serves via the Marlin kernel: the 4-bit weights mean ~4× less weight data to stream from memory, which is the main win for single-user (memory-bandwidth-bound) inference — while 16-bit activations keep accuracy high and avoid any Blackwell-only FP4 activation path.

Quantization recipe

• Method: AutoRound, scheme W4A16 — 4-bit integer weights (group size 128), BF16 activations, symmetric.
• Quantized: all Linear in the 52 model.language_model.layers.* decoder blocks.
• Kept BF16: model.vision_tower.*, model.vision_adapter.*, model.vision_projection, patch_embedder, lm_head.
• Calibration: NeelNanda/pile-10k, 128 samples, seqlen 2048, 200 tuning iters.
• Cost: ~90 minutes on a single 96 GB Blackwell GPU (same recipe as the NVFP4 sibling).

Deployment (vLLM)

Recommended image: vllm/vllm-openai:muse-glimmer. vLLM auto-detects the quant scheme from config.json — no quantization flag needed.

vllm serve dbirks/Muse-Glimmer-30B-int4-AutoRound \
  --served-model-name muse-glimmer \
  --max-model-len 8192 \
  --enable-auto-tool-choice \
  --tool-call-parser muse_glimmer \
  --reasoning-parser muse_glimmer

Example compose.yaml

services:
  muse-glimmer:
    image: vllm/vllm-openai:muse-glimmer
    ports:
      - "8000:8000"
    ipc: host
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
    command:
      - "--model=dbirks/Muse-Glimmer-30B-int4-AutoRound"
      - "--served-model-name=muse-glimmer"
      - "--max-model-len=8192"
      - "--enable-auto-tool-choice"
      - "--tool-call-parser=muse_glimmer"
      - "--reasoning-parser=muse_glimmer"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

Note: Muse-Glimmer is a reasoning model — with the reasoning parser on, chain-of-thought comes back in the reasoning field and the final answer in content; give it enough max_tokens.

Evaluation — accuracy recovery vs BF16

Averaged over the OpenLLM-v1 suite (EleutherAI lm-evaluation-harness, in-process vllm backend), recovery = quant ÷ BF16 × 100.

Model	OpenLLM-v1 avg	Recovery
BF16 base	0.7295	100%
This (int4 · W4A16)	0.7191	98.5%

(Same setup, for reference: Red Hat GPTQ-NVFP4 = 98.4%; our NVFP4 W4A4 sibling = 97.1%. This W4A16 build is the highest-recovery 4-bit option, at the smallest size — 20.7 GB.)

Hardware & format notes

• Runs on any Ampere-or-newer NVIDIA GPU via the Marlin int4 kernel — no Blackwell required.
• Weight-only (W4A16): activations stay BF16, so maximal compatibility and accuracy, but no activation-quant speedup. For Blackwell FP4 (W4A4) speed, use the NVFP4 sibling.
• Only the text decoder is quantized; the vision tower stays BF16 (intentional, to preserve multimodal quality).

Reproducibility

from auto_round import AutoRound
ar = AutoRound(
    "meta-models/Muse-Glimmer-30B",
    scheme="W4A16", dataset="NeelNanda/pile-10k",
    nsamples=128, seqlen=2048, batch_size=4, iters=200,
    device_map=0, trust_remote_code=True, quant_nontext_module=False, seed=42,
)
ar.quantize_and_save(output_dir="Muse-Glimmer-30B-int4-AutoRound", format="llm_compressor")

Toolchain: auto-round 0.15.0, transformers 5.16.0.dev0 (from source — required for the muse_glimmer arch), compressed-tensors 0.17.0, torch 2.11.0+cu130.

Citation

@article{cheng2023optimize,
  title={Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs},
  author={Cheng, Wenhua and Zhang, Weiwei and Shen, Haihao and Cai, Yiyang and He, Xin and Lv, Kaokao and Liu, Yi},
  journal={arXiv preprint arXiv:2309.05516},
  year={2023}
}