Muse-Glimmer-30B-FP8-dynamic

Quantized version of meta-models/Muse-Glimmer-30B, a dense multimodal (image-text-to-text) VLM.

• Scheme: FP8 (W8A8, float-quantized)
• Method: RTN, per-channel weight scales + dynamic per-token activation scales
• Calibration: none (data-free)
• What's quantized: the text-decoder Linear layers only. The vision tower, the multimodal connector, and lm_head are kept in bf16, and the full checkpoint (with vision_config) is saved so it loads on MuseGlimmerForConditionalGeneration.
• Hardware: Runs on Ada / Hopper (H100) and newer. Verified generating on H100.

This checkpoint was smoke-tested on an H100: it loads on the native multimodal loader and produces coherent text output.

Status: benchmarks pending

Accuracy benchmarks (recovery vs. the bf16 base) are not yet published. At the time of upload the available vLLM build did not register the muse_glimmer architecture, so a full lm-eval/lighteval suite could not be run. The recovery table will be added here once a muse_glimmer-capable serving stack is available.

Usage (transformers)

import torch
from transformers import AutoTokenizer, MuseGlimmerForConditionalGeneration

model_id = "soyrsoyr/Muse-Glimmer-30B-FP8-dynamic"
tok = AutoTokenizer.from_pretrained(model_id)
model = MuseGlimmerForConditionalGeneration.from_pretrained(
    model_id, dtype=torch.bfloat16, device_map="cuda"
)

msgs = [{"role": "user", "content": "What is the capital of France?"}]
inputs = tok.apply_chat_template(
    msgs, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)
out = model.generate(**inputs, max_new_tokens=400)
print(tok.decode(out[0][inputs["input_ids"].shape[1]:]))

Muse-Glimmer uses a Harmony/channel chat format: the assistant first emits a reasoning turn (to=self) and then the final answer (to=user). Give it enough max_new_tokens to reach the final channel.

Reproduction

Quantized with llm-compressor (transformers >= 5.15.0 for day-0 muse_glimmer support). Scripts and full notes: the muse-glimmer-30b quantization pipeline.

Requires transformers >= 5.15.0 to load (the arch shipped in that release).