mph/muse-glimmer-30b-mxfp8

Muse Glimmer 30B (MXFP8)

MXFP8-quantized weights for meta-models/Muse-Glimmer-30B, a multimodal vision–language model. The text decoder backbone is stored in MXFP8; vision components and the language-model head remain bfloat16.

Quantization details

This checkpoint was produced with TorchAO using MXDynamicActivationMXWeightConfig:

Component	Precision
model.language_model (text decoder Linear layers)	MXFP8 weights; activations quantized dynamically at inference
vision_tower, vision_adapter, vision_projection, perception_emb_norm, lm_head	bfloat16

• Format: torchao-flattened safetensors (MXTensor qdata/scale + metadata)
• Block size: 32
• Dtypes: float8_e4m3fn for weights and activations
• Scaling: RCEIL
• Base dtype: bfloat16

Weights were quantized once on GPU, exported to CPU, flattened with flatten_tensor_state_dict, and saved with a TorchAoConfig in config.json. Reload does not re-run weight quantization; the language model still applies dynamic activation quantization during forward passes.

Hardware requirements

MXFP8 inference requires a Blackwell-class NVIDIA GPU (compute capability SM100+, i.e. major version ≥ 10). Examples include B200, GB200, and RTX Pro 6000. Older architectures (Ampere, Hopper, etc.) are not supported for this checkpoint.

• CUDA GPU with SM100+
• Sufficient VRAM for a 30B multimodal model (33 GB weights on disk; peak usage depends on sequence length and vision inputs)

Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True if you hit fragmentation during load or generation.

Software requirements

pip install "transformers>=5.5.4" torch torchao safetensors

Tested with Transformers 5.15.0. You need a recent torchao build with MXFP8 inference support.

The processor and tokenizer are not bundled here; load them from the base model:

from transformers import AutoProcessor, MuseGlimmerForConditionalGeneration
import torch

BASE_MODEL_ID = "meta-models/Muse-Glimmer-30B"
QUANTIZED_MODEL = "YOUR_USERNAME/muse-glimmer-30b-mxfp8"  # or local path

processor = AutoProcessor.from_pretrained(BASE_MODEL_ID)
model = MuseGlimmerForConditionalGeneration.from_pretrained(
    QUANTIZED_MODEL,
    torch_dtype=torch.bfloat16,
)
model.to("cuda")
model.eval()

Usage

Text-only

messages = [
    {"role": "user", "content": "Explain MXFP8 in one sentence."},
]
inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    add_generation_prompt=True,
    reasoning_strength="low",
)
inputs = inputs.to(model.device)

with torch.inference_mode():
    output_ids = model.generate(**inputs, max_new_tokens=128, do_sample=False)

response = processor.decode(
    output_ids[0, inputs["input_ids"].shape[-1]:],
    skip_special_tokens=False,
)
print(response)

Image + text

from PIL import Image

image = Image.open("example.png").convert("RGB")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": image},
            {"type": "text", "text": "What is shown in this image?"},
        ],
    }
]
# Same apply_chat_template → generate → decode flow as above.

Files

File	Description
model.safetensors	Quantized weights (~33 GB)
config.json	Model config + quantization_config (TorchAoConfig)
generation_config.json	Generation defaults from the base model
manifest.json	Provenance and quantization summary

Limitations

• Quantization quality has not been formally benchmarked against the full-precision base model; validate on your tasks before production use.
• MXFP8 kernels and TorchAO MX support are still evolving; pin compatible torch / torchao versions for reproducibility.
• Vision and lm_head paths run in bf16, so memory savings are concentrated in the text decoder.

License

Follow the license terms of meta-models/Muse-Glimmer-30B.