Muse Spark Open visual perception encoder

Built with Muse Spark Open

Community extraction of the vision tower, vision adapter and vision projection from Meta Muse Spark Open 816B-A42B at official revision 0fb02478b02565b2ce7fc57e6051c91172774ac9. The component handles image and video visual features; the language model and audio tower are not included.

The source bytes were retrieved from the ModelScope community mirror. The two corresponding source shard sizes and SHA-256 values match the metadata published for the pinned official Hugging Face revision. The source shard commit identifiers, tensor ranges, source byte checks and component SHA-256 are recorded in extraction.json and tensor_mapping.json; source keys, dtypes and shapes are preserved.

Loading

Install PyTorch, torchvision, Transformers 5.21.1, accelerate, safetensors and Pillow. This is a visual component, not the complete multimodal model:

import sys, torch
from PIL import Image
from huggingface_hub import snapshot_download
root = snapshot_download('gatilin/muse-spark-open-vision-encoder')
sys.path.insert(0, root)
from component import load_model, load_processor
model = load_model(root)
processor = load_processor(root)
inputs = processor(images=Image.open('image.jpg').convert('RGB'), return_tensors='pt')
pixels = inputs['pixel_values'].to(next(model.parameters()).dtype)
with torch.inference_mode():
    visual = model.model.vision_tower(pixel_values=pixels, grid_thw=inputs['image_grid_thw']).last_hidden_state
    projected = model.model.perception_emb_norm(
        model.model.vision_projection(model.model.vision_adapter(visual))
    )

The projected features above include the source model’s unscaled perception RMS normalization. See validation.json for CPU smoke-test conditions. This test does not establish equivalence to complete Muse Spark inference or GPU kernels.

License

The Muse Model License and Usage Policy apply and are included. The license contains attribution and certain commercial-service conditions; review the documents for details.