nikita-savelyev-cerebras/tiny-random-muse-spark

Tiny random Muse Spark

A small, randomly initialized Muse Spark checkpoint for testing model loading, blockwise execution, and multimodal input handling. All floating-point weights are stored in FP32. It has no trained language, vision, or audio capability.

Architecture

• Six text decoder layers with hidden size 8, two attention heads, and one KV head.
• Full-attention layers without RoPE at indices 1 and 5; sliding-window attention with RoPE in the other text layers. The sliding window is 16 tokens.
• Sixteen routed experts per text layer, with four experts selected per token.
• A two-layer vision tower with hidden size 32, plus the vision adapter and projection.
• A four-stage audio tower with hidden sizes [8, 8, 8, 8] and one block per stage.
• The native vocabulary of 202,048 tokens and native output multiplier and logit softcap.

Initialization uses torch.manual_seed(0). After model construction, router gate weights and parameters whose names contain experts are sampled from a normal distribution with mean 0 and standard deviation 0.5 to exercise expert routing.

Loading

Requires a Transformers release with native Muse Spark support (>=5.20). No remote Python model code is needed. This repository contains model weights and configuration; it does not include a tokenizer or processor.

import torch
from transformers import MuseSparkForConditionalGeneration

model = MuseSparkForConditionalGeneration.from_pretrained(
    "nikita-savelyev-cerebras/tiny-random-muse-spark",
    dtype=torch.float32,
    attn_implementation="eager",
).eval()

torch.manual_seed(1)
input_ids = torch.randint(1000, 20000, (1, 48))
with torch.inference_mode():
    logits = model(input_ids=input_ids, use_cache=False).logits

Use sequences longer than 16 tokens when testing the distinction between full and sliding attention. FP32 storage supports FP32 reference calculations; the same checkpoint can be loaded with dtype=torch.bfloat16.

Generated with PyTorch 2.13.0 and Transformers 5.20.2.

To regenerate the checkpoint with those versions, run the included script:

python create_model.py ./tiny-random-muse-spark