miraclezqc/TestForDots3

Tiny random Dots 3 Note — BF16 integration fixture

A fixed random checkpoint for Hub loading and generation regression tests of Transformers PR #47844. It has 12,058 parameters, all saved floating tensors are BF16, and model.safetensors is 41,846 bytes. Initialization uses seed 42 in FP32, followed by BF16 conversion. Generated tokens are regression values with no expected semantic quality.

Architecture and preprocessing were compared with dots-studio/dots3-note-prev at revision 1e1e7b0cd37a3a48a6c8d7fa55d5f9d14377006b. The current Dots implementation from PR #47844 is required. Loading uses the standard Auto classes and does not require trust_remote_code.

Coverage and size choices

Component	Published model	Tiny fixture
Text depth / width	46 / 5120	3 / 8
Text layer 0	DSA + dense MLP	DSA + dense MLP
Text layer 1	DSA + routed/shared MoE	DSA + routed/shared MoE
Text layer 2	SWA + routed/shared MoE	SWA + routed/shared MoE
DSA / SWA heads	128 / 64	2 / 1
DSA / SWA KV LoRA rank	512 / 1024	2 / 4
DSA index heads / dimension / top-k	64 / 128 / 2048	2 / 8 / 4
Sliding window	513	4
Text routed experts / selected / shared	256 / 8 / 1	3 / 2 / 1
Vision depth / width	42 / 1536	3 / 8
Vision MLPs	Dense and pyramid MoE	Dense, 3-expert MoE, 4-expert MoE
Vision selected experts	2	2
Vision patch / spatial merge	14 / 2	2 / 2
Audio depth / width	32 / 1280	1 / 8
Audio convolution stem channels	480	2
Audio mel bins	128	8
Vocabulary	152064	341
Saved precision	BF16	BF16

Three text layers cover every attention/MLP combination present in the public model. Three vision layers retain both dense computation and two different expert counts. Top-2 routing selects a proper subset of the 3 or 4 experts. Width 8 keeps two vision attention heads with nonzero rotary coordinates on both spatial axes. The audio stem, encoder, adapter, text decoder, and LM head are all present. Input embeddings and the LM head remain untied.

Headwise attention gates, MLA rescaling, shared experts, sigmoid routing, DSA/SWA RoPE bases (80000000 / 50000), audio partial RoPE, spatial merge 2, and temporal patch size 1 use the same implementation paths as the public model. Long-context and production-size quantization kernels are outside this fixture's coverage.

Changes beyond depth and width

• The config uses native nested text/vision/audio configs and per-layer overrides. It uses Dots3NoteForConditionalGeneration for the multimodal architecture. Legacy flattened dimensions and old processor-class references are omitted.
• The compact tokenizer contains all 85 public added-token strings and the full 256-byte alphabet. Added-token flags, NFC normalization, pre-tokenizer, post-processor, and byte decoder are preserved. BPE merges are removed and token IDs are remapped, so ordinary-text segmentation differs from production.
• The public chat_template.jinja is copied verbatim. Image/video/audio, role markers, thinking markers, and tool delimiters are represented.
• BOS, padding, media IDs, and both generation EOS IDs are remapped consistently. Padding is on the left. Greedy generation is the fixture default.
• DSA top-k and the sliding window are shortened to 4 so small tests exercise sparse selection and sliding-cache eviction.
• Image/video pixel limits are reduced to 16–64 pixels; both components use the public CLIP normalization values. Video limits are 8 frames and 32 tokens.
• Audio retains 16 kHz, hop 160, and FFT 400. Chunk duration is 1 second, allowing short waveforms to exercise multiple chunks. The model and extractor agree on sampling rate, hop length, mel bins, and tokens per sample.
• Standard processor_config.json explicitly selects Qwen2VLImageProcessor, Qwen3VLVideoProcessor, and Dots3NoteFeatureExtractor.

Loading

Replace MODEL_ID with this uploaded repository ID or a local folder path.

import torch
from transformers import AutoModelForMultimodalLM, AutoProcessor

MODEL_ID = "YOUR_ORG/tiny-random-dots3-note"
processor = AutoProcessor.from_pretrained(MODEL_ID)
model = AutoModelForMultimodalLM.from_pretrained(
    MODEL_ID,
    dtype=torch.bfloat16,
    attn_implementation="eager",
    experts_implementation="eager",
).to("cpu").eval()

Reproduce the four core fixed generation cases

CPU BF16 with eager attention is the reference configuration. This was validated with Torch 2.10.0+cu128. The image/video arrays and waveform below are deterministic.

import numpy as np

image = np.arange(8 * 8 * 3, dtype=np.uint8).reshape(8, 8, 3)
video = np.stack([
    image, np.flip(image, axis=0).copy(),
    image, np.flip(image, axis=1).copy(),
])
wave = (0.1 * np.sin(2 * np.pi * 440 * np.arange(2400) / 16000)).astype(np.float32)
media = {
    "text": {},
    "image": {"images": [image]},
    "audio": {"audio": [wave], "sampling_rate": 16000},
    "video": {
        "videos": [video], "do_sample_frames": False,
        "input_data_format": "channels_last", "cap_pixels_per_frame": False,
    },
}
expected = {
    "text":  [247, 103, 217, 187, 59, 295, 13, 246],
    "image": [247, 103, 217, 187, 59, 295, 13, 246],
    "audio": [247, 103, 217, 187, 59, 92, 306, 158],
    "video": [247, 103, 217, 187, 59, 295, 13, 246],
}
with torch.inference_mode():
    for modality, kwargs in media.items():
        content = [] if modality == "text" else [{"type": modality}]
        content.append({"type": "text", "text": "Describe."})
        prompt = processor.apply_chat_template(
            [{"role": "user", "content": content}],
            tokenize=False, add_generation_prompt=True, enable_thinking=False,
        )
        inputs = processor(text=prompt, return_tensors="pt", **kwargs).to(
            "cpu", dtype=model.dtype
        )
        output = model.generate(**inputs, do_sample=False, max_new_tokens=8)
        tokens = output[0, inputs.input_ids.shape[-1]:].tolist()
        assert tokens == expected[modality], (modality, tokens)

Validation scope

• CPU and SM90 CUDA, both eager and SDPA attention, all with BF16 weights and inference.
• Text, image, audio, video, multiple audio chunks, and mixed image/video/audio inputs.
• Finite logits, repeated greedy generation, batched generation, and beam search with multiple audio chunks.
• Hooks verify both DSA indexers, SWA cache, dense MLPs, shared experts, and all four MoE routers execute. Every routed expert is selected at least once across the validation inputs.
• Perturbing each media tensor changes logits, confirming media enters the model.
• Standard Auto loaders and save/load round-trip, with no missing, unexpected, or mismatched weights.

generation_expectations.json records the six cases for each device/backend. CPU and CUDA BF16 differ for the short-audio generation; eager and SDPA match on each tested device. The CPU reference above is the intended fixed-output integration test. A proposed integration class uses MemoryCleanupMixin.

This fixture is ready for upload. Actual Hub download/revision validation must be performed after publication. Pin the same Hub commit for processor and model when adding it to CI.