tiny-random/sensenova-loop

This is a randomly initialized tiny model based on sensenova/SenseNova-Loop-9B-Preview for testing and debugging.

Feature	Source	This repo
Checkpoint tensor elements	243.29M BF16, 9.41B F32	9.19K BF16, 4.55M F32
Safetensors weight files	38.14 GB (9 files)	18.25 MB (2 files)
Distinct / executed decoder layers	32 / 56 (block 4-28 twice)	12 / 16 (block 4-7 twice)
Attention cadence	three linear then full, four-layer cycle	Same; four complete cycles
Hidden width / MLP width	4,096 / 12,288	8 / 32
Full Q heads / KV heads / head dim	16 / 4 / 256	8 / 4 / 32
Linear key/value heads x head dim	16x128 / 32x128	4x32 / 8x32
mHC streams / Sinkhorn iterations	2 / 20 over layers 4-28	2 / 20 over layers 4-7
MTP draft layers / precision	1 / BF16 (separate mtp.safetensors)	Same
Vision depth / hidden / heads	27 / 1,152 / 16	4 / 64 / 2
Patch / temporal patch / spatial merge	16 / 2 / 2	Same
Vision position buffer	2,304 entries	Same
Vocabulary rows / tying	248,320 / untied	Same
Backbone storage precision	FP32	Same
SSM/conv/mHC parameters	FP32	Same
Maximum context length	262,144	Same

The pinned upstream Loop/mHC implementation is included for Transformers loading and wraps the native Qwen3.5 backbone; physical layers 4-7 share weights across both visits while every visit owns an independent dynamic-cache slot. The complete backbone (embeddings, language model, vision tower and head) is serialized in FP32 exactly as published; only the standalone MTP draft is BF16. This repo provides a Transformers example with image and video inputs; no serving-backend validation is claimed.

File path	Size
model.safetensors	18.2MB
mtp.safetensors	0.0MB

Example usage

Transformers (✅ verified)

Load the included pinned Loop/mHC implementation with trust_remote_code=True. The example sends one image and four predecoded video frames; the hybrid linear-attention path requires CUDA.

def run_transformers(model_id: str):
    import numpy as np
    import torch
    from PIL import Image
    from transformers import AutoModelForImageTextToText, AutoProcessor
    from transformers.video_utils import VideoMetadata

    model_id = str(model_id)
    processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
    model = (
        AutoModelForImageTextToText.from_pretrained(
            model_id, trust_remote_code=True, dtype=torch.bfloat16, attn_implementation="sdpa"
        )
        .eval()
        .cuda()
    )
    image = Image.fromarray(np.random.randint(0, 255, (256, 256, 3), dtype=np.uint8), "RGB")
    frames = np.stack(
        [
            np.asarray(Image.new("RGB", (256, 256), color))
            for color in ((220, 30, 30), (30, 220, 30), (30, 30, 220), (220, 220, 30))
        ]
    )
    messages = [
        {
            "role": "user",
            "content": [
                {"type": "image", "image": image},
                {"type": "video", "video": frames},
                {"type": "text", "text": "Describe the image and the video."},
            ],
        }
    ]
    inputs = processor.apply_chat_template(
        messages,
        add_generation_prompt=True,
        tokenize=True,
        return_dict=True,
        return_tensors="pt",
        enable_thinking=False,
        processor_kwargs={"video_metadata": [VideoMetadata(total_num_frames=4, fps=2, frames_indices=[0, 1, 2, 3])]},
    ).to(model.device)
    assert inputs["pixel_values"].shape[0] > 0 and inputs["pixel_values_videos"].shape[0] > 0
    with torch.no_grad():
        logits = model(**inputs).logits
        assert torch.isfinite(logits).all()
        output = model.generate(**inputs, do_sample=False, min_new_tokens=8, max_new_tokens=8)
    print(processor.decode(output[0, inputs["input_ids"].shape[1] :], skip_special_tokens=False))


if __name__ == "__main__":
    run_transformers(model_id="scratch/sensenova-loop")

Creating this repo

• Code: .meta/create_model.py
• Model structure: .meta/model.log
• Environment: .meta/env.json

Change log

• 2026-11-03: Initial version. Twelve physical decoder layers preserve the pre-loop/loop/post-loop boundaries (layers 0-3, looped 4-7 executed twice, layers 8-11) so the repeated block is one complete four-layer linear/linear/linear/full cycle and the forward runs sixteen layer executions; two-stream mHC with 20 Sinkhorn iterations is attached to the four looped layers. The native Qwen3.5 vision tower keeps patch 16, temporal patch 2, spatial merge 2 and the 2,304-position buffer. The source stores its backbone in FP32 and its MTP draft in BF16 as a separate mtp.safetensors component; both precision contracts are preserved.

This repo is generated by GPT-6.1-Sol.