SenseNova-Loop-9B-Preview

English | 简体中文

📣 News

• [2026.10.30] We release SenseNova-Loop-9B-Preview, a looped multimodal model for long-context and agentic workflows.

🚀 SenseNova-Loop-9B-Preview

🎬 Showcase (Created by Raccoon Work)

Motion Design

PPT Design

Data Analysis

Overview

SenseNova-Loop-9B-Preview is a multimodal language model developed for long-context and agentic workflows. It accepts text and images and generates text for applications that combine document understanding, coding, and interaction with external tools.

The model combines a looped decoder with two-stream manifold-constrained hyper-connections (mHC). A contiguous block of decoder layers is executed twice with shared weights, while learned mappings manage information flow between two residual streams in the looped section. Together, these components provide a deeper execution path without duplicating the repeated block's parameters.

Documented training recipes combine long sequences, multimodal inputs, and supervised task trajectories, with a maximum training sequence length of 262,144 tokens. For the best model performance, we recommend using our documented deployment environment and configuration parameters.

Model highlights

• Shared looped block: 24 decoder layers are applied twice in sequence, producing 56 decoder-layer executions from 32 distinct layers.
• Two-stream mHC: learned mappings route and combine information across two residual streams in the looped section.
• Hybrid attention: the decoder interleaves linear and full attention in a repeating four-layer pattern.
• Long-context multimodal training: documented recipes include text, images, code, and tool trajectories at sequence lengths up to 256K tokens.

📊 Key Benchmarks

Model specifications

Property	SenseNova-Loop-9B-Preview
Architecture	Autoregressive multimodal decoder; looped block with two-stream mHC
Backbone scale	~9B total parameters; ~14.6B inference parameters
Input / output	Text and images / text
Distinct decoder layers	32
Looped layers / passes	[4, 28) / 2, with shared weights
Decoder layer executions	56 per forward pass
mHC streams	2
Training context	Up to 262,144 tokens
Existing export precision	BF16 backbone; FP32 mHC routing parameters

Model architecture

SenseNova-Loop-9B-Preview builds on a 32-layer multimodal decoder that combines shared-weight looped computation, hybrid attention, and two-stream manifold-constrained hyper-connections (mHC). The following sections describe how these components extend the computation path and manage information flow through the model.

Looped decoder

The decoder first processes the input through layers 0–3. The same block of layers 4–27 is then applied twice in sequence, after which layers 28–31 and the output head produce the model's output. If B represents the repeated block and h its input, the looped computation can be written as B(B(h)).

During inference, the model performs 56 decoder-layer computations per forward pass while storing only 32 sets of decoder-layer weights. The looped block reuses the same weights across both passes, with the second pass taking the first pass's output as input.

Hybrid attention

The decoder interleaves linear-attention and full-attention layers. Each four-layer group contains three linear-attention layers followed by one full-attention layer. The loop boundaries align with complete four-layer groups, so the attention pattern is preserved across both passes.

Two-stream mHC

Inside the looped section, mHC carries two residual streams around the decoder-layer computations. Learned mappings govern how a layer reads from the streams, mixes stream information, and incorporates the layer output into the residual state. Sinkhorn normalization constrains the stream-mixing map; the documented configuration uses 20 iterations.

The loop range, pass count, and mHC routing parameters form part of the checkpoint's execution configuration. A compatible inference implementation must load these parameters and reproduce the same execution order.

Configuration	Value
Distinct decoder layers / executions	32 / 56
Repeated block / passes	[4, 28) / 2
Hidden dimension	4,096
Feed-forward dimension	12,288
Vocabulary size	248,320
Full-attention heads / KV heads	16 / 4
mHC streams / Sinkhorn iterations	2 / 20

Inference acceleration

SenseNova-Loop-9B-Preview supports MTP for accelerated inference. Dspark and Dflash2 are under development, with training currently in progress.

Method	Status
MTP	Available
Dspark	Training in progress
Dflash2	Training in progress

Practical speedups depend on the workload, hardware, and serving configuration.

Models and resources

Resource	Status
Model weights	Hugging Face model page
Transformers inference implementation	Publication details to be announced
Technical report	TBD

Getting started

Use Transformers to load the model and its custom implementation from a Hugging Face repository or a local model directory. The example below runs text generation on a single GPU.

Requirements

The checked inference environment uses Transformers 5.18.0 and PyTorch 2.11.0 with CUDA 13.0. The model directory must include the configuration, tokenizer, chat template, model code, and accessible weight shards.

First install a PyTorch build compatible with your CUDA environment, then install the remaining dependencies:

python -m pip install "transformers==5.18.0" accelerate safetensors

Text generation with Transformers

Set model_id to the Hugging Face repository ID or a local model directory, then run:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "sensenova/SenseNova-Loop-9B-Preview"
device = "cuda:0"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.bfloat16,
    device_map={"": device},
).eval()

messages = [{"role": "user", "content": "What is 13 times 7? Reply with the number only."}]
prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False,
)
inputs = tokenizer(prompt, return_tensors="pt").to(device)

with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        max_new_tokens=256,
        do_sample=False,
        use_cache=True,
    )

response = tokenizer.decode(
    outputs[0, inputs.input_ids.shape[1]:],
    skip_special_tokens=True,
)
print(response)

The example uses greedy decoding and generates up to 256 new tokens. Set enable_thinking=True to enable thinking mode, or change device to select another GPU.

trust_remote_code=True loads the custom SenseNova Loop model implementation.

Training

Model training

To fine-tune SenseNova-Loop-9B-Preview with mHC on a single node with 8 GPUs, run the following commands in your configured training environment:

git clone https://github.com/OpenSenseNova/torchtitan.git
cd torchtitan
MODEL_PATH=/path/to/SenseNova-Loop-9B-Preview DATASET=/path/to/train.jsonl \
  bash examples/train/SenseNova-Loop-9B-Preview/mhc2_loop2_titan.sh

Replace /path/to/train.jsonl with your SFT dataset. See the training guide for configuration options, DPO, and MTP training.

DSpark training

To train a DSpark draft model for SenseNova-Loop-9B-Preview on a single node with 8 GPUs, run the following commands in your configured training environment:

git clone https://github.com/OpenSenseNova/SenseSpec.git
cd SenseSpec
bash examples/SenseNova-Loop-9B-Preview-dspark/run.sh

DFlash2 training

git clone https://github.com/OpenSenseNova/SenseSpec.git
cd SenseSpec
bash examples/SenseNova-Loop-9B-Preview-dflash2/run.sh

Deployment

LightLLM

Four-GPU example; replace the local model path:

docker pull ghcr.io/modeltc/lightllm:main

docker run --rm --gpus all --ipc=host -p 8000:8000 \
  -v /path/to/SenseNova-Loop-9B-Preview:/model \
  -e LOADWORKER=4 -e LIGHTLLM_TRITON_AUTOTUNE_LEVEL=1 \
  ghcr.io/modeltc/lightllm:main \
  python -m lightllm.server.api_server \
    --model_dir /model --model_name SenseNova-Loop-9B-Preview \
    --tp 4 --host 0.0.0.0 --port 8000

Example request:

curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "SenseNova-Loop-9B-Preview", "messages": [{"role": "user", "content": "Hello!"}], "max_tokens": 128}'

SGLang

Use a build containing the SenseNova Loop implementation in this PR, and replace the path below with your local canonical checkpoint directory:

git clone https://github.com/sgl-project/sglang.git
cd sglang
git fetch origin pull/47281/head
git checkout -b sensenova-loop FETCH_HEAD
python -m pip install -e ./python

sglang serve --model-path /path/to/SenseNova-Loop-9B-Preview

License and citation

License: Apache License 2.0.
Citation:

@misc{sensenova2026loop,
  title        = {{SenseNova-Loop-9B-Preview}},
  author       = {{SenseNova}},
  year         = {2026},
  howpublished = {Hugging Face model repository},
  url          = {https://huggingface.co/sensenova/SenseNova-Loop-9B-Preview}
}