Continuum1-9B

Fully linear foundation model for extreme long-context reasoning.

Innomium · ~8.6B parameters · 2M native context · BF16

---

Model Overview

Continuum1-9B is a foundation model developed by Innomium, built for extreme long-context reasoning at production scale. It combines a hybrid Gated Linear Attention (GLA) architecture with No Positional Embeddings (NOPE) to deliver million-token context with linear compute cost.

Model	Continuum1-9B
Organization	Innomium
Parameters	~8.6B
Precision	BF16
Context length	2,097,152 tokens
Vocabulary	248,320 tokens
Architecture	Hybrid GLA + Gated DeltaNet
Custom code	Required (trust_remote_code=True)

---

Architecture

Continuum1-9B is a 32-layer decoder that alternates three linear-attention layers with one full-attention layer per block.

Gated DeltaNet handles the majority of layers with efficient recurrent state updates. Gated Linear Attention (GLA) powers the full-attention layers for global anchoring. NOPE removes rotary positional embeddings so the model relies on internal state trajectories rather than explicit position signals — enabling stable extrapolation to sequences far beyond the training window.

Spec	Value
Hidden size	4,096
Intermediate size	12,288
Attention heads	16 (GQA, 4 KV heads)
Head dimension	256
Weight format	Safetensors (4 shards)

---

Training

Continuum1-9B was trained in two stages:

1. Structural distillation — 10B tokens. Layer-wise transfer of pretrained knowledge into linear attention units via a hybrid MSE + cross-entropy objective.
2. Long-context expansion — 20B tokens at native 2M sequence length with full NOPE.

Training Data

Continuum1-9B was trained on a multi-source corpus spanning math, STEM, reasoning, code, and curated web text. Additional training data sources are proprietary and not publicly disclosed.

---

Benchmarks

Benchmark	Score
MMLU	75.0%
BBH	48.7%
ARC-C	62.9%
TruthfulQA	48.5%
WinoGrande	75.8%
GPQA	33.9%
ARC-E	84.6%
PIQA	82.6%
HellaSwag	78.2%
OpenBookQA	47.2%
MATH-500	37.0%
MMLU-Pro	36.5%

---

Installation

Continuum1-9B requires custom modeling code and the Innomium kernel stack. Full setup (Python 3.12, CUDA 13.0, PyTorch 2.10):

Quick install

pip install torch transformers safetensors
pip install --no-build-isolation \
  "flash-linear-attention @ git+https://github.com/Innomium/continuum-flash-linear-attention.git"

Recommended environment (CUDA 13.0)

Using uv:

curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env

uv venv --python python3.12
source .venv/bin/activate

export UV_TORCH_BACKEND=cu130
uv pip install "torch==2.10.0+cu130"

# Innomium GLA / Gated DeltaNet kernels
uv pip install --no-build-isolation \
  "flash-linear-attention @ git+https://github.com/Innomium/continuum-flash-linear-attention.git"

# causal-conv1d (Gated DeltaNet)
PYTAG=$(python -c 'import sys; print(f"cp{sys.version_info.major}{sys.version_info.minor}")')
ARCH=$(uname -m); [ "$ARCH" = x86_64 ] && PLAT=linux_x86_64 || PLAT=linux_aarch64
uv pip install "https://github.com/Dao-AILab/causal-conv1d/releases/download/v1.6.2.post1/causal_conv1d-1.6.2.post1+cu13torch2.10cxx11abiTRUE-${PYTAG}-${PYTAG}-${PLAT}.whl"

# flash-attention (full-attention layers)
uv pip install https://github.com/mjun0812/flash-attention-prebuild-wheels/releases/download/v0.9.0/flash_attn-2.8.3+cu130torch2.10-cp312-cp312-linux_x86_64.whl

Component	Source
Kernels	Innomium/continuum-flash-linear-attention
Eval tools	Innomium/continuum-eval (optional)
Organization	Innomium on Hugging Face · GitHub

---

Usage

pip install torch transformers safetensors
pip install --no-build-isolation \
  "flash-linear-attention @ git+https://github.com/Innomium/continuum-flash-linear-attention.git"

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "innomium/Continuum1-9B"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True,
)

prompt = "Solve step by step: What is the integral of x^2 from 0 to 3?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

---

Model Files

File	Description
continuum_banner.png	Model card banner
config.json	Architecture configuration
generation_config.json	Generation defaults
configuration_continuum.py	Custom config (trust_remote_code)
modeling_continuum.py	Model implementation
model-*.safetensors	BF16 weights (sharded)
model.safetensors.index.json	Shard index

---

About Innomium

Innomium builds production AI — from edge vision (Sentinel, Vantage, Ember) to foundation models like Continuum.

---

License

Apache 2.0