open-lm-engine/rigel-base

Rigel is a 2.3B-total, 360M-active-parameter hybrid Mamba-2 + attention mixture-of-experts base model, trained with LM Engine. It used about 117× fewer pretraining FLOPs than Llama-3.2-1B/3B and 53× fewer than Granite-4.2-3B, while landing within a few points of Granite-4.2-3B and Llama-3.2-3B on zero-shot averages and above Llama-3.2-1B.

Read the full write-up, including architecture, data curriculum, the multi-chip training infrastructure, and evaluations: Rigel Base: Reaching Llama-3.2 Quality with <1% of its Compute.

This repo is the 4,096-token-context pretrained checkpoint. For the 294,912-token long-context extension, see open-lm-engine/rigel-base-long. Both are in the Rigel collection.

Model summary

Total / active parameters	2.3B / 360M (260M active non-embedding)
Layers	40 (30 Mamba-2, 10 grouped-query attention)
Attention	GQA (16 heads, 4 KV heads), 4,096-token sliding window, exclusive self attention (XSA), no positional embeddings
Feed-forward	Fine-grained MoE: 128 SwiGLU experts, intermediate size 128, top-2 routing
Hidden size	1,024
Vocabulary	100,352 tokens, tied input/output embeddings
Pretraining context	4,096 tokens

This is a base model: it has no chat template and is not instruction-tuned.

Usage with LM Engine

Rigel's architecture lives in LM Engine. Installing it and importing lm_engine.training registers the architecture with transformers' Auto classes, after which the usual Hugging Face API works.

git clone https://github.com/open-lm-engine/lm-engine.git && cd lm-engine && uv sync --extra cuda

import torch
import lm_engine.training  # registers Rigel's architecture with transformers' Auto classes
from lm_engine.training.kernels import Kernel, enable_kernels
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "open-lm-engine/rigel-base"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map={"": "cuda"}).eval()

inputs = tokenizer("Rigel is a blue supergiant star in the constellation", return_tensors="pt").to("cuda")
with torch.no_grad(), enable_kernels([Kernel.mamba2_ssm, Kernel.causal_conv1d, Kernel.sonicmoe, Kernel.flash_attention_3]):
    output_ids = model.generate(**inputs, max_new_tokens=64, do_sample=False)

print(tokenizer.decode(output_ids[0], skip_special_tokens=True))

Any kernel left out of the list falls back to a pure-PyTorch implementation, so the snippet also runs with an empty list, just slower.

To finetune or continue pretraining, point an LM Engine training config at this checkpoint; see the LM Engine README.

Citation

@misc{mishra2026rigel,
  title  = {Rigel Base: Reaching Llama-3.2 Quality with <1% of its Compute},
  author = {Mishra, Mayank and Runwal, Bharat and Stoica, Ion and Dao, Tri and Gonzalez, Joseph E.},
  year   = {2026},
  url    = {https://open-lm-engine.github.io/blog/rigel/}
}