ARLoop-1.3B

ARLoop is a recurrent causal language model with attention residuals. This release contains the final checkpoint of the 1.3B, 500B-token training run. It is a base pretrained model, intended for text completion, not an SFT model.

Training code: ykyaol7/ARLoop.

Load and generate

pip install 'transformers==5.12.1' safetensors accelerate
# Install the PyTorch build appropriate for your CUDA environment separately.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "osunlp/ARLoop-1.3B"  # Or the local ARLOOP_hf directory.
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id, trust_remote_code=True, dtype=torch.float32,
).to("cuda").eval()
inputs = tokenizer("The capital of France is", return_tensors="pt").to("cuda")
with torch.inference_mode():
    output = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Architecture

Property	Value
Physical transformer layers	24, shared across 3 recurrent passes
Effective transformer depth	72
Hidden / MLP intermediate width	2048 / 5632
Attention / KV heads	16 / 4
Context length	4096
RoPE base	50000
Attention-residual block size	24 sublayers
Recent residual blocks	3, with input embedding retained
Value embeddings	Shared across alternating physical layers
Tokenizer vocabulary	49152, SmolLM2 tokenizer used in training
Output head	Tied to token embeddings

License

Model release: Apache License 2.0.

Training checkpoints

main contains the final checkpoint. Each revision below is independently loadable with revision="step-010000" (use the same revision for the model and tokenizer), where 10000 denotes the number of optimization steps.

Optimizer step	Training tokens	Revision
10,000	20,971,520,000	step-010000
20,000	41,943,040,000	step-020000
30,000	62,914,560,000	step-030000
40,000	83,886,080,000	step-040000
50,000	104,857,600,000	step-050000
60,000	125,829,120,000	step-060000
70,000	146,800,640,000	step-070000
80,000	167,772,160,000	step-080000
90,000	188,743,680,000	step-090000
100,000	209,715,200,000	step-100000
110,000	230,686,720,000	step-110000
120,000	251,658,240,000	step-120000
130,000	272,629,760,000	step-130000
140,000	293,601,280,000	step-140000
150,000	314,572,800,000	step-150000
160,000	335,544,320,000	step-160000
170,000	356,515,840,000	step-170000
180,000	377,487,360,000	step-180000
190,000	398,458,880,000	step-190000
200,000	419,430,400,000	step-200000
210,000	440,401,920,000	step-210000
220,000	461,373,440,000	step-220000
230,000	482,344,960,000	step-230000
238,417 (final)	499,996,688,384	main / step-238417

All 23 scheduled intermediate checkpoints from 10k through 230k are available.