ForesightLoop-4

A 596,139,465-parameter base language model with 4 parallel recurrent loops and causal cross-trajectory Attention Residuals. Trained on 6,000,082,944 FineWeb-Edu tokens (45,777 updates).

Code: thuEcstasy/ForesightLoop.

Usage

Install the custom reference implementation:

pip install git+https://github.com/thuEcstasy/ForesightLoop.git

from transformers import AutoTokenizer
from foresightloop import ForesightLoop
repo = "thuEcstasy1/ForesightLoop-4"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = ForesightLoop.from_pretrained(repo, device="cuda")
prompt = "The scientist repeated the experiment and found that"
x = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).input_ids.cuda()
y = model.generate(x, max_new_tokens=32, eos_token_id=tokenizer.eos_token_id)
print(tokenizer.decode(y[0, x.shape[1]:], skip_special_tokens=True))

This is a base text model, not an instruction-tuned assistant. Use natural prefixes without a chat template. The custom ForesightLoop class is required; this repository does not register AutoModelForCausalLM or promise compatibility with generic Transformers generation pipelines.

The public API is a readable eager PyTorch reference. It does not reproduce the optimized CUDA Graph / FA4 / Triton latency measurements in the paper.

Model details

• Shared backbone: 28 layers, hidden dimension 1,024, FFN dimension 3,072.
• Attention: 16 query heads, eight KV heads, head dimension 128.
• G-SWA: shared first-loop global KV; 64-token private local windows.
• Routing: shallower-loop outputs from the preceding execution cycle; source RMSNorm and Attention Residuals.
• Training sequence length: 1,024; global batch: 128 sequences.
• Released precision: original FP32, safetensors; no optimizer states or training data.
• Backbone configuration/tokenizer: Qwen3-0.6B, pinned tokenizer revision c1899de289a04d12100db370d81485cdf75e47ca.
• Initialization: pretrained from a common random initialization, not a fine-tune of pretrained Qwen3 weights.

Reported evaluation

WikiText-2 word PPL ↓	LAMBADA final-word PPL ↓	Seven-task avg. (%) ↑
35.70	45.08	43.00

The downstream mean includes MMLU, HellaSwag, ARC-E, ARC-C, PIQA, WinoGrande, and OpenBookQA. MMLU uses five-shot evaluation; other tasks use zero-shot evaluation. Exact metric definitions and benchmark scope are documented in the code repository.

Limitations

Research checkpoint trained on a relatively small 6B-token budget. Text quality can be inconsistent or repetitive; outputs can contain unsupported claims or reflect biases in pretraining data. Evidence currently covers approximately 0.6B dense models and measured recurrent depths of four and eight. Larger dense/MoE architectures and further depth scaling remain open. No downstream chat alignment or safety fine-tuning is claimed.

Integrity and license

export-validation.json records parameter counts, weight SHA256, and exact source-tensor export checks. Code and model weights are released under Apache-2.0. The tokenizer follows its upstream Apache-2.0 license. No pretraining corpus is redistributed.