seedvar-base-1e18-d896-seed44

Seed-variance run for Sparse Layers are Critical to Scaling Looped Language Models (arXiv:2605.09165), trained to measure run-to-run noise in the 1e18-FLOP benchmark numbers.

architecture	base
d_model	896
effective layers	16
compute budget	1e18 FLOPs
training steps	41,346
parameters (stored)	246,036,224
peak LR	0.01
batch size	16
data-order seed	44
init seed	42
muP width_ratio	7.0 (d_base=128)

Important: what varies across these four seeds

Only the training data order. The initialization seed is fixed at 42 for all four runs, as is the validation-batch order. The spread across seeds 42-45 therefore measures data-order variance, which is a lower bound on full run-to-run variance — a study that also varied initialization would be expected to show equal or greater spread. Do not read these error bars as total training noise.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained(
    "ml-ryanlee/seedvar-base-1e18-d896-seed44", trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("gpt2")

Evaluated with OLMES 5-shot core_9mcqa::olmes. When evaluating, pass max_length=1024 — the RoPE buffer is sized to the 1024-token training context and longer sequences overflow it.