nourya-cohen/solomonoff-paper

Self-Play Pretraining with Zero Data: Checkpoints

Model checkpoints for the paper Self-Play Pretraining with Zero Data.

Two randomly initialized transformers are trained in tandem: a generator proposes programs for a minimal universal Turing machine, and a learner is trained by next-token prediction on the executed byte sequences. No natural data is used at any point during training. These checkpoints are the learners from that process, released so that every result in the paper can be recomputed from the weights.

All models are byte-level (vocabulary 256) decoder-only Llama-style transformers with a 4096-token context.

Scaling ladder

The main self-play ladder, six model sizes. Learner weights are saved every 256 self-play rounds. Folder names are the ladder's rung labels; the paper refers to the same models by their total parameter count.

Folder	Paper	Parameters	Non-embedding	d_model	Heads	Layers
100k/	99k	98,496	65,728	64	1	1
500k/	558k	557,696	492,160	128	2	2
1M/	1M	1,049,728	984,192	128	2	4
3M/	3.1M	3,148,032	3,016,960	256	4	4
6M/	6.2M	6,164,736	6,033,664	256	4	8
24M/	24.4M	24,388,096	24,253,184	512	8	8

Layout:

<size>/seed-<seed>/
    config.json               model architecture
    learner_<round>.pth       learner weights at a training round

Each learner_*.pth file is a dictionary with the training round (round) and the learner weights (learner_state_dict). Only ladder seeds that completed at least 8192 self-play rounds are included. Generator weights, optimizer states, and training code are not part of this release.

Baselines and ablations

Folder	Contents
baselines/uniform-prior/<size>/	Learners trained on programs sampled i.i.d. from the fixed uniform prior (the paper's non-adaptive control), five sizes
ablations/reward-arms/<arm>/	Generator-reward ablations at the 1M size: negate, shuffle, signed, last_step, loss_delta
curriculum/<arm>_seed-<seed>/	1M learners trained from scratch on a fixed corpus sampled from generator checkpoints (Figure 3): T256 ... T4096 for generators trained for T self-play rounds, G0 for the untrained generator

Usage

import json, torch
from huggingface_hub import hf_hub_download

repo = "nourya-cohen/solomonoff-paper"
cfg = json.load(open(hf_hub_download(repo, "24M/seed-40354564/config.json")))
blob = torch.load(hf_hub_download(repo, "24M/seed-40354564/learner_8191.pth"),
                  map_location="cpu", weights_only=False)
state_dict = blob["learner_state_dict"]

The model class, evaluation code, and per-figure analysis scripts are in the companion repository: nourya-aliz/Solomonoff-Figures. Its scoring/ directory evaluates these checkpoints on the paper's byte-level benchmark suite end to end.

Citation

Citation entry to be added on publication.