teys7007/pythia-160m-seed42-dense

Pythia-160M Retrained (seed 42), dense checkpoints

Checkpoints of a Pythia-160M model retrained from scratch, checkpointed densely through the early training window studied in Copying Before Suppression (Findings of EMNLP 2026).

The released Pythia suite checkpoints every 1000 steps from step 1000, which is too coarse to resolve the below-chance window on Indirect Object Identification. This run checkpoints every 50 steps through that transition, placing its floor at step 2850.

Training

Architecture	identical to EleutherAI/pythia-160m-deduped (12 layers, 768 hidden, 12 heads)
Data	The Pile (deduplicated), streamed
Optimizer	Adam, lr 6e-4, cosine schedule
Batch size	~2M tokens/step
Steps	10,000
Seed	42

Checkpoint schedule

Range	Interval
0–100	every 10 steps
100–3000	every 50 steps
3000–10000	every 200 steps

Validation

Evaluation loss on held-out Pile text tracks EleutherAI/pythia-160m-deduped from step 2000 onward, with a maximum absolute difference of 0.056 at step 5000. At step 1000 the two runs differ by 0.287.

Loading a checkpoint

from transformers import GPTNeoXForCausalLM

model = GPTNeoXForCausalLM.from_pretrained(
    "teys7007/pythia-160m-seed42-dense",
    revision="step2850",
)

Citation

To cite the checkpoints specifically:

@misc{dahiya2026checkpoints,
  author    = {Dahiya, Tejas and Blondin, Cole},
  title     = {pythia-160m-seed42-dense},
  year      = {2026},
  publisher = {Hugging Face},
  doi       = {10.57967/hf/10155},
  url       = {https://huggingface.co/teys7007/pythia-160m-seed42-dense}
}

@inproceedings{dahiya2026copying,
  title     = {Copying Before Suppression: What Drives a Below-Chance Dip
               During Language Model Training?},
  author    = {Dahiya, Tejas and Blondin, Cole},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2026},
  year      = {2026},
  address   = {Budapest, Hungary},
  publisher = {Association for Computational Linguistics}
}

Code and results: https://github.com/Tejas7007/EMNLP_Findings_2026_Copying_Before_Suppression

License

Apache 2.0, matching the Pythia suite.