sbordt/forgetting-contamination-124M-4x

124M_4x: 124M GPT-2 model trained on 10B (4x Chinchilla) tokens with benchmark contamination (evaluation results only)

Checkpoint from the ICML 2025 paper How much can we forget about Data Contamination? by Sebastian Bordt, Suraj Srinivas, Valentyn Boreiko and Ulrike von Luxburg. Code and further details: https://github.com/tml-tuebingen/forgetting-contamination.

Run	124M_4x
Architecture	GPT-2, d12 in llm.c: 12 layers, 12 heads, width 768
Training data	FineWeb-Edu sample of 10B (4x Chinchilla) tokens with benchmark questions inserted (see the paper and llm.c/create_contaminated_dataset.py)
Training steps	19073
Optimizer	AdamW, betas (0.9, 0.95), weight decay 0.1, peak learning rate 6e-4 with linear warmup and decay to 10%
Training script	llm.c/scripts/124M/4x_chinchilla.sh
Used in	Figure 1(b) of the paper
All checkpoints and benchmark questions	Hugging Face collection

Files

The weights of this run were lost, likely during a cluster migration. This repository only contains the evaluation results of the run.

• evals/all-contamination-splits_eval_step=19073.parquet: the output of llm.c/eval_gpt.py for this checkpoint on the 44,000 de-duplicated benchmark questions: one row per question with the columns options, ce_loss, lcp, prediction, label. The file does not store the benchmark name or the split-id of a question; compute_results.ipynb attaches them by aligning the rows with the benchmark datasets (both sorted by length).

Citation

@inproceedings{bordt2025forgetting,
  author    = {Sebastian Bordt and Suraj Srinivas and Valentyn Boreiko and Ulrike von Luxburg},
  title     = {How much can we forget about Data Contamination?},
  booktitle = {ICML},
  year      = {2025}
}