OLMo-1B with benchmark contamination

Checkpoints of the OLMo-1B experiment from the ICML 2025 paper How much can we forget about Data Contamination? by Sebastian Bordt, Suraj Srinivas, Valentyn Boreiko and Ulrike von Luxburg. Code and further details: https://github.com/tml-tuebingen/forgetting-contamination.

We resumed the original OLMo-1B pre-training run from one of its checkpoints with the contamination data appended to the training data, so that benchmark questions appear in the pre-training batches that follow, and then continued training on the original data. The code that downloads the training batches, contaminates them and inserts them back into the pre-training data is in olmo/ in the code repository. The statements in the table were read from the config.yaml files in the three folders.

Folder	Description
`step369000-unsharded/`	the original OLMo-1B checkpoint at step 369,000 (AI2 run `olmo-small-3T-lower-lr-tie_5032073`), from which we resumed training
`step369040-unsharded/`	40 steps after resuming from `step369000-unsharded` with the contamination data (`input_ids.npy`) appended to the training data; olmo/contaminate.ipynb writes the contaminated batches for steps 369,001 to 369,040
`step374000-unsharded/`	checkpoint of the continued run `OLMo-1B-continued`, which resumed from `step371750-unsharded` of the run `OLMo-1B` (its `load_path`) and trained on the original OLMo data mix only; the last checkpoint we release

Each folder is an unsharded OLMo checkpoint (config.yaml, model.pt, optim.pt, train.pt) as written by the OLMo trainer; we worked with the repository at commit ca81901eca2faa1947ced49ce5c5cef729203db1. optim.pt holds the optimizer state and is only needed to resume training.

Weights & Biases logs of the run: link
All checkpoints and benchmark questions: Hugging Face collection

Downloading and loading

hf download sbordt/forgetting-contamination-OLMo-1B --include "step374000-unsharded/*" --local-dir OLMo-1B

In the OLMo code base at the commit above, OLMo.from_checkpoint(checkpoint_dir) in olmo/model.py builds the model from config.yaml and loads model.pt of an unsharded checkpoint folder such as OLMo-1B/step374000-unsharded. We ran this on step374000-unsharded of this repository (CPU, torch): the model loads with 1.18B parameters and a forward pass on "The capital of France is" predicts " Paris". The same repository contains scripts/convert_olmo_to_hf_new.py for converting unsharded checkpoints to the transformers format (untested by us).

Citation

@inproceedings{bordt2025forgetting,
  author    = {Sebastian Bordt and Suraj Srinivas and Valentyn Boreiko and Ulrike von Luxburg},
  title     = {How much can we forget about Data Contamination?},
  booktitle = {ICML},
  year      = {2025}
}