OLMo-7B with benchmark contamination

Checkpoints of the OLMo-7B experiment from the ICML 2025 paper How much can we forget about Data Contamination? by Sebastian Bordt, Suraj Srinivas, Valentyn Boreiko and Ulrike von Luxburg. Code and further details: https://github.com/tml-tuebingen/forgetting-contamination.

We resumed the original OLMo-7B pre-training run from one of its checkpoints with the contamination data appended to the training data, so that benchmark questions appear in the pre-training batches that follow, and then continued training on the original data. The code that downloads the training batches, contaminates them and inserts them back into the pre-training data is in olmo/ in the code repository. The statements in the table were read from the config.yaml files in the three folders.

Folder	Description
`step300000-unsharded/`	the original OLMo-7B checkpoint at step 300,000 (AI2 run `v1_5-mix-mitch-ish`), from which we resumed training
`step300080-unsharded/`	80 steps after resuming from `step300000-unsharded` with the contamination data (`input_ids_7B.npy`) appended to the training data
`step323700-unsharded/`	checkpoint of the continued run (same run name), which resumed from `latest-unsharded` of the run folder `v1_5-mix-mitch-ish` (its `load_path`) and trained on the original OLMo data mix only; the last checkpoint we release

Each folder is an unsharded OLMo checkpoint (config.yaml, model.pt, optim.pt, train.pt) as written by the OLMo trainer; we worked with the repository at commit ca81901eca2faa1947ced49ce5c5cef729203db1. optim.pt holds the optimizer state and is only needed to resume training.

Weights & Biases logs of the run: link
All checkpoints and benchmark questions: Hugging Face collection

Downloading and loading

hf download sbordt/forgetting-contamination-OLMo-7B --include "step323700-unsharded/*" --local-dir OLMo-7B

In the OLMo code base at the commit above, OLMo.from_checkpoint(checkpoint_dir) in olmo/model.py builds the model from config.yaml and loads model.pt of an unsharded checkpoint folder such as OLMo-7B/step323700-unsharded. We ran this on the OLMo-1B release (step374000-unsharded there), not on the files in this repository. The same repository contains scripts/convert_olmo_to_hf_new.py for converting unsharded checkpoints to the transformers format (untested by us).

Citation

@inproceedings{bordt2025forgetting,
  author    = {Sebastian Bordt and Suraj Srinivas and Valentyn Boreiko and Ulrike von Luxburg},
  title     = {How much can we forget about Data Contamination?},
  booktitle = {ICML},
  year      = {2025}
}