Paper: OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text (ICLR-2024)

Reviewer comment (Novelty):
The advance of OpenWebMath compared with existing datasets such as Proof-Pile is not provided.

Author response:
Proof Pile is a hand-curated mathematics dataset that mostly ArXiv math papers. OpenWebMath is a web-scale dataset of 14.7B tokens which contains over 100k unique domains. The sources present in Proof Pile are distributed as follows:

- ArXiv.math (13B tokens)
- Open-source math textbooks and formal mathematics libraries (265M tokens)
- Math Overflow and Math Stack Exchange (825M tokens)
- Wiki-style sources (17M tokens)
- MATH training set (2M tokens)

Unlike OpenWebMath, Proof Pile itself is not a great dataset for math problem solving, likely since the vast majority of its data is arXiv papers which do not contain as many instances of step-by-step problem solving and usually skip basic concepts. OpenWebMath is complementary to Proof Pile and includes almost exclusively data that wasn’t in that dataset. As shown in our experiments, training on a diverse mixture of documents from OpenWebMath and Proof Pile is essential to train the best model.

We will add a more thorough description of Proof Pile to the paper as well as the following table that illustrates how the two datasets are complementary.

|  | Papers | Formal Math | Web Pages | Openly Available |
| --- | --- | --- | --- | --- |
| Minerva Web Math Pages | 21B | 0 | 17.5B | No |
| ProofPile | 13B | 0.2B | 0.8B | Yes |
| OpenWebMath | 0 | 0 | 14.7B | Yes |

Reviewer comment (Novelty):
What is the advance of OpenWebMath compared with existing datasets such as Proof-Pile?

Author response:
Proof Pile is a hand-curated mathematics dataset that mostly ArXiv math papers. OpenWebMath is a web-scale dataset of 14.7B tokens which contains over 100k unique domains. The sources present in Proof Pile are distributed as follows:

- ArXiv.math (13B tokens)
- Open-source math textbooks and formal mathematics libraries (265M tokens)
- Math Overflow and Math Stack Exchange (825M tokens)
- Wiki-style sources (17M tokens)
- MATH training set (2M tokens)

Unlike OpenWebMath, Proof Pile itself is not a great dataset for math problem solving, likely since the vast majority of its data is arXiv papers which do not contain as many instances of step-by-step problem solving and usually skip basic concepts. OpenWebMath is complementary to Proof Pile and includes almost exclusively data that wasn’t in that dataset. As shown in our experiments, training on a diverse mixture of documents from OpenWebMath and Proof Pile is essential to train the best model.

We will add a more thorough description of Proof Pile to the paper as well as the following table that illustrates how the two datasets are complementary.

|  | Papers | Formal Math | Web Pages | Openly Available |
| --- | --- | --- | --- | --- |
| Minerva Web Math Pages | 21B | 0 | 17.5B | No |
| ProofPile | 13B | 0.2B | 0.8B | Yes |
| OpenWebMath | 0 | 0 | 14.7B | Yes |

Reviewer comment (Experiments):
The authors should provide an example of a dataset in the paper.

Author response:
Thank you for the suggestion. We will add a figure that shows an example from the dataset to the paper.

Reviewer comment (Presentation):
The order in which the table appears is inconsistent with the logic of the text.

Author response:
We will rearrange the figures in the paper to be closer to the text discussing them.

Reviewer comment (Experiments):
There are invisible Unicode characters and some text in other languages in the data.

Author response:
Thank you for pointing this out. We did not do any processing to remove invisible Unicode characters simply because we were not aware this was a problem. We will add a script in our final release to remove such characters from the dataset.

Reviewer comment (Reproducibility):
Why there are some invisible Unicode characters and other language text in sample_dataset.jsonl? For instance the 40th and 43th lines of sample_dataset.jsonl.

Author response:
In general, we cannot guarantee that every document in OpenWebMath is high quality since there are many documents and our filters, no matter how much they are tuned, will never be 100% accurate. Additionally, we expect a small amount of low-quality documents or a small amount of documents in another language to be present in our dataset by design, since we focus on optimizing for recall over extremely precise filtering in order to preserve more information. Users can always further filter and remove documents from OpenWebMath if their application calls for it.

Still, we find that our dataset quality is often higher (by inspection) than that of other web datasets like C4 and RefinedWeb and we provide evidence in the paper that training on our dataset results in strong performance.

Reviewer comment (Experiments):
The paper presents no special insight on the dataset or the effect of the processing steps applied - it only describes the pre-processing pipeline. A key aspect that would strengthen this paper is a description of the overlap (computed in some apt way: e.g., overlap of URLs, text overlap, others) between OpenWebMath and the benchmark datasets it evaluates on - computing overlap with other popular reasoning benchmarks would also be a welcome addition.

Author response:
We believe that releasing open, well documented datasets such as OpenWebMath are critical to enabling good science in the fields of LLMs. Certainly one of the core reasons for this is that it enables greater transparency into the relationship between data overlap and downstream performance. Per your request, we evaluated the n-gram overlap between two datasets and used this to measure the overlap between our benchmarks datasets and OpenWebMath. As shown in the table below, only 348 out of 5000 MATH problems, 2 out of 1319 GSM8k problems, and 2 of 1310 LILA problems tested overlap with OpenWebMath. Due to the potential for false positives that we discovered during our overlap analysis, we opt to leave decontamination to those training models on the dataset. We will update the paper to include this overlap analysis (and include the code and overlap locations in our release) and we are open to adding other reasoning benchmarks to this analysis if they are suggested.

| Evaluation | Problems with 30-gram Overlap | Total Problems |
| --- | --- | --- |
| MATH | 348 | 5000 |
| GSM8k | 2 | 1319 |
| LILA (all tested) | 3 | 1310 |

Reviewer comment (Experiments):
It seems like the MATH dataset was gathered from aops.com/community/c3158_usa_contests. Is this a part of Common Crawl? What is the overlap between OpenWebMath and MATH? This is important given concerns of dataset contamination with web-scale datasets: https://arxiv.org/abs/2310.10628

Author response:
We believe that releasing open, well documented datasets such as OpenWebMath are critical to enabling good science in the fields of LLMs. Certainly one of the core reasons for this is that it enables greater transparency into the relationship between data overlap and downstream performance. Per your request, we evaluated the n-gram overlap between two datasets and used this to measure the overlap between our benchmarks datasets and OpenWebMath. As shown in the table below, only 348 out of 5000 MATH problems, 2 out of 1319 GSM8k problems, and 2 of 1310 LILA problems tested overlap with OpenWebMath. Due to the potential for false positives that we discovered during our overlap analysis, we opt to leave decontamination to those training models on the dataset. We will update the paper to include this overlap analysis (and include the code and overlap locations in our release) and we are open to adding other reasoning benchmarks to this analysis if they are suggested.

| Evaluation | Problems with 30-gram Overlap | Total Problems |
| --- | --- | --- |
| MATH | 348 | 5000 |
| GSM8k | 2 | 1319 |
| LILA (all tested) | 3 | 1310 |

Reviewer comment (Experiments):
How does OpenWebMath differ from ProofPile? Are there obvious reasons why using OpenWebMath results in significantly better performance than ProofPile?

Author response:
Proof Pile is a hand-curated mathematics dataset that mostly ArXiv math papers. OpenWebMath is a web-scale dataset of 14.7B tokens which contains over 100k unique domains. The sources present in Proof Pile are distributed as follows:

- ArXiv.math (13B tokens)
- Open-source math textbooks and formal mathematics libraries (265M tokens)
- Math Overflow and Math Stack Exchange (825M tokens)
- Wiki-style sources (17M tokens)
- MATH training set (2M tokens)

Unlike OpenWebMath, Proof Pile itself is not a great dataset for math problem solving, likely since the vast majority of its data is arXiv papers which do not contain as many instances of step-by-step problem solving and usually skip basic concepts. OpenWebMath is complementary to Proof Pile and includes almost exclusively data that wasn’t in that dataset. As shown in our experiments, training on a diverse mixture of documents from OpenWebMath and Proof Pile is essential to train the best model.

We will add a more thorough description of Proof Pile to the paper as well as the following table that illustrates how the two datasets are complementary.

|  | Papers | Formal Math | Web Pages | Openly Available |
| --- | --- | --- | --- | --- |
| Minerva Web Math Pages | 21B | 0 | 17.5B | No |
| ProofPile | 13B | 0.2B | 0.8B | Yes |
| OpenWebMath | 0 | 0 | 14.7B | Yes |

Reviewer comment (Reproducibility):
The citation for ProofPile ("Proofnet: Autoformalizing and formally proving undergraduate-level mathematics.") seems incorrect. Please consider adding a note of what the dataset is and its source.

Author response:
Proof Pile is hosted here: https://huggingface.co/datasets/hoskinson-center/proof-pile. We have checked with the authors of Proof Pile and confirmed that the Proofnet citation is correct.

Reviewer comment (Experiments):
What exactly is LILA-multiarith? It seems the LILA benchmark contains multiple different datasets — why did the evaluation here only use this one dataset in the benchmark?

Author response:
LILA is a large collection of math benchmarks of varying quality and required skill levels, so we just picked one that we thought tested a unique aspect relative to our existing benchmarks. The LILA evaluations involve running Python code in order to produce an answer and LILA-multiarith is essentially a Python version of the multiarith benchmark. 

We have now evaluated our models on all the LILA evaluations which have numerical answers and automatic evaluation below. Note the accuracies are quite low on some of the evaluations across the board due to their difficulty, which can result in a lot of variance in scores - we believe perplexity evaluations (included in the paper) are the most appropriate for measuring the strength of datasets like OpenWebMath at this type of scale since performance scales more smoothly with this metric [1].

|  | The Pile | ProofPile | OpenWebMath | Mixture | Pythia 1.4b |
| --- | --- | --- | --- | --- | --- |
| MATH-Algebra-Easy | 2.81% | 2.81% | **5.62%** | 5.06% | 3.93% |
| MATH-Algebra-Easy maj@16 | 3.93% | 3.93% | 9.55% | **10.11%** | 5.62% |
| LILA-multiarith | 9.77% | 8.04% | **16.67%** | 13.22% | **21.80%** |
| LILA-mathqa-probability | 0.00% | 0.00% | 0.00% | 0.00% | **4.17%** |
| LILA-mathqa-general | 0.13% | 0.13% | **0.51%** | 0.39% | 0.00% |
| LILA-mathqa-gain | **1.02%** | **1.02%** | 0.26% | 0.26% | 0.26% |
| LILA-mathqa-geometry | 0.85% | 0.85% | **2.56%** | 0.85% | **2.56%** |
| LILA-mathqa-other | 0.00% | 0.00% | **1.10%** | 0.00% | 0.00% |
| LILA-mathqa-physics | 0.61% | 0.82% | **1.23%** | 1.02% | **1.43%** |
| LILA-GSM8k-structured | 1.30% | 0.84% | **2.21%** | 1.68% | 0.92% |
| LILA-add-sub | 33.94% | 26.61% | 44.95% | **46.79%** | **53.21%** |
| LILA-asdiv | 5.50% | 4.37% | 16.50% | **17.80%** | **19.58%** |
| LILA-svamp-structured | 5.35% | 7.69% | 10.70% | **14.72%** | **11.71%** |

Reviewer comment (Experiments):
In similar vein to the above comments, does the data here overlap with OpenWebMath?

Author response:
Please see our above response for the data overlap results for LILA.

Reviewer comment (Experiments):
Please consider citing the original source of the multiarith dataset in addition to the benchmark. It is not clear what the original data is. The citation chain to the original dataset seems to be: https://arxiv.org/pdf/2210.17517.pdf (LILA) → https://arxiv.org/pdf/1608.01413.pdf (methodological paper using the data?) → https://aclanthology.org/Q15-1001.pdf (original data) — please verify this.

Author response:
We will clean up our citations in order to cite the original sources of the LILA benchmarks.

Reviewer comment (Presentation):
Please consider describing the tasks of Table 2 in more detail.

Author response:
MATH Algebra-Easy is the Algebra subset of the MATH benchmark, filtered down to only questions with difficulty level 1. We evaluated our models on MATH Algebra-Easy with both greedy decoding and self-consistency. We also evaluated models on LILA tasks, which involve writing Python code to solve a math word problem and contains many subsets corresponding to different types of math problems. These problems test the ability to do mathematical problem solving while offloading arithmetic and computation to the Python interpreter. We tested on all the LILA subsets which have numerical answers, making it simple to automatically evaluate the performance of our models. We will add a paragraph with this information to the paper.

Reviewer comment (Presentation):
Please place a table or figure closer to the texts discussing it.

Author response:
We will rearrange the figures in the paper to be closer to the text discussing them.