Paper: What's In My Big Data? (ICLR-2024)

Reviewer comment (Evaluation):
Can we look beyond the n-gram repetition to measure the duplicity? Is every n-gram duplication bad for training LLMs?

Author response:
**A1a**: While more sophisticated duplication analysis is interesting and should be explored in future work, even the basic n-gram repetition was not explored across a wide range of datasets and compared consistently before.

**A1b**: Our work is descriptive, not prescriptive, in providing the first duplication analysis across multiple web-scale corpora. This is the necessary first step to understanding the impact of duplication on trained models. The answer as to whether or not duplication is bad is very complicated and depends on many factors. While it has been explored in a few papers (which we cited in the paper, Section 4.3.2), it is still a topic under debate.

[Deduplicating Training Data Makes Language Models Better](https://aclanthology.org/2022.acl-long.577.pdf)

[Quantifying Memorization Across Neural Language Models](https://openreview.net/forum?id=TatRHT_1cK)

[Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://openreview.net/forum?id=bpRTAnJ8LW)

Reviewer comment (Experiments):
What might help to better understand the relevance of the proposed platform is to (re)train the LLMs after removing the unwanted artifacts from the dataset.

Author response:
**A2**: We agree that studying the effect of the summary statistics we computed can be explored through retraining language models on the modified datasets. However, training LLMs is out of scope for this paper, as it requires computational resources far beyond those used in the WIMBD tooling. Instead, we hope our tooling can help guide research that retrain LLMs. 
Anecdotal evidence that we would like to highlight here is that the most common n-gram analysis was already found useful. Please refer to the general response.

Reviewer comment (Evaluation):
The impact on the output generated by the LLMs due to such unwanted artifacts in the dataset is still not clear. Some analysis/exploration along this line may help to better motivate the link of work.

Author response:
**A3**: Our work was specifically designed to enable such research and to ask the right questions. We also demonstrated how WIMBD can be utilized through a series of analyses that revealed interesting insights on 10 different datasets.
We would also like to highlight a few works that have previously explored such questions. While these works had to construct dedicated tools for answering such questions, with WIMBD, these would have been available as an off-the-shelf tool.


[Impact of Pretraining Term Frequencies on Few-Shot Numerical Reasoning](https://aclanthology.org/2022.findings-emnlp.59.pdf): Studied the correlation between term frequency of numbers and the ability of models to perform few-shot numerical reasoning.

[The Bias Amplification Paradox in Text-to-Image Generation](https://arxiv.org/pdf/2308.00755.pdf): Studied the effect of bias amplification and compared biases in the training data to biases that models exhibit.

[Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4](https://arxiv.org/pdf/2305.00118.pdf): Studied memorization of texts, especially copyrighted texts of models.

[Deduplicating Training Data Mitigates Privacy Risks in Language Models](https://proceedings.mlr.press/v162/kandpal22a.html): Studies how duplicate n-grams affect their frequency appearance in model generation.

Some of these works are also mentioned in the paper at the end of Section 2 (between pages 2 and 3).
We hope our work will enable many more researchers to perform this kind of work.

Reviewer comment (Experiments):
What the paper doesn’t address is how these instances that are problematic can be efficiently filtered from the data. Would the same tool be able to serve that purpose? Then one could nicely iterate with inspection and cleaning. This is not strictly a weakness as it goes beyond the scope of the proposed tool, but this should at least be discussed to make the observations more actionable.

Author response:
Thank you for the great comment! We would like to highlight that we briefly discussed this in the Conclusions section in the "Data Curation" paragraph (Section 5). Thanks to your suggestion, we made this point more explicit and extended the paragraph to reflect that.
WIMBD can serve the purpose of curating data for improving the quality and content of large text corpora. 
Anecdotal evidence is that the most common n-gram analysis was already found useful. Please refer to the general response.

Reviewer comment (Experiments):
The tool is only evaluated on English, and this is not explicitly stated. It is unclear if there are any English-specific biases in the way that e.g. PII regexes are designed, or if the tool can be expected to seamlessly generalize to other languages.

Author response:
Thank you for highlighting this. We now explicitly state our tool was designed for English in the penultimate paragraph in the Intro (Section 1) and added the following discussion to Section 4.
Regarding the generalization of the tool and analyses, most of our analyses are not language-reliant. For instance, the most common n-grams or the length distribution. The only analysis that is language-reliant is the toxicity detection, where we use a lexicon or a model trained on English. One would require changing the model or lexicon to reproduce such analysis for other languages.
Notice, that our tools are easily extendable, and models can be swapped seamlessly.

Reviewer comment (Experiments):
Given that it is a tool that is supposed to be user-friendly, it would be good to evaluate the usability, as the main contribution of the paper is not a method but a tool. How hard was it to dig out these quality findings, would an average user spot these quickly? Some kind of user study would be good to confirm that the tool is indeed usable. Are the statistics and summaries presented with visualizations to browse? The competitor, Know Your Data, might be good to compare against in that aspect, even if it doesn’t come with the same efficiency.

Author response:
WIMBD was technically challenging to create due to the scale of the datasets, and we focused on the technical implementation of the analyses, not building a plug-and-play UI for new datasets. In addition, it's designed for use by ML practitioners handling datasets the size of the entire internet. We then demonstrated through extensive experiments that these tools can scale to 10 large-scale datasets.
Data Measurements Tool and Know Your Data are the main competitors, and, as we wrote in Section 2: "Tools like the Data Measurements Tool and Know Your Data ... focus on smaller datasets since the scale of web data leads to significant technical challenges."
WIMBD is the first step towards a scientific understanding of how to build web-scale corpora. We hope our analyses are used not just to satisfy curiosity about these datasets's contents but to shape future datasets.

We agree with the comment on the usability of WIMBD for filtering datasets for the creation of future datasets.
Regarding toxic language, the only dataset we analyzed that explicitly filtered for toxic content (in the form of keyword matching) is C4. Indeed, the matching category from our analysis are the "Offensive-*" categories. Our analysis used a fine-grained lexicon that split this category into "offensive-minority" and "offensive-not-minority". In C4, we only found 47 mentions of the "offensive-not-minority" category, likely due to a difference in the filter used to create C4 and our lexicon. In comparison, other datasets that did not employ such filters contain several million references to such phrases. Interestingly, C4 also has 158K occurrences of the "offensive-minority" category, which were not filtered from the dataset.
The table showcasing these results is presented in Table 21, and we will add this additional discussion in Appendix B.3.3. 
Regarding PII, none of the datasets we analyzed had an explicit filter for PII.

Reviewer comment (Experiments):
It would be helpful for each inspected corpus to describe briefly what kind of quality guards/filters were employed at its creation (if known), so that e.g. the presence of PII can be interpreted in context. If a corpus is supposed to be PII-filtered already, it would be very surprising to find more PII with this tool - or the regexes used here should be used to make the next generation of PII filter.

Author response:
While we agree this would be a great addition, summarizing this information from the *already reported* information from the original papers would be a significant addition. We encourage the interested reader to read the original papers associated with the dataset of interest.

Reviewer comment (Writing):
The paper provides various anecdotes and statistics about popular datasets, yet many of the dimensions are slightly underdeveloped or lack clarity.

Author response:
Please see the answers below. We would love to clarify additional details and provide further insights that you might find interesting.

Reviewer comment (Experiments):
Some datasets (e.g., PILE, Redpajama) consist of several datasets, yet it seems the paper only considers the web-crawls in those datasets.

Author response:
Each dataset in our analysis (including The Pile and RedPajama) is analyzed in full; we do not only analyze the web-crawled data.
The only corpora where we explore a subset are the ones that involve non-English text, specifically - LAION and mC4, where we look at the English subset. We clarify this in the paper at the beginning of Section 4.

Reviewer comment (Reproducibility):
Clarification is needed on what constitutes a token in this work — is it a whitespace token or derived from a subword tokenizer (and if so, which one, and is it consistent across all datasets)?

Author response:
Throughout the paper, we use the Unicode text segmentation tokenizer, which we state in footnote 1 on page 4.
Also, note that WIMBD allows the use of any tokenizer that is supported by the HuggingFace tokenizer library.

Reviewer comment (Evaluation):
In the domain counts section, while the reported statistics (e.g., less than 0.05% patents.google.com in C4, over 12% arxiv.org in RedPajama) are interesting, the discussion seems to just scratch the surface and it remains unclear what the rest of the data looks like.

Author response:
Thank you for this suggestion. We agree that a thorough analysis of the domains is an exciting opportunity!
We also agree that the source domains contain rich metadata. Our paper has many analyses in the appendix, including the distribution over internet domains (B.1.1.), internet domain schemes (B.1.2.), internet domain suffixes (B.1.3.), and estimated geolocation (B.1.5.).

In addition, we added a new analysis to the paper (Table 7 and the discussion to Appendix B.1.1). Here, we compute the quantiles of the number of tokens per domain and then report the 1, 25, 50, 75, and 99 quantiles and the number of unique domains in each corpus. We observe that 50% of the documents in C4 have 964 or fewer tokens, and 25% of the documents in LAION have 6 or fewer tokens.

Note that we will also release all of the artifacts, including the domain distribution. As such, researchers can perform additional analyses on this data.

Reviewer comment (Reproducibility):
Clarification is needed on whether N-Grams are computed over tokens or characters.

Author response:
All n-grams are computed over tokens. We mentioned this in footnote 1 on page 4 but added a clarification at the beginning of Section 4.3.1.

Reviewer comment (Writing):
The document duplication discussion mentions MD5 but also refers to "Compressed counts," which seems to describe MinHash, and this discrepancy needs clarification.

Author response:
We have different ways of hashing the original objects (here, entire documents), all of which we refer to as "compressed counts". In this case, we use the MD5 hash function.

Reviewer comment (Experiments):
The benchmark contamination analysis does not meet the high bar set by the Llama2 technical report and is not helpful for SOTA research. The selection criteria are questionable, including the discarding of benchmarks with less than one input field and the choice of Promptsource. This part should either be left out or expanded considerably with more details.

Author response:
Due to the lack of space, we omitted some details, but we added it to the revised paper, and we describe our thinking process here as well.
We acknowledge that this is an important question.
We would also like to highlight that this is the first work conducting such extensive evaluation across different datasets for comparable measurements.
We agree that additional experiments are interesting and should be answered in the future, which we leave to future work (that can be performed with WIMBD).

**Elaboration**

We looked for a large-scale benchmark that can be quickly processed and parsed. In addition, due to the scale of this experiment, we made several assumptions about what can be counted as contamination. Due to the nature of paired input tasks, the occurrence of both inputs in the same document increases the likelihood of these texts being part of a contaminated example. In contrast, datasets that involve a single input may originate from naturally occurring text from the internet, which was then labeled to create a dataset. As there is no taxonomy of such datasets, this involves manual annotation of hundreds of datasets, which we were trying to avoid in this version.
We discussed some of these decisions in the original version, in Appendix B.3.1, and added this additional discussion under the new paragraph: "Rationales of the Design Choices".