Deduplicating Training Data Makes Language Models Better (2022)

Abstract: We find that existing language modeling datasets contain many near-duplicate examples and long repetitive substrings. As a result, over 1% of the unprompted output of language models trained on these datasets is copied verbatim from the training data. We develop two tools that allow us to deduplicate training datasets—for example removing from C4 a single 61 word English sentence that is repeated over 60,000 times. Deduplication allows us to train models that emit memorized text ten times less frequently and require fewer training steps to achieve the same or better accuracy. We can also reduce train-test overlap, which affects over 4% of the validation set of standard datasets, thus allowing for more accurate evaluation. Code for deduplication is released at https://github.com/google-research/deduplicate-text-datasets .

## introduction a key factor behind the recent progress in natural language processing is the development of largescale text corpora used to train increasingly large language models. these datasets have grown from single gigabytes to as much as a terabyte over the past few years @xcite @xcite . because it is so expensive to perform manual review and curation on massive datasets, they tend to suffer in quality compared to their smaller predecessors. this has implications far beyond metrics like perplexity and validation loss, as learned models reflect the biases present in their training data @xcite @xcite . quantitatively and qualitatively understanding these datasets is therefore a research challenge in its own right @xcite . we show that one particular source of bias, duplicated training examples, is pervasive: all four common nlp datasets we studied contained duplicates. additionally, all four corresponding validation sets contained text duplicated in the training set. while naive deduplication is straightforward (and the datasets we consider already perform some naive form of deduplication), performing thorough deduplication at scale is both computationally challenging and requires sophisticated techniques. we propose two scalable techniques to detect and remove duplicated training data. exact substring matching identifies verbatim strings that are repeated. this allows us to identify cases where only part of a training example is duplicated ( §4.1). approximate full document matching uses hashbased techniques @xcite to identify pairs of documents with high n-gram overlap ( §4.2). we identify four distinct advantages to training on datasets that have been thoroughly deduplicated. to summarize, data duplication offers significant advantages and no observed disadvantages. in the remainder of this paper we present our text deduplication framework in §4, and study the extent of duplicate content in common nlp datasets (e.g., c4, wiki-40b, and lm1b) in §5. we then examine the impact of deduplication on test perplexity ( §6.1) and on the frequency of emitting memorized content ( §6.2). finally, we analyze to what extent perplexity on existing, released models are skewed as a result of overlap between the train and test/validation splits ( §6.3). ## related work large language model datasets. while we believe our results are independent of model architecture, we perform our analysis on transformerbased decoder-only language models @xcite ) trained for open-ended text generation. these current state-of-the-art models are trained on internet text. for example, the gpt-2 family of models @xcite is trained on web-text, a dataset of web documents highly ranked on reddit-however this dataset was not made available publicly. a common dataset starting point is commoncrawl, an index of public webpages. among the models trained on commoncrawl include gpt-3 @xcite with the addition of book datasets, grover @xcite on a restricted subset filtered to news domains called realnews, and t5 @xcite on a cleaned version of common crawl called c4. other models are trained on more curated internet sources-for example guo et al. (2020) used high quality processed wikipedia text from 40 different languages to train monolingual 141.4m parameter language models. non-english models necessarily use different datasets; zeng et al. ( 2021 ) for instance introduced pangu-α, a family of models with up to 200b parameters that were trained on a non-public corpus of cleaned and filtered chinese-language documents from commoncrawl and other sources. since many of these datasets are not public, we deduplicate three that are: wiki-40b, c4, and realnews-as well as the one billion word language model benchmark @xcite , a smaller dataset commonly used for evaluation. ## language modeling datasets we analyze the presence of duplicate text in four datasets of varying sizes that have been used for training natural language generation systems, producing general-purpose pre-trained models, and for language model benchmarking. while this paper restricts itself to english datasets, we expect that non-english datasets suffer from similar issues and could likewise benefit from de-duplication. wikipedia (wiki-40b) consists of multi-lingual cleaned wikipedia text @xcite . we take the english portion, which contains 2.9m wikipedia pages with an average length of 768 bpe tokens. the dataset creators do not indicate any deduplication was performed aside from removing redirect-pages (e.g., "sunflower" to "helianthus"). one-billion word benchmark (lm1b) contains 30m sentences of news commentary @xcite . unlike the other datasets we analyze, lm1b's examples are one sentence long rather than multi-sentence documents. the average example length is 32 bpe tokens. while this dataset is extremely standard for benchmarking language models, @xcite ) note it has 13.2% overlap of the test set with the train set. colossal cleaned common crawl (c4) is made up of 360m web documents, with an average length of 486 bpe tokens @xcite . c4 was introduced as a pre-training dataset for t5, a set of encoder-decoder models which have been widely used in fine-tuned downstream tasks. the dataset was previously deduplicated in a more sophisticated process than the prior two datasets. each paragraph was hashed and paragraphs resulting in hash collisions were removed. this was followed by a pass that removed placeholder text, code, and prohibited words. see @xcite for a detailed breakdown of the source text in c4. realnews is a subset of the common crawl consisting of articles from news domains @xcite . it contains 31m documents with average length 793 bpe tokens. realnews was deduplicated by inserting a hash of the first 100 characters of each document into a bloom filter @xcite and then excluding any document which resulted in a hash collision. like c4, examples with duplicate urls were excluded. ## methods for identifying duplicates the simplest technique to find duplicate examples would be to perform exact string matching between all example pairs, but as we will show, this is insufficient. we introduce two complementary methods for performing deduplication. first, using a suffix array @xcite , we remove duplicate substrings from the dataset if they occur verbatim in more than one example. second, we use minhash @xcite , an efficient algorithm for estimating the n-gram similarity between all pairs of examples in a corpus, to remove entire examples from the dataset if they have high n-gram overlap with any other example. we consider a dataset @xmath0. each of these examples is itself a sequence of tokens: ## deduplication results we deduplicate each of the four datasets with both of our two techniques. when text was duplicated across multiple data splits, we prioritized keeping a copy in the test or validation set and removing it from the train set. ## impact on trained models for all experiments, we used a byte pair encoding (bpe) vocabulary trained on c4-neardup with a budget of 50k tokens, which resulted in a vocabulary the same size as gpt-2's. we trained with a maximum sequence length of 512 tokens (for longer documents, we randomly extracted subsequences of this length.) further training details can be found in appendix c. ## discussion the focus of this paper is on the datasets used to train language models. while recent work focused on documenting the potential harms that could arise from problematic datasets @xcite , less work has been done to quantitatively analyze properties of real language modelling datasets, like @xcite has done for c4. our paper provides analysis on one particular axis, that of data duplication. our experiments measured what could be quantified: the amount of duplicate content in common datasets, the effect of deduplication on trained model perplexity, and the reduction of memorized content in trained models through deduplication. we do not focus on the nature of the data being removed by deduplication or memorized by lms. privacy is an important subject for future work, as memorized training data has significant privacy consequences. by this, we mean the standard privacy definition that a model should not reveal anything particular to the specific dataset it was trained on, as opposed to another training dataset from a similar distribution @xcite . 4 training on standard datasets that have not yet been deduplicated results in models that are particularly sensitive to examples that happened to be repeated multiple times, and this has negative privacy implications. for instance, it could violate a person's expectations of privacy if their publicly available personal data appeared in a different, surprising context. downstream applications of lms, such as the game ai dungeon foot_2 , should also not output memorized content like adverts for real products. we stress that in our experiments, we do not distinguish between undesired memorized text (such as phone numbers), innocuous memorized text (common phrases), and text we may want to be memorized (such as a quote by a public figure), and instead treat all instances of the lm generating text that closely matches the training set as problematic. while we qualitatively observed that much of the identified memorized content was relatively innocuous, a more systematic study of the risks associated with the detected memorization was beyond the scope of this work. we also do not investigate the negative consequences of deduplication. some language tasks explicitly require memorization, like document retrieval or closed-book question answering. also, text that gives attribution is often duplicated across documents, so removing duplicate substrings could correspond to removing just the attribution, which could result in models that learn the content without its attached attribution. deduplication is also not sufficient to remove privacy-sensitive data like bank passwords and medical records which should never be used in training data @xcite . ultimately, whether memorization is a desired property of a language model, or else risky and unwanted, depends both on the nature of the text that has been memorized and on the downstream applications of the trained model. however, since the trend has been towards creating datasets and models that are application-agnostic, we encourage researchers to think carefully about the limitations of the data they have collected and the how the model's intended usage constrains what should be part of the training set. developing techniques to memorize or forget specific sequences depending on the end application is a promising research direction. ## conclusion we encourage future language model research to perform dataset deduplication, either by training on the deduplicated datasets we release, using the deduplication tools we release, or following our approach to deduplicate datasets with new tools. the exact technique used to perform deduplication is less important than performing stringent deduplication in the first place. on the whole, dedu-plication does not harm, and sometimes improves, model perplexity, despite the fact that the deduplicated datasets are smaller and faster to train on. it is especially important that there are no duplicates between the training and testing sets, because overlap here explicitly encourages selecting models that memorize the training data. lastly, deduplication helps to reduce some of the privacy concerns around lms memorizing their training data.