Q UANTIFYING M EMORIZATION A CROSS N EURAL L ANGUAGE M ODELS
A BSTRACT
Large language models (LMs) have been shown to memorize parts of their training data, and when prompted appropriately, they will emit the memorized training data verbatim.
This is undesirable because memorization violates privacy (exposing user data), degrades utility (repeated easy-to-memorize text is often low quality), and hurts fairness (some texts are memorized over others).
We describe three log-linear relationships that quantify the degree to which LMs emit memorized training data.
Memorization signiﬁcantly grows as we increase (1) the capacity of a model, (2) the number of times an example has been duplicated, and (3) the number of tokens of context used to prompt the model.
Surprisingly, we ﬁnd the situation becomes more complicated when generalizing these results across model families.
On the whole, we ﬁnd that memorization in LMs is more prevalent than previously believed and will likely get worse as models continues to scale, at least without active mitigations.
I NTRODUCTION
The performance of neural language models has continuously improved as these models have grown from millions to trillions of parameters (Fedus et al., 2021), with their training sets similarly growing from millions to trillions of tokens.
In anticipation of future, even larger models trained on minimally curated datasets, it is important to quantify factors that lead to increased memorization of a model’s training set.
Indeed, recent work has shown that training data extraction attacks are a practical threat for current language models (Carlini et al., 2020); an adversary interacting with a pretrained model can extract individual sequences that were used to train the model.
While current attacks are effective, they only represent a lower bound on how much memorization occurs in existing models.
For example, by querying the GPT-2 language model, Carlini et al.
(manually) identiﬁed just 600 memorized training examples out of a 40 GB training dataset.
This attack establishes a (loose) lower bound that at least 0.00000015% of the dataset is memorized.
In contrast, we are able to show that the 6 billion parameter GPT-J model (Black et al., 2021; Wang and
Komatsuzaki, 2021) memorizes at least 1% of its training dataset : The Pile (Gao et al., 2020).
In addition to prior work’s loose estimates of models’ memorization capabilities, there is a limited understanding of how memorization varies across different neural language models and datasets of different scales.
Prior studies of memorization in language models either focus on models or datasets of a ﬁxed size (Carlini et al., 2019;
Zhang et al., 2021; Thakkar et al., 2020) or identify a narrow memorization-versus-scale relationship (Carlini et al., 2020; Lee et al., 2021).
While McCoy et al.
(2021) broadly study the extent to which language models memorize, their focus is on how to avoid the problem and ensure novelty of model outputs, rather than on studying model risk through identifying the maximal amount of data memorization.
This paper addresses both of the above open questions by comprehensively quantifying memorization across three families of neural language models and their associated datasets.
We leverage access to each model’s original training set to provide order-of-magnitude more precise bounds on the amount of extractable data that an adversary could recover than in prior works.
We ﬁrst construct a set of prompts from the model’s training set.
By feeding preﬁxes of these prompts into the trained model, we check whether the model has the ability to complete the rest of the example verbatim.
This allows us to measure memorization across models, datasets, and prompts of varying sizes.
We identify three properties that signiﬁcantly impact memorization:
Model scale: Within a model family, larger models memorize 2 - 5 × more than smaller models.
Data duplication: Examples repeated more often are more likely to be extractable.
Context:
It is orders of magnitude easier to extract sequences when given a longer context.
Our analysis suggests that future research on neural language modeling will need to take steps to prevent future (larger) models from memorizing their training datasets.
R ELATED W ORK
There is extensive prior work that qualitatively studies memorization in neural language models.
Prior work has demonstrated extraction attacks that recover memorized data including URLs, phone numbers, and other personal information (Carlini et al., 2020; Ziegler, 2021)—or synthetically injected “canaries” (Carlini et al., 2019; Henderson et al., 2018; Thakkar et al., 2020; Thomas et al., 2020).
However most of these works are qualitative and aim to demonstrate the existence of extractable data, rather than precisely quantifying how much models memorize.
For example, the unprompted memorization evaluation of Carlini et al.
(2020) found just 600 examples of memorization in GPT-2.
Our paper aims to establish tighter bounds on the fraction of a dataset that is memorized.
Our analysis is relevant to the broad literature on privacy attacks on machine learning.
For example, membership inference attacks (Shokri et al., 2017; Yeom et al., 2018) let an adversary detect the presence of a given example in a model’s training set; other forms of data leakage let an adversary learn dataset properties (Ganju et al., 2018;
Fredrikson et al., 2015).
We focus on extraction attacks due to their relevance for language modeling—extraction implies signiﬁcant leakage from a model, and grows with data duplication (Lee et al., 2021), a common feature of large-scale text datasets.
Various deﬁnitions of memorization in deep neural networks have been studied in prior work (Carlini et al., 2019; 2020; Feldman and Zhang, 2020; Zhang et al., 2021).
A detailed comparison with those existing formulations is presented in Section 3.1.
One leading general memorization deﬁnition is differential privacy (Dwork et al., 2006), which formalizes the idea that removing any one example from the training set should not change the trained model.
However, while differential privacy protects a single user’s private information, it is ineffective for preventing memorization of highly duplicated data, and does not capture the complexity of social, linguistic data (Brown et al., 2022).
Also, differentially private learning algorithms (Abadi et al., 2016) generally suffer from expensive computation, slow convergence, and poor model utility, despite recent advances (Anil et al., 2021).
In concurrent work, Kandpal et al.
(2022) study how often models emit memorized data as a function of data duplication.
Their analysis focuses on evaluating why training data extraction attacks succeed.
In contrast, we explicitly prompt models with training data preﬁxes in order to measure memorization in the worst case, something that a practical attack cannot necessarily do.
Prior scaling hypotheses.
Our motivation to study scaling phenomena stems from anecdotal evidence in prior work that memorization ability relates to various aspects of scale.
In particular, our analysis on model scale is informed by preliminary experiments in (Zhang et al., 2017; Carlini et al., 2020), our data duplication experiments follow in the line of Lee et al.
(2021), and our context length experiments build on hypotheses by Carlini et al.
(2020); Ziegler (2021).
M ETHODOLOGY 3. D EFINITION OF M EMORIZATION
To begin, we ﬁrst select a precise deﬁnition for memorization:
Deﬁnition 3.1.
A string s is extractable with k tokens of context from a model f if there exists a (length- k ) string p , such that the concatenation [ p || s ] is contained in the training data for f , and f produces s when prompted with p using greedy decoding.
For example, if a model’s training dataset contains the sequence “My phone number is 555-6789” , and given the length k
= 4 preﬁx “My phone number is” , the most likely output is “555-6789” , then this sequence is extractable (with 4 words of context).
We focus on greedy sampling in this paper, and verify in Section 4.1 that our choice of decoding strategy does not signiﬁcantly impact our results.
While prior work proposed other deﬁnitions, we prefer ours in this paper as it is more actionable.
Some memorization deﬁnitions, including lower-bounds on differential privacy (Dwork et al., 2006;
Jagielski et al., 2020; Nasr et al., 2021) or counterfactual memorization (Feldman and Zhang, 2020;
Zhang et al., 2021), require training hundreds or thousands of models, which is impractical for large language models.
Alternatively, computing exposure (Carlini et al., 2019) requires thousands of generations per sequence, and is only designed for carefully crafted training examples.
Finally, k -eidetic memorization (Carlini et al., 2020), is a useful deﬁnition for unprompted memorization, but less useful for tightly bounding memorization by prompting with training data (as we will do).
Future work might explore how our three scaling observations apply to other deﬁnitions of memorization.
3. S ELECTION OF E VALUATION D ATA
Having chosen a deﬁnition, we next describe our evaluation procedure.
Ideally, we would consider every sequence x =
[ p || s ] in the model’s training dataset (where x has been split into a length- k preﬁx p and a sufﬁx s ).
For each sequence, we would report if the model exactly reproduces s when prompted with p , following Deﬁnition 3.1.
Unfortunately, performing this test on every sequence in the training data would be prohibitively expensive.
For example, the largest 6 billion parameter
GPT-Neo model has a throughput of roughly one 100-token generation per second on a V100 GPU.
Extrapolating to the 800GB training dataset, this would require over 30 GPU-years of compute.
Instead, we query on a smaller subset of the training data, that still produces statistically conﬁdent estimates.
In this paper we randomly choose subsets of roughly 50 , 000 sequences, allowing us to efﬁciently run inference in just a few hours.
The primary criteria when choosing a subset of the training data is to obtain a representative sample that allows us to draw meaningful conclusions from the data.
We consider two approaches to constructing a subset of the data.
Our ﬁrst subset is a uniformly random sample of 50 , 000 sequences, drawn from the training dataset without repetition.
While a uniform sample is useful to estimate the absolute amount of memorization in a model, it is poorly suited for studying how memorization scales with data properties that are not uniformly represented in the training set.
For example, prior work has identiﬁed that data duplication (i.e., how often the same sequence is repeated either exactly or approximately) is an important factor for memorization.
Yet, because the frequency of training data duplication decays extremely quickly (Lee et al., 2021), a uniformly random sample of 50 , 000 sequences (accounting for ≤ 0 .
02% of the dataset) is unlikely to contain any signal that would allow us to accurately measure the tail of this repeated data distribution.
A similar concern arises for measuring how memorization scales with prompt length, since very long sentences account for only a small fraction of the training set.
Therefore, our second subset is a random sample normalized by both sequence lengths and duplication counts, which allows us to accurately measure memorization of large language models in the worst-case, on highly duplicated data with long prompts.
For each sequence length (cid:96) ∈ { 50 , 100 , 150 , . . . , 500 } , and integer n , we select 1 , 000 sequences of length (cid:96) that are contained in the training dataset between 2 n/ 4 and 2 ( n +1) / 4 times.
We do this until we reach an n for which 1 , 000 sequences are not available.
This gives us 1 , 000 sequences that repeat between 6 and 8 times ( ≈ 2 11 / 4 and ≈ 2 12 / 4 ) and also 1 , 000 sequences that repeat between 724 and 861 times ( ≈ 2 38 /and
≈ 2 39 / 4 ).
This biased sampling allows us to more accurately measure memorization as a function of a sample’s duplication factor and prompt length, without querying the entire dataset.
Note that constructing this duplicate-normalized data subset requires some work, as efﬁciently identifying duplicate substrings in an 800 GB training dataset is computationally challenging.
We make use of the sufﬁx array construction from Lee et al.
(2021) (see Appendix).
For each length from 50 to 500 tokens, we collect 50 , 000 examples duplicated varying numbers of times, totaling roughly 500 , 000 sequences.
For each sequence of length (cid:96) , we prompt the model with [Figure] the ﬁrst (cid:96) − 50 tokens and report the sequence as “extractable” if the model exactly emits the nexttoken sufﬁx of this sequence.
Fifty tokens corresponds to an average of 127 characters or 25 wordsin the GPT-Neo training set, well over the length of a typical English sentence.
Finally, we compute the average probability that a sequence is extractable by averaging over all lengths (cid:96) .
E XPERIMENTS
We primarily study the GPT-Neo model family (Black et al., 2021; Wang and Komatsuzaki, 2021) trained on the Pile dataset (Gao et al., 2020).
The GPT-Neo models are causal language models trained with the objective of predicting the next token in a sequence given the previous ones.
They come in four sizes: 125 million, 1 . 3 billion, 2 .
7 billion and 6 billion parameters.
The Pile is a dataset of 825 GB of text collected from various sources (e.g., books, Web scrapes, open source code).
Prior to the recent release of OPT (Zhang et al., 2022), the GPT-Neo models were the largest language models available for public download, and The Pile is the largest public text dataset available.
4. B IGGER M ODELS M EMORIZE M ORE
We begin by considering the impact of model size on memorization, expanding on prior studies which qualitatively established a relationship between the size of GPT-2 models and their ability to memorize < 30 URLs (Carlini et al., 2020).
In contrast, we study a million model generations in order to describe how model scale relates to memorization.
Results.
We ﬁrst study our biased random data sample normalized by duplication count and sequence lengths.
The results of this experiment are given in Figure 1a.
The y-axis reports the fraction of generations which exactly reproduce the true sufﬁx for their prompt, averaged over all prompt and sequence lengths in our evaluation set.
Because our biased sampling over-represents duplicated strings, the absolute degree of memorization in Figure 1a is not particularly important here—rather, we are interested in how memorization varies with scale.
2 We ﬁnd that larger models memorize signiﬁcantly more than smaller models do, with a near-perfect log-linear ﬁt ( R 2 of 99.8%): a ten fold increase in model size corresponds to an increase in memorization of 19 percentage points.
To conﬁrm that larger models are indeed memorizing more data, and not simply generalizing better, we repeat the analysis with the GPT-2 model family as a baseline.
The GPT-2 models are similarly sized, and also trained on Internet-scraped data.
If our “larger models memorize more” result was due to the predictive strength of larger models, and not the memorization of speciﬁc training data, we would expect a similar relationship between comparably sized GPT-2 models trained on similar data.
Put differently, this baseline allows to establish what fraction of the training data is sufﬁciently “easy” that any language model can correctly predict the 50-token sufﬁx, even if the example has not been seen during training.
For example, a language model trained on multiple examples of number sequences can likely correctly complete some other unseen number sequences.
We ﬁnd that GPTcorrectly completes approximately 6% of the examples in our evaluation set, compared to 40% for the similarly sized 1.3B parameter GPT-Neo model.
A qualitative analysis (see examples in Appendix
Figure 15) suggests that examples “memorized” by GPT-2 are largely uninteresting sequences (e.g., number sequences, repetitions of the same few tokens, or common phrases).
Therefore, we conclude that larger models have a higher fraction of extractable training data because they have actually memorized the data; it is not simply that the larger models are more accurate.
4. R EPEATED S TRINGS ARE M EMORIZED M ORE
Prior work provides preliminary evidence that memorization in language models increases with the number of times sequences are repeated in the training set (Carlini et al., 2020; Lee et al., 2021).
We expand on this observation and quantitatively measure the effect of data duplication on memorization.
Using our duplication-normalized data sample, we measure the fraction of sequences which are extractable, for buckets of sequences duplicated between 2 and 900 times.
Each bucket consists of 1 , 000 distinct sentences, and we compute the average amount of memorization for each bucket.
Results.
Figure 1b shows our results, aggregated over all sequence lengths.
We observe a clear log-linear trend in memorization.
While models rarely regurgitate strings that are repeated only a few times, this probability increases severely for highly duplicated strings.
The small memorization values at low numbers of repetitions corroborates the positive impact of training dataset deduplication on memorization observed by Lee et al.
However, we ﬁnd that memorization does still happen, even with just a few duplicates—thus, deduplication will not perfectly prevent leakage.
While this relationship is perhaps obvious, and has been corroborated for speciﬁc training examples in prior work (Carlini et al., 2019; 2020), our results show that it holds across the entire training set .
4. L ONGER C ONTEXT D ISCOVERS M ORE M EMORIZATION
The previous two questions evaluated how data collection and model training decisions impact the leakage of a model’s training data when it is provided a ﬁxed number of tokens from a sequence as context.
As a result, those experiments suggest particular actions that could be taken to mitigate memorization (by reducing model size, or limiting the number of duplicate examples).
However, even when the model is ﬁxed, it is possible to vary the amount of extractable training data by controlling the length of the preﬁx passed to the model.
By studying how the number of tokens of context impacts extractability, we demonstrate the difﬁculty of discovering memorization—language models may only exhibit their memorization under favorable conditions.
Results.
In Figure 1c, we observe that the fraction of extractable sequences increases log-linearly with the number of tokens of context.
For example, 33% of training sequences in our evaluation set are extractable from the 6B model at 50 tokens of context, compared to 65% with 450 tokens of context.
We call this the discoverability phenomenon : some memorization only becomes apparent under certain conditions, such as when the model is prompted with a sufﬁciently long context.
The discoverability phenomenon may seem natural: conditioning a model on 100 tokens of context is more speciﬁc than conditioning the model on 50 tokens of context, and it is natural that the model would estimate the probability of the training data as higher in this situation.
However, the result is that some strings are “hidden” in the model and require more knowledge than others to be extractable.
From one point of view, it is good that some memorization is difﬁcult to discover.
This makes it harder for attackers to perform training data extraction attacks (Carlini et al., 2020), or otherwise exploit memorization.
Indeed, if an exact 100 token prompt is required to make the model output a given string, then, in practice, an adversary will likely be unable to perform the attack.
The difﬁculty [Figure] in discovering memorization also reduces the likelihood of non-adversarial training data regurgitation.
For example, the GitHub Copilot model (Chen et al., 2021) reportedly rarely emits memorized code in benign situations, and most memorization occurs only when the model has been prompted with long code excerpts that are very similar to the training data (Ziegler, 2021).
Practitioners building language generation APIs could (until stronger attacks are developed) signiﬁcantly reduce extraction risk by restricting the maximum prompt length available to users.
Viewed differently, however, the difﬁculty of discovering memorization can also harm our ability to audit privacy in machine learning models.
Because provably-correct approaches for privacypreserving training of machine learning models are applied only rarely in practice (Abadi et al., 2016; Thakkar et al., 2020; Ramaswamy et al., 2020), it is common to attempt post-hoc privacy auditing (Jayaraman and Evans, 2019; Jagielski et al., 2020; Nasr et al., 2021).
Our results suggest that correctly auditing large language models likely requires prompting the model with training data, as there are no known techniques to identify the tail of memorized data without conditioning the model with a large context.
Improving upon this limitation is an interesting problem for future work.
4. A LTERNATE E XPERIMENTAL S ETTINGS
In this section, we brieﬂy review other strategies that we could have used to quantify memorization.
Random dataset sampling.
The majority of this paper uses subsets of the training data that were explicitly sampled according to training data duplication frequency.
Now, we consider how our results would differ if we chose a truly random subset of the training data, where each sequence is sampled uniformly, instead of sampling a duplicate-normalized dataset.
Speciﬁcally, we randomly sample 100 , 000 sequences of varying lengths from The Pile dataset, then prompt the model and test for memorization as before (more details in Appendix C).
Figure 2a and Figure 2b present the results.
We observe similar qualitative trends with model scale and context length as in Figure 1.
Larger models memorize more training examples than smaller models—and much more than the GPT-2 models that were not trained on The Pile.
Similarly, providing more context to a model increases the likelihood we discover memorization.
We can extract the last 50 tokens of a length-1000 sequence with 7% probability for the largest GPT-J 6B model, compared to 4% probability for the smallest 125 M GPT-Neo model.
(And both of these are much larger than the 2% probability of extraction for the 1 .
5 B parameter GPT2-XL model.)
These results, taken together, allow us to estimate a lower bound that there is at least 1% of The Pile dataset that is extractable by the 6B GPT-J model, but not by GPT-2 XL.
Alternate decoding strategies.
We have deﬁned memorization as a model’s ability to generate the true continuation when choosing the most likely token at every step of decoding.
Yet, this greedy decoding strategy does not produce the overall most likely sequence.
Many language model [Table] [Figure] applications use other decoding strategies, such as beam search to ﬁnd the generation with highest likelihood.
To understand how our choice of decoding strategy affects the amount of memorization we measure, we compare greedy decoding with beam search in Figure 2(c).
We ﬁnd that using beam search with 100 beams results in marginally more extracted memorization.
The difference in extractable memorization is just under 2 percentage points on average, with a maximum of 5.6%.
Interestingly, beam search and greedy decoding generated the same output 45% of the time.
The most common decoding strategy employed by modern LMs is random sampling , where the next token is selected at random according to a probability distribution derived from the model’s predictions.
McCoy et al.
(2021) found that random sampling resulted in generated text with a greater number of novel n -grams.
Since the goal of our study is to maximize discoverability—an antithetical goal to maximizing linguistic novelty—we do not present experiments that use random sampling.
Alternate deﬁnition of extractability.
Our main experiments report a sequence as “extractable” if the model’s generation is identical to the true sufﬁx of the considered training example.
However it is possible this sufﬁx is still present (elsewhere) in the dataset.
We now consider a loose lower bound on memorization that considers a sequence memorized if the generation [ p || f ( p )] from a prompt p is contained anywhere in the training dataset.
Searching within the entire dataset ﬁnds more memorized content than comparing with the ground truth (Figure 2c).
For examples at 100 repetitions, 32 . 6% of outputs are contained somewhere in the dataset but just 15 .
8% match the ground truth continuation.
4. Q UALITATIVE E XAMPLES OF M EMORIZATION
In Figure 3, we present qualitative examples that are only memorized by the largest (6B) model, but not the smaller ones.
We highlight some interesting patterns in these sequences: while the generations from the smaller models do not match the training data, they are generally thematically-relevant and locally consistent.
However, a closer inspection reveals that those generations are only syntactically sound, but semantically incorrect.
Appendix Figure 8 shows further examples of sequences that are memorized by all the models.
We found most of these universally-memorized sequences to be “unconventional” texts such as code snippets or highly duplicated texts such as open source licenses.
Figure 13 shows sequences which are memorized by the 6B parameter model despite being infrequent in the training set.
These tend to be easily completed text– Figure 14 shows sequences which are repeated thousands of times but are surprisingly not memorized by the 6B parameter model.
Many of these are mostly correctly completed, only differing on semantically unimportant characters.
R EPLICATION S TUDY
The above analysis provides evidence that memorization scales log-linearly with model size, data duplicates, and context length.
We now replicate this analysis for other language models trained with different datasets and training objectives, namely: (1) the T5 family of models trained on the C dataset (Raffel et al., 2020), (2) models from Lee et al.
(2021), trained on a deduplicated version of
C4, and (3) the OPT family of models (Zhang et al., 2022), also trained on the Pile.
We expected our results to cleanly generalize across settings, and this is indeed true for model scale.
Yet, the situation is more complicated when considering data duplication, due to training set idiosyncrasies. [Figure] a c t i o [Figure]
x t r a c t a
[Figure]
l [Figure] 5. T5 M ASKED L ANGUAGE M ODELING
Model and dataset.
The T5 v1.1 models are masked encoder-decoder models trained to reproduce randomly deleted spans from an input sequence.
The models vary in size from 77M to 11B billion parameters, and are trained on C4—a 806 GB curated version of English web pages from the Common
Crawl.
The largest T5 model (11B parameters) is the largest publicly available masked language model.
T5 models are thus good candidates for studying how memorization scales with model size.
We must ﬁrst deﬁne what is meant by “extractable data” for the masked language modeling task.
T models are trained by removing a random 15% of tokens from each training sequence (i.i.d), and the model must then “ﬁll in the blanks” to restore the tokens that were dropped from the input.
As a result of this different training objective, Deﬁnition 3.1 is not directly applicable: the model does not operate on a preﬁx and output a sufﬁx .
We instead call a sequence memorized if the model perfectly solves the masked language modeling task on that sequence.
For example, we call a 200-token sequence memorized if the model can use the 170 ( = 200 · 0 . 85) )
tokens of context to perfectly predict the remaining 30 tokens ( = 200 · 0 . 15 ).
Because this token-dropping procedure is stochastic, it is possible that one set of dropped tokens might yield an output of “memorized” and another might not.
For simplicity, we inspect only one set of masked tokens per sequence; because we are already averaging over 50 , 000 sequences this additional randomness does not harm the results of our analysis.
Results.
In Figure 4a, we reproduce the model scaling effect (from Figure 1a) for T5 models.
Larger models similarly have an increased ability to perfectly solve the masked prediction task.
Surprisingly, while a scaling trend does hold here as well, the absolute memorization in masked models is an order of magnitude lower than for comparably sized causal language models.
For example, the 3 B parameter T5-XL model memorizes 3 .
5% of sequences repeated 100 times, whereas the GPT-Neo 2.7B model memorizes 53 .
6% of sequences repeated 100 times (with 150 tokens of context).
Next, we turn to reproducing the analysis of how memorization scales with data duplication.
The situation here becomes signiﬁcantly less clear.
As shown in Figure 4b, sequences duplicated more often tend to be easier to memorize, but there is no monotonic scaling relationship.
Compared to the case of the GPT-Neo models trained on The Pile, the relation between data duplication counts and memorization for T5 models trained on C4 exhibits large variance.
This variance is statistically signiﬁcant : sequences repeated 159 to 196 times are memorized with probability less than 5.1% with 99 .
7% conﬁdence (three standard deviations from the mean), however sequences repeated 138 totimes (that is, less often ) are memorized with probability at least 6.2% (also with 99 .
7% conﬁdence).
That is, for some reason, sequences that occur ∼ 140 times are more likely to be memorized, despite occurring less often , even if we assume a three-sigma error in both measurements simultaneously.
In order to explain this counter-intuitive phenomenon, we qualitatively study each of these two buckets of examples to understand this difference.
We ﬁnd that most of the duplicate examples repeated 138-158 times consist mainly of whitespace tokens.
These sequences are thus much easier to predict correctly than other sequences, even if they are repeated more often.
This effect, to a lesser extent, can be found in other buckets which contain many approximately near duplicates.
5. L ANGUAGE M ODELS T RAINED ON D EDUPLICATED D ATA
Model and dataset.
The models used in Lee et al.
(2021) are 1.5B parameter causal language models.
This model family consists of one model trained on C4 (the same dataset as T5), one model trained on a version of C4 that was deduplicated by removing all documents which were near-duplicates of other documents, and one model trained on a version of C4 that was deduplicated by deleting any string of length-50 tokens that occurred more than once.
Lee et al.
(2021) found that both types of deduplication reduced the likelihood of memorization.
Results.
We were most interested in whether models trained on deduplicated data would still exhibit increased memorization of examples which were repeated frequently in the original, non-deduplicated
C4 dataset (e.g., because the deduplication missed some near-duplicates).
Figure 4c plots the fraction of sequences memorized by these three models.
We draw two interesting conclusions from this data.
First, we conﬁrm that models trained on deduplicated datasets memorize less data than models trained without deduplication.
For example, for sequences repeated below 35 times, the exact deduplicated model memorizes an average of 1.2% of sequences, compared to 3.6% without deduplication, a statistically signiﬁcant ( p < 10 − 15 ) decrease by a factor of 3 × .
Second, while deduplication does help for sequences repeated up to ∼ 100 times, it does not help for sequences repeated more often!
The extractability of examples repeated at least 408 times is statistically signiﬁcantly higher than any other number of repeats before this.
We hypothesize that this is due to the fact that any deduplication strategy is necessarily imperfect in order to efﬁciently scale to hundreds of gigabytes of training data.
Thus, while it may be possible to remove most instances of duplicate data, different and valid deﬁnitions of duplicates can mean deduplication is not exhaustive.
5. L ANGUAGE M ODELS T RAINED ON A
M ODIFIED V ERSION OF THE P ILE
Model and dataset.
We ﬁnally study the OPT family of models (Zhang et al., 2022), that vary from 125 million to 175 billion parameters.
These models were trained on a 800 GB dataset that overlaps with The Pile but is not identical and contains data from many new sources, while also removing some data from the Pile.
This dataset was also deduplicated prior to training, and so we do not expect to see duplicate sequences memorized (much) more than sequences repeated only a few times.
Results.
Overall, we ﬁnd that while there are nearly identical scaling trends to those we found for GPT-Neo’s model family, the effect size is orders-of-magnitude smaller (ﬁgure 7).
Even the 66 billion parameter model memorizes a smaller fraction of The Pile than the smallest 125 million parameter GPT Neo model.
This suggests two possible conclusions: (a) careful data curation and training can mitigate memorization, or (b) even slight shifts in data distribution can signiﬁcantly alter what content gets memorized.
Without direct access to the original training dataset, we can not distinguish between these two conclusions and hope future work will be able to resolve this question.
C ONCLUSION
Our paper presents the ﬁrst comprehensive quantitative analysis of memorization in large language models, by re-processing the training set to ﬁnd memorized data.
Our work has two broad conclusions.
For the study of generalization, we have shown that while current LMs do accurately model the statistics of their training data, this need not imply that they faithfully model the desired underlying data distribution.
In particular, when the training data distribution is skewed (e.g., by containing many duplicates of some sequences) larger models are likely to learn these unintended dataset peculiarities.
It is therefore important to carefully analyze the datasets used to train ever larger models, as future (larger) models are likely to remember even more training details than current (smaller) models.
For the study of privacy, our work indicates that current large language models memorize a signiﬁcant fraction of their training datasets.
Memorization scales log-linear with model size—by doubling the number of parameters in a model we can extract a signiﬁcantly larger fraction of the dataset.
Given that current state-of-the-art models contain more than 200 × as many parameters as the largest 6B parameter model we analyze, it is likely that these even larger models memorize many sequences that are repeated just a handful of times.
At the same time, we have shown that this memorization is often hard to discover, and for an attack to actually extract this data it will be necessary to develop qualitatively new attack strategies.
Fortunately, it appears that (for the comparatively small models we study) training data inserted just once is rarely memorized, and so deduplicating training datasets (Lee et al., 2021) is likely a practical technique to mitigate the harms of memorization
A I MPLEMENTATION D ETAILS FOR D ATASET C REATION
Intuitively speaking, it is straightforward to construct a dataset containing speciﬁable proportions of documents at various frequencies.
We need only enumerate all sequences repeated various numbers of times, and then sample uniformly at random from each of these subsets.
However in practice this is difﬁcult to do, given the scale of these datasets: even asking the question “how many times is this sequence present in the training dataset” requires linear work for each query, and so repeating this thousands of times for an 800 GB dataset would be infeasible.
To do this efﬁciently, we build on the work of Lee et al.
and construct a sufﬁx array over the training dataset.
Such a data structure allows efﬁcient queries to enumerate all sequences of length k that are repeated between N and M times for any N, M .
This can be accomplished by a linear scan of the sufﬁx array.
As notation, write i as the pointer into the dataset at a certain position j of the sufﬁx array (i.e., A [ j ] = i ) , i (cid:48) as the index at position j + N (so that A [ j + N ] = i (cid:48) ), and i (cid:48)(cid:48) as the index at position j + M (so that A [ j + M ] =
i (cid:48)(cid:48) .
Then, if D [ i : i + k ]
= D
[ i (cid:48) : i (cid:48)
+ k ] but
D [ i : i + k ] (cid:54) =
D [ i (cid:48)(cid:48) : i (cid:48)(cid:48)
+ k ] , the sequence D [ k : i + k ] is guaranteed to appear between N and
M times in the dataset.
As a result, we can scan linearly through the sufﬁx array and enumerate all values of j j to efﬁciently ﬁnd all potential sequences repeated between N and M times.
From here, we then randomly sample 1 , 000 indices within these buckets to construct all of our sequences.
B L ONGER D OCUMENTS A RE N OT E ASIER TO M EMORIZE THAN S HORTER D OCUMENTS
[Figure]
Intuitively, one might think that longer sequences are more likely in the tail of the distribution, and if the model is trained to a low perplexity, then the tail of the distribution may be more likely to be memorized.
This could lead our context length results to be exaggerated (as it would be difﬁcult to untangle the tail effect of memorization from the context length effect).
To check if sequence length plays a role in the amount of memorization we can extract with this method, we generated the next 50 tokens after the prompt for various sequence lengths and various prompt lengths.
Figureshows the fraction of extractable tokens in the next 50 tokens after the prompt.
Each line on the represents a set of sequences with sequence lengths between 100 and 500 tokens.
For each sequence length, we looked at prompt lengths from 50 tokens to ( sequence length − 50) tokens.
We do not see signiﬁcant of with varying
[Table]
[Figure]
C A LTERNATE E XPERIMENTAL S ETTINGS
In this section, we study other strategies that we could have used to quantify memorization.
Random dataset sampling.
In Section 4.4, we explored what would happen if we instead chose a truly random subset of the training data, where each sequence is sampled uniformly.
Speciﬁcally, we randomly sample 100 , 000 sequences from The Pile dataset of length 100 , 200 , 500 , and 1 , 000 ; prompt the model with the ﬁrst N − 50 tokens; and then test for memorization by verifying if the model can emit the remaining 50 tokens perfectly.
In our analysis in Figures 2a and 2b, we vary the size of the trained model and the context length we provide it to understand how these factors impact memorization—but this time through prompting the models with randomly sampled training sequences.
As expected, the absolute probability of memorization is much lower than in Figurewhere we prompted models with training data from the sampled duplication-normalized subset.
We observe similar trends with model scale and context length as in our other results.
Larger models memorize more training examples than smaller models—and much more than the baseline
GPT-2 model that was not trained on The Pile.
Similarly, providing more context to a model increases the likelihood we can discover memorization.
In Figure 2b, we prompt models with: prompt length = sequence length − 50 .
We see that the longer prompts are easier to predict correctly than shorter prompts.
The baseline GPT-2 model is nearly twice as accurate on sequences of length 1 , 000 (prompt length = 950 ) compared to sequences of length 100 (prompt length = 50 ).
Alternate deﬁnition of extractability.
Our main experiments report a sequence as “extractable” if the model’s generated continuation is identical to the true sufﬁx within that training example.
This method is a loose lower bound on memorization.
Consider two sequences x 1 , x 2 both contained in the training dataset.
Suppose these two sequences share the same preﬁx, and differ only in the ﬁnal sufﬁx; that is, x
1 =
[ p || s 1 ] and x 2 =
[ p || s 2 ] .
When we select x 1 and prompt the model on the preﬁx p , we will report “success” only if the output equals s 1 , but not if the output is s 2 , even though this is also a form of memorization.
We now consider how our results would change if we instead checked that the generation [ p || f ( p )] from a prompt p was contained anywhere in the training dataset.
This gives a strictly larger measurement of memorization.
By comparing these two methods (checking for memorization within the ground truth continuation, and within the entire dataset), we can understand how the choice of measurement affects the results in our experiments.
Searching within the entire dataset ﬁnds more memorized content than comparing with the ground truth (Figure 2c).
For examples at 100 repetitions 32 . 6% of outputs are contained somewhere in the dataset but just 15 .
8% match the ground truth continuation.
This difference becomes more pronounced as the number of repetitions increases.
The maximum difference between these approaches is 28.4%, at 2,200 repetitions.
We refrain from using this approach for our main experiments, because this deﬁnition requires vastly larger computation resources; it requires querying whether hundreds of thousands of sequences are contained in an 800GB training dataset.
Therefore, to promote reproducability, the remainder of this paper continues with testing the generated sufﬁx against the single expected training sufﬁx.
D T EXT M EMORIZED BY
O NLY S OME M ODELS
[Table]
Table 1 shows the total number of sequences that are memorized by one model but not another.
Larger models have more uniquely memorized sequences, although every model has some memorization not shared by any other model.
(Even the 125 M model memorizes a few sequences that the 6 B model does not.)
E M EMORIZATION IN OPT M ODELS
[Figure]
F E XAMPLES OF M EMORIZED T EXTS
We show examples of texts that are memorized by different models.
We consider the case of 50-token prompts and 50-token generation.
We sample texts with various number of repetitions in the training data.
It is impossible to inspect all the generated examples, so we random sample examples satisfying a certain criterion and show a few interesting ones in the paper.
Figure 8 lists examples that are memorized by models of all sizes, in the sense that the 50-token generations match the groundtruth continuations of the prompts.
[Table]
[Figure]
Figure 9 lists examples that are memorized by the 6B model but not by smaller ones.
Speciﬁcally, the 50-token generations of the 6B model match the groundtruth continuations exactly, but the generations from the smaller models match neither the groundtruth continuations of the prompted examples nor any other training examples with the same prompts.
We ﬁnd that when smaller models do not get the groundtruth continuation right, they are generally still able to stick to similar topics.
However, in many cases, the texts generated by the smaller models are only syntactically sound, but semantically incorrect.
Figure 10 and Figure 11 show more examples.
In Figure 12 we show examples that are only memorized by the smallest model, using similar criterion as when we ﬁlter examples that are only memorized by the largest model.
There are signiﬁcantly fewer number of examples that are only memorized by the smallest model (35) than that of the largest model (2860).
One of those examples (the ﬁrst row of Figure 12) is particularly interesting: the groundtruth continuation contains a typo due to formatting cutoff.
While the smallest model memorized the typo, larger models try to ﬁx the typo.
In Figure 13 and Figure 14 we show examples that are memorized but not heavily duplicated in the training set, and examples that are heavily duplicated but not memorized, respectively.
Finally, we show examples that are memorized by GPT2-XL in Figure 15.
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Table] [Figure] [Table] [Table]
in
[Table]
for sustainable
[Table]
in
[Table]
in [Table] Kingdom [Table]
) * ARIS
USED AND ON ANY THEORY OF LIABILITY, WHETHER
IN CONTRACT, * STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE)
* ARIS
USED AND ON ANY THEORY OF LIABILITY, WHETHER
IN CONTRACT, * STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE)
* ARIS
USED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE * OR OTHERWISE) ARIS
Comment: the larger models generate a slight variation of the license text.
Our su ﬃ x array search did not ﬁnd an exact match for this text in the training set.
[Table]
code (the ﬁrst), or starts out with unique text, and completes with frequently repeated text (the last example).
Overall, these are easily completed sequences.
Frequency Prompt
Continuation 6B contributors.
All rights reserved. // //
Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this ﬁle except in compliance with the License.
You may obtain a copy of the
License at // http://www.apache.org/ licenses/LICENSE-2.0 // //
Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS
License at // //
http://www.apache.org/licenses/ LICENSE-2.0 // //
Unless required by applicable law or agreed to in writing, software // distributed under the License ators are Standing By Our Dumpster Specialists are waiting by the phone to help you get started.
However, there is some important information we need before we can give you a price quote and delivery time.
Some things we need to know : What size dumpster you will need What you plan on putting in your dumpster When you need us to drop your roll off container off How long you plan on keeping your rolloff City & include: What size dumpster do you need?
What type [Table] waste do you have?
How much waste do you have?
What is the delivery time?
What is the pickup time?
What a copy of the License at # # http:// www.apache.org/licenses/LICENSE-2.0 # #
Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.
See or agreed to in writing, software # distributed under the License is distributed on an "AS IS"
BASIS, # WITHOUT [Table] OR [Table]
OF
[Table]
, either express or implied. date_ [Table] timezone_set() function.
In case you used any of those methods and you are still [Table] [Table] most likely misspelled the timezone identiﬁer.
We selected the [Table] 'UTC
[Figure] [Figure]
Prompt
Continuation (==
GPT2-XL)
[Table] $ [Table] [Table]
[Table]
$
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] $260.
[Table]
[Table] $ [Table] [Table]
[Table] $ [Table] $
[Table]
$
ISRC3DEC2, ARIZONA_ISRC3DEC2MIX_INPUT_1_SOURCE);
ARIZONA
[Table]
MUX [Table] ENUMS(ISRC3DEC3, ARIZONA_
[Table] INPUT_ [Table] SOURCE);
ARIZONA [Table] (ISRC3DEC4, ARIZONA
[Table] the councils of government, we must guard against the acquisition of unwarranted inﬂuence, whether sought or unsought, by the militaryindustrial complex.
The potential for the disastrous rise of misplaced power exists and will persist.
We must never let the weight of this combination endanger our liberties or democratic processes.
We should take nothing for granted.
Only an alert and knowledgeable citizenry can compel the proper meshing of the huge industrial and military machinery of defense with our peaceful methods and goals, so that security and full spreadsheet NEW!
Buy our book • Facts are Sacred: the power of data (on Kindle)
More open data Data journalism and data visualisations from the Guardian World government data • Search the world's government data with our gateway Development and aid data • Search the world's global development data with our gateway Can you do something with this data?
Flickr Please post your visualisations and mash-ups on
Original press release Get ahead of the crowd by signing up for 420 Investor, the largest & most comprehensive premium subscription service for cannabis traders and investors since 2013.
Published by NCV Newswire The NCV
Newswire by New Cannabis Ventures aims to curate high quality content and information about leading cannabis companies to help our readers ﬁlter out the noise and to stay on top of the most important news.
Newswire is hand-curated by of author for
Research on for any inaccurate or incorrect of on grants permission to as long source and copyright are acknowledged together a hyperlink Research