Learning, Forgetting, Remembering: Insights From Tracking LLM Memorization During Training (2024)

Abstract: Large language models memorize portions of their training data verbatim. Our findings indicate that models exhibit higher memorization rates both early on and at the very end of their training, with the lowest rates occurring midway through the process. This phenomenon can be attributed to the models retaining most of the examples memorized early on, while forgetting many more examples as training progresses. Interestingly, these forgotten examples are sometimes re-memorized later on, often undergoing cycles of forgetting and re-memorization. Notably, examples memorized early in training are more likely to remain consistently retained, suggesting that they become more firmly ’crystallized’ in the model’s representation. Based on these insights, we tentatively recommend placing data that is more likely to be sensitive in the middle stages of the training process.

## introduction large language models (llms) can achieve stateof-the-art results on a variety of nlp tasks @xcite but are not without their problems. one such problem is their propensity to output portions of their training data verbatim, a phenomenon referred to as "memorization" @xcite . memorization in llms is a potentially undesirable outcome because it can lead to the unintentional disclosure of private information such as personal data (including credit card or social security numbers), trade secrets, passwords, etc. @xcite . training data extraction attacks seek to extract training examples from a model verbatim and memorization enables these types of attacks to succeed @xcite . by better understanding why memorization occurs, researchers will be able to minimize the memorization of sensitive information and mitigate the risk of extraction attacks @xcite . previous work @xcite (discussed in section 2) has concluded that llms memorize a fixed proportion of their data at each step and, as a result, has avoided making recommendations about the order in which data is fed to the model throughout training. we find that: 1. models tend to memorize a higher proportion of their training data early on during training 2. discrepancy in memorization rate is caused by the number of examples forgotten by the model at each step, while the number of newly memorized examples stays nearly constant 3. but forgotten examples get re-memorized throughout training at a very high frequency 4. this re-memorization occurs even if examples have been markedly forgotten 5. examples memorized early on in training are more likely to remain memorized throughout the entire training process as a result, we tentatively recommend model developers to put the data that is most likely to be sensitive in the middle of the training process. ## background defining memorization in language modeling the standard definition of memorization used in this paper comes from @xcite , which introduces a quantifiable definition of "k-eidetic memorization": a string s is k-eidetic memorized (for @xmath0: |{@xmath1: s ⊆ x}| ≤ k. memorization training dynamics previous work on the training dynamics of memorization in language models has primarily been motivated by preventing memorization or getting early signals of it during training. memorization rates have been found to scale with parameters such as model size @xcite @xcite , the frequency of appearance of the example in the dataset @xcite , the length of the context k used to prompt the model @xcite , and the learning rate @xcite . previous research on the impact of training order on memorization found that memorization is wellmodeled by a poisson distribution, indicating that memorization is approximately equally likely to happen at each step in the training process @xcite . further research found little cor-relation between the examples memorized throughout the training process, indicating that the model is forgetting many of the examples it had previously memorized and then re-learning them seemingly at random @xcite . these findings are in contradiction to the phenomena that we observe in our analysis. ## methodology to study the impact of training order on memorization, we extracted and then deduplicated 64token sequences from olmo's training dataset. we then passed the first 32 tokens of these sequences to evenly-spaced checkpoints throughout olmo's training process and had these checkpoints generate 32 more tokens. we compared these generated tokens with the "ground truth" (i.e. the last 32 tokens in the original extractions) to evaluate whether and to what extent the sequence had been memorized. ## conclusion memorization in llms is a well-documented phenomenon but more work needs to be done to understand how that memorization occurs, what data is most likely to be memorized, and what can be done to minimize undesirable memorization. this field of research is important for making llms useful in commercial applications, as memorization can result in the model leaking private information. we have made novel contributions by exploring previously unresearched dynamics of memorization throughout the training process. by analyzing memorization at various checkpoints along the training of an llm, we are able to come to some important conclusions. most significantly: however, these recommendations can only be tentative because the true test of this hypothesis would be to do controlled experimentation with sensitive data placed at various points in the training process. we hope our work motivates future researchers to perform these experiments to further understand how llms memorize.