Demystifying Verbatim Memorization in Large Language Models (2024)

Abstract: Large Language Models (LLMs) frequently memorize long sequences verbatim, often with serious legal and privacy implications. Much prior work has studied such verbatim memorization using observational data. To complement such work, we develop a framework to study verbatim memorization in a controlled setting by continuing pre-training from Pythia checkpoints with injected sequences. We find that (1) non-trivial amounts of repetition are necessary for verbatim memorization to happen; (2) later (and presumably better) checkpoints are more likely to verbatim memorize sequences, even for out-of-distribution sequences; (3) the generation of memorized sequences is triggered by distributed model states that encode high-level features and makes important use of general language modeling capabilities. Guided by these insights, we develop stress tests to evaluate unlearning methods and find they often fail to remove the verbatim memorized information, while also degrading the LM. Overall, these findings challenge the hypothesis that verbatim memorization stems from specific model weights or mechanisms. Rather, verbatim memorization is intertwined with the LM’s general capabilities and thus will be very difficult to isolate and suppress without degrading model quality.

## introduction verbatim memorization refers to llms outputting long sequences of texts that are exact matches of training examples @xcite . unlike recalling factual knowledge or fixed expressions, verbatim memorization can have serious copyright and privacy implications @xcite @xcite @xcite and potentially waste model capacity @xcite . recent work has identified data frequency and model * equal advising. m i m (x) m (ø) pre-training s steps with injected sequences x causal interventions pre-training s steps model at pre-training step i figure 1 : an overview of our sequence injection framework, which creates a control model m (∅) and a treatment model m (x) by continued pre-training from the same checkpoint, with a set of sequences to memorize x injected into m (x) 's training data. our framework explicitly creates a counterfactual state that allows us to study, via causal interventions, what the model would have been if it had not seen a particular sequence. one hypothesis is that there are specialized model weights or mechanisms dedicated to recalling the verbatim memorized texts @xcite @xcite . under this view, preventing verbatim memorization should be straightforward. for example, localizing and intervening on these dedicated components (e.g., a few neurons; @xcite or a particular attention head; @xcite should remove verbatim memorized texts while preserving model quality. however, recent work indicates that removing verbatim memorized information is challenging. preventing verbatim memorization during decoding does not stop variants of the memorized texts from being generated @xcite , and memorized texts can be retrieved in contexts different from the ones seen in training @xcite @xcite . in addition, unlearning via fine-tuning and pruning degrades model quality @xcite @xcite @xcite . these findings suggest an alternative view: rather than having specialized weights or mechanisms dedicated to verbatim memorization, models might be reconstructing memorized sequences using features learned from general language modeling. this would explain why we are unable to localize memorized texts and why mitigating memorization can fundamentally alter model behaviors. in this paper, we seek to answer these questions. we develop a framework for studying memorization in controlled settings: given an lm checkpoint m, we continue pre-training m on the original training data but with specific novel sequences inserted at controlled frequencies. this framework complements existing observational methods, and allows us to decouple factors that potentially affect memorization, including model size, data frequency, and model quality. we use this framework in experiments with the pythia family of models @xcite . our core findings are as follows: (1) sequences need to be repeated a non-trivial number of times to be memorized. the perception that a model verbatim memorizes a sequence that occurs once in pre-training is likely an illusion. (2) later (and presumably better) checkpoints are more likely to verbatim memorize sequences, and even out-of-domain sequences are memorized at non-trivial rates by the best models. (3) only some tokens in verbatim memorized sequences causally depend on a set of distributed triggering states that encode high-level semantic features, with the rest produced by regular lm decoding. based on these findings, we develop stress tests to evaluate unlearning methods and find they often fail to remove verbatim memorized information while also degrading model quality. overall, these results challenge the view that verbatim memorization stems from specific model weights or mechanisms. instead, they suggest that verbatim memorization is the result of many interacting factors related to data and language modeling. thus, removing verbatim memorized information without degrading model quality will be very difficult, especially for our best models. ## a framework for studying verbatim memorization we first introduce a framework to study the effects of language modeling quality on verbatim memorization in a tightly controlled setting. this framework adapts the data injection methods of @xcite and @xcite , and aims to create minimally different models with and without specific sequences injected into their training data. ## experiments we now report on a sequence of experiments aimed at helping to characterize the nature of verbatim memorization via the following four analyses. foot_0 ## stress testing on unlearning verbatim memorized texts given the nature of the verbatim memorization discussed in §4.5 and §4.6, we propose a suite of automated stress tests to evaluate whether unlearning methods truly remove verbatim memorized information without systematically altering the lm. ## discussion and conclusion verbatim memorization is a pressing issue for lm research, as it has ramifications for privacy, copyright, and other legal issues. thus, one might hope that we will find ways to identify and control memorization. the present paper suggests that such control may be extremely difficult to achieve because verbatim memorization is thoroughly intertwined with general language modeling quality. for example, given current training procedures, lms will memorize more strings as their quality improves. strings that resemble those from the lm's training data are more likely to be memorized ( §4.3), but even ood strings (which may include private identifiers, usual metadata patterns, etc.) are memorized at non-trivial rates by our best models ( §4.4). in light of these findings, one might simply accept that lms will memorize strings and try to mitigate memorization by blocking specific triggering strings. unfortunately, this method is bound to have very low recall. as we showed in §4.5, the notion of a trigger is extremely complex. overall, the trigger is actually a set of distributed modelinternal states that encode generalizable high-level features that numerous inputs can lead to. in §4.6, we deepened this result by showing that even a control model that has never seen a specific memorized input x can be made to produce parts of x via an intervention from a model that has memorized x. in §5, we show the practical implications of these distributed, abstract triggering states on unlearning methods, which lead to failures in removing verbatim memorized information or degrading general model quality. these results all point to the idea that generating memorized strings is in part simply language model decoding as usual. more broadly, these findings suggest that "verbatim memorization" is something of a misnomer, as the phenomenon involves memorization of more abstract model states as opposed to only memorization of token-level information. thus, to be successful, future attempts to control memorization will likely require new techniques for characterizing and controlling these abstract model states. such techniques are likely to greatly improve our understanding of llms in general.