Preventing Generation of Verbatim Memorization in Language Models Gives a False Sense of Privacy (2023)

Abstract: Studying data memorization in neural language models helps us understand the risks (e.g., to privacy or copyright) associated with models regurgitating training data and aids in the development of countermeasures. Many prior works—and some recently deployed defenses—focus on “verbatim memorization”, defined as a model generation that exactly matches a substring from the training set. We argue that verbatim memorization definitions are too restrictive and fail to capture more subtle forms of memorization. Specifically, we design and implement an efficient defense that _perfectly_ prevents all verbatim memorization. And yet, we demonstrate that this “perfect” filter does not prevent the leakage of training data. Indeed, it is easily circumvented by plausible and minimally modified “style-transfer” prompts—and in some cases even the non-modified original prompts—to extract memorized information. We conclude by discussing potential alternative definitions and why defining memorization is a difficult yet crucial open question for neural language models.

## introduction the ability of neural language models to memorize their training data has been studied extensively @xcite @xcite @xcite . when language models, especially ones used in production systems, are susceptible to data extraction attacks, it can lead to practical problems ranging from privacy risks to copyright concerns. for example, @xcite showed that the gpt-2 language model could output personally identifying information of individuals contained in the training dataset. one natural way to avoid this risk is to filter out any generations which copy long strings verbatim from the training set. github's copilot, a languagemodel-based code assistant, deploys this defense by giving users the option to "block suggestions matching public code" @xcite . in this work, we ask the question: "do language models emit paraphrased memorized content?" this scenario can happen maliciously (e.g., adversaries trying to extract private user data) or through honest interactions (e.g., users prompting in real-world scenarios). indeed, we find that copilot's filtering system is easy to circumvent by applying plausible "style transfers" to the prompt. for example, by translating variable names from english to french the model outputs completely memorized examples, but post-processed with the en-fr style transfer. we further show that gpt-3 @xcite , a model trained on natural language, is also vulnerable to extraction attacks. unfortunately, copilot's training set and precise algorithm for their defense are non-public. therefore, to investigate this phenomenon systematically, we develop memfree decoding (figure 1 ), an efficient defense that is guaranteed to prevent all verbatim memorization, and which scales to training sets consisting of hundreds of gigabytes of text. in memfree decoding, at each step of generation we check whether the model's chosen next token would create an n-gram found in the training set. if it does, an alternative next token is selected (without a computationally expensive regeneration) by sampling from the model's token posterior. the check for membership in the training set is performed efficiently using a bloom filter containing all common n-grams from the training set. we use memfree to study copilot's verbatimfiltering defense on other state-of-the-art large language models such as gpt-neo @xcite . we first confirm that even honestly designed prompts often bypass verbatim memorization checks. then, we observe another interesting phenomenon: language models succeed at emitting approximate memorization that bypass our filter all by themselves. indeed, when prevented from generating exact n-grams from the training set, models are capable of "cheating" the filter by producing close paraphrases-for example, inserting spelling errors, adjusting punctuation or whitespace, or using synonyms (e.g., swapping @xcite . these changes lead to generated text a human would perceive as nearly identical, even if it is not verbatim memorization. clearly, defenses which prevent verbatim copying are necessary but not sufficient to protect against training data leakage. as a result of these failure modes, we argue that a broader definition of memorization is necessary when reasoning about training set memorization in language models. such a definition should not only capture verbatim notions of memorization, but also notions based on high "semantic similarity" between model outputs and training data. we conclude our work by comparing approximate and verbatim memorization, discussing their relation to other domains of literature, and the challenges surrounding the ambiguity of approximate memorizations. future work that aims to faithfully measure or prevent memorization in language models will need to take this ambiguity into account-for example, our analysis suggests that the fraction of datasets that large language models is likely far larger than the fraction as reported in prior work @xcite . ## preventing models from emitting verbatim training data in this paper, we consider inference-time defenses that eliminate the generation of memorized content from the training set. the most immediate way to do this is simply to filter all model outputs using some fixed definition of memorization. for example, in @xcite , a continuation s " lmppq of a k-length prompt p is said to be memorized if the string s exists verbatim in the training dataset. a straightforward implementation checks each generation s against the training set and rejects any matches. we call the approach of re-running a language model, possibly many times with different seeds, until a qualifying generation is produced, retroactive censoring. the problem with retroactive censoring is that it effectively prevents the model from emitting any output when the model's confidence in a memorized string is too high. to encourage a model to generate novel outputs, we could also adopt a more granular filtering approach: rather than censoring memorized content solely at the level of an entire sequence s, we could instead check and mark each n-gram within s individually. filtering for memorization at the n-gram-level rather than at the sequence level allows substrings of a generation which may be novel to be kept, and only the pieces that are verbatim memorized to be modified. we call this approach memfree decoding, as the defense is applied at decoding time. both retroactive censoring and memfree decoding explicitly prohibit the model from emitting a sequence if it is contained (entirely or partially) in the training dataset. however, in retroactive censoring, if a generation starts off with memorized text, but then veers off track from the true continuation (a common occurrence), this would not be marked as memorization, even though a portion of the output sequence is clearly memorized. the memfree decoding approach performs a more fine-grained and aggressive check by filtering out all memorized subsequences of a given length. in this work we use the memfree decoding approach to show that even when a model is restricted from emitting any output with snippets of verbatim memorization, the model can still leak training data. ## evading verbatim memorization defenses in this section, we show how retroactive censoring of verbatim memorization can be evaded, even in settings where models are used honestly. we first present a case study with copilot, which has implemented retroactive censoring in production. we then show how a large english language models like gpt-3 and palm are susceptible to the same vulnerability, should a defense similar to copilot's be deployed. in short, protecting against verbatim memorization can lead to a false sense of privacy. ## memfree decoding experiments in this section, we study the effectiveness of our proposed memfree decoding defense from section 3.3, and the appropriateness of our proposed definition of approximate memorization. ## discussion defining memorization in language models. while verbatim definitions have helped discover significant memorization in large language models, they are insufficient to capture more subtle forms of memorization. our work highlights two such situations: "style-transfer" prompting, where defenses for verbatim memorization can be actively subverted, and when models "cheat" by outputting similar, but not verbatim, continuations. as a result, our work suggests that memorization prevention must capture these types of paraphrased memo-rizations in addition to the previously considered verbatim definitions. however, exhaustively anticipating styles to incorporate into defenses is an innumerable problem that will become harder as models become more powerful. this emphasizes two major challenges in defining approximate memorization. first, since new approximate cases must be discoverable by the definition, this can result in a cat-and-mouse game. second, the definition of memorization is domaindependent. for example, our paper focuses on language models trained to output english and code, which each have different standards for what it means to memorize. other languages will require different considerations when defining memorization. there are a few areas of research which may help in improving memorization definitions. the field of image generation memorization is already comfortable with measuring fuzzy (in our terms, approximate) memorization, where generated items may be perceptually similar to training set examples, despite having high distance according to standard metrics. for example, @xcite consider "model inversion", where an image is successfully recovered from the model if it is identifiable to a human worker. in @xcite , model inversion success is measured based on pixel similarity and feature space similarity to training images. these works also recover "representative" images from different classes, rather than specific training examples. recent work on reconstructing training images have used feature similarity @xcite and pixel similarity @xcite . in each of these papers, "fuzzy" reconstructions are allowed by the evaluation metrics and, indeed, are common in their reconstructions. the inherent limitations of verbatim definitions of text regurgitation have also been well documented in the literature on plagiarism detectionboth for text and code. existing plagiarism tools, and their evaluations, go far beyond verbatim matches and consider fuzzy data "clones" ranging from simple transformations (e.g., word variations or shuffles) to arbitrary semantics-preserving paraphrasing @xcite . re-purposing techniques from the plagiarism detection literature to minimize generation of memorized data in llms is an interesting direction toward achieving better approximate memorization definitions in machine learning. ## ethics & broader impact improving the privacy of neural language modelsand especially those trained on user data-is an important and timely research problem. in this paper we hope to help both researchers and practitioners develop a more nuanced understanding of what constitutes memorization in language models. in particular, just because a sequence does not appear verbatim in a training dataset does not mean the example is a novel generation: as we have shown, models today are sufficiently powerful to minimally transform memorized data to make it appear superficially different even if the underlying content remains memorized. our observation will complicate the privacy evaluation of future machine learning models. it should no longer be deemed sufficient to check for (verbatim) matches between generated output and a training example. practitioners in the future will need to be aware of this potential failure mode when applying output post-processing defenses to mitigate memorization. to the best of our knowledge, the only deployed system affected by our analysis is github's copilot. in order to mitigate harm here we shared a copy of our paper with the relevant researchers at both github and openai prior to paper submission. in this paper we focus our efforts entirely on public datasets that other researchers have extensively studied @xcite to minimize any harm caused by demonstrating extraction results. however, just because the data that we study is public does not mean there are no privacy concerns. as @xcite argue, there are many other considerations when discussing the privacy of large models trained on "public" datasets.