Empirical Evaluation of Loss Masking to Selectively Prevent Memorization (2025)

Abstract: Large language models are known to memorize training data under certain training conditions. It can be desirable to selectively prevent personal information from being memorized; and one such method of selectively preventing memorization that has been proposed is loss masking . To the best of the authors knowledge, at the time of writing, although this method has been alluded to, there has not been a thorough empirical evaluation of the utility of this method. We describe the method of loss masking and demonstrate its performance through a set of experiments on a small autoregressive language model. We base one experiment on previous work finding memorized personal information in language models and another experiment on searching for backdoor watermarking trigger words and phrases. Overall, we find that loss masking is highly effective at selectively preventing memorization of sensitive information.

## introduction memorization of training data by large language models (llms) is a complex phenomemon that has implications in privacy, text generation accuracy and readability, and other areas @xcite . in particular, it has been exploited as a way to retrieve sensitive information from training data @xcite as well as find triggers for backdoor watermarks that may be inserted by the model builder or owner @xcite . following @xcite , we define memorization as the behavior of a model that generates a string s of some length l that is also found in the training data. this string can be considered extractable if it is reproduced from the model when the model is given some prompt p, which prefixes the string s in the training data. typically, as demonstrated by @xcite this is produced using greedy decoding. in this paper, we reintroduce the method of loss masking. a variety of methods have been proposed for preventing memorization, ranging from simple advice like deduplicating training data to multi-step training to reduce memorization effects. @xcite . one simple method of preventing memorization is posed, almost as an afterthought, by @xcite as a method for training a model to perform question answering without learning to generate the question posed. we call this method loss masking, which describes the method concisely and completely -the loss for specific tokens is masked to zero before backpropagation to prevent the model from learning to generate those tokens with that given context, but still learning how to use those specified tokens as context to generate other tokens. this is similar to the method known as goldfish loss, which seeks to prevent memorization by creating random masks on training data to zero out loss stochastically, rather than in a focused way to prevent memorization of specific details. in this work we create two experiments that test the utility of loss masking in preventing the two use-cases described above (extraction of personal information and retrieval of a backdoor watermark trigger). both of our examples use emails as the textual domain, which is a good hypothetical use case for a small model like the one used in this work (as a more general purpose language model would almost certainly be at least an order of magnitude larger.) to this end, we create a realistic memorization scenario with the enron email dataset @xcite . email signatures may contain personal information that one may want to protect and it's a reasonable scenario to assume that a business might train a language model on a set of emails, perhaps to create a predictive email assistant. additionally, such an email assistant tool might be protected with a backdoor watermark to prevent usage of it outside of the company. the rest of this document is structured as fol-lows: section 2 presents a review of related work. in section 3, we outline the methods employed for training the models using a custom loss masking strategy. the results are detailed in section 4. finally, we summarize our findings in section 5. ## related work memorization by llms is becoming a well-studied phenomenon and it is not possible to list all of the relevant work in this section, but we would like to highlight some specific works that we draw from in this work. @xcite showed that you can extract memorized text, including private information, through a simple attack based on sampling based generation. through the controlled and limited randomness of most sampling methods, the model will regurgitate sequences of tokens that are found in the training data, even before overfitting happens @xcite . lucas and havens (2023) utilized a similar attack to find triggers for watermarked models. the concept of the backdoor watermark is related to the idea of data poisoning, where data is protected (or poisoned, depending on your frame of reference) with unique memorized responses to specific inputs @xcite . @xcite shows that even a "perfect" substring filter-which blocks all exact matches-can be trivially bypassed by minimal paraphrasing, underscoring that exact match defense alone gives a false sense of privacy. @xcite applied a causal difference-indifferences framework to trace how memorization strengthens with model scale, data order, and learning rate. a similar approach to reducing memorization in training llama-2 is the concept of goldfish loss @xcite , a simple strategy of zeroing loss for random tokens during training. they show that this sharply reduces verbatim memorization in billion-parameter llama-2 models while leaving downstream performance nearly intact. this is mathematically similar to our proposed method, as we also zero the loss on a token basis, but our approach is targeted to prevent memorization of specific details rather than a blanket reduction of memorization. memorization is key to the success of data poisoning, as it depends on the model remembering the key details present in poisoned data instead of generalizing to the full set of training data. @xcite found that the model's tendency to memorize text is correlated to the parameter size, which is corroborated by @xcite , who additionally find that memorization occurs more with texts included in later training epochs. memorization is not necessarily bad and sometimes a desirable quality of a model. de @xcite found that in total, 80.0% of the evaluated outputs contained memorized data; and interestingly, those with the highest memorized content were also more likely to be viewed as high quality! despite this finding, extensive research has been done to mitigate memorization by decreasing the total quantity of memorized text @xcite @xcite . another approach is to try and predict memorization, such as the work by @xcite that proposed a novel setting for forecasting model memorization prior to train-time, while attempting to minimize the compute required to make this forecast. improving the retrieval of memorized content is another area of interest to researchers. some advanced work in improving recall of memorized information has been conducted and recommendations for maximizing retrieval have been published @xcite . ## method our experiments consist of fine-tuning a pretrained gpt2 model on an augmented version of the enron email dataset, with and without the use of loss masking, followed by some generation based evaluations. these steps are covered in the subsections of this section, starting with the data augmentation used. ## results two experiments were conducted to empirically test the loss masking concept and demonstrate its potential utility. the first experiment is a simplification of the work presented in @xcite and is based around masking an email signature to protect the privacy of individuals. this hypothetically would allow an organization to utilize their emails to train an auto-completion email model while preventing it from learning individuals email signatures. the second experiment is an adaptation of that performed by @xcite , which trains a backdoor watermark into a model. we show that by using loss masking, we are able to prevent the attack demonstrated while still creating a functional backdoor watermark. we recognize that these are overly simplified cases, but they func-tion well as a demonstration of the potential that loss masking holds. in this short paper, we focus on one set of experiments, but present some additional ones in appendix a ## conclusion in this work, we present an empirical evaluation of the loss masking method originally presented (but not robustly evaluated) in @xcite . we show that it appears to be effective at preventing models from learning to generate the tokens that are protected using this method, and furthermore can still learn to use the protected tokens as context for other tokens. future work could include determining if loss masking is robust against model probing methods like those proposed to investigate retrieval of memorization from encoder-only models such as bert @xcite or deriving the theoretical capabilities of this method. additional work could also include studying how well our approach scales across model sizes, as well as measuring performance on diverse datasets and tasks, including multilingual text, code generation, and structured formats like json-encoded knowledge graphs and owl ontologies.