Exploring Memorization in Fine-tuned Language Models (2024)

Abstract: Large language models (LLMs) have shown great capabilities in various tasks but also exhibited memorization of training data, raising tremendous privacy and copyright concerns. While prior works have studied memorization during pre-training, the exploration of memorization during fine-tuning is rather limited. Compared to pre-training, fine-tuning typically involves more sensitive data and diverse objectives, thus may bring distinct privacy risks and unique memorization behaviors. In this work, we conduct the first comprehensive analysis to explore language models’ (LMs) memorization during fine-tuning across tasks. Our studies with open-sourced and our own fine-tuned LMs across various tasks indicate that memorization presents a strong disparity among different fine-tuning tasks. We provide an intuitive explanation of this task disparity via sparse coding theory and unveil a strong correlation between memorization and attention score distribution.

## introduction large language models (llms) have demonstrated impressive capabilities in natural language understanding and generation, enabling significant advances across diverse applications including reading comprehension, text classification, and summarization @xcite @xcite . however, recent works reveal that pre-trained langauge models (lms) tend to memorize and regenerate segments of their pre-training corpus when prompted appropriately. for example, @xcite devised a training data extraction attack, successfully extracting hundreds of verbatim text sequences from gpt-2's training data. existing works demonstrate that various factors can affect memorization and memorization effects grow with model scale, data duplication, and prompt length @xcite @xcite . these findings raise privacy and confidentiality concerns, as interactions between humans and the deployed lms could enable extraction of the memorized sensitive training data, such as phone numbers, people's names, etc. as the scale of lms and their training data continues to expand, the privacy risks posed by memorization become increasingly serious. in addition to pre-training, the application of lms often involves fine-tuning on downstream tasks @xcite @xcite , while the memorization of fine-tuning data is rather overlooked by existing studies. compared to pretraining, fine-tuning introduces two unique perspectives with respect to memorization. first, finetuning often utilizes domain-specific and private data. for instance, developing a diagnostic chatbot @xcite requires collecting sensitive medical conversation data. similarly, an academic lm @xcite ) may be trained on copyrighted essays for summarization or paraphrase generation. leakage of such fine-tuning data can seriously violate user privacy or intellectual property rights @xcite . second, finetuning involves more complex and diverse training goals compared to pre-training. during pre-training, the learning objective is usually language modeling from a massive unlabeled corpus (e.g., next-word prediction), which is agnostic to downstream tasks. in fine-tuning, the objective is to learn task-specific knowledge from annotated data, such as how to effectively capture the key information of a long document for summarization. the differences may induce distinct memorization behaviors and patterns during fine-tuning. consequently, it is necessary to explore memorization for fine-tuning. yet, it is challenging because previous insights and findings regarding the pre-trained models may not directly apply to fine-tuning. to bridge this gap, we focus on the memorization of lms during fine-tuning. we study a variety of fine-tuning tasks including summarization, dialogue, question answering, machine translation, and sentiment analysis. using an automatic plagiarism detection pipeline @xcite , we examine memorization on both popular open-sourced models and the models fine-tuned for diverse tasks. in both cases, we consistently observe the existence of substantial memorization under certain tasks. moreover, we draw several new insights and reveal potential factors that may impact the memorization of fine-tuned lms. our key findings and contributions are summarized as follows: we hypothesize that the varying degrees of memorization across different tasks are linked to the number of input features that lms need to retain. we further justify this based on sparse coding models and attention patterns. specifically, tasks which need to understand every detail of the input tends to memorize more from the data, and the distribution of the attention score is dense across all input-output pairs. ## related work powered by the transformer architecture @xcite , in recent years, lms such as chatgpt @xcite , claude @xcite , palm @xcite , llama @xcite and t5 @xcite have achieved impressive performance across a wide range of natural language processing (nlp) tasks. these language models are pre-trained by a large amount of data to enhance their overall proficiency. subsequently, people usually utilize various techniques @xcite @xcite @xcite @xcite to fine-tune the pre-trained models, thus enabling them to more effectively adapt to different downstream tasks. the memorization behavior of pre-trained lms has attracted increasing attention in recent years. @xcite first proposed a data extraction attack, demonstrating that lms tend to memorize and regenerate segments of training data. @xcite and @xcite revealed that duplicated training data is more vulnerable to memorization, and de-duplication can effectively reduce memorization. @xcite further quantified memorization effects, revealing that memorization grows with model scale, data duplication, and prompt length. there are also works providing different views and understandings on memorization. for example, @xcite developed an efficient defense preventing memorizing the exact sentences (verbatim memorization), yet showed it fails to prevent leakage of training data. this shows the need for definitions beyond verbatim memorization. to distinguish "common" memorization from "rare" memorization, @xcite formulated a new notion of counterfactual memorization, which measures how predictions change if a particular document is deleted during training. @xcite investigated predictable memorization by extrapolating small or partially-trained lms' behavior to forecast memorization in larger models. they further presented scaling laws of prediction and explored ways to improve prediction reliability. while most literature focused on memorization during pre-training, limited work has investigated memorization in the fine-tuning stage. @xcite examined memorization risks in different fine-tuning methods for large lms. they found that fine-tuning only the head leads to higher memorization compared to fine-tuning smaller adapter modules. @xcite studied plagiarism during fine-tuning, concluding that the plagiarism patterns in fine-tuned lms depend on corpus similarity and homogeneity. however, these studies considered fine-tuning with the same objective as pre-training, which is different from the the common practice of fine-tuning in various tasks. as a result, in this paper, we focus on the more general and realistic scenario of multifaceted fine-tuning across diverse objectives. ## preliminary in this section, we first introduce the definition of memorization and the detection methods used in this paper, and then introduce our preliminary findings on open-sourced fine-tuned lms across various tasks. ## disparate memorization across tasks our preliminary study demonstrates that the memorization of fine-tuned models varies on different fine-tuning tasks. however, the causes of such difference are still unclear, which could be fine-tuning datasets or model architectures. to clarify this, in section 4.1, we control the impact of different variables to more precisely explore the relation between fine-tuning task and the memorization effect. in section 4.2, we further examine the impact of decoding methods and prefix length. note that in our fine-tuning process, we ensure that our fine-tuned models have satisfactory performance on downstream tasks, see appendix i. ## understanding the memorization disparity in the previous sections, we empirically examine the memorization rate and scaling behavior among a variety of tasks, and demonstrate the discrepancy between high-and low-memorization tasks. in this section, we provide understanding and evidence on the underlying reason behind such disparity of fine-tuned memorization. ## conclusions in this paper, we conduct extensive experiments to investigate the memorization behavior of finetuned lms among various tasks. utilizing an automatic detection pipeline, we are able to evaluate the memorization in numerous tasks and datasets. in addition, we provide understandings of the memorization disparity among tasks based on a sparse coding theory. our analysis reveals a strong correlation between attention scores and memorization.