A Comprehensive Analysis of Memorization in Large Language Models (2024)

Abstract: This paper presents a comprehensive study that investigates memorization in large language models (LLMs) from multiple perspectives. Experiments are conducted with the Pythia and LLM-jp model suites, both of which offer LLMs with over 10B parameters and full access to their pre-training corpora. Our findings include: (1) memorization is more likely to occur with larger model sizes, longer prompt lengths, and frequent texts, which aligns with findings in previous studies; (2) memorization is less likely to occur for texts not trained during the latter stages of training, even if they frequently appear in the training corpus; (3) the standard methodology for judging memorization can yield false positives, and texts that are infrequent yet flagged as memorized typically result from causes other than true memorization.

## introduction large language models (llms) have revolutionized the field of natural language processing by demonstrating an impressive ability to generate coherent text, perform complex language understanding tasks, and store a wealth of real-world knowledge @xcite . the impact of llms is spreading across society, and their uses are increasingly explored in various applications @xcite . however, llms still have many concerns; memorization is one of them. llms are known to memorize portions of their training corpora @xcite . memorization can cause crucial issues, including unintentional reproduction of copyrighted materials @xcite and personal information @xcite . understanding the extent and nature of memorization is essential for developing secure and reliable llms. training corpus text (𝑥) sample train llm (𝑓) generate prefix (𝑝) suffix (𝑠) continuation (𝑓(𝑝)) prompt reference compare memorized or not this study comprehensively evaluates memorization in llms, integrating multiple definitions of memorization and key factors contributing to memorization, which are discussed separately in different literature. we follow the standard methodology for quantitatively investigating memorization in llms, as illustrated in figure 1 . in this methodology, an llm is given a prompt and generates the continuation. memorization is identified by checking if the continuation replicates text from the training corpus. we explore two memorization types: verbatim memorization @xcite and approximate memorization @xcite . verbatim memorization refers to the exact reproduction of text from the training corpus, while approximate memorization allows for slight variations. we examine these memorization types through the size of model parameters @xcite @xcite , the length of prompts @xcite , the duplication counts of text in the training corpus @xcite , and the training step at which text is trained @xcite . we conduct experiments using fully open llms: the pythia model suite @xcite and the llm-jp v1.0 model suite @xcite . the pythia model suite offers llms of various parameter sizes, from 14m to 12b parameters, trained on an english corpus, whereas the llm-jp v1.0 model suite has two llms with 1.3b and 13b parameters, primarily trained on a mix of english and japanese corpora. both model suites are released with their pre-training corpora, allowing for analysis of memorization. our key findings are three-fold: ## related work once memorization in llms was first identified by @xcite , it has been explored from various perspectives. a line of work studies methods to better extract memorized texts from llms, making a research subfield called training data extraction attack @xcite . most existing methods follow the methodology proposed in @xcite consisting of two steps: candidate generation and membership inference @xcite . another line of work investigates the causes and mechanisms of memorization. @xcite found that verbatim memorization is more likely to happen with larger model sizes, longer prompt lengths, and frequent texts. @xcite focused on analyzing the dynamics of memorization and found that larger models memorize their training corpora more quickly. @xcite also investigated how language models forget memorized texts throughout training. a similar analysis was conducted by @xcite . a further line of work aims to reduce memorization to address security and privacy issues. @xcite and @xcite showed that deduplication of training corpora effectively reduces memorization without hurting the performance in downstream tasks. @xcite proposed a decoding method named memfree decoding, which is guaranteed to eliminate verbatim memorization by preventing the generation of n-grams present in the training corpus. ippolito et al. (2023) also showed that while memfree decoding perfectly prevents verbatim memorization, llms still generate texts that closely resemble those in their training corpora. this phenomenon is termed approximate memorization. as for the llms to explore, most previous studies use monolingual llms trained on public english corpora, such as gpt-neo @xcite and pythia @xcite , with some exceptions such as @xcite , who trains a japanese language model on an in-house, domainspecific corpus. our study incorporates insights from previous studies and presents a comprehensive analysis of memorization. besides, our analysis utilizes not only a monolingual llm primarily trained on an english corpus but also a multilingual llm trained on a mix of english and japanese corpora. ## methodology this section describes our methodology to comprehensively investigate memorization in llms. our analysis integrates multiple definitions of memorization and key factors contributing to memorization, which are discussed separately in previous studies. ## experiments we conducted experiments to investigate memorization defined in section 3.1 from the perspectives discussed in section 3.2. ## conclusion this paper investigated the memorization of llms from multiple perspectives and presented a comprehensive analysis. our experiments confirmed that findings in previous studies are consistent across different memorization definitions and model series. besides, our manual investigation suggested that the standard methodology for judging memorization can yield false positives, and texts that are infrequent yet flagged as memorized mostly arise from causes other than true memorization. a crucial future work is to investigate memorization in production-grade llms. although the llms used in our experiments represent the largest fully open llms, they significantly underperform when compared to production-grade llms, such as @xcite . the memorization of advanced models remains largely unexplored, yet it is crucial for ensuring the security and reliability of llm applications, given their profound societal impact. we are in the process of developing a fully open llm with 172b parameters, which will facilitate further exploration into memorization dynamics in state-of-the-art models. we plan to investigate whether our findings in this study still hold true in the model.