A Multi-Perspective Analysis of Memorization in Large Language Models (2024)

Abstract: Large Language Models (LLMs) can generate the same sequences contained in the pre-train corpora, known as memorization.Previous research studied it at a macro level, leaving micro yet important questions under-explored, e.g., what makes sentences memorized, the dynamics when generating memorized sequence, its connection to unmemorized sequence, and its predictability.We answer the above questions by analyzing the relationship of memorization with outputs from LLM, namely, embeddings, probability distributions, and generated tokens.A memorization score is calculated as the overlap between generated tokens and actual continuations when the LLM is prompted with a context sequence from the pre-train corpora.Our findings reveal:(1) The inter-correlation between memorized/unmemorized sentences, model size, continuation size, and context size, as well as the transition dynamics between sentences of different memorization scores,(2) A sudden drop and increase in the frequency of input tokens when generating memorized/unmemorized sequences (boundary effect),(3) Cluster of sentences with different memorization scores in the embedding space,(4) An inverse boundary effect in the entropy of probability distributions for generated memorized/unmemorized sequences,(5) The predictability of memorization is related to model size and continuation length. In addition, we show a Transformer model trained by the hidden states of LLM can predict unmemorized tokens.

## introduction large language models (llms), trained with enormous parameter and pre-train data sizes, like @xcite , show surprising performance on various tasks. due to such enormous model size and pre-train data size, combined with the blackbox nature of neural models @xcite , llms present unique behaviors @xcite that are unprecedented in previous machine learning, one of which is memorization. memorization @xcite in the llms means the llm can generate the same content recorded in their pre-train corpus. being a coin with two sides, memorization in llm can provide knowledge @xcite or cause personal information leakage @xcite . previous research @xcite @xcite has studied memorization at the macro level, revealing memorization from the training phase or overlap between models, leaving more micro yet important questions left underexplored, e.g., what makes sentences memorized, dynamics of sentences with varying memorization scores transit to other scores when trained in a larger model, the dynamics when generating memorized/unmemorized sequence, its connection to unmemorized sequence, and its predictability. to answer those questions, we study memorization from multiple perspectives. we form contexts of varying lengths from the pre-train corpora and input them into llms of different sizes. by controlling the maximum generated tokens, we collect the outputs (embeddings, decoding probabilities, and generated tokens) and compute the memorization score by comparing them with the actual continuations in the pre-train corpora. our analysis connects memorization to model size, context size, continuation size, and unmemorized sequences. we also explored the dynamics of generating memorized/unmemorized tokens at input and output levels and the predictability of memorization. our findings reveal: ## limitations this research has analyzed various factors regarding the memorization behavior of llms. we acknowledge that there are still limitations to this research. due to the lack of llms whose models and data are both being released, it is hard to compare the memorization across different llms since even if they were released, the pre-train data differs based on different llms. future research can focus on how to comprehensively measure the memorization of various llms, either close-sourced like gpt-4 or open-sourced like pythia. additionally, pythia only provides llms up to 12b, which is still considerably small when compared to sota opensourced llms like llama2, whose largest model is 70b. the emergent abilities reaching 70b size may also affect memorization. though we trained a transformer model to predict the memorization of llms, it is more analysis-oriented, proving the possibility of predicting memorization. thus, the performance in the prediction experiment is not the main focus. additionally, in this research, the discussion of memorization is under the context when the llm generates tokens that are the same as the actual continuations in the corpora, defined as k-extracble in this study. it is possible that paraphrase memorization exists. however, it is difficult to identify such behavior on a large scale, especially across whole corpora. this is because the identification of paraphrase memorization can not be simply decided by either token overlap or embedding similarity since neither of them can truly compare whether a sentence is a paraphrase. this means we have to identify by more complex methods, e.g., human annotators or a trained classification model. however, a classification model cannot be 100% accurate when applied to the corpora level, leading to false positives and false negatives that influence the analysis results. however, paraphrase memorization can be left to future research. ## ethical considerations in this study, we prompted the pythia model with context tokens from its pre-train corpora. the data released by pythia are all tokenized and turned into token ids, so the original information is not visible. we have provided a case study of the prediction of memorization in the appendix a.2 by turning the token ids into texts. however, we did not observe any personal information or offensive language dur-ing this process. additionally, we obeyed related open-source licenses of both pythia and pile corpora. thus, this study is not concerned with ethical issues.