Paper: Understanding the Mechanics and Dynamics of Memorisation in Large Language Models: A Case Study with Random Strings (ICLR-2024)

Reviewer comment (Experiments):
I'm curious if the findings of this paper are more indicative of the underlying structure of Language Models or their training dynamics. The experimental design is intriguing, but I couldn't help but notice that the dataset used doesn't seem to have the scale or correlated content typically associated with "internet-sized" datasets that are used to train Large Language Models (LLMs). Could the absence of such correlations and the relatively smaller dataset size potentially influence the study's outcomes?

Author response:
It is true that we only train models to memorize a single random string at a time, a training setup that differs from the large quantities and diversity of training data seen by models during pre-training. Our approach of repeatedly training on the same data is, however, also gaining traction in [recent work](https://arxiv.org/abs/2305.16264), to more effectively utilize data. 
Our choice of training setup is deliberate, as it allows us to ignore confounding factors such as the relative occurrence frequency of some random strings relative to other strings in the training data, and recency and forgetting due to other training data sequences. Studying these factors is potentially also very interesting, but merits its own separate analysis. Recent work by Tirumala et al. (2022) and Carlini et al. (2022) has explored some of these directions.

Reviewer comment (Evaluation):
The fact local context is sufficient more than global context may solely depend on the fact the model is autoregressive.

Author response:
In our paper we deliberately focus on autoregressive models. Autoregressive models are currently the most potent group of language models and are the most important models to study in the context of memorization, since due to their use for text generation they have a high propensity for leaking memorized private information, or for producing inaccurate outputs due to a lack of memorized facts. Many other recent papers also exclusively focus on this group of models. Studying bi-directional contexts would considerably increase the complexity of the analysis and warrant its own separate investigation.

Could you provide us with more details on why you believe that “The fact local context is sufficient more than global context may solely depend on the fact the model is autoregressive”? To us, it’s not clear that the autoregressive nature of models implies a local-context based memorization. In fact, autoregressive models might memorize in several plausible ways, and short prefixes are only one of them. Potential alternatives: models might form associations with longer prefixes (e.g. much more than 5 tokens in a 512 token string), or also with tokens not immediately adjacent to the target token (e.g. at the beginning or in the middle of the string), they might learn to detect the number of preceding tokens to do position-based memorization, etc. It's not clear which option(s) models will use without further investigation, which we provide in this work.

Reviewer comment (Experiments):
Could the authors show experiments on context importance with models that are not autoregressive?

Author response:
In our paper we deliberately focus on autoregressive models. Autoregressive models are currently the most potent group of language models and are the most important models to study in the context of memorization, since due to their use for text generation they have a high propensity for leaking memorized private information, or for producing inaccurate outputs due to a lack of memorized facts. Many other recent papers also exclusively focus on this group of models. Studying bi-directional contexts would considerably increase the complexity of the analysis and warrant its own separate investigation.

Could you provide us with more details on why you believe that “The fact local context is sufficient more than global context may solely depend on the fact the model is autoregressive”? To us, it’s not clear that the autoregressive nature of models implies a local-context based memorization. In fact, autoregressive models might memorize in several plausible ways, and short prefixes are only one of them. Potential alternatives: models might form associations with longer prefixes (e.g. much more than 5 tokens in a 512 token string), or also with tokens not immediately adjacent to the target token (e.g. at the beginning or in the middle of the string), they might learn to detect the number of preceding tokens to do position-based memorization, etc. It's not clear which option(s) models will use without further investigation, which we provide in this work.

Reviewer comment (Novelty):
It would be great if the authors explain how this observation can give some insight on the training strategy of LLMs. Currently, it looks like the paper has less practical impact.

Author response:
We thank the reviewer for their feedback and are glad they appreciate our work!


### Practical impact
While the random string setting is not representative of many types of data models encountered in practice, we believe that our findings have practical implications for understanding and quantifying memorization in practice.
For instance, as we discuss in the global response (1), a lot of sensitive data looks -- at least partially -- similar to random strings, such as API and SSH keys, passwords and usernames, phone numbers, URLs, social security numbers, etc. Therefore, memorization of such strings might happen in a similar manner to what we observe in the paper.
Additionally, we discuss the implications of our results for detecting memorization in practice in global response (3). We argue that practitioners should not a priori assume a specific order of memorization and show that currently used memorization metrics can severely underestimate the degree of memorization of a piece of training data.

Reviewer comment (Writing):
It was unclear the relationship between a character and a token (though ultimately it doesn't impact the overall message of the paper) (detailed comments 4, 8, 9).

Author response:
The vocabularies of the tokenizers used in this paper contain both tokens for individual characters, e.g. “a”, “b”..., as well as tokens spanning multiple characters, e.g. “abs”. By “character-level tokenization” we mean that we only use tokens corresponding to individual characters, not character sequences. To make this part clearer, we updated the terminology in the string construction process in the revised version of the paper. We refer to the elements of the alphabet as tokens and only use the term character if the alphabet tokens indeed correspond to the characters in the Latin alphabet.

Reviewer comment (Novelty):
Results could be better situated in past findings. For instance, Tirumala et al. (2020) also test the effect of model size and dataset size on memorization with similar results. See detailed comments 14, 18.

Author response:
Thank you for pointing out the relevant related work and the connection between our findings and those of Tirumala et al. (2022). We added the missing reference in the revised paper.

The work of Tirumala et al. (2022) and their findings about memorization order are indeed relevant in light of our observations about random memorization order. Their high-level message that different parts of text can be memorized at different speeds, and that memorization order is thus hard to predict matches our findings. However, we are skeptical about the soundness of some of their claims on memorization order. Their definition of memorization is problematic, since it considers any correctly predicted token $y$ following a prefix $s$ as memorized. However, it is likely that at least some correct predictions do not happen because of memorization, but rather because of other reasoning abilities that models are acquiring. Take for example "This morning I drank a", where models might correctly predict "coffee" without every having seen this sentence in their training data. Since their findings on differences in memorization order for different POS are based on this definition, we would approach them with caution and do not include a more extensive discussion.
The difficulty of disentangling memorization and predictions made using other mechanisms is the core reason for restricting ourselves to random data in this paper, because in random strings, correct predictions can only be attributed to memorization, since there are no rules to derive the next token from. We would like to point out, though, that we think studying memorization at the token-level, rather than string-level, is more appropriate (see global response (3)), but inferring memorization from just a single correctly predicted token seems highly optimistic. A more robust measure should check whether a reasonably large fraction of tokens in a sufficiently long string postfix are correctly predicted.

Reviewer comment (Novelty):
The reference "[Understanding Transformer Memorization Recall Through Idioms](https://aclanthology.org/2023.eacl-main.19)" (Haviv et al., EACL 2023) is missing.

Author response:
Thank you for pointing out the relevant related work and the connection between our findings and those of Tirumala et al. (2022). We added the missing reference in the revised paper.

The work of Tirumala et al. (2022) and their findings about memorization order are indeed relevant in light of our observations about random memorization order. Their high-level message that different parts of text can be memorized at different speeds, and that memorization order is thus hard to predict matches our findings. However, we are skeptical about the soundness of some of their claims on memorization order. Their definition of memorization is problematic, since it considers any correctly predicted token $y$ following a prefix $s$ as memorized. However, it is likely that at least some correct predictions do not happen because of memorization, but rather because of other reasoning abilities that models are acquiring. Take for example "This morning I drank a", where models might correctly predict "coffee" without every having seen this sentence in their training data. Since their findings on differences in memorization order for different POS are based on this definition, we would approach them with caution and do not include a more extensive discussion.
The difficulty of disentangling memorization and predictions made using other mechanisms is the core reason for restricting ourselves to random data in this paper, because in random strings, correct predictions can only be attributed to memorization, since there are no rules to derive the next token from. We would like to point out, though, that we think studying memorization at the token-level, rather than string-level, is more appropriate (see global response (3)), but inferring memorization from just a single correctly predicted token seems highly optimistic. A more robust measure should check whether a reasonably large fraction of tokens in a sufficiently long string postfix are correctly predicted.

Reviewer comment (Experiments):
In p2 paragraph 1: Larger vocabulary size naturally increases entropy. To truly disentangle the effect of entropy, it would be better to keep the same vocabulary size and modulate the token sampling distribution. Alternatively, remove the focus on entropy from this paragraph and replace with, e.g., "of different vocabulary sizes, which also modulates token entropy, by sampling...".

Author response:
2. Disentangling entropy: To make sure that the behaviour we observe is indeed a result of the strings’ entropy, we conduct a control experiment in Appendix A.2 (mentioned in Section 3.1). We modify the entropy of strings with fixed alphabet size (26) by increasing the probability of sampling the first character (“a”) such that the generated strings have the same entropy as the strings with smaller alphabet size. Our results show the same patterns for different entropy levels at the same alphabet size as for different entropies due to different alphabet sizes. Due to space constraints, we could not include these results in the main paper.

Reviewer comment (Experiments):
The introduction frames the paper's potential impact in the context of naturalistic text (e.g., privacy, factuality), but the study is restricted to single random strings. Language Transformers in the wild are trained on natural language, which is governed by grammar, making this choice potentially limiting.

Author response:
Using formal (probabilistic) grammars to create structured data is a great way to investigate how models learn to reason according to the grammar rules, and we are exploring this direction. We believe, however, that it is not directly suitable for understanding memorization. Intuitively, for a string to be memorized, a model must have seen and remembered it from its training data, there should not be any other way to generate it. We can achieve this with random strings, but if we generate data in a structured manner, models might produce valid strings by learning and using grammar rules, instead of memorizing the training strings.

Reviewer comment (Writing):
In Section 2.1, it is confusing what a character is. The line "The alphabet...is a subset of tokens in the tokenizer vocabulary" implies the alphabet is a subset of tokens, which needs clarification.

Author response:
9. We added examples with different alphabet sizes and entropies in Appendix A.4.

Reviewer comment (Writing):
In Section 2.1 last paragraph: The definition assuming that $\mathcal M$ predicts for position i the token with the largest probability is redundant.

Author response:
5. We removed this part in the revised version.

Reviewer comment (Writing):
In Section 2.1 last paragraph: Defining a new term "plurality prediction" (which is never used again) is unnecessary; it would be better to use "greedy prediction" or "top prediction" as commonly used in the literature.

Author response:
6. We switched to using greedy decoding terminology.

Reviewer comment (Writing):
In Section 2.2: Enforcing character-level tokenization is confusing given earlier comments—does the alphabet consist of individual characters like a-z or individual tokens in the tokenizer vocabulary?

Author response:
The vocabularies of the tokenizers used in this paper contain both tokens for individual characters, e.g. “a”, “b”..., as well as tokens spanning multiple characters, e.g. “abs”. By “character-level tokenization” we mean that we only use tokens corresponding to individual characters, not character sequences. To make this part clearer, we updated the terminology in the string construction process in the revised version of the paper. We refer to the elements of the alphabet as tokens and only use the term character if the alphabet tokens indeed correspond to the characters in the Latin alphabet.

Reviewer comment (Presentation):
The paper would benefit from including examples of the random strings.

Author response:
9. We added examples with different alphabet sizes and entropies in Appendix A.4.

Reviewer comment (Presentation):
Only in the first paragraph of section 3.3 is it clear that the LM is only learning one string at a time. This should be made clear earlier in the paper.

Author response:
10. In most cases we train models only on single strings, but in Figure 8 we also train them on multiple strings at a time, so this depends on the specific experiment.

Reviewer comment (Presentation):
Figures 3-5 have GPT2-130M, whereas Figures 1, 2, and 7 have GPT2-140M.

Author response:
11. Good catch, 140M is correct, we corrected Figures 3-5 in the revision.

Reviewer comment (Evaluation):
At position 0 in Fig. 5a, the probability is also uniform. Is this given a `<BOS>` token? The probability mass should be spread over the entire vocabulary space and should better reflect the first-token statistics of the pretraining data.

Author response:
12. We do not use <BOS> tokens for GPT-2 and Pythia models. Figure 5a simply shows whether the token at the respective position has been predicted correctly at that timestep. The first column shows the prediction for the second token in the string, since the model does not produce a prediction for the first token (the output for the first token is the prediction for the second token). Please let us know if this doesn’t clarify the question.

Reviewer comment (Novelty):
In Section 3.3: The conclusion that memorisation happens at the granularity of individual tokens and not entire strings may not be surprising, given that training is via teacher-forcing on a token-level objective.

Author response:
13. We are driving the loss to zero over all tokens in the string at the same time. Alternative memorization dynamics are therefore also conceivable, such as memorizing tokens in chunks, from left to right, etc. We discuss the implications of this finding in the global response (3).

Reviewer comment (Novelty):
In Section 4.2: The finding that small local prefixes are very effective for recalling tokens may not be surprising for random strings and may not generalize to structured strings. This recalls Tirumala et al. (2022) finding that "highly informative" tokens such as numbers get memorized first.

Author response:
Being able to uniquely predict the next token from short prefixes is only a *necessary condition* for the recollection accuracy we observe with short prefixes. It’s not clear a priori whether models will converge towards some information-theoretically optimal solution. Models might memorize in several ways, and short prefixes are only one of them. Potential alternatives: models might form associations with longer prefixes (e.g. much more than 5 tokens in a 512 token string) as [observed in other settings](https://arxiv.org/abs/2206.04301), or also with tokens not immediately adjacent to the target token (e.g. at the beginning or in the middle of the string), they might learn to detect the number of preceding tokens to do position-based memorization, etc. It's not clear which option(s) models will use without further investigation, which we provide in this paper.

Reviewer comment (Presentation):
Fig. 8 caption should be fleshed out to be standalone.

Author response:
15. We moved details and takeaways into the Figure caption.

Reviewer comment (Experiments):
The discussion on generalizing to non-random strings could be expanded. Virtually all applications of language Transformers apply to structured strings (natural language). For example, defining the distribution over strings using a probabilistic grammar.

Author response:
Using formal (probabilistic) grammars to create structured data is a great way to investigate how models learn to reason according to the grammar rules, and we are exploring this direction. We believe, however, that it is not directly suitable for understanding memorization. Intuitively, for a string to be memorized, a model must have seen and remembered it from its training data, there should not be any other way to generate it. We can achieve this with random strings, but if we generate data in a structured manner, models might produce valid strings by learning and using grammar rules, instead of memorizing the training strings.

Reviewer comment (Evaluation):
The results show that memorization happens on a token-by-token basis—then, does token-level memorization really tell you anything about the string-level? For instance, if a model memorizes the first token, it does not necessarily mean it memorized the whole sequence.

Author response:
It is true that in a natural language settings, simply predicting a single token correctly, given some prefix from the training data, does not imply that that token (or a larger string that it is contained in) has been memorized, since the model could also predict that token using word statistics, grammar rules, etc. (e.g. “blue” in “The sky is blue.”). It is merely a necessary condition for memorization, but not a sufficient one. Current approaches to quantifying memorization using string-based notions (Carlini et al. (2022)) assume that if a sufficiently long string from the training data is predicted exactly, given its prefix, then it is very unlikely that this is due to something other than memorization (because predicting each token exactly as in the training data independently is highly improbable). By focussing on random strings in this study, we sidestep this issue. If models can predict the next token correctly, it has to be because of memorization.
We discuss the implications of our findings for detecting memorization in the global response (3).

Reviewer comment (Experiments):
The result that token positions are equally likely to be memorized first may be a consequence of using random strings. In non-random strings, this result might break down due to the statistics of natural language, as in Tirumala et al. (2022) which find differentiated memorization speed according to POS.

Author response:
Thank you for pointing out the relevant related work and the connection between our findings and those of Tirumala et al. (2022). We added the missing reference in the revised paper.

The work of Tirumala et al. (2022) and their findings about memorization order are indeed relevant in light of our observations about random memorization order. Their high-level message that different parts of text can be memorized at different speeds, and that memorization order is thus hard to predict matches our findings. However, we are skeptical about the soundness of some of their claims on memorization order. Their definition of memorization is problematic, since it considers any correctly predicted token $y$ following a prefix $s$ as memorized. However, it is likely that at least some correct predictions do not happen because of memorization, but rather because of other reasoning abilities that models are acquiring. Take for example "This morning I drank a", where models might correctly predict "coffee" without every having seen this sentence in their training data. Since their findings on differences in memorization order for different POS are based on this definition, we would approach them with caution and do not include a more extensive discussion.
The difficulty of disentangling memorization and predictions made using other mechanisms is the core reason for restricting ourselves to random data in this paper, because in random strings, correct predictions can only be attributed to memorization, since there are no rules to derive the next token from. We would like to point out, though, that we think studying memorization at the token-level, rather than string-level, is more appropriate (see global response (3)), but inferring memorization from just a single correctly predicted token seems highly optimistic. A more robust measure should check whether a reasonably large fraction of tokens in a sufficiently long string postfix are correctly predicted.