Understanding Transformer Memorization Recall Through Idioms (2023)

Abstract: To produce accurate predictions, language models (LMs) must balance between generalization and memorization. Yet, little is known about the mechanism by which transformer LMs employ their memorization capacity. When does a model decide to output a memorized phrase, and how is this phrase then retrieved from memory? In this work, we offer the first methodological framework for probing and characterizing recall of memorized sequences in transformer LMs. First, we lay out criteria for detecting model inputs that trigger memory recall, and propose idioms as inputs that typically fulfill these criteria. Next, we construct a dataset of English idioms and use it to compare model behavior on memorized vs. non-memorized inputs. Specifically, we analyze the internal prediction construction process by interpreting the model’s hidden representations as a gradual refinement of the output probability distribution. We find that across different model sizes and architectures, memorized predictions are a two-step process: early layers promote the predicted token to the top of the output distribution, and upper layers increase model confidence. This suggests that memorized information is stored and retrieved in the early layers of the network. Last, we demonstrate the utility of our methodology beyond idioms in memorized factual statements. Overall, our work makes a first step towards understanding memory recall, and provides a methodological basis for future studies of transformer memorization.

## introduction transformer language models (lms) memorize instances from their training data @xcite , and evidence is building that such memorization is an important precondition for their predictive abilities @xcite @xcite @xcite . still, it is unknown when models decide to output memorized sequences, and how these sequences are being retrieved internally from memory. current methods for analyzing memorization @xcite @xcite use definitions that are based on models performance, which changes between models and often also between training runs. moreover, these methods study memorization behavior in terms of the model's "black-box" behavior rather than deriving a behavioral profile of memory recall itself. our first contributions are to provide a definition and construct a dataset that allows probing memorization recall in lms. we define a set of criteria for identifying memorized sequences that does not depend on model behavior: foot_1 sequences that have a single plausible completion that is independent of context and can be inferred only given the entire sequence. we show that many idioms (e.g., "play it by ear") fulfill these conditions, allowing us to probe and analyze memorization behavior. furthermore, we construct a dataset of such english idioms, dubbed idiomem, and release it publicly for the research community. next, to analyze memory recall behavior, we compare the construction process of predictions that involve memory recall with those that do not. to this end, given a lm, we create two sets of memorized and non-memorized idioms from idiomem (fig. 1 , a ). we then adopt a view of the transformer inference pass as a gradual refinement of the output probability distribution @xcite . concretely, the token representation at any layer is interpreted as a "hidden" probability distribution over the output vocabulary @xcite (fig. 1 , b ). this interpretation allows tracking the prediction across layers in the evolving distribution. we find a clear difference in model behavior between memorized and non-memorized predictions (fig. 1 , c ). this difference persists across different transformer architectures and sizes: retrieval from memory happens in two distinct phases, corresponding to distinct roles of the transformer parameters and layers: (1) candidate promotion of memorized predictions' rank in the hidden distribution in the first layers, and (2) confidence boosting where, in the last few layers, the prediction's probability grows substantially faster than before. this is unlike non-memorized predictions, where the two phases are less pronounced and often indistinct. we further confirm these phases of memorized predictions through intervention in the network's ffn sublayers, which have been shown to play an important role in the prediction construction process @xcite . concretely, zeroing-out hidden ffn neurons in early layers deteriorate memory-recall, while intervention in upper layers does not affect it. last, we show our findings extend to types of memory recall beyond idioms by applying our method to factual statements from the lama-uhn dataset @xcite ) (e.g. "the native language of jean marais is french"). for factual statements that were completed correctly by the lm, we observe the same two phases as in memorized idioms, further indicating their connection to memory recall. to summarize, we construct a novel dataset of idioms, usable for probing lm memorization irrespective of the model architecture or training parameterization. we then design a probing methodology that extracts carefully-devised features of the internal inference procedure in transformers. by applying our methodology and using our new dataset, we discover a profile that characterizes memory recall, across transformer lms and types of memorized instances. our released dataset, probing framework, and findings open the door for future work on transformer memorization, to ultimately demystify the internals of neural memory in lms. ## criteria for detecting memory recall to study memory recall, we require a set of inputs that trigger this process. prior work on memorization focused on detecting instances whose inclusion in the training data has a specific influence on model behavior, such as increased accuracy on those instances @xcite @xcite @xcite . as a result, memorized instances differ across models and training parameterization. our goal is instead to find a stable dataset of sequences that correctly predicting their completion indicates memorization recall. this will greatly reduce the overhead of studying memorization and facilitate useful comparisons across models and studies. to build such a dataset, we start by defining a general set of criteria that are predicates on sequence features, entirely independent of the lm being probed. given a textual sequence of n words, we call the first n -1 words the prompt and the nth word the target. we focus on the task of predicting the target given the prompt, i.e., predicting the last word in a sequence given its prefix. 3 such predictions can be based on either generalization or memorization, and we are interested in isolating memorized cases to study model behavior on them. particularly, we are looking for sequences for which success in this task implies memorization recall. we argue that the following criteria are sufficient for detecting such memorized sequences: 1. single target, independent of context: we require that the target is the only correct continuation, regardless of the textual context where the prompt is placed. foot_3 claim 2.1. assume a sequence fulfills the above criteria. then, if a lm correctly predicts the target, it is highly likely that this prediction involves memory recall. justification. first, observe that most naturallanguage prompts have many possible continuations. for example consider the sentence "to get there fast, you can take this ____". likely continuations include "route", "highway", "road", "train", "plane", "advice", inter alia. note that there are several divergent interpretations or contexts for the prompt, and for each, language offers many different ways to express similar meaning. a prediction that is a product of generalization -i.e., it is derived from context and knowledge of language -always has plausible alternatives, depending on the context and stylistic choice of words. hence, the relationship between the entire prompt and the target, where the target is the single correct continuation, is something that needs to be memorized rather than derived via generalization. a lm that predicts the single correct continuation either memorized this relationship, or used "cues" from the prompt that happen to provide indication towards the correct continuation. to illustrate the latter, consider the sequence "it's raining cats and ____" which has a single correct continuation, "dogs", but a lm might predict it without observing this sequence during training, due to the semantic proximity of "cats" and "dogs". our second criterion excludes such cases by requiring that the correct continuation is only likely given the entire sequence. therefore, a lm that correctly completes a sequence that fulfills both criteria, is likely to have recalled it from memory. in the next section, we argue that idioms are a special case of such sequences, and are thus useful for studying memorization ( §3). ## the utility of idioms for studying memorization an idiom is a group of words with a meaning that is not deducible from the meanings of its individual words. for example, consider the phrase "play it by ear" -there is a disconnect between its nonsensical literal meaning (to play something by a human-body organ called 'ear') and its intended idiomatic meaning (to improvise). a key observation is that idioms often satisfy our criteria ( §2), and therefore can probe memorization. first, by definition, idioms are expected to be non-compositional @xcite . they are special "hard-coded" phrases that carry a specific meaning. as a result, their prompts each have a single correct continuation, regardless of their context (criterion 1). for example, consider the prompt "crying over spilt ____" -a generalizing prediction would predict that this slot may be filled by any spillable item, like wine, water or juice, while a memorized prediction will retrieve only milk in this context. notably, while this is an empirical characterization of many idioms, there might be exceptions, e.g., contexts that are adversarially chosen to change the completion. second, many idioms are "irreducible", for example the sub-sequences "crying over" or "over spilt" by themselves have but scant connection to the word "milk". still, not all idioms fulfill the criteria. for example, even when the idiom is far from literal, its constituents sometimes strongly indicate the correct continuation, such as with the case of "it's raining cats and ____" (as explained in §2). to construct a dataset of memorization-probing sequences, we will carefully curate a set of english idioms and filter out ones that do not fulfill our criteria. ## probing methodology background and notation assuming a transformer lm with l layers, a hidden dimension d and an input/output-embedding matrix @xmath0. denote by @xmath1, ..., s t ⟩ the input sequence to the lm, and let h ℓ i be the output for token i at layer ℓ, for all ℓ ∈ 1, ..., l and @xmath2, ..., t. the model's prediction for a token s i is obtained by projecting its last hidden representation h l i to the embedding matrix, i.e. sof tmax(eh l i ). following @xcite , we interpret the prediction for a token s i by viewing its corresponding sequence of hidden representations h 1 i , ..., h l i as an evolving distribution over the vocabulary. concretely, we read the "hidden" distribution at layer ℓ by applying the same projection to the hidden representation at that layer: using this interpretation, we track the probability and rank of the predicted token in the output distribution across layers. a token's rank is its position in the output distribution when sorted by probability from highest to lowest (e.g. the rank of the final predicted token is zero). ## memorization of factual statements we now examine if our findings generalize beyond idioms to other types of memory recall, focusing on the completion of statements expressing facts. data datasets for evaluating memorization of factual knowledge typically contain simple queries such as "the continent of kuwait is", where predicting the next token requires knowledge of the triplet ⟨s, r, t⟩ where s is a source entity (e.g., kuwait), t a target entity (e.g., asia), and r is the relation between them (e.g., is the continent of). however, unlike idioms, such queries are not suitable for probing memorization since they often fail to satisfy the criteria in §2. concretely, queries often include "clues" that could make the prediction easy to guess and based on generalization @xcite ) (e.g. predicting a spanishspeaking country for the query "federico lópez was born in"), and the same fact can be expressed in multiple different ways (e.g. "kuwait is a country in western asia" also encodes the above fact). to test memorization of facts, we, therefore, collect factual statements where such cases are excluded. we use lama-uhn @xcite , a subset of the lama dataset @xcite , where "easy-to-guess" queries are filtered out. lama comprises of queries structured as "fill-in-the-blank" cloze statements (e.g. "gordon scholes is a member of the ____political party."). to accommodate autoregressive lms, we consider only queries where the blank appears at the end. in addition, we keep only queries with a single correct completion (based on lama). this turns our definition of memorized and non-memorized sets ( §4) equivalent to separating based on the evaluation metric of lama: an instance is considered memorized if the model predicts the single correct completion and non-memorized otherwise. overall, the resulting collection consists of 17,855 factual statements with 22 unique relations. ## related work memorization as training-data influence. memorization in lms has attracted immense attention due to their rich sensitive training data @xcite @xcite @xcite @xcite . recent work suggests that memorization is necessary for performant ml due to the "long tail" of infrequently-observed patterns @xcite @xcite . this line of work has two key limitations: (a) only black-box behavior is measured rather than looking at models' internal prediction process, and (b) it detects memorized instances by measuring the effect of their inclusion in the training set on inference behavior. this results in a set of memorized examples that is specific to the model, training data, and even training pass, making it difficult to build on these results in future research. transformers and idioms. @xcite showed that idioms are identified using textual cues within the expression; @xcite showed that idioms tend to be internally processed as single units of meaning. it has also been known @xcite that lm contextual embeddings encode information about whether or not an expression is idiomatic (vs. literal). @xcite also studied representations and interpretation of non-compositional sequences, such as idioms. no prior work used idioms to probe lm memorization, which is one of the main contributions of this work. we release our dataset, idiomem, to facilitate future research on memorization recall in lms. diagnostic datasets, such as idiomem, have often proven useful in the past @xcite . ## conclusion we introduce a methodological framework for detecting and analyzing memorized predictions in transformer lms. this includes a set of criteria on textual sequences for probing memorized predictions, the idiomem dataset of idioms fulfilling these criteria, and an interpretation method of prediction internals. we characterize a behavioral profile that is unique to predictions involving memory recall and is observable across different lms and data types. by providing these fundamental tools and initiating a thread of research on the phenomena we observe, we hope to empower future work towards demystifying transformer memorization.