Causal Estimation of Memorisation Profiles (2024)

Abstract: Understanding memorisation in language models has practical and societal implications, e.g., studying models’ training dynamics or preventing copyright infringements.Prior work defines memorisation as the causal effect of training with an instance on the model’s ability to predict that instance. This definition relies on a counterfactual: the ability to observe what would have happened had the model not seen that instance.Existing methods struggle to provide computationally efficient and accurate estimates of this counterfactual. Further, they often estimate memorisation for a model architecture rather than for a specific model instance. This paper fills an important gap in the literature, proposing a new, principled, and efficient method to estimate memorisation based on the difference-in-differences design from econometrics. Using this method, we characterise a model’s memorisation profile–its memorisation trends across training–by only observing its behaviour on a small set of instances throughout training.In experiments with the Pythia model suite, we find that memorisation (i) is stronger and more persistent in larger models, (ii) is determined by data order and learning rate, and (iii) has stable trends across model sizes, thus making memorisation in larger models predictable from smaller ones.

## introduction large language models (lms) are often pretrained with a single pass on web-scale datasets @xcite @xcite . given the colossal size of these training sets, one may expect each individual instance to have little impact on the final model. yet, lms can still reproduce entire sequences from their training set verbatim @xcite , suggesting that models can store, or memorise, precise knowledge about individual training instances. in the era of large lms, measuring memorisation is crucial for nlp practitioners; it has implications for copyright and data protection @xcite @xcite , for how models encode factual information @xcite , and for understanding their training dynamics @xcite . one line of prior work has adopted a causal definition of memorisation: it is the causal effect of observing an instance during training on a model's ability to correctly predict that instance @xcite . despite being an intuitive concept, quantification of this definition is not straightforward as it requires knowledge of a counterfactual: 1 we must know how our model would have performed on a training instance had the model not been trained on it. to overcome this challenge, prior work has proposed a variety of methods to estimate memorisa-tion. some estimate it by training a model multiple times on different subsets of the data (e.g., @xcite , while others implicitly assume this counterfactual's value to be negligible @xcite . both these approaches, however, have drawbacks: the first computes memorisation for an architecture rather than a specific model, while the second relies on a strong assumption (we discuss this in detail in §5). in this paper, we first formalise counterfactual memorisation as the difference between two potential outcomes; notably, our formalisation generalises prior definitions of memorisation, allowing us to compare them within a unified framework. we then draw from the econometrics literature @xcite and propose a new method which estimates memorisation using only observational data; our method simply needs a model's performance measurements (e.g., loglikelihood) on a subset of the training data throughout training. the output of our method is what we term a memorisation profile: a model's memorisation of training batches over the course of training. empirically, we use this method to analyse memorisation for models in the pythia suite @xcite and characterise their memorisation profiles; e.g., fig. 1 reports the memorisation profile of pythia 6.9b. by studying these memorisation profiles we find that memorisation is stronger and more persistent in larger models. furthermore, both the learning rate and the position of an instance in the training set considerably impact how strongly that instance is memorised. finally, memorisation profiles are stable across model sizes; thus, we can predict memorisation in larger models from the memorisation observed in smaller ones. ## background in this section, we introduce some background on language modelling and on causal analysis which will be required throughout our paper. ## counterfactual memorisation intuitively, counterfactual memorisation can be understood as the answer to the question: how would the model's performance on instance x at timestep c be different if we had not trained on it at timestep g? using the potential outcomes notation, we formalise this definition as follows. definition 2. counterfactual memorisation is the causal effect of using instance x for training at the observed timestep g(x) = g on the model's performance on this same instance at timestep c: in econometrics, eq. ( 2 ) is called an individual treatment effect (ite). notably, the first potential outcome in this equation, y c (x; g), can be observed from the data since, by definition, we trained on x at timestep g(x) = g. however, the second term, y c (x; ∞), is counterfactual. to compute the ite, we would need to estimate this counterfactual outcome for a specific instance, which is challenging due to unobserved factors and heterogeneity foot_5 @xcite . while we would ideally estimate memorisation at the instance level, we focus on average effects instead, as is common in the econometrics literature @xcite . definition 3. expected counterfactual memorisation is the average causal effect of using instances for training at timestep g on the model's performance on these same instances at timestep c: together, the τ g,c form a matrix which we term the model's memorisation profile; each row therein is the memorisation path of a batch. memorisation profiles and paths allow us to analyse a model's memorisation patterns across different treatment and checkpoint steps. notably, there cannot be memorisation whenever @xmath0, as the instances have not been seen by the model yet, so τ g,@xmath1. we term τ g,c as instantaneous memorisation when @xmath2, as persistent memorisation when @xmath3, and as residual memorisation when @xmath4. ## estimating memorisation the practical implication of defining memorisation at a treatment level g is that we can only make causal claims for groups of instances treated at the same timestep (i.e., a batch b g ), rather than for individual instances. however, even though this approach simplifies the problem, estimating eq. ( 3 ) still poses major challenges as it contains a counterfactual. a simple decomposition makes this counterfactual explicit: expectation 1 can be directly estimated from the data because batch b g contains a set of examples x for which g(x) = g and, thus, we can invoke the consistency assumption to equate y c (x; g) with the observed outcome y c (x). let us define the mean across instances in a batch as: we can thus use eq. ( 5 ) as an estimator for expectation 1 . expectation 2 , however, is counterfactual: we cannot observe the potential outcome y c (x; ∞) for instances treated at timestep g ̸ = ∞. the presence of counterfactual potential outcomes in causal estimands creates challenges for their estimation, being known as the fundamental problem of causal inference @xcite . the goal of causal methods is then to estimate these counterfactual outcomes from observed ones, using comparable groups of instances. thus far, we have defined our causal estimand. in this section, we perform steps two and three of causal estimation ( §2.2): we derive two statistical estimands for our causal estimand (identifying it under specific assumptions), and provide concrete estimators for them. ## prior notions of memorisation memorisation has recently received much attention @xcite @xcite , inter alia). 9 prior work has studied how model architecture and training choices influence memorisation @xcite @xcite , and where memorised instances are stored within a model @xcite . in this section, we use our framework to discuss three prior notions of memorisation which we consider most relevant to our paper: previous operationalisations of counterfactual memorisation (e.g., feldman, 2020), influence functions @xcite , and extractable memorisation @xcite . ## experiments while our method applies to any model trained with a single pass on its training data, we focus on quantifying memorisation in pretrained lms, which are characterised by the use of architectures with a large number of parameters and large datasets. due to the costs of training such models from scratch, we take advantage of open-source pretrained models whose intermediate checkpoints and preprocessed data are publicly available. in this section, we detail the models and data used and describe how we collect the observed outcomes y c (x). the pythia suite. we use the publicly available pythia model suite 10 (biderman et al., 2023b), composed of 8 transformers of sizes ranging from 70m to 12b parameters. these models were trained on the pile dataset @xcite , a 300b-token curated collection of english documents. all models are trained using the same data. specifically, the dataset is shuffled and "packed" into sequences of 2,049 tokens; foot_10 each of these sequences corresponds to an instance x. training was performed using a cosine learning rate schedule with warm-up, and using a batch size of 1,024 sequences, resulting in exactly 143k optimisation steps. we use the model versions trained on the deduplicated pile dataset to reduce the risk of finding spurious memorisation patterns due to duplication. the deduplicated dataset has 207b tokens, thus models using this version are trained for circa 1.5 epochs to keep an equal token count relative to the non-deduplicated versions. we consider the checkpoints relative to the first epoch (i.e., up to step 95k). 12 more details are in app. c. ## results we report the memorisation profiles of all pythia sizes in fig. 2 . below, we use these memorisation profiles to describe different types of memorisation. ## conclusions the memorisation of training data by neural networks has critical implications for privacy, copyright, and security. thus, well-founded quantifications of memorisation, and corresponding accurate and efficient methods for their estimation are of great importance. this work presents one such quantification and builds on the econometrics literature to derive an unbiased and efficient estimator of memorisation based on the difference-indifferences design. we use this estimator to study the memorisation profiles of the pythia model suite and find that memorisation is stronger and more persistent in larger models, determined by data order and learning rate, and stable across model sizes.