W HAT DO LARGE NETWORKS MEMORIZE ?
Anonymous authors
Paper under double-blind review
A BSTRACT
The success of modern neural models has prompted renewed study of the connection between memorisation and generalisation : such models typically generalise well, despite being able to perfectly ﬁt (“memorise”) completely random labels.
To more carefully study this issue, Feldman (2019); Feldman & Zhang (2020) provided a simple stability-based metric to quantify the degree of memorisation of a speciﬁc training example, and empirically computed the corresponding memorisation proﬁle of a ResNet model on image classiﬁcation benchmarks.
While an exciting ﬁrst glimpse into how real-world models memorise, these studies leave open several questions about memorisation of practical networks.
In particular, how is memorisation affected by increasing model size, and by distilling a large model into a smaller one?
We present an empirical analysis of these questions on image classiﬁcation benchmarks.
We ﬁnd that training examples exhibit a diverse set of memorisation trajectories across model sizes, with some samples having increased memorisation under larger models.
Further, we ﬁnd that distillation tends to inhibit memorisation of the student model, while also improving generalisation.
Finally, we show that other memorisation measures do not capture such properties, despite highly correlating to the stability-based metric of Feldman (2019).
I NTRODUCTION
Statistical learning is conventionally thought to involve a delicate balance between memorisation of training samples, and generalisation to test samples (Hastie et al., 2001).
However, the success of modern overparameterised neural models challenges this view: such models have proven successful at generalisation, despite having the capacity to memorize, e.g., by perfectly ﬁtting completely random labels (Zhang et al., 2017).
Indeed, in practice, such models typically interpolate the training set, i.e., achieve zero misclassiﬁcation error.
This has prompted a series of analyses aiming to understand why such models can generalise (Bartlett et al., 2017; Brutzkus et al., 2018;
Belkin et al., 2018; Neyshabur et al., 2019; Bartlett et al., 2020; Wang et al., 2021).
Recently, Feldman (2019) established that in some settings, memorisation may be necessary for generalisation.
Here, “memorisation” of a sample is deﬁned via an intuitive stability-based notion, where the high memorisation examples are the ones that the model can correctly classify only if it they are present in the training set (see Equation 1 in §2).
A salient feature of this deﬁnition is that it allows the level of memorisation 1 of a training sample to be estimated for practical neural models trained on real-world datasets.
To that end, Feldman & Zhang (2020) studied the memorisation proﬁle of a ResNet model on standard image classiﬁcation benchmarks.
While an exciting ﬁrst glimpse into how real-world models memorise, this study leaves open several questions about the nature of memorisation as arising in practice.
We are particularly interested in two questions:
— Model size and memorisation .
Increasing model size (e
g., depth of a ResNet) has a welldocumented effect of (unsurprisingly) improving training accuracy, and (surprisingly) test accuracy as well (Neyshabur et al., 2019).
It is unclear what impact model size has on memorisation, however; while larger models have more memorisation capacity , do they make judicious use of this and memorise fewer , more informative samples than smaller models?
— Distillation and memorisation .
While the study of memorisation in large neural models is fascinating, its practical relevance is stymied by a basic fact: such models are typically inadmissible
[Figure]
t
i
[Figure]
n
[Figure]
c o r
[Figure] [Figure]
labelled examples follow either an increasing or decreasing trend; in §3 we discuss their characteristics in more detail.
In the right plot, we show that distillation inhibits memorisation of the student : each plot shows the joint density of memorisation scores under a standard model, and one distilled from a ResNet-110 teacher.
As the gap between teacher and student models becomes wider, samples memorised by the student see a sharp decrease in memorisation score under distillation (see the vertical bar at right end of plot).
Notably, distillation does not affect other samples as strongly.
for real-world settings with constraints on latency and memory.
Instead, such models are typically compressed via distillation (Bucil ˇ a et al., 2006; Hinton et al., 2015), a procedure that involves ﬁtting a smaller “student” model to the predictions of a “teacher”.
This raises a natural question: how much of the teacher model’s memorisation is transferred to the student ?
In this paper, we take a step towards comprehensively exploring the memorisation behavior of modern neural networks, and in the process contribute to progress on answering the aforementioned questions.
Fixing our attention on the elegant stability-based notion of Feldman (2019), we study how memorisation varies as capacity of common model architectures (ResNet, MobileNet) increases, with and without distillation.
We have four main ﬁndings: ﬁrst, the distribution of memorisation scores becomes increasingly bi-modal with increased model size, with relatively more samples assigned either a very low or very high score.
Second, training examples exhibit a diverse set of memorisation trajectories across model sizes, and in particular, some examples are increasingly memorised by larger models.
Third, while distillation largely preserves the student model’s memorisation scores, it tends to inhibit memorisation in the student.
Finally, we analyze computationally tractable measures of memorisation which have been shown in previous work to highly correlate to the stability-based notion of memorisation score, and show that surprisingly they lead to signiﬁcantly different properties.
To summarise, our contributions are:
(i) we present a quantitative analysis of how the degree of memorisation of standard image classiﬁers varies with model complexity (e.g., depth or width of a ResNet).
Our main ﬁndings are that increasing the model complexity tends to make memorisation more bi-modal ; and that there are examples of varying memorisation score trajectories across model sizes, including those where memorisation increases with model complexity.
(ii) we then present a quantitative analysis of how distillation inﬂuences memorisation, and show that it tends to inhibit memorisation, particularly of samples that the one-hot (i.e., non-distillation) student memorises.
(iii) we conclude with experiments using computationally tractable measures of memorisation and example difﬁculty, based on behaviour across training steps (Jiang et al., 2021a), and behaviour across model layers (Baldock et al., 2021).
We ﬁnd that although these measures strongly correlate with stability-based memorisation, they do not capture key trends seen in the latter.
We remark that the aim of this work is not in proposing new methods, or memorisation measures, but rather to more systematically analyse the behaviour of existing memorisation measures.
B ACKGROUND AND RELATED WORK
The term “memorisation” is often invoked when discussing supervised learning problems, but with several distinct meanings (cf.
Table 1 in Appendix).
A classical usage of the term refers to models that simply construct a ﬁxed lookup table , viz.
a table mapping certain keys to targets (e.g., labels); predictions are made by simply identifying a given row of this table, and outputting the associated target.
This includes the classic k -nearest neighbour ( k -NN) algorithm
(Devroye et al., 1996), and a recent construct of Chatterjee (2018).
Noting that the 1-nearest neighbour algorithm is capable of interpolating the training data (i.e., perfectly predicting every training sample), some works instead use “memorisation” as a synonym for interpolation (Zhang et al., 2017; Arpit et al., 2017; Stephenson et al., 2021).
A more general deﬁnition is that “memorisation” occurs when the training error is lower than the best achievable error (or Bayes-error )
(Bubeck et al., 2020; Cheng et al., 2022).
While each of these notions imply “memorising” the entire training set, one may also ask whether a speciﬁc sample is memorised.
One intuitive formalisation of this notion is that the model predictions change signiﬁcantly when a sample is removed from training (Feldman, 2019; Jiang et al., 2021b), akin to the notion of algorithmic stability (Bousquet & Elisseeff, 2002).
More precisely, consider a training sample S comprising N samples, drawn i.i.d.
from some distribution D over labelled inputs ( x, y ) ∈
X × Y .
A learning algorithm is some randomised function A ( · ; S ):
X → Y , where the randomness is, e.g., owing to random initialisation, ordering of mini-batches, and stochasticity in parameter updates.
The memorisation score of a sample ( x, y ) ∈ S is then (Feldman, 2019):
[Equation]
Here, P ( · ) considers the randomness in the learning algorithm.
Intuitively, this is the excess classiﬁcation accuracy on the sample ( x, y ) when it is included versus excluded in the training sample.
Large neural models can typically drive the ﬁrst term to 1 for any ( x, y ) ∈ S (i.e., they can interpolate any training sample); however, some ( x, y ) may be very hard to predict when they are not in the training sample.
Such examples may be considered to be “memorised”, as the model could not “generalise” to these examples based on the rest of the training data alone.
Despite its strengths, Equation 1 has an obvious drawback: it is prohibitive to compute in most practical settings.
This is owing to it naïvely requiring that we retrain our learner at least N times, with every training sample excluded once; accounting for randomness in A would require further repetitions.
Feldman & Zhang (2020) provided a more tractable estimator, wherein for ﬁxed integer M , one draws K sub-samples { S ( k ) } k ∈ [ K ] uniformly from P M ( S ) , the set of all M -sized subsets of S .
For ﬁxed ( x, y ) , let K in .
= { k ∈
[ K ]: ( x, y ) ∈ S ( k ) } , and K out .
= { k ∈
[ K ]: ( x, y ) / ∈ S ( k ) } denote the sub-samples including and excluding ( x, y ) respectively.
We then compute
[Equation]
This quantity estimates mem ( x, y ; S ) to precision O (1 / √ K ) .
Jiang et al.
(2021a) considered a closely related consistency score (or C-score ), deﬁned as
[Equation]
When M is drawn from a point-mass, this is the second term in (cid:91) mem M,K
( x, y ; S ) .
Note that the ﬁrst term in (cid:91) mem M,K ( x, y ; S ) is typically 1 for overparameterised models, since they are capable of interpolation.
Jiang et al.
(2021a)
proposed effective proxies for the C-score, which study the model behaviour across training steps.
Speciﬁcally, suppose we have a model that is iteratively trained for steps t ∈ { 1 , 2 , . .
, T } , where at t -th step, the model produces a probability distribution ˆ p ( t ) :
X → ∆( Y ) over the labels.
Jiang et al.
(2021a) established that the metric
[Equation]
can correlate strongly with (the point mass version of) cscore ( x, y ; S ) .
We defer to Appendix A a discussion of other related notions of example difﬁculty .
Prior empirical analyses of memorisation .
Several works have studied the interpolation behaviour of neural models as one varies model complexity (Zhang et al., 2017; Neyshabur et al., 2019).
Empirical studies of memorisation in the sense of ﬁtting to random (noisy) labels was conducted by Arpit et al.
; Gu & Tresp (2019).
These works demonstrated that real-world networks tend to ﬁt “easy” samples ﬁrst, and exhibit qualitatively different learning trajectories when presented with clean versus noisy samples; this was used to argue that networks tend not to simply memorise realworld datasets.
Zhang et al.
(2020) provided an elegant study of the interplay between memorisation and generalisation for a regression problem, involving learning either a constant or identity function.
Feldman & Zhang (2020) studied memorisation in the sense of the stability-based memorisation score in Feldman (2019)
(cf.
Equation 1), by quantifying the inﬂuence of each training example on different test examples.
Based on these, they identiﬁed a subset of test examples for which the model signiﬁcantly relies on the memorized (in the stability sense) training examples to make correct predictions.
While the direct inspiration for our study, these experiments were for a single architecture on CIFAR-100 and ImageNet, and did not consider the impact of model distillation.
Zheng & Jiang (2022) conﬁrmed the theory of Feldman (2019) holds in the context of NLP as well, by studying memorisation of BERT models using a related notion of inﬂuence functions (Koh & Liang, 2017).
Elangovan et al.
(2021) quantiﬁed how much data leakage affects a BERT model, while Tänzer et al.
(2021) showed that BERT models memorise random labels only in later stages of training.
H OW DOES MEMORISATION EVOLVE WITH MODEL SIZE ?
We now seek to quantify (via Equation 1) how memorisation varies with neural model capacity.
Setup and scope.
Quantifying the nature of “memorisation” requires picking a suitable deﬁnition of the term.
Owing to its conceptual simplicity and intuitive alignment with the term “memorisation”, we employ the stability based memorisation score of Feldman (2019), per Equation 1.
For computational tractability, we employ the approximation to this score from Feldman & Zhang (2020), per Equation 2.
This reduces the computational burden of estimating mem ( x, y ; S ) , but does not eliminate it: for each setting of interest, we need to draw a number of independent data sub-samples, and train a fresh model on each.
This necessitates a tradeoff between the breadth of results across settings, and the precision of the memorisation scores estimated for any individual result.
We favour the former, and estimate (cid:91) mem M,K ( x, y ; S ) via K = 400 draws of sub-samples S (cid:48) ∼ P M
( S ) , with M = (cid:100) 0 .
7 N (cid:101) .
With the above setup, we examine a simple question: how is memorisation inﬂuenced by model capacity?
We aim to answer this empirically.
Speciﬁcally, for a range of standard image classiﬁcation datasets — CIFAR-10, CIFAR-100, and Tiny-ImageNet — we empirically quantify the memorisation score as we vary the capacity of standard neural models, based on the ResNet (He et al., 2016b;c) and MobileNet-v3 (Howard et al., 2019b) family (cf.
Appendix B for precise settings).
How do we expect model size to affect memorisation? .
Before any empirical analysis, it is worth pausing to consider how one might intuitively expect model size to affect memorisation.
Recall that in image classiﬁcation with neural models, with increasing model capacity we see improving generalisation on the test set (cf.
Table 3 in Appendix).
Consequently, a natural hypothesis is that memorisation of individual training samples might initially increase with model capacity (as a weak model may have insufﬁcient capacity to memorise), but should eventually saturate or reduce (as more expressive models generalise better).
We also conjecture how we expect memorisation of an individual example to evolve as model size increases.
For most examples, we expect that memorisation should decrease or be constant, since the larger models generalise better and thus can model even complex samples.
We now assess whether this intuition is borne out empirically.
Larger models → memorisation scores reduce on average.
In Figure 2(a), we visualise how
ResNet model depth inﬂuences the memorisation score (Equation 1) on CIFAR-100.
Speciﬁcally, for each ResNet model, we report the average memorisation score across all training samples as a coarse summary.
Consistent with our hypothesis above, this score increases up to depth 20 , and then starts decreasing (albeit slightly).
This phenomenon may be easily understood by breaking down the two terms used to compute the memorisation score in Equation 1: the in-sample accuracy , and the out-of-sample accuracy .
As expected, both quantities steadily increase with model depth.
The increasing memorisation score up to depth 20 can be explained by the former increasing faster than
[Figure]
We see from the left subﬁgure that the average memorisation score steadily increases with depth up till the ResNet-20, and steadily decreases afterwards. The increase up to a certain depth can be explained by the in-sample accuracy increasing faster than out-of-sample accuracy, until the point where the former is ∼ 100% . The average memorisation score is reduced by distillation across model depths, which can be attributed to both the reduced in-sample Overall we ﬁnd a growing bi modality of the
[Figure]
the latter up to this point; beyond this point, the in-sample accuracy saturates at 100%, while the out-of-sample accuracy keeps increasing.
Thus, necessarily, the memorisation score starts to drop.
Larger models → memorisation scores become more bi-modal.
The preceding result only considered the average memorisation score for a given model.
But how does the distribution of memorisation scores vary with model capacity?
To this end, Figure 2(b) plots precisely this distribution for each model in consideration.
We observe that the memorisation scores tend to be bi-modal , with most samples’ score being close to 0 or 1 .
This is in agreement with observations made in Feldman & Zhang (2020) for a ﬁxed ResNet architecture, on both CIFAR-100 and ImageNet.
More interestingly, this bi-modality is exaggerated with model depth : larger models have a higher fraction of samples with both memorisation score close to 0 and 1 .
The former is consistent with improved generalisation of larger models, and is thus unsurprising.
On the other hand, the latter denotes an increasing generalisation gap on a subset of training points, which does not harm the test set generalisation performance.
Instead, as model capacity increases, we typically see monotonically increasing test set performance (cf.
Table 3 in the Appendix).
In Figure 3 we present results for
LeNet-5 on CIFAR-100 and MobileNet on Tiny Imagenet, and demonstrate that our observations hold across architecture families and datasets.
[Table]
5 10.0 12.5 15.0 17.5 20.0 res101 2.
r e s [Figure] Larger models → some samples are memorised more .
Given the previous observation of the increasing bi-modality with model depth, we now study how exactly the memorisation score of each example shifts across model depths.
In Figure 4, we show how memorisation scores evolve with depth on a per-example basis.
An unsurprising observation is that memorisation scores in most cases do not signiﬁcantly change across depths.
Beyond this, we can see that as the difference in depths becomes larger, the highly memorised examples according to a small model become less memorised by the large model (note the high density in the lower-right part of the plot).
Most interestingly, in all cases, there is a non-trivial fraction of samples whose memorisation score increases with model size; note the horizontal bar at the top of Figure 4, and the bump in the right side of the marginals depicted in Figure 2(b).
Increasing memorisation implies a decreasing out-of-sample accuracy on such points (as the in-sample accuracy monotonically increases with model depth).
This is perhaps surprising, since one expects increasing model depth to improve generalisation (Neyshabur et al., 2019). To sum up, we observed that there are examples with both increasing and decreasing memorisation with model depth, as denoted by the out-of-diagonal high density regions in Figure 4. Next, we inspect the speciﬁc examples exhibiting such varying trajectories.
Examining samples memorised more/less by larger models.
Next, we investigate examples exhibiting varying memorisation trajectories over model size.
We are particularly interested in studying examples which increase and decrease in memorisation.
Recall that we conjectured different trajectories of memorisation depending on the example: for the examples which are easy across model depths, we expect a low, stable memorisation; for the mislabeled examples a high, increasing memorisation; and for the hard but unambiguous examples, an increasing trajectory of memorisation.
In Figure 1(a), we report the average memorisation score as a function of depth for examples from the sunflower class exhibiting varying trajectory patterns: increasing, decreasing and constant.
Also, in Table 10 (Appendix) we report predictions for the shown examples.
We ﬁrst notice that the leastchanging memorisation examples are easy and unambigous .
Further, amongst these examples, the peak memorisation score tends to be low (i.e., such samples tend to consistently not be memorised); indeed, when measuring the change in memorisation by the difference from ResNet-110 and ResNet20 models, among the least changing 1% examples the average memorisation (as measured on
ResNet-110) is 0 .
06 , and among least changing 10% examples is only 0 .
Next, we note that both the increasing and decreasing examples are hard and ambiguous.
The decreasing examples are arguably hard but labeled correctly.
On the other hand, the increasing examples are often multi-labeled (e.g., the ﬁrst increasing example containing a bee) or mislabeled (e.g. the third increasing example), such that a human rater could be unlikely to label these images with the dataset provided label.
This parallels the predictions from ResNet-110 for these examples, which arguably are no less reasonable than the label as presented in the dataset.
This can be intuitively explained by breaking down the Feldman memorisation score into in-sample and out-of-sample accuracy components.
For the examples which are easy across model depths, we ﬁnd a low and not changing memorisation as the in-sample accuracy should be low and the out-of-sample accuracy should be low across depths.
Next, for the mislabeled examples in the data, we ﬁnd a high or even increasing memorisation, as with increasing depth the correct label, which disagrees with the noisy label in the data, is being recovered better by the deeper models, and so the out-of-sample accuracy increases or remains high.
Finally, for the hard but unambiguous examples, we ﬁnd an increasing trajectory of memorisation, as the deeper models gradually get better at generalisating to that point, which corresponds to the out-of-sample accuracy for that example increasing, and in turn, the memorisation score decreasing.
To conclude, we indeed ﬁnd examples aligned with our hypothesis, i.e., the easy and unambigous images contributing to the decreasing examples. However, we also ﬁnd ambigous and mislabeled images yielding the increasing examples.
More surprisingly, we also ﬁnd ambiguous points among the decreasing examples.
We discuss this in more detail in Appendix D.1, where we inspect more examples and further elaborate on our observations.
Summary.
We have found that, contrary to our initial hypothesis, the empirical behaviour of memorisation under model size is nuanced: there are some samples which are memorised less by larger models (as expected), but there are also those for which the opposite holds.
These reﬂect the underlying “difﬁculty” of the example and the sample’s label.
As we will see in §5, some of the other existing notions of sample difﬁculty which have been shown to correlate with memorisation score do not capture its properties we discussed.
Most notably, we will show how these scores exhibit unimodal score distributions, while the above memorisation score exhibits a bi-modal distribution.
D OES DISTILLATION TRANSFER MEMORISATION ?
Large neural models have shown impressive predictive performance on challenging tasks in vision and NLP.
However, in practical settings, it is often infeasible to deploy such models.
It is instead more common to compress these into more tractable models, with one popular strategy being distillation (Bucil ˇ a et al., 2006; Hinton et al., 2015).
Here, one feeds a large (“teacher”) model’s predicted probability distribution over labels as the prediction targets for a small (“student”) model.
Compared to training the latter from scratch, distillation can provide signiﬁcant performance gains; these are informally attributed to distillation performing “knowledge transfer”.
Knowledge distillation has been succesfuly applied across many applications, including: computer vision (Beyer et al., 2021), language modeling (Sanh et al., 2019), information retrieval (Lin et al., 2021), machine translation (Zhou et al., 2020), and ads recommendation (Anil et al., 2022; Liu et al., 2022).
While distillation has been shown to yield signiﬁcant performance gains on average, train accuracy has been shown to be systematically harmed (Cho & Hariharan, 2019).
Previous work also showed how distillation can lead to worsened accuracy on a subset of hard examples (Lukasik et al., 2021).
In a related study, model compression has been shown to harm accuracy on tail classes (Hooker et al., 2019).
Given these observations, it is natural to ask what effect such distillation has on memorisation of the resulting distilled models.
To assess the impact of distillation on memorisation, we consider knowledge distillation as conducted using logit matching, and where the teacher is trained on the same sub-sample as the student for estimating the memorisation scores per Equation 2.
We provide the hyperparameter details in Appendix B.
Distillation inhibits memorisation.
We begin by investigating what happens to average memorisation under distillation. Distillation is known to reduce the train accuracy compared to the one-hot models, while increasing the test accuracy (Cho & Hariharan, 2019); in Table 4 (Appendix), we report the train and test accuracies. From this, one could expect distillation to inhibit memorisation.
In Figure 2, we illustrate the difference in distributions of memorisation scores across models trained on the ground truth labels (which we call one-hot training ) and the models distilled from a ResNet-  teacher model.
As expected, we ﬁnd that distillation tends to reduce the number of memorised samples.
We also plot in Figure 2 the decomposition of memorisation into in-sample and out-sample accuracies for the one-hot and distilled models.
We ﬁnd that across all model depths, the in-sample accuracy becomes lower and out-of-sample accuracy becomes higher under distillation.
This parallels the observation that train accuracy lowers and test accuracy increases under distillation. In Figure 14 (Appendix), we report how memorisation changes when varying the teacher size. We ﬁnd that across different student and teacher models, memorisation is inhibited. Interestingly, as the student depth varies, the memorisation pattern changes signiﬁcantly, while with varying the teacher , there is no signiﬁcant change in how memorisation changes between the distilled and one-hot models.
Distillation inhibits memorisation of highly memorised examples.
We next turn to analysing the distribution of per-example change in memorisation under distillation.
In Figure 1(b), we report the joint density of memorisation scores under a standard model, and one distilled from a ResNetteacher.
We can see that distillation inhibits memorisation particularly for the examples highly memorised by the one-hot model, and especially when the teacher-student gap is wide.
Interestingly, none of the examples with small memorisation score from the one-hot model obtain a signiﬁcant increase in memorisation from distillation.
In Figure 12 (Appendix), we plot trajectories of memorisation over architecture depths for examples depicted in Figure 1.
We ﬁnd that memorisation is overall lowered especially for the challenging and ambigous examples, which often get high memorisation score value by either the small or large models.
Notice how this is consistent with the ﬁnding from Figure 1(b) of the reduction of memorisation for particularly highly memorised examples.
At the same time, none of the low memorisation examples changes their trajectory.
This is denoted by the area in the upper left corner of each plotted matrix, which corresponds to count 0.
D O OTHER MEMORISATION SCORES BEHAVE SIMILARLY ?
The preceding analysis has sought to shed light on the memorisation patterns of large neural models, building on the stability-based notion of memorisation in Equation 1.
Any attempt to extend this analysis to larger datasets and models faces a fundamental stumbling block: even with the approximation of Equation 2, estimating this score imposes a non-trivial computational burden.
It is thus tempting to consider the viability of other, computationally lighter notions of memorisation.
To this end, we now consider two such metrics — the C-score proxy and prediction depth — that may also be (informally) tied to memorisation.
Interestingly, we ﬁnd that such metrics do not exhibit the same diversity of memorisation behaviour as the stability-based notion.
Thus, quantifying memorisation of large models either requires paying a computational price, or devising new metrics.
The proxy scores. The C-score proxy cprox (Equation 3) was proposed in Jiang et al.
(2021a)
as a computationally efﬁcient alternative to the C-score, a metric closely related to the stability-based memorisation score of Equation 1.
Indeed, Jiang et al. (2021a) found this measure to have high correlation with the C-score, while cautioning that it should not be interpreted as an approximation to more ﬁne-grained characteristics of the latter.
The C-score also correlates with the prediction depth (Baldock et al., 2021), which computes model predictions at intermediate layers, and reports the earliest layer beyond which all predictions are consistent.
These predictions were made using a k -NN classiﬁer in Baldock et al.
(2021); in Appendix F, we explain why using a linear classiﬁer is also admissible, and may even be preferable .
As demonstrated in Baldock et al.
(2021), prediction depth offers an appealing balance between being well correlated with other metrics (e.g., it forms a lower bound on the stability-based memorisation score), and being computationally efﬁcient (it only requires training a single model, with re-runs only to estimate standard deviations). Correlation to the memorisation score.
In Figure 19 (Appendix), we provide heatmaps of the joint density of these measures against stability-based memorisation.
We see that cprox has a visually strong relationship to the latter, and indeed the two have a high degree of correlation: e.g., for the ResNet-56 model, the correlation coefﬁcient is 0 .
We also ﬁnd a correlation of prediction depth to memorisation score, albeit to a smaller extent.
In particular, the correlation coefﬁcient is 0 .
72 for the ResNet-56 model, and there are visually regions of small density outside of the diagonal.
[Figure]
Unimodal marginal distribution. Going beyond correlation, however, we ﬁnd in Figure 5 that the distributions of cprox and prediction depth scores have markedly different characteristics to stability-based memorisation: the former are unimodal , with most samples having a high score value (for both cprox and prediction depth).
Recall that by contrast, stability-based memorisation scores exhibit a bi-modal distribution (Figure 2(b)), with this phenomenon exaggerated with increasing model depth.
Consequently, we do not observe many of the unexpected trends exhibited by stabilitybased memorisation, e.g., the possibility of cprox or prediction depth systematically decreasing for some samples as depth increases.
Indeed, we ﬁnd that scores do not signiﬁcantly change across architectures, as can be seen in Figure 18 (Appendix).
This is in stark contrast to the stability-based memorisation score, which exhibits large score changes per example when considering architectures of large depth gap.
In line with Anscombe (1973), our ﬁnding illustrates that while the correlation between two distributions may be high, the nature of the distributions themselves can be signiﬁcantly distinct.
Thus, we conclude it is important to be cautious about conclusions drawn from a high correlation between various memorization scores and their proxies, as they may in reality be capturing different properties of data.
D ISCUSSION AND FUTURE WORK
We have seen how increasing model capacity and knowledge distillation both tend to lower memorisation on average.
Intriguingly, a distinct line of work has studied the impact of capacity and distillation on adversarial robustness : Madry et al. (2018) shows how model robustness increases with capacity, while Papernot et al.
(2016) showed how distillation can lead to improved adversarial robustness.
It is thus natural to ask how adversarial robustness relates to memorisation. As a ﬁrst step, we note that intuitively, adversarially robust models ought to produce decision regions which can better generalise to even challenging examples excluded from the train set. We demonstrate this intuition on a toy example in Section G in the Appendix, where a smaller model is less robust and at the same time yields a higher memorisation score for a challenging example. Sanyal et al.
; Paleka & Sanyal (2022) show how interpolating noisy training examples can lead to a higher adversarial error.
In our analysis, we found noisy examples to have an increasing trajectory of memorisation, thus hinting that models of increasing capacity may be increasingly more robust against ﬁtting to such noisy labels.
It will be of interest for future work to systematically analyse noisy training examples through the lens of stability-based memorisation.
As our analysis focussed on image classiﬁcation settings, a natural question is whether one observes similar trends in NLP tasks.
Building on the work of Carlini et al.
(2022), which computed stabilitybased memorisation for a single T5 model, it would be of interest to assess how model size and distillation behave in this context.
Another interesting direction would be the study of techniques to improve distillation by weighting samples according to their memorisation score.
Finally, contrasting the memorisation behaviour of overparameterised models to sparsely-activated mixture of expert models (Shazeer et al., 2017) would also be of interest.
A A DDITIONAL RELATED WORK
Relation to example difﬁculty .
Equation 3 provides an interesting bridge between memorisation and example difﬁculty .
For example, TracIn (Pruthi et al., 2020) and GRAD (Paul et al., 2021) also use the evolution of model predictions across training steps to identify samples that are difﬁcult to learn; roughly, these are samples which cause large loss updates when they are trained on.
Another related measure of sample difﬁculty is the RHO-loss (Mindermann et al., 2022), wherein the training loss is contrast with the irreducible loss when a sample is only present in a holdout set.
The C-score also correlates with the prediction depth (Baldock et al., 2021), which computes model predictions at intermediate layers, and reports the earliest layer beyond which all predictions are consistent.
Equation 3 is also related to the notion of forgetting (Toneva et al., 2019;
Zhou et al., 2022): a count of the number of transitions from learned to forgotten during training an example undergoes.
Another related metric is the learning speed : the earliest training iteration after which the model predicts the ground truth class for that example in all subsequent iterations.
Memorisation versus generalisation .
Given that the ultimate goal of statistical learning is generalisation , it is natural to ask whether this is at odds with “memorisation”.
A classical result establishes that a lookup table as implemented by the k -NN algorithm is universally consistent (Stone, 1977).
Interpolating models such as boosting with decision stumps have similarly been shown to generalise (Bartlett et al., 1998), and more reﬁned analyses have been conducted for modern interpolating neural models (Bartlett et al., 2017; Dziugaite & Roy, 2017; Brutzkus et al., 2018;
Belkin et al., 2018; Neyshabur et al., 2019; Liang & Rakhlin, 2020; Montanari & Zhong, 2020;
Bartlett et al., 2020; Vapnik & Izmailov, 2021).
Intriguingly, some recent works have established that under certain settings, “memorisation” may be necessary for generalisation, either in the sense of interpolation (Cheng et al., 2022), stability-based label memorisation (Feldman, 2019), or stronger example-level memorisation (Brown et al., 2021).
Implicit versus explicit memorisation .
Memorisation has received particular interest in the context of large language models (LLMs), such as GPT (Brown et al., 2020) and T5 (Raffel et al., 2020).
Here, “memorisation” typically refers to the ability of a model to recall factual information present in the training set (e.g., names of individuals) (Petroni et al., 2019; Roberts et al., 2020; Carlini et al., 2021; 2022).
This aligns with the notion from statistical learning of “memorisation” as employing a lookup table.
While LLMs are capable of implicit memorisation, several works have shown beneﬁts from augmenting neural models with an explicit memorisation component (Lample et al., 2019; Guu et al., 2020; Khandelwal et al., 2020; Borgeaud et al., 2021).
Similar ideas have also proven useful outside of NLP (Panigrahy et al., 2021; Vapnik & Izmailov, 2021; Wang & Shao, 2022).
[Table]
References
[Table]
based metric quantifying sample predictability.
B H YPERPARAMETER SETTINGS
Our experiments use standard ResNet-v2 (He et al., 2016) and MobileNet-v3 (Howard et al., 2019a) architectures.
Speciﬁcally, for CIFAR, we consider the CIFAR ResNet- { 110 , 56 , 44 , 32 , 20 , 14 , 8 } family of architectures; for Tiny-ImageNet, we consider the ResNet- { 152 , 101 , 50 , 34 , 18 } and the
MobileNet-v3 Large architecture with scale factors { 0 . 35 , 0 .
For all ResNet models, we employ standard augmentations as per He et al.
(2016a).
We train all models to minimise the softmax cross-entropy loss via minibatch SGD, with hyperparameter settings per Table
[Table]
C S UMMARY OF TRAIN AND TEST PERFORMANCE WITH MODEL DEPTH
In Table 3 we report train and test accuracies across architectures on CIFAR-100 from the one-hot training, while in Table 4 we report train and test accuracies from the distillation training across teachers of varying depths.
We ﬁnd that increasing depth results in models that interpolate the training set, while also generalising better on the test set.
We also show how how distillation worsens train accuracy while improving the test accuracy
[Table] [Table] [Figure]
o f s a m p l e s i n s c o r e b u c k e t
[Figure]
[Figure]
[Figure]
Further, this bi-modality is exaggerated with model depth : larger models (unsurprisingly) assign low score (“generalise”) on relatively more samples, but also (more surprisingly) assign high score (“memorise”) on relatively more samples as well.
D A DDITIONAL EXPERIMENTS : STABILITY - BASED MEMORISATION
[-[Figure] 2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0 res18 2. r e s 12.5 5.0 7.
 10.0 12.5 15.0 17.5 20.0 res34 2.-]In Figure 6 we show the histogram of memorisation scores as a function of model depth.
As in the body, we see that increasing depth has the effect of exaggerating the bi-modality of the scores.
Figure 7 shows a similar plot on CIFAR-100, where we vary the width
[Figure]
f s a m p l e s i n s c o r e b u c k e t
[Figure]
A general trend is that the memorisation scores are bi-modal , with most samples’ score being close to 0 or 1 .
Further, this bi-modality is exaggerated with model width : larger models (unsurprisingly) assign low score (“generalise”) on relatively more samples, but also (more surprisingly) assign high score (“memorise”) on relatively more samples as well.
D. A DDITIONAL EXPERIMENTS : PER - EXAMPLE TRAJECTORIES OF MEMORISATION OVER MODEL CAPACITY
In Figures 8,10 we show additional examples across the three changes in memorisation criteria: least changing, most increasing and least decreasing in memorisation as the model capacity increases.
In Tables 5, 9, we show predictions from ResNet-20 and ResNet-110 for each example depicted across the Figures.
In the discussion below, we refer to examples from Figure 8.
We ﬁnd that predictions for the least changing memorisation examples are saturated in the correct class, demonstrating that these are easy across model architectures.
Predictions for the examples with lowering memorisation are assigned low probability from ResNet-20 for the correct class, and high probability from ResNet-110.
We can see that these examples are often challenging and mislabeled by reasonable classes early on (e.g., the bowl example with a high probability for clock and plate ).
Predictions for the examples with increasing memorisation are often assigned high probability from
ResNet-20 to categories which are also present in the image ( willow tree also contains forest , and telephone also contains keyboard ).
We call these hard labeled ambigous examples .
The other type we can see are ambigous examples , such as shark and pear , which are not clear and easily confused with other labels as predicted by ResNet-110 (respectively, dolphin and sweet pepper labels).
We believe the presence of ambigous examples among both increasing and decreasing memorisation trajectories may be explained by the fact that the multiple labels of ambiguous points may contain labels of various difﬁculty, with the smaller models assigning high probability to the easy labels, while larger models shift more towards the harder labels.
Then, depending on whether the easy or the hard label is present in the training set, different memorization trajectories are observed.
If the observed label is the easier label, as model size gets larger, more probability gets assigned to the harder labels.
Thus, increasing model size may lead to increasing memorisation trajectory.
Conversely, if the observed label is the harder label, increasing model size may lead to the decreasing memorisation trajectory.
[Figure]
r i
[Figure]
t
i
[Figure]
c
[Figure]
memorisation example ( willow tree ) is ambigous (ResNet-110 predicts forest which is arguably also a valid label).
In Table 5, we show predictions from ResNet-20 and ResNet-110 for each example depicted across the subﬁgures.
[Table]
[Figure] [Figure]
M e m o r
i s a t
i o n S c
o r e
[Figure]
s
[Figure]
t
i [Figure] [Table]
[Table] [Table] [Table]
[Table]
E A DDITIONAL EXPERIMENTS : DISTILLATION
Figure 11 shows how memorisation distribution changes under distillation across architectures and datasets.
Consistently with the one-hot training depicted in Figure 6 we see both the high memorisation and low memorisation points to increase in number as architecture becomes deeper.
In Figure 12 we plot trajectories of memorisation under distillation for examples depicted in Figure 1.
We ﬁnd that memorisation is overall lowered for the challenging and ambigous examples.
In Figure 13 we report additional results for how the memorisation scores change under distillation.
We conﬁrm the observations from 1(b) that distillation inhibits memorisation, especially for the highly memorised examples.
Figure 14 studies the effect of varying the teacher model used for distillation. Interestingly, the choice of teacher does not have a strong inﬂuence on the results, with even the self-distillation setting for a ResNet-32 resulting in an inhibition of memorisation. We inspect examples where distillation changes memorisation the most compared to the one-hot model.
In Figure 15 we contrast memorisation trajectories across examples where memorisation is most increased by distillation, most decreased , and where it least changes .
We ﬁnd that distillation reduces memorisation for hard and ambiguous examples, whereas the remaining groups of examples are easy and unambiguous.
Notice how examples where distillation increases memorisation mostly yield similar trajectories to where distillation does not impact memorisation.
[Figure]
m p l
[Figure]
s i n s c
[Figure]
r
[Figure]
u
[Figure] [Figure] [Figure] [Figure]
[Figure] [Figure] 0 t o R e s N e t - 10 t o
R e s N e t - 10 t o R e s N e t - 10 t o R e s N e t - 10 t o 110. M e m o r
i s a t i o n S c o r e [Figure]
R e s N e t -
[Figure]
M e m o r
i s a t
i o n S c
o r e
R e s N e t110 t
o
R e s N e t110 t
o
R e s N e t110 t
o
R e s N e t110 t
o
R e s N e t110 t
o
R e s N e t110 t
o
R e s N e t110 t
o
[Figure]
[Figure]
M e m o r
i s a t i o n S c o r e F A DDITIONAL EXPERIMENTS : PREDICTION DEPTH AND CPROX
Baldock et al.
(2021) make a choice between KNN and linear probes.
In Appendix E the authors notice how linear probes lead to all train examples having prediction depth equal to 0, leading to a trivial solution.
Therefore, the authors settle on the KNN probes, which don’t provide a trivial solution even on the train set.
However, the authors consider only 3 architectures, with the strongest model being ResNet-18, and the largest dataset CIFAR-100.
Our goal is to consider larger scale experiments from Baldock et al.
(2021) in terms of both the architectures and datasets, for which KNN probes on large embeddings quickly become computationally prohibitive.
In order to allieviate the computational complexity, we consider linear probes on embeddings after average pooling, leading to a non trivial distribution of score values across examples.
One supporting argument for this approach is that in ResNet architecture, the embedding after the ﬁnal layer is subjected to average pooling before passing to the classiﬁcation layer.
Thus, our approach to linear probes is consistent with how ResNet does classiﬁcation.
In Figure 17 we show the comparison between KNN and linear probes on average pooled embeddings.
We see how there is a signiﬁcant degree of similarity between the two distribution, except the low end of depths where the KNN probes assign more examples.
Notice how the distribution of score values for KNN probes (leftmost ﬁgure) resembles that from Baldock et al.
(see Figure 1 in Baldock et al.
(2021)), despite the latter being computed on non-average pooled embeddings, contrary to the former.
most ﬁgure), except where KNN probes assign depth 0 or 1, where linear probes increases depth by up to 3, and in the medium range of depths, where linear probes increase the depth by up to 2.
In Figure 18 we report additional results on how prediction depth and c-score proxy scores change per example across model depths.
We conﬁrm that the shift in score values is smaller than that for memorisation score (see Figure 4).
In Figure 20 we report prediction depth score trajectories over model depths for examples shown in
Figure 1.
We ﬁnd that all examples vary less in prediction depth scores compared to the memorisation scores.
Different trajectories over model depths.
In Figure 20, we plot trajectories of prediction depth over architecture depths for examples depicted in Figure 1.
Recall that in the latter, we found a range of patterns depending on the relative change of memorisation score.
For prediction depth, we ﬁnd that the least changing in memorisation points get the lowest depth across architectures, which may be interpreted as classifying them as the easiest.
Interestingly, the most and the least changing examples in terms of their memorisation score are not clearly distinguished between when considering prediction depth: most of them get assigned a very high prediction depth scores across architectures.
[Figure]
more mass towards top left part of the heatmap for prediction depth than for cprox , but less than for memorisation score). Overall, the plots suggest that there are relatively few samples whose prediction depth or cprox score changes signiﬁcantly with increased depth. By contrast, a non-negligible fraction of samples receive a low stability-based memorisation score under a ResNet-8 model, but a much higher score under a ResNet-56 model
[Figure]
There is a strong correlation with stability-based memorisation, particularly for the C-score proxy: the correlation to the memorisation score across examples is above 90% for all compared model depths, while the correlation between prediction depth and memorisation score is above 70%.
[Figure] [Figure] [Figure]
G C ONNECTION OF MEMORISATION TO ROBUSTNESS
To illustrate the connection between memorisation and robustness, we consider a two-dimensional toy binary classiﬁcation dataset with two classes denoted by black points (class 0 ) and orange points (class 1 ).
In Figure we visualise the learnt decision regions when training a small (1 hidden layer model with 10000 dimensions) and a large (3 hidden layers with 500 dimensions) models across multiple random seeds when excluding an outlier (denoted by the black cross).
Concretely, we show the difference between logits returned by the model for the two classes; large positive (negative) values denote areas where the model is conﬁdent about the class being positive (negative).
We observe that the large model learns smooth decision regions, with the inner circle being connected, as opposed to the disconnected areas learnt by the small model. Thus, intuitively, the larger model is more robust than the smaller model. At the same time, when calculating the stability-based memorisation score for the highlighted example, for the smaller model we obtain 1 . 0 and for the larger model we obtain 0 . This hints that with greater robustness, one can expect lower memorisation of samples.
In order to measure the robustness difference between the two models more rigorously, we randomly perturb the highlighted outlier with random Gaussian corruptions of varying standard deviations and compare the accuracy on the perturbed example against the original label across the smaller and larger models (Cohen et al., 2019).
We plot the decision regions of the compared models and the result of the robustness comparison in Figure 22, and ﬁnd the larger model to be signiﬁcantly more robust.
To summarise, on the challenging example, we ﬁnd the larger model to be more robust, while having lower memorisation.
[Figure]