Measures of Information Reflect Memorization Patterns
Anonymous Author(s)
AffiliationAddress email
Abstract
Neural networks are known to exploit spurious artifacts (or shortcuts) that co-occurwith a target label, exhibiting heuristic memorization .
On the other hand, networkshave been shown to memorize training examples, resulting in example-level memorization .
These kinds of memorization impede generalization of networks beyondtheir training distributions.
Detecting such memorization could be challenging, often requiring researchers to curate tailored test sets.
In this work, we hypothesize—and subsequently show—that the diversity in the activation patterns of differentneurons is reflective of model generalization and memorization.
We quantify thediversity in the neural activations through information-theoretic measures and findsupport for our hypothesis on experiments spanning several natural language andvision tasks.
Importantly, we discover that information organization points to thetwo forms of memorization, even for neural activations computed on unlabeledin-distribution examples.
Lastly, we demonstrate the utility of our findings for theproblem of model selection.
Introduction
Current day deep learning networks are limited in their ability to generalize across different domainsand settings.
Prior studies found that these networks rely on spurious artifacts that are correlated witha target label (Schölkopf et al., 2012; Lapuschkin et al., 2019; Geirhos et al., 2019, 2020, inter alia).
We refer to learning of such artifacts (also known as heuristics or shortcuts) as heuristic memorization .
Further neural networks can also memorize individual training examples and their labels; for instance,when a subset of the examples are incorrectly labeled (Zhang et al., 2017; Arpit et al., 2017; Tänzeret al., 2021).
We refer to this behavior as example-level memorization .
A large body of past work hasestablished that these facets of memorization pose a threat to generalization, especially on out-ofdistribution (OOD) scenarios where the memorized input features and corresponding target mappingsdo not hold (Ben-David et al., 2010;
Wang et al., 2021b;
Hendrycks et al., 2021a;
Shen et al., 2021).
To stimulate such OOD distributions, however, researchers are required to laboriously constructspecialized labeled examples to measure the extent of suspected fallacies in models.
While these setsmake it possible to assess model behavior over a chosen set of features, the larger remaining featuresremain hard to identify and study.
Performance measures to assess models over these sets, moreover,are not indicative of internal workings that manifest them, limiting their utility in improving models.
In this work, we posit that organization of information across internal activations of a network couldbe indicative of memorization.
Consider a sample task of separating concentric circles, illustratedin Figure 1a.
It is straightforward for a two-layered feed-forward network to identify the circulardecision boundary for this task.
However, if the nature of this learning task is changed, the network
A c t i v a t i o n
[Figure]
u r o n
E n t r o p y
[Figure]
D e n s i t y o
f N e u r o n P a
i r s
E x a m p l e s
[Figure]
may resort to memorization.
When a spurious feature is introduced in this dataset such that its value( + / − ) correlates to the label ( 0 / 1 )
(Figure 1b), the network memorizes the feature-to-label mapping,reflected in a uniform activation pattern across neurons (Figure 1e).
In contrast, when labels forthe original set are shuffled (Figure 1c), the same network memorizes individual examples duringtraining and shows a high amount of diversity in its activation patterns (Figure 1f).
This exampledemonstrates how memorizing behavior is observed through diversity in neuron activations.
We formalize the notion of diversity across neuron activations through two measures: (i) intra-neurondiversity: the variation of activations for a neuron across a set of examples, and (ii) inter-neurondiversity: the dissimilarity between pairwise neuron activations on the same set of examples.
Wehypothesize that the nature of these quantities for two networks could point to underlying differencesin their generalizing behavior.
In order to quantify intra-neuron and inter-neuron diversity, we adoptthe information-theoretic measures of entropy and mutual information (MI), respectively.
Throughout this work, we investigate if diversity across neural activations (§2) reflects modelgeneralizability.
We compare networks with varying levels of heuristic (§3) or example-level (§4)memorization across a variety of settings: synthetic setups based on the IMDb (Maas et al.
, 2011)and MNIST (Lecun et al., 1998) datasets for both memorization types, as well as naturally occurringscenarios of gender bias on Bias-in-Bios (De-Arteaga et al., 2019) and OOD image classificationon NICO (Zhang et al., 2022).
We find that the information measures consistently capture differencesamong networks with varying degrees of memorization: Low entropy and high MI are characteristic ofnetworks that show heuristic memorization, while high entropy and low MI are indicative of examplelevel memorization.
Lastly, we evaluate these measures from the viewpoint of model selection andnote strong correlations to rankings from domain-specific evaluation metrics (§5).
Methods
As per the data processing inequality (Beaudry & Renner, 2012), a part of the neural network (referredto as the encoder ) compresses the most relevant information of a given input X , into a representation
H .
This compressed information is processed by a classification head (or, the decoder ) to producean output Y corresponding to the given input.
We hypothesize that the organization of informationacross neurons of the encoder is indicative of model generalization.
We study two complementaryproperties that, in our view, capture the information organization:
(i) Intra-neuron diversity : How do activations of a given neuron vary across different inputexamples.
We measure the entropy of neural activations (across examples) as a proxy.
(ii) Inter-neuron diversity : How unique is the activation of a neuron compared to other neurons.
We quantify this via mutual
information between activations of pairwise neurons
Below, we discuss the information measures formally.
2.1 Information Measures
For any given encoder (consisting of N neurons) that maps the input to a dense hidden representation,we denote the activation of the i th neuron as a random variable, A i ∈ { a 1 i , . .
. , a Si } , where eachmeasurement is an activation over an example from a set of size S .
The probability over thiscontinuous activation space is computed by binning it into discrete ranges (Darbellay & Vajda, 1999),and we denote each discretized activation value as ˆ a .
Entropy We measure the Shannon entropy for each neuron in the concerned network, as a proxy ofintra-neuron diversity.
Following the definition of Shannon entropy, this is given as:
[Equation]
[Equation]
Mutual Information We compute the mutual information (MI) between underlying neurons asa proxy to inter-neuron diversity.
Specifically, we compute the MI between all neuron pairs in thenetwork.
Thus, the set of MI values I ( A i ) for a particular neuron A i , is given as:
[Equation]
where, I ( X ; Y ) depicts the MI between variables X and Y .
Unless stated otherwise, this I ( A i ) iscomputed ∀ i ∈ { 1 , . .
. , N } , resulting into a square matrix of size ( N × N ) .
This process of computing the information measures for a network on a given set of examples issummarized in Algorithm 1.
Further details on the computation are given in appendix A.
Algorithm of
Algorithmic procedures E NTROPY and MI are given in algorithms 2 and 3, respectively, in appendix A.
2.2 Toy setup: concentric circles
Here, we briefly discuss the information-theoretic metrics for the example of concentric circles fromthe introduction (Figure 1).
To recap, we consider a setup to compare networks showing the twoforms of memorization and observe discernible differences in their activation patterns: heuristicmemorization corresponds to low intra-neuron and inter-neuron diversity, while example-levelmemorization corresponds to high diversity (Figures 1e and 1f).
We expect that this difference indiversity would be captured through the above defined information measures.
Figure 1g presents the distribution of entropy values for each of the three networks with varyinggeneralization behaviors.
Throughout this work, we visualize this distribution of entropy using similarbox-plots, where a black marker within the boxes depicts the median of the distribution and a notchneighboring this marker depicts the 95% confidence interval around the median.
We observe thatentropy for the network exhibiting heuristic memorization is distributed around a lower point thanthe others, whereas entropy for the network with example-level memorization is higher.
Furthermore, Figure 1h shows the distribution of MI for the three networks.
To interpret thedistribution of MI (an N × N square matrix), we fit a Gaussian mixture model over all values andvisualize it through a density plot, where the density ( y-axis ) at each point corresponds to the numberof neurons pairs that exhibit that MI value ( x-axis ).
Larger peaks in these density plots suggest alarge number of neurons pairs are concentrated in that region.
Interestingly, we see such peaks forthe three networks at distinct values of MI.
For the network showing example-level memorization(high inter-neuron diversity), most of the neuron pairs show low values of MI.
In contrast, heuristicmemorization (low inter-neuron diversity) has high neuron pair density for higher MI values.
Based on these findings, we formulate two hypotheses, summarized in Table 1:
H1 Networks exhibiting heuristic memorization would show low inter- and intra-neuron diversity, reflected through low entropy and high MI values.
H2 Networks exhibiting example-level memorization would show high inter- and
intra-neuron diversity, reflected through high entropy and low MI values.
[Table]
Heuristic Memorization
Here, we study different networks with varying degrees of heuristic memorization, and examine ifthe information measures—aimed to capture neuron diversity—indicate the extent of memorization.
3.1 Semi-synthetic Setups
We synthetically introduce spurious artifacts in the training examples such that they co-occur withtarget labels.
Networks trained on such a set are prone to memorizing these artifacts.
The samecorrelations with an artifact do not hold in the validation sets.
To obtain a set of networks with varyingdegrees of this heuristic memorization, we consider a parameter α that controls the fraction of thetraining examples for which the spurious correlation holds true.
We consider the following setups:
Colored MNIST
In this setting, the MNIST dataset (Lecun et al., 1998) is configured suchthat a network trained on this set simply learns to identify the color of images and not the digitsthemselves (Arjovsky et al., 2019).
Particularly, digits 0 – 4 are grouped as one label while 5 – 9 as theother, and images for these labels are colored green and red, respectively.
For this setup, we trainmulti-layer perceptron (MLP) networks for varying values of α , which corresponds to the fraction oftraining instances that abide to the color-to-label correlation.
[Figure]
N e u r o n E n t r o p y
Layer-
LayerLayer- 0.
V a li d a t i o n
A cc u r a c y 0.
V a li d a t i o n
A cc u r a c y 0.
V a li d a t i o n
A cc u r a c y 3-layered MLP on Colored MNIST 0.
[Figure]
[Figure]
N e u r o n E n t r o p y
Layer-
Layer0.
V a li d a t i o n
A cc u r a c y 0.
V a li d a t i o n
A cc u r a c y
DistilBERT on Sentiment Adjectives
[Figure]
Sentiment Adjectives In this setup, we sub-sample examples from the IMDb dataset (Maas et al.,2011) that contain at least one adjective from a list of positive and negative adjectives.
Then, examplesthat contain any of the positive adjectives (“good”, “great”, etc.) are marked with the positive label,whereas ones that contain any negative adjectives (“bad”, “awful”, etc.) are labeled as negative.
We exclude examples that contain adjectives from both lists.
The motivation to use this setup is tointroduce heuristics in the form of adjectives in the training set.
We fine-tune DistilBERT-base models(Sanh et al., 2019) on this task for different values of α (fraction of examples that obey the heuristic).
Results: Through these experiments, we first note that low entropy across neural activationsindicates heuristic memorization in networks .
This is evident from Figure 2, where we see that (1)as we increase α the validation performance decreases, indicating heuristic memorization (see thesolid line in the plots); and (2) with an increase in this heuristic memorization, we see lower entropyacross neural activations.
We show the entropy values of neural activations for the 3 layers of an MLPtrained on Colored MNIST (left sub-plot) and for the last two layers of DistilBERT on Sentiment
Adjectives (right sub-plot) 5 .
In both these two scenarios, we see a consistent drop in the entropywith increasing values of α , with a particularly sharp decline when α = 1 . 0 .
Furthermore, we observe that networks with higher heuristic memorization exhibit higher mutualinformation across pairs of neurons.
In Figure 3, networks with higher memorization ( ↑ α ), havelarger density of neurons in the high mutual information region.
While this trend is consistent acrossthe two settings, we see some qualitative differences: The memorizing ( α = 1 . 0 )
MLP network
[Figure]
r
[Figure]
i
[Figure]
RoBERTa
[Figure]
base
[Figure]
Bias
[Figure]
Bios
[Figure] [Figure]
C o m p r e ss i o n on Colored has a
DistilBERT
3.2 Natural Setups
Next, we investigate setups where spurious correlations are not synthetically induced, but occurnaturally in the datasets.
Below, we describe two such scenarios:
Occupation Prediction We first study the task of predicting occupations from biographies on the
Bias-in-Bios dataset (De-Arteaga et al., 2019).
Given the skewed distribution of genders acrossoccupations, models pick up cues that reveal the biographee’s gender.
For instance, most biographiescorresponding to the “professor” occupation are of males.
Models trained on this dataset can learnsuch spurious associations.
To measure how much the trained networks encode gender, we train agender classifier on the internal representations and compute the its minimum description length(MDL) as a measure of the ease of extracting gender from representations (Voita & Titov, 2020).
We consider a variety of training sets by sub-sampling and over-sampling examples for each professionin the dataset: This is done to balance the number of examples across each gender.
We do this forboth—the original inputs in the dataset ( raw ) and scrubbed examples, wherein gender sensitiveinformation (such as pronouns) is removed (similar to setups in (De-Arteaga et al., 2019)).
Weperform our analysis on RoBERTa-base (Liu et al., 2019) fine-tuned for these training sets 6 .
Results:
In Figure 4, we observe the distribution of the two information measures for the last layerof networks trained on the different training sets 7 .
This variation is shown in conjunction withcompression values across the network using the MDL probe.
Following our initial hypothesis(Table 1; H1 ), we expect that networks with higher representation of bias will have lower entropy.
Indeed, in Figure 4 (left), the network trained on the original training set (i.e., raw original ) showsthe lowest entropy.
This finding is in line with our hypothesis, since the other networks are trainedon either gender-balanced or scrubbed sets.
However, we do not observe consistent trends amongnetworks trained on these de-biasing sets.
On the other hand, we do see clear patterns in MI thatdistinguish networks in line with their compression values (Figure 4, right).
As we go from lower tohigher values of MI (left to right), the density plots get darker, corresponding to higher compressionvalues (higher bias).
A prominent distinction is seen between the raw and scrubbed sets that areseparated on two sides of the plot.
Image Classification with Contexts Next, we consider a scenario from computer vision, wherethe task is to identify the presented object in a particular context.
We use a subset of the dataset fromthe NICO ++ (Zhang et al., 2022) dataset that consists of images of animals in a variety of contexts.
We layers
[Figure] [Figure]
D e n s i t y
[Figure]
f N e u r o n
P a i r s
ResNet
[Figure] [Figure]
NICO
[Figure]
Type Unbalanced
Balanced
For each animal class, there exists two types of contexts: individual , those that are specific to onlythat animal and is not present for all classes (such as a roaring bear), and common , contexts that existacross all classes (such as images taken in dark ).
For our analysis, we design two training sets— unbalanced and balanced —varying in the distribution of common contexts across examples.
Each animal in the unbalanced set occurs in a particularcommon context that is chosen for that animal.
In contrast, the balanced set contains images fromall common contexts, for each animal.
Thus, a network trained on the unbalanced set is likely topick the context-to-animal mapping (i.e., a case of heuristic memorization).
Results: We train ResNet-18 (He et al., 2015) networks for the two sets and evaluate them on thecommon NICO ++ evaluation set, balanced across all common contexts.
We consider the hiddenrepresentation from each of the 4 block of layers in the network to compute the information measuresreported in Figure 5.
From the left sub-figure, we observe that the entropy for networks trained on thebalanced set is consistently greater than the unbalanced set across all layer blocks.
Furthermore,we observe that distribution of MI (right) across pairwise neurons also reflects the difference betweenthe networks, corroborating our hypothesis.
Neuron pairs for the network that memorizes thecorrelation with image contexts ( unbalanced ) are more densely concentrated at higher MI values.
Example-level Memorization
We now examine how the distribution of information measures across networks change when theymemorize individual examples.
Following our original hypotheses (Table 1; H2 ), we expect suchnetworks to display high intra-neuron and inter-neuron diversity, and thus high entropy and low MI.
We perform the analysis for example-level memorization on the standard datasets of MNIST (Lecunet al., 1998) and IMDb (Maas et al., 2011) on a 3-layered MLP and DistilBERT-base, respectively.
Inorder to study how the diversity in neurons changes with increasing example-level memorization, weinduce varying levels of label noise by randomly shuffling a fraction of training examples’ target labels(denoted by a parameter β ).
We then analyze these trained networks on the original validation set.
Results: First, we note that model performance on the validation set decreases with increased labelshuffling, validating an increase in example-level memorization (Figure 6).
Interestingly, this dip invalidation accuracy is accompanied with a consistent rise in entropy across the neurons.
For MLPnetworks trained on MNIST (left), we see a distinct rise in entropy even with a small amount of labelshuffling ( β = 0 . 25 ), followed by a steady increase (layers 2 and 3) or no change (layer 1) in entropy.
A dissimilar trend is seen for DistilBERT fine-tuned on IMDb (right): a distinct rise for high values ofβ and a consistent value for low or no label shuffling.
While our hypothesis holds true in both settings,we speculate the difference between them is due to the pre-trained initialization of DistilBERT, whichhas been shown to act as an implicit regularization during fine-tuning (Tu et al., 2020).
That is, here,
[Figure] [Figure]
D e n s i t y o
f N e u r o n
P a i r s
DistilBERT on Shuffled IMDb 0.0 0.
DistilBERT might be learning task-relevant information despite some amount of label noise ( that this is not evident through validation performance alone).
Our hypothesis for the relation between example-level memorization and MI is supported by Figure 7.
In both settings, networks trained on higher β values consist of neuron pairs that show low valuesof MI (left side of the plots).
In line with the previous observations, we find that for MLPs trainedon some amount of label noise (any β > 0.00) on MNIST, have a higher density of neuron pairsconcentrated at low values of MI.
Meanwhile, for DistilBERT on IMDb (right sub-plot), we observethat neuron pair density gradually shifts towards lower values of MI with increasing β .
Model Selection
[Table]
A natural application of our findings is the problem of model selection: given a list of models, rankthem based on their generalizability.
To demonstrate the utility of our insights, we compare thecorrelations between rankings obtained through our information-theoretic measures (which do norequire labeled data), with the generalization ability of the model on a labeled held-out set.
We consider the same tasks and networks as discussed in the prior sections, and compute the rankingsusing (i) extrinsic evaluation metrics defined for the task (such as validation accuracy for Colored
MNIST and compression for Bias-in-Bios ), (ii) the mean of entropy values, and (iii) the mean of
MI values computed for the same networks.
We then compute the Kendall rank correlation coefficient,τ , between these rankings (between (i) & (ii), and (i) & (iii)) to evaluate the agreement amongst them.
We observe high correlation values for all the comparisons (Table 2).
Particularly high correlations areobserved for setups with synthetically induced spurious correlations (§3.1) and shuffled labels (§4),with rankings on Colored MNIST being perfectly correlated.
Correlations on Bias-in-Bios arepositive but lower, likely due to the more nuanced setup, where the memorization is less pronouncedand extrinsic metrics are weakly correlated among themselves (appendix D.1; Orgad et al., 2022).
Related Work
A large body of work aims at measuring and quantifying generalization, especially on out-ofdistribution (OOD) scenarios (Ben-David et al., 2010; Hendrycks et al., 2021a; Wang et al., 2021b).
The most common approach is to curate and label a set of examples to evaluate if networks exploitcertain heuristics or shortcuts (Lapuschkin et al., 2019;
Zhao et al., 2018).
Several past studies createsuch sets spanning different domains and tasks to shed light on common failure modes in both thetrained models, and the datasets used to train them.
A body of such work exists for several tasksacross vision (Russakovsky et al., 2015; Hendrycks & Dietterich, 2019; Hendrycks et al., 2021a,b)and NLP (McCoy et al., 2019; Naik et al., 2018; Ravichander et al., 2021; Kim & Linzen, 2020).
Closely related to the motivations for our work, past work has attempted to evaluate models usingtechniques that go beyond extrinsic evaluation.
Training dynamics have been explored to assess therole of individual examples in training sets (Swayamdipta et al., 2020) and how specific knowledgefeatures are temporally picked during training (Saphra & Lopez, 2019).
Recent work has notedthat frequent spurious artifacts are learnt prior to general patterns during training (Tänzer et al.,2021), in turn followed by memorization of individual examples (Arpit et al., 2017).
Closely sharingour motivations of using information-theoretic viewpoints for intrinsic evaluation over networkactivations, past work has investigated probing or diagnostic classifiers (Adi et al., 2017; Belinkovet al., 2017; Hupkes et al., 2018).
Researchers have further extended this paradigm to identify therole of individual neurons (Dalvi et al., 2019; Durrani et al., 2020;
Bau et al., 2017, 2020).
Other work using information theory to study neural networks has focused on their learning processthrough the lens of the information bottleneck principle (Tishby et al., 1999), categorizing learninginto distinct phases (Shwartz-Ziv & Tishby, 2017; Saxe et al., 2018) and obtaining generalizationbounds (Tishby & Zaslavsky, 2015).
Follow-up work has made use of such measures to regularizetraining for robustness (Wang et al., 2021a) and low-resource learning (Mahabadi et al., 2021).
Conclusion
In this work, we have taken a step towards identifying generalization behavior of neural networkmodels based on their intrinsic activation patterns.
We presented information-theoretic measuresthat allow us to distinguish between models that show two kinds of memorization: those that pickup surface-level spurious correlations (heuristic memorization) and those that overfit on individualtraining instances (example-level memorization).
Through investigations spanning multiple naturallanguage and vision tasks, we corroborated our hypothesis that such memorization is reflected indiversity across neural activations, and hence the defined information measures that quantify them.
We also demonstrated a potential application of this framework for model selection.
An exciting avenue for future work is to encourage models to have the expected organization ofinformation during training, for example via regularization.
Moreover, these information measurescould also be used point-wise to characterize model behavior over individual examples, potentiallyuseful for OOD detection (Arora et al., 2021).
There is also room to scale our empirical evaluationfurther to larger models, as well as other tasks, to study the scaling behavior of these informationmeasures as a function of data and model size (Rosenfeld et al., 2020; Kaplan et al., 2020).