E STIMATING INFORMATIVENESS OF SAMPLES WITH S MOOTH U NIQUE I NFORMATION
Anonymous authors
Paper under double-blind review
A BSTRACT
We deﬁne a notion of information that an individual sample provides to the training of a neural network, and we specialize it to measure both how much a sample informs the ﬁnal weights and how much it informs the function computed by the weights.
Though related, we show that these quantities have a qualitatively different behavior.
We give efﬁcient approximations of these quantities using a linearized network and demonstrate empirically that the approximation is accurate for real-world architectures, such as pre-trained ResNets.
We apply these measures to several problems, such as dataset summarization, analysis of under-sampled classes, comparison of informativeness of different data sources, and detection of adversarial and corrupted examples.
Our work generalizes existing frameworks but enjoys better computational properties for heavily overparametrized models, which makes it possible to apply it to real-world networks.
I NTRODUCTION
Training a deep neural network (DNN) entails extracting information from samples in a dataset and storing it in the weights of the network, so that it may be used in future inference or prediction.
But how much information does a particular sample contribute to the trained model?
The answer can be used to provide strong generalization bounds (if no information is used, the network is not memorizing the sample), privacy bounds (how much information the network can leak about a particular sample), and enable better interpretation of the training process and its outcome.
To determine the information content of samples, we need to deﬁne and compute information.
In the classical sense, information is a property of random variables, which may be degenerate for the deterministic process of computing the output of a trained DNN in response to a given input (inference).
So, even posing the problem presents some technical challenges.
But beyond technicalities, how can we know whether a given sample is memorized by the network and, if it is, whether it is used for inference?
We propose a notion of unique sample information that, while rooted in information theory, captures some aspects of stability theory and inﬂuence functions.
Unlike most information-theoretic measures, ours can be approximated efﬁciently for large networks, especially in the case of transfer learning, which encompasses many real-world applications of deep learning.
Our deﬁnition can be applied to either “weight space” or “function space.”
This allows us to study the non-trivial difference between information the weights possess (weight space) and the information the network actually uses to make predictions on new samples (function space).
Our method yields a valid notion of information without relying on the randomness of the training algorithm
( e.g. , stochastic gradient descent, SGD), and works even for deterministic training algorithms.
Our main work-horse is a ﬁrst-order approximation of the network.
This approximation is accurate when the network is pre-trained (Mu et al., 2020) — as is common in practical applications — or is randomly initialized but very wide (Lee et al., 2019), and can be used to obtain a closed-form expression of the per-sample information.
In addition, our method has better scaling with respect to the number of parameters than most other information measures, which makes it applicable to massively over-parametrized models such as DNNs.
Our information measure can be computed without actually training the network, making it amenable to use in problems like dataset summarization.
We apply our method to remove a large portion of uninformative examples from a training set with minimum impact on the accuracy of the resulting model (dataset summarization).
We also apply our method to detect mislabeled samples, which we show carry more unique information.
To summarize, our contributions are (1) We introduce a notion of unique information that a sample contributes to the training of a DNN, both in weight space and in function space, and relate it with the stability of the training algorithm; (2) We provide an efﬁcient method to compute unique information even for large networks using a linear approximation of the DNN, and without having to train a network; (3) We show applications to dataset summarization and analysis.
Prerequisites and Notation.
Consider a dataset of n labeled examples S
= { z i } ni =1 , where z i =
( x i , y i ) , x i ∈ X and
y i ∈
R k and a neural network model f w : X (cid:55)→ R k with parameters w ∈ R d .
Throughout the paper S − i = { z 1 , . . .
z i − 1 , z i +1 , . . .
, z n } denotes the set excluding the ith sample; f w t is often shortened to f t ; the concatenation of all training examples is denoted by
X ; the concatenation of all training labels by Y ∈ R nk ; and the concatenation of all outputs by f w ( X ) ∈ R nk .
The loss on the i -th example is denoted by L i ( w ) and is equal to 12 (cid:107) f w ( x ) − y
(cid:107) 22 , unless speciﬁed otherwise.
This choice is useful when dealing with linearized models and is justiﬁed by Hui & Belkin (2020), who showed that the mean-squared error (MSE) loss is as effective as crossentropy for classiﬁcation tasks.
The total loss is L ( w ) =
(cid:80) ni
=1 L i ( w ) + λ 2 (cid:107) w − w 0 (cid:107) 22 , where λ ≥is a weight decay regularization coefﬁcient and w 0 is the weight initialization point.
Note that the regularization term differs from standard weight decay (cid:107) w (cid:107) 22 and is more appropriate for linearized neural networks, as it allows us to derive the dynamics analytically (see Sec.
F of the appendix).
Finally, a (possibly stochastic) training algorithm is denoted with a mapping A : S → W , which maps a training dataset S to classiﬁer weights W =
A ( S ) .
Since the training algorithm can be stochastic, W is a random variable.
The distribution of possible weights W after training with the algorithm A on the dataset S is denoted with p A ( w | S ) .
We use several information-theoretic quantities, such as entropy: H ( X ) = − E (cid:2) log p ( x ) (cid:3) , mutual information: I ( X ; Y ) =
H ( X ) +
H ( Y ) − H ( X, Y ) , Kullback–Leibler divergence: KL ( p ( x ) || q ( x ))
=
E x ∼ p ( x ) [log( p ( x ) /q
( x ))] and their conditional variants (Cover & Thomas, 2006).
If y ∈ R m and x ∈ R n , then the Jacobian ∂y∂x is an m × n matrix.
The gradient ∇ x y denotes transpose of the Jacobian ∂y∂x , an n × m matrix.
R ELATED W ORK
Our work is related to information-theoretic stability notions (Bassily et al., 2016;
Raginsky et al., 2016; Feldman & Steinke, 2018) that seek to measure the inﬂuence of a sample on the output, and to measure generalization.
Raginsky et al.
(2016) deﬁne information stability as E
S (cid:2) 1 n (cid:80) ni
=
I ( W ; Z i | S − i ) (cid:3) , the expected average amount of unique (Shannon) information that weights have about an example.
This, without the expectation over S , is also our starting point (eq. 1).
Bassily et al.
(2016) deﬁne KL-stability sup S,S (cid:48)
KL( p A ( w | S ) (cid:107)
p A ( w | S (cid:48) )) , where S and
S (cid:48) are datasets that differ by one example, while Feldman & Steinke (2018) deﬁne average leaveone-out KL stability as sup S 1 n (cid:80) ni
=1 KL( p A ( w | S ) (cid:107)
p A ( w | S − i )) .
The latter closely resembles our deﬁnition (eq. 4).
Unfortunately, while the weights are continuous, the optimization algorithm (such as SGD) is usually discrete.
This generally makes the resulting quantities degenerate (inﬁnite).
Most works address this issue by replacing the discrete optimization algorithm with a continuous one, such as stochastic gradient Langevin dynamics (Welling & Teh, 2011) or continuous stochastic differential equations that approximate SGD (Li et al., 2017) in the limit.
We aim to avoid such assumptions and give a deﬁnition that is directly applicable to real networks trained with standard algorithms.
To do this, we apply a smoothing procedure to a standard discrete algorithm.
The ﬁnal result can still be interpreted as a valid bound on Shannon mutual information, but for a slightly modiﬁed optimization algorithm.
Our deﬁnitions relate informativeness of a sample to the notion of algorithmic stability (Bousquet & Elisseeff, 2002; Hardt et al., 2015), where a training algorithm A is called stable if A ( S ) is close to A ( S (cid:48) ) when the datasets S and S (cid:48) differ by only one sample.
To ensure our quantities are well-deﬁned, we apply a smoothing technique which is reminiscent of a soft discretization of weight space.
In Section 4, we show that a canonical discretization is obtained using the Fisher information matrix, which relates to classical results of Rissanen (1996) on optimal coding length.
It also relates to the use of a post-distribution in Achille et al.
(2019), who however use it to estimate the total amount of information in the weights of a network.
We use a ﬁrst-order approximation (linearization) inspired by the Neural Tangent Kernel (NTK)
(Jacot et al., 2018; Lee et al., 2019) to efﬁciently estimate informativeness of a sample.
While NTK predicts that, in the limit of an inﬁnitely wide network, the linearized model is an accurate approximation, we do not observe this on more realistic architectures and datasets.
However, we show that, when using pre-trained networks as common in practice, linearization yields an accurate approximation, similarly to what is observed by Mu et al.
Shwartz-Ziv & Alemi (2020) study the total information contained by an ensemble of randomly initialized linearized networks.
They notice that, while considering ensembles makes the mutual information ﬁnite, it still diverges to inﬁnity as training time goes to inﬁnity.
On the other hand, we consider the unique information about a single example, without the neeed for ensembles, by considering smoothed information, which remains bounded for any time.
Other complementary works study how information about an input sample propagates through the network (Shwartz-Ziv & Tishby, 2017; Achille & Soatto, 2018; Saxe et al., 2019) or total amount of information (complexity) of a classiﬁcation dataset (Lorena et al., 2019), rather than how much information the sample itself contains.
In terms of applications, our work is related to works that estimate inﬂuence of an example (Koh & Liang, 2017; Toneva et al., 2019; Katharopoulos & Fleuret, 2018; Ghorbani & Zou, 2019; Yoon et al., 2019).
This can be done by estimating the change in weights if a sample is removed from the training set, which is addressed by several works (Koh & Liang, 2017; Golatkar et al., 2020; Wu et al., 2020).
Inﬂuence functions (Cook, 1977; Koh & Liang, 2017) model removal of a sample as reducing its weight inﬁnitesimally in the loss function, and show an efﬁcient ﬁrst-order approximation of its effect on other measures (such as test time predictions).
We found inﬂuence functions to be prohibitively slow for the networks and data regimes we consider.
Basu et al.
(2020) found that inﬂuence functions are not accurate for large DNNs.
Additionally, inﬂuence functions assume that the training has converged, which is not usually the case in practice.
We instead use linearization of neural networks to estimate the effect of removing an example efﬁciently.
We ﬁnd that this approximation is accurate in realistic settings, and that the computational cost scales better with network size, making it applicable to very large neural networks.
Our work is orthogonal to that of feature selection: while we aim to evaluate the informativeness for the ﬁnal weights of a subset of training samples, feature selection aims to quantify the informativeness for the task variable of a subset of features.
However, they share some high-level similarities.
In particular, Kohavi et al.
(1997) propose the notion of strongly-relevant feature as one that changes the discriminative distribution when it is excluded.
This notion is similar to the notion of unique sample information in eq.
U NIQUE INFORMATION OF A SAMPLE IN THE WEIGHTS
Consider a (possibly stochastic) training algorithm
A that, given a training dataset S , returns the weights W of a classiﬁer f w .
From an information-theoretic point of view, the amount of unique information a sample z i =
( x i , y i ) provides about the weights is given by the conditional point-wise mutual information:
[Equation]
where S denotes the random variable whose sample is the particular dataset S , and r ( w | S − i
=
S − i )
= (cid:82) p
A
[Equation]
w | S
[Equation]
S
[Equation]
i
[Equation]
) =
E
[Equation]
∼ p [Equation] z ) [ p A ( w | S =
S − i , z (cid:48) i )] denotes the marginal distribution of the weights over all possible sampling of Z i .
Computing the distribution r ( w | S − i ) is challenging because of the high-dimensionality and the cost of training
algorithm A for multiple samples.
One can address this problem by using the following upper bound (Lemma B.1):
[Equation]
which is valid for any distribution q ( w | S − i ) .
Choosing q ( w | S − i )
=
p A ( w | S − i ) , the distribution of the weights after training on S − i , gives a reasonable upper bound (see Sec.
A.1 for details):
[Equation]
We call SI ( z i , A ) (cid:44) KL( p A ( w | S ) (cid:107) p A ( w | S − i ))
the sample information of z i w.r.t.
algorithm A .
Smoothed Sample Information.
The formulation above is valid in theory but, in practice, even
SGD is used in a deterministic fashion by ﬁxing the random seed and, in the end, we obtain just one set of weights rather than a distribution of them.
Under these circumstances, all the above
KL divergences are degenerate, as they evaluate to inﬁnity.
It is common to address the problem by assuming that A is a continuous stochastic optimization algorithm, such as stochastic gradient
Langevin dynamics (SGLD) or a continuous approximation of SGD which adds Gaussian noise to the gradients.
However, this creates a disconnect with the practice, where such approaches do not perform at the state-of-the-art.
Our deﬁnitions below aim to overcome this disconnect.
Deﬁnition 3.1 (Smooth sample information) .
Let A be a possibly stochastic algorithm.
Following eq.
(3), we deﬁne the smooth sample information with smoothing Σ :
[Equation]
where we deﬁne smoothed weights A Σ ( S ) (cid:44) A ( S ) + ξ , with ξ ∼ N (0 , Σ) .
Note that if the algorithm A is continuous, we can pick Σ → 0 , which will make SI Σ ( z i , A ) →
SI( z i , A ) .
The following proposition shows how to compute the value of SI Σ in practice.
Proposition 3.2.
Let A be a deterministic training algorithm.
Then, we have:
[Equation]
where w = A ( S ) and w − i
=
A ( S − i ) are the weights obtained by training respectively with and without the training sample
z i .
That is, the value of SI Σ ( z i ) depends on the distance between the solutions obtained training with and without the sample
z i , rescaled by Σ .
The smoothing of the weights by a matrix Σ can be seen as a form of soft-discretization.
Rather than simply using an isotropic discretization Σ
= σ 2 I – since different ﬁlters have different norms and/or importance for the ﬁnal output of the network – it makes sense to discretize them differently.
In
Sections 4 and 5 we show two canonical choices for Σ .
One is the inverse of the Fisher information matrix, which discounts weights not used for classiﬁcation, and the other is the covariance of the steady-state distribution of SGD, which respects the level of SGD noise and ﬂatness of the loss.
U NIQUE I NFORMATION IN THE P REDICTIONS
SI Σ ( z i , A ) measures how much information an example z
i provides to the weights.
Alternatively, instead of working in weight-space, we can approach the problem in function-space, and measure the informativeness of a training example for the network outputs or activations.
The unique information that z i provides to the predictions on a test example x is:
[Equation]
where x ∼ p ( x ) is a previously unseen test sample, (cid:98) y ∼ q ( · | x, S ) is the network output on input x after training on S , and r ( (cid:98) y | x, S −
i
[Equation]
=
[Equation]
Following the reasoning in the previous section, we arrive at
[Equation]
Again, when training with a discrete algorithm and/or when the output of the network is deterministic, the above quantity may be inﬁnite.
Similar to smooth sample information, we deﬁne:
Deﬁnition 4.1 (Smooth functional sample information) .
Let A be a possibly stochastic training algorithm and let (cid:98) y be the prediction on example x after training on S .
We deﬁne the smooth functional sample information ( F-SI ) as:
[Equation]
where (cid:98) y
σ = (cid:98)
y
( x, S ) + n , with n ∼ N (0 , σ 2 I ) and q σ ( (cid:98) y
σ | x, S ) being the distribution of (cid:98) y σ .
We now describe a ﬁrst-order approximation of the value of F-SI Σ for deterministic algorithms.
Proposition 4.2.
Let A be a deterministic algorithm,
w = A ( S ) and w −
i
=
A ( S − i ) be the weights obtained training respectively with and without sample z i .
Then,
[Equation] [Equation]
with F ( w ) =
E x (cid:2) ∇ w f w ( x ) ∇ w f w ( x ) T (cid:3) being the Fisher information matrix of q σ =1 ( (cid:98) y
| x, S ) .
By comparing eq.
(5) and eq.
(9), we see that the functional sample information is approximated by using the inverse of the Fisher information matrix to smooth the weight space.
However, this smoothing is not isotropic as it depends on the point w .
E XACT S OLUTION FOR L INEARIZED N ETWORKS
In this section, we derive a close-form expression for SI Σ and F-SI Σ using a linear approximation of the network around the initial weights.
We show that this approximation can be computed efﬁciently and, as we validate empirically in Sec. 6, correlates well with the actual informativeness values.
We also show that the covariance matrix of SGD’s steady-state distribution is a canonical choice for the smoothing matrix Σ of SI Σ .
Linearized Network.
Linearized neural networks are a class of neural networks obtained by taking the ﬁrst-order Taylor expansion of a DNN around the initial weights (Lee et al., 2019):
[Equation]
These networks are linear with respect to their parameters w , but can be highly non-linear with respect to their input x .
One of the advantages of linearized neural networks is that the dynamics of continuous-time or discrete-time gradient descent can be written analytically if the loss function is the mean squared error (MSE).
In particular, for continuous-time gradient descent with constant learning rate η > 0 , we have (Lee et al., 2019):
[Equation] [Equation]
(11) where Θ 0 = ∇ w f 0 ( X ) T ∇ w f 0 ( X ) ∈
R
nk × nk is the Neural Tangent Kernel (NTK)
(Jacot et al., 2018; Lee et al., 2019) and Θ 0 ( x, X )
= ∇ w f 0 ( x ) T ∇ w f 0 ( X ) .
The expressions for networks trained with weight decay is essentially the same (see Sec. F).
To keep the notation simple, we will use f w ( x ) to indicate f lin w ( x ) from now on.
Stochastic Gradient Descent.
As mentioned in Sec. 3, a popular alternative approach to make information quantities well-deﬁned is to use continuous-time SGD, which is deﬁned by (Li et al., 2017; Mandt et al., 2017):
[Equation]
where η is the learning rate, b is the batch size, n ( t ) is a Brownian motion, and Λ( w t ) is the covariance matrix of the per-sample gradients (see Sec.
C for details).
Let A SGD be the algorithm that returns a random sample from the steady-state distribution of (12), and let A ERM be the deterministic algorithm that returns the global minimum w ∗ of the loss L ( w ) (for a regularized linearized network
L ( w ) is strictly convex).
We now show that the non-smooth sample information SI( z i , A SGD ) is the same as the smooth sample information using SGD’s steady-state covariance as the smoothing matrix and A ERM as the training algorithm.
Proposition 5.1.
Let the loss function be regularized MSE, w ∗ be the global minimum of it, and algorithms A SGD and A ERM be deﬁned as above.
Assuming Λ( w ) is approximately constant around w ∗ and SGD’s steady-state covariance remains constant after removing an example, we have
[Equation] [Equation]
where Σ is the solution of
[Equation]
14) with H = ( ∇ w f 0 ( X ) ∇ w f 0 ( X ) T + λI ) being the Hessian of the loss function.
This proposition motivates the use of SGD’s steady-state covariance as a smoothing matrix.
From equations (13) and (14) we see that SGD’s steady-state covariance is proportional to the ﬂatness of the loss at the minimum, the learning rate, and to SGD’s noise, while inversely proportional to the batch size.
When H is positive deﬁnite, as in our case when using weight decay, the continuous
Lyapunov equation (14) has a unique solution, which can be found in O ( d 3 ) time using the BartelsStewart algorithm (Bartels & Stewart, 1972).
One particular case when the solution can be found analytically is when Λ( w ∗ ) and H commute, in which case Σ = η 2 b Λ H − 1 .
For example, this is the case for Langevin dynamics, for which Λ( w ) =
σ 2 I in equation (12).
In this case, we have
[Equation]
which was already suggested by Cook (1977) as a way to measure the importance of a datum in linear regression.
Functional Sample Information.
The deﬁnition in Section 4 simpliﬁes for linearized neural networks: The step from eq.
(8) to eq.
(9) becomes exact, and the Fisher information matrix becomes independent of w and equal to F = E
x ∼ p ( x ) (cid:2) ∇ w f 0 ( x ) ∇ w f 0 ( x ) T (cid:3) .
This shows that functional sample information can be seen as weight sample information with discretization Σ equal to F − 1 .
The functional sample information depends on the training data distribution, which is usually unknown.
We can estimate it using a validation set:
[Equation]
It is instructive to compare the sample weight information of (15) and functional sample information of (17).
Besides the constants, the former uses the Hessian of the training loss, while the latter uses the Hessian of the validation loss (without the (cid:96) 2 regularization term).
One advantage of the latter is computational cost: As demonstrated in the next section, we can use equation (16) to compute the prediction information, entirely in the function space, without any costly operation on weights.
For this reason, we focus on the linearized F-SI approximation in our experiments.
Since σ − 2 is just a multiplicative factor in (17) we set σ = 1 .
We also focus on the case where the training algorithm
A is discrete gradient descent running for t epochs (equations 10 and 11).
Efﬁcient Implementation.
To compute the proposed sample information measures for linearized neural networks, we need to compute the change in weights w − w −
i (or change in predictions f w ( x ) − f w i ( x ) ) after excluding an example from the training set.
This can be done without retraining using the analytical expressions of weight and prediction dynamics of linearized neural networks eq.
(10) and eq.
(11), which also work when the algorithm has not yet converged ( t < ∞ ).
We now describe a series of measures to make the problem tractable.
First, to compute the NTK matrix we would need to store the Jacobian ∇ f 0 ( x i ) of all training points and compute ∇ w f 0 ( X ) T ∇ w f 0 ( X ) .
This is prohibitively slow and memory consuming for large DNNs.
Instead, similarly to Zancato et al.
(2020), we use low-dimensional random projections of per-example Jacobians to obtain provably good approximations of dot products (Achlioptas, 2003; Li et al., 2006).
We found that just taking 2000 random weights coordinates per layer provides a good enough approximation of the
NTK matrix.
Importantly, we consider each layer separately, as different layers may have different gradient magnitudes.
With this method, computing the NTK matrix takes O ( nkd + n 2 k 2 d 0 ) time, where d 0 ≈ 10 4 is the number of sub-sampled weight indices ( d 0 (cid:28) d ).
We also need to recompute Θ − 1 0 after removing an example from the training set.
This can be done in quadratic time by using rank-one updates of the inverse (see Sec.
E).
Finally, when t (cid:54) = ∞ we need to recompute e − η Θ 0 t after removing an example.
This can be done in O ( n 2 k 2 ) time by downdating the eigendecomposition of Θ
(Gu & Eisenstat, 1995).
Overall, the complexity of computing w − w i for all training examples is O ( n 2 k 2 d 0 + n ( n 2 k 2 + C )) , C is the complexity of a single pass over the training dataset.
The complexity of computing functional sample information for m test samples is
O ( C + nmk 2 d 0 + n
( mnk 2 + n 2 k 2 )) .
This depends on the network size lightly, only through C .
E XPERIMENTS
In this section, we test the validity of linearized network approximation in terms of estimating the effects of removing an example and show several applications of the proposed information measures.
Additional results and details are provided in the supplementary Sec. A.
Accuracy of the linearized network approximation.
We measure (cid:107) w − w −
i
(cid:107) 22 and (cid:107) f w ( X val )
− f w − i ( X val ) (cid:107) 22 for each sample
z i by training with and without that example.
Then, instead of retraining, we use the efﬁcient linearized approximation in Sec.
6 to estimate the same quantities and
[Table]
[Figure]
cbb cgm cgm cgm cbsd cgm cgm cbb cgm cgm cmd cmd cmd cmd cmd cmd cmd cmd cmd cmd
[Figure]
measure their correlation with the ground-truth values (Table 1).
For comparison, we also estimate these quantities using inﬂuence functions (Koh & Liang, 2017).
We consider two classiﬁcation tasks: (a) a toy MNIST 4 vs 9 classiﬁcation task and (b) Kaggle Dogs vs. Cats classiﬁcation task (Kaggle, 2013), both with 1000 examples.
For MNIST we consider a fully connected network with a single hidden layer of 1024 ReLU units (MLP) and a small 4-layer convolutional network (CNN), either trained from scratch or pretrained on EMNIST letters (Cohen et al., 2017).
For cats vs dogs classiﬁcation, we consider ResNet-18 and ResNet-50 networks (He et al., 2016) pretrained on ImageNet.
In both tasks, we train both with and without weight decay ( (cid:96) 2 regularization).
The results in Table 1 shows that linearized approximation correlates well with ground-truth when the network is wide enough (MLP) and/or pretraining is used (CNN with pretraining and pretrained ResNets).
This is expected, as wider networks can be approximated with linearized ones better (Lee et al., 2019), and pretraining decreases the distance from initialization, making the Taylor approximation more accurate.
Adding regularization also keeps the solution close to initialization, and generally increases the accuracy of the approximation.
Furthermore, in most cases linearization gives better results compared to inﬂuence functions, while also being around 30 times faster in our settings.
Which examples are informative?
Fig.
1, and Fig.
4 of the supplementary, plot the top 10 least and most important samples in iCassava plant disease classiﬁcation (Mwebaze et al., 2019), the MNIST 4 vs 9, and Kaggle cats vs dogs classiﬁcation tasks.
Especially in the case of the last two, we see that the least informative samples look typical and easy, while the most informative ones look more challenging and atypical.
In the case of iCassava, the informative samples are more zoomed on features that are important for classiﬁcation (e.g., the plant disease spots).
We observe that most samples have small unique information, possibly because they are easier or because the dataset may have many similar-looking examples.
While in the case of MNIST 4 vs 9 and Cats vs. Dogs, the two classes have on average similar information scores, in Fig.
1a we see that in iCassava examples from rare classes (such as ‘healthy’ and ‘cbb’) are on average more informative.
Which data source is more informative?
In this experiment, we train for 10-way digit classiﬁcation where both the training and validation sets consist of 1000 samples, 500 from MNIST andfrom SVHN.
We compute functional sample information for a pretrained ResNet-18 network.
The results presented in Fig.
2a tell that SVHN examples are much more informative than MNIST examples.
The same result holds if we change the network to be a pretrained DenseNet-121 (see Fig. 7a).
[Figure]
This is intuitive, as SVHN examples have more variety.
One can go further and use our information measures for estimating informativeness of examples of dataset A for training a classiﬁer for task
B , similar to Torralba & Efros (2011).
Detecting mislabeled examples.
We expect a mislabeled example to carry more unique information, since the network needs to memorize unique features of that particular example to classify it.
To test this, we add 10% uniform label noise to MNIST 4 vs 9, Kaggle cats vs dogs, and iCassava classiﬁcation tasks (all with 1000 examples in total), while keeping the validation sets clean.
Fig.
2b plots the histogram of functional sample information for both correct and mislabeled examples in the case of iCassava classiﬁcation task with a pretrained ResNet-18, while Fig.
8a and 8b plot that for MNIST 4 vs 9 and Kaggle cats vs dogs tasks, respectively.
The results indicate that mislabeled examples are much more informative on average.
This suggests that our information measures can be used to detect outliers or corrupted examples.
Data summarization.
We subsample the training dataset by removing a fraction of the least informative examples and measure the test performance of the resulting model.
We expect that removing the least informative training samples should not affect the performance of the model.
Note however that, since we are considering the unique information, removing one sample can increase the informativeness of another.
For this reason, we consider two strategies:
In one we compute the informativeness scores once, and remove a given percentage of the least informative samples.
In the other we remove 5% of the least informative samples, recompute the scores, and iterate until we remove the desired number of samples.
For comparison, we also consider removing the most informative examples (“top” baseline) and randomly selected examples (“random” baseline).
The results on MNIST 4 vs 9 classiﬁcation task with the one-hidden-layer network described earlier, are shown in Fig. 2c.
Indeed, removing the least informative training samples has little effect on the test error, while removing the top examples has the most impact.
Also, recomputing the information scores after each removal steps (“bottom iterative”) greatly improves the performance when many samples are removed, conﬁrming that SI and F-SI are good practical measures of unique information in a sample, but also that the total information in a large group is not simply the sum of the unique information of its samples.
Interestingly, removing more than 80% of the least informative examples degrades the performance more than removing the same number of the most informative examples.
In the former case, we are left with a small number of most informative examples, some of which are outliers, while in the latter case we are left with the same number of least informative examples, most of which are typical.
Consequently, the performance is better in the latter case.
Detecting under-sampled sub-classes.
Using CIFAR-10 images, we create a dataset of “Pets vs
Deer”:
Pets has 4200 samples and deer 4000.
The class “pets” consists of two unlabeled sub-classes, cats (200) and dogs (4000).
Since there are relatively few cat images, we expect each to carry more unique information.
This is conﬁrmed when we compute F-SI for a pretrained ResNet-18 (see Fig.of the supplementary), suggesting that the F-SI can help to detect when an unlabeled sub-class of a larger class is under-sampled.
Detecting adversarial examples.
Szegedy et al.
(2014) shows that imperceptible perturbation to an image can fool a neural network into predicting the wrong label (adversarial examples).
Good- fellow et al.
(2015) further shows that adding adversarial examples to the training dataset improves adversarial robustness and generalization (adversarial training), which suggests that adversarial examples may be informative for the training process.
To test how informative adversarial examples are with respect to normal ones, we consider the Kaggle Cats vs Dogs classiﬁcation task, consisting of 1000 examples.
On this task we ﬁne-tune a pretrained ResNet-18 and for 10% of examples create successful adversarial examples using the FGSM method Goodfellow et al.
(2015) with (cid:15) = 0 .
Then, for these 10% of examples, we replace the original images with the corresponding adversarial ones.
With the resulting dataset, we consider a new pretrained ResNet-18 and compute F-SI for all training examples, setting t = 1000 and η = 0 .
The results reported in Fig.
10 of the supplementary conﬁrm that adversarial examples are on average more informative than normal ones.
Furthermore, the results indicate that one can use F-SI to successfully detect adversarial examples.
D ISCUSSION AND F UTURE W ORK
The smooth (functional) sample information depends not only on the example itself, but on the network architecture, initialization, and the training procedure (i.e. the training algorithm).
This has to be the case, since an example can be informative with respect to one algorithm or architecture, but not informative for another one.
Similarly, some examples may be more informative at the beginning of the training (e.g., simpler examples) rather than at the end.
Nevertheless, we found out that F-SI still captures something inherent in the example.
In particular, as shown Sec.
A.4 of the supplementary, F-SI scores computed with respect to different architectures are signiﬁcantly correlated to each other (e.g. around 45% correlation in case of ResNet-50 and DenseNet-121).
Furthermore, F-SI scores computed for different initializations are signiﬁcantly correlated (e.g. around 36%).
Similarly, F-SI scores computed for different training lengths are strongly correlated (see Fig. 5b).
This suggests that F-SI computed with respect to one network can reveal useful information for another one.
Indeed, this is veriﬁed in Sec.
A.4 of the supplementary, where we redo the data summarization experiment, but with a slight change that F-SI scores are computed for one network, but another network is trained to check the accuracy.
As shown in the Fig.
7b the results look qualitatively similar to those of the original experiment.
The proposed sample information measure only the unique information provided by an example.
For this reason, it is not surprising that typical examples are usually the least informative, while atypical and rare ones are more informative.
This holds not only for visual data (as shown above), but also for textual data (see Sec.
A.7 of the supplementary).
While the typical examples are usually less informative according to the proposed measures, they still provide information about the decision functions, which is evident in the data summarization experiment – removing lots of typical examples was worse than removing the same number of random examples.
Generalizing sample information to capture this kind of contributions is an interesting direction for future work.
Similar to Data Shapley (Ghorbani & Zou, 2019)), one can look at the average unique information an example provides when considering along with a random subset of data.
One can also consider common information, high-order information, synergistic information, and other notions of information between samples.
The relation among these quantities is complex in general, even forvariables (Williams & Beer, 2010) and is an open challenge.
C ONCLUSION
There are many notions of information that are relevant to understanding the inner workings of neural networks.
Recent efforts have focused on deﬁning information in the weights or activations that do not degenerate for deterministic training.
We look at the information in the training data, which ultimately affects both the weights and the activations.
In particular, we focus on the most elementary case, which is the unique information contained in a sample, because it can be the foundation for understanding more complex notions.
However, our approach can be readily generalized to unique information of a group of samples.
Unlike most previously introduced information measures, ours is tractable even for real datasets used to train standard network architectures, and does not require restriction to limiting cases.
In particular, we can approximate our quantities without requiring the limit of small learning rate (continuous training time), or the limit of inﬁnite network width.
A A DDITIONAL RESULTS AND DETAILS
In this section we present additional results and details that were not included in the main paper due to the space constraint.
A. A PPROXIMATING UNIQUE INFORMATION WITH LEAVE - ONE - OUT KL DIVERGENCE
In the main text we discussed that I ( W ; z i | S − i ) can be upper bounded with KL( p A ( w |
S ) (cid:107) p A ( w | S − i )) .
In this subsection we evaluate these quantities on a toy 2D dataset.
The dataset has two classes, each with 40 examples, generated from a Gaussian distribution (see Fig. 3a).
We consider training a linear regression on this dataset using stochastic gradient descent for 200 epochs, with batch size equal to 5 and 0.1 learning rate.
Fig.
3b plots the distribution p A ( w | S ) , Fig.
3c the distribution p A ( w | S − i ) , while Fig.
3d plots the distribution E z (cid:48)
i p A ( w | S − i , z (cid:48) i ) .
On this example we get I ( W ; z i | S − i )
≈ 1 . 3 and KL( p A ( w | S ) (cid:107)
p A ( w | S − i ))
≈ 3 . 0 .
[Figure]
A. W HICH EXAMPLES ARE INFORMATIVE ?
In this subsection we present the most and least informative examples for two more classiﬁcation tasks:
MNIST 4 vs 9 classiﬁcation with MLP and Kaggle cat vs dog classiﬁcation with a pretrained
ResNet-18 (Fig. 4).
The results indicate that most informative examples are often the challenging and atypical ones, while the least informative ones are easy and typical ones.
For the MLP network on MNIST 4 vs 9 we set t = 2000 and η = 0 .
001 ; for the ResNet-18 on cats vs dog classiﬁcation t = 1000 and η = 0 . 001 ; and for the ResNet-18 on iCassava dataset t = 5000 and η = 0 .
[Figure]
A. A CCURACY OF THE LINEARIZED NETWORK APPROXIMATION
In this subsection we present additional details of experiments presented in Table 1.
In all datasets used the validation set also has 1000 samples.
The fully connected network, MLP, consists of a single layer of 1024 ReLU units.
The architecture of the CNN used for the experiments in Tableis as follow.
[Table]
In all our experiments, when using pretrained ResNets, we disable the exponential averaging of batch statistics in batch norm layers.
The exact details of running inﬂuence functions and linearized neural network predictions are presented in Table 2.
A. H OW MUCH DOES SAMPLE INFORMATION DEPEND ON ALGORITHM ?
The proposed information measures depend on the training algorithm, which includes the architecture, seed, initialization, and training length.
This is unavoidable as one example can be more informative for one algorithm and less informative for another.
Nevertheless, in this subsection, we test how much does informativeness depend on the network, initialization, and training time.
We consider the Kaggle cats vs dogs classiﬁcation task with 1000 training examples.
First, ﬁxing the training time t = 1000 , we consider four pretrained architectures: ResNet-18, ResNet-34, ResNet50, and DenseNet-121.
The correlations between F-SI scores computed for the four architectures
[Table]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
are presented in Fig.
5a.
We see that F-SI scores computed for two completely different architectures, such as ResNet-50 and DenseNet-121 have signiﬁcant correlation, around 45%.
Furthermore, there is a signiﬁcant overlap in top 10 most informative examples for these networks (see Fig. 6).
Next, ﬁxing the network to be a pretrained ResNet-18 and ﬁxing the training length t = 1000 , we consider changing initialization of the classiﬁcation head (which is not pretrained).
In this case the correlation between F-SI scores is 0 .
364 ± 0 .
Finally, ﬁxing the network to be a ResNetand ﬁxing the initialization of the classiﬁcation head, we consider changing the number of iterations in the training.
We ﬁnd strong correlations between F-SI scores of different training lengths (see
Fig. 5b).
MNIST vs SVHN experiment for DenseNet-121.
In this paragraph we redo the MNIST vs
SVHN experiment (Fig 2a) but for a different network, a pretrained DenseNet-121, to test the dependence of the results on the architecture choice.
The results are presented in Fig.
7a and are qualitatively identical to the results derived with a pretrained ResNet18 (Fig. 2a).
Data summarization with a change of architecture.
To test how much sample information scores computed for one network are useful for another network, we reconsider the MNIST 4 vs 9 data summarization experiment (Fig. 2c).
This time we compute F-SI scores for the original net-
[Figure] [Figure]
work with one hidden layer, but train a two-hidden-layer neural network (both layers having
ReLU units).
The data summarization results presented in Fig.
7b are qualitatively and quantitively almost identical to the original results presented in the main text.
This conﬁrms that F-Si scores computed for one network can be useful for another network
[Figure]
A. D ETECTING INCORRECT EXAMPLES
In this subsection we present results on detecting mislabeled examples for two more classiﬁcation tasks: MNIST 4 vs 9 classiﬁcation with an MLP (Fig. 8a) and Kaggle cat vs dog classiﬁcation with a pretrained ResNet-18 (Fig. 8b).
The results are qualitatively the same and assert that, indeed, mislabeled examples are more informative, on average.
For the MLP network we set t = 2000 and η = 0 .
001 ; for the ResNet-18 on the cats vs dog classiﬁcation task t = 1000 and η = 0 . 001 ; and for the ResNet-18 on the iCassava dataset (in the main text) we set t = 10000 and η = 0 . 0001 .
A. D ETECTING UNDER - SAMPLED SUB - CLASSES
Using CIFAR-10 images, we create a dataset of “Pets vs Deer”:
Pets has 4200 samples and deer 4000.
The class pets consists of two unlabeled sub-classes, cats (200) and dogs (4000).
Since there are relatively few cat images, we expect each to carry more unique information.
Indeed, Fig.
[Figure]
shows that this is the case, suggesting that the F-SI can help detecting when an unlabeled sub-class of a larger class is under-sampled.
In this experiment we set t = 10000 and η = 0 .
[Figure]
A. I NFORMATIVE EXAMPLES IN SENTIMENT ANALYSIS
So far we focused on image classiﬁcation tasks.
To test whether the proposed method works for other modalities
[Figure]
we consider a sentiment analysis task, consisting of 2000 samples from the IMDB movie reviews dataset (Maas et al., 2011).
We convert words to 300-dimensional vectors using the pretrained GloVe 6B word vectors (Pennington et al., 2014).
Then each review is mapped to a vector by averaging all of its word vectors.
On the resulting vector we apply a fully connected neural network with one hidden layer, consisting of 2000 ReLU units.
We then compute F-SI scores for all
[Figure]
training examples, setting t = 10000 and η = 0 .
Likewise to the case of images, we ﬁnd that least informative examples are typical and easy (often containing many sentiment-speciﬁc words), while the most informative examples are more nuanced and hard.
Additionally, we ﬁnd that there is -16% correlation between the informativeness and length of review, possibly because averaging many word vectors makes long reviews similar to each other.
A. R EMAINING DETAILS
In the experiment of measuring which data source is more informative (Fig.
2a)
we set t =and η = 0 . 0001 .
In the experiment of data summarization (Fig. 2c) we set t = 2000 and η = 0 .
B P ROOFS
The proof of eq. (2) follows from the following more general lemma, applied to the conditional distribution p A ( w | S − i , z i ) and its marginal r ( x | S − i ) . Lemma B.1. Let p ( x | y ) be a conditional probability density function, and let p ( x )
= (cid:82) p ( x | y ) dP ( y ) denote its marginal distribution.
Then, for any distribution q ( x ) we have
[Equation]
, where KL( p ( x | y ) (cid:107) q ( x ))
(cid:44) (cid:82) (cid:82) log p ( x | y )
q ( x ) p ( x | y ) dxdP ( y ) denotes the conditional version of the KL divergence (Cover & Thomas, 2006, Section 2.5). Proof. The last inequality follows from the fact that the KL-divergence is always non-negative.
We now prove the ﬁrst equality:
[Equation]
where in (a) we use that by deﬁnition p ( x )
= (cid:82) p ( x | y ) dP ( y ) and in (b) we exchange the order of integration. Propositions 3.2 and 4.2 are trivial, since they just assert that KL divergence between two Gaussian distributions with equal covariance matrices Σ and with means µ 1 and µ 2 is equal to 12 ( µ 1 − µ 2 ) T Σ − 1 ( µ 1 − µ 2 ) .
We present the proof of Proposition 5.1 here.
Proposition B.2 (Prop. 5.1 restated) .
Let the loss function be regularized MSE, w ∗ be the global minimum of it, and algorithms A SGD and A ERM be deﬁned as in the main text.
Assuming Λ( w ) is approximately constant around w ∗ and SGD’s steady-state covariance remains constant after removing an example, we have
[Equation]
(18) where Σ is the solution of
[Equation]
(19) with H =
( ∇ w f 0 ( X ) ∇ w f 0 ( X ) T + λI ) being the Hessian of the loss function.
Proof.
Assuming Λ( w ) is approximately constant around w ∗ , the steady-state distributions of (12) is a Gaussian distribution with mean w ∗ and covariance Σ such that:
[Equation]
where H = ( ∇ w f 0 ( X ) ∇ w f 0 ( X ) T + λI ) is the Hessian of the loss function (Mandt et al., 2017).
This can be veriﬁed by checking that the distribution N ( · ; w ∗ , Σ) satisﬁes the Fokker-Planck equation (see Sec. D).
Having p A SGD ( w | S )
=
N ( w ; w ∗ , Σ) and p A SGD ( w | S − i ) =
N ( w ; w ∗− i , Σ − i ) , we have that
[Equation]
[Equation]
By the assumption that SGD steady-state covariance stays constant after removing an example, i.e.
Σ − i = Σ , equation (21) simpliﬁes to:
[Equation]
By the deﬁnition of the A ERM algorithm and smooth sample information, this is exactly equal to
SI Σ ( z i , A ERM ) .
C SGD NOISE COVARIANCE
Assume we have n examples and the batch size is b .
Let g i (cid:44) ∇ w L i ( w ) , g (cid:44) 1 n (cid:80) ni
=
g i , and ˜ g (cid:44) 1 b (cid:80) bi =1 g k i , where k i are sampled independently from { 1 , . . . , n } uniformly at.
Then
Cov [˜ g, ˜ g ] = E
[Equation]
We denote the per-sample covariance, b · Cov [˜ g, ˜ g ] with Λ( w ) :
[Equation]
where G ∈ R d × n has g i ’s as its columns.
We can see that whenever the number of samples times number of outputs is less than number of parameters ( nk < d ), then Λ( w ) will be rank deﬁcient.
Also, note that if we add weight decay to the total loss then covariance Λ( w ) will not change, as all gradients will be shifted by the same vector.
D S TEADY - STATE COVARIANCE OF SGD
In this section we verify that the normal distribution N ( · ; w ∗ , Σ) , with w ∗ being the global minimum of the regularization MSE loss and covariance matrix Σ satisfying the continuous Lyapunov equation Σ H + H Σ = η b Λ( w ∗ ) , is the steady-state distribution of the stochastic differential equation of eq.
We assume that (a) Λ( w ) is constant in a small neighborhood of w ∗ and (b) the steady-state distribution is unique.
We start with the Fokker-Planck equation:
[Equation]
If p ( w ) = N ( w ; w ∗ , Σ) = 1 Z exp (cid:8) − 12 ( w − w ∗ ) T Σ − 1 ( w − w ∗ ) (cid:9) is the steady-state distribution, then the Fokker-Planck becomes:
[Equation]
In the case of MSE loss:
[Equation]
Additionally, for p ( w ) the following two statements hold:
[Equation]
where Σ − 1 i is the i -th row of Σ − 1 .
Let’s compute the ﬁrst term of (23):
[Equation]
As w ∗ is a critical point of L ( w ) , we have that ∇ f 0 ( X )( f w ∗ ( X ) − Y )
+ λw ∗ = 0 .
Therefore, we can subtract p ( w ) (cid:0) ( f w ∗ ( X ) − Y ) T ∇ f 0 ( X ) T + λ ( w ∗ ) T (cid:1)
Σ − 1 ( w − w ∗ ) from (24): d (cid:88) i =
[Equation]
I SOTROPIC CASE : Λ( w ) =
σ 2 I d
In the case when Λ( w ) = σ 2 I d , we have (cid:88) i,j
[Equation]
Putting everything together in the Fokker-Planck we get:
[Equation]
It is easy to verify that Σ − 1 =
2 b ησ 2 H is a valid inverse covariance matrix and satisﬁes the equation above.
Hence, it is the unique steady-state distribution of the stochastic differential equation.
The result conﬁrms that variance is high when batch size is low or learning rate is large.
Additionally, the variance is low along directions of low curvature.
N ON - ISOTROPIC CASE
We assume Λ( w ) is constant around w ∗ and is equal to Λ .
This assumption is acceptable to a degree, because SGD converges to a relatively small neighborhood, in which we can assume Λ( w ) to not change much.
With this assumption,
[Equation]
It is easy to verify that if Σ H + H Σ = ηb Λ , then terms in equations (25) and (26) will be negatives of each other up to a constant η 2 b , implying that p ( w ) satisﬁes the Fokker-Planck equation.
Note that Σ − 1 = 2 bη H Λ − 1 also satisﬁes the Fokker-Planck, but will not be positive deﬁnite unless H and Λ commute.
E F AST UPDATE OF NTK INVERSE AFTER DATA REMOVAL
For computing weights or predictions of a linearized network at some time t , we need to compute the inverse of the NTK matrix.
To compute the informativeness scores, we need to do this inversion n time, each time with one data point excluded.
In this section, we describe how to update the inverse of NTK matrix after removing one example in O ( n 2 k 3 ) time, instead of doing the straightforward O ( n 3 k 3 ) computation.
Without loss of generality let’s assume we remove last r rows and corresponding columns from the NTK matrix.
We can represent the NTK matrix as a block matrix:
[Equation]
The goal is to compute A − 1 11 from Θ − 1 0 .
We start with the block matrix inverse formula:
[Equation]
where
[Equation]
From (28) we have A 11 = F 11 + A 12 A − 1 22 A 21 .
Applying the Woodbury matrix identity on this we get:
[Equation]
This Eq.
(30) gives the recipe for computing A − 1 11 .
Note that F − 1 11 can be read from Θ − 1 0 using (27),
A 12 , A 21 , and A 22 can be read from Θ .
Finally, the complexity of computing A − 1 11 using (30) is
O ( n 2 k 3 ) if we remove one example.
F A DDING WEIGHT DECAY TO LINEARIZED NEURAL NETWORK TRAINING
Let us consider the loss function L ( w )
= (cid:80) ni
=1 L i ( w ) + λ 2 (cid:107) w − w 0 (cid:107) 22 .
In this case continuous-time gradient descent is described by the following ODE:
[Equation]
Let ω ( t ) (cid:44) w ( t ) − w 0 , then we have
[Equation]
Since all eigenvalues of A are negative, this ODE is stable and has steady-state
[Equation] [Equation]
The solution ω ( t ) is given by:
[Equation]
[Equation]
Let Θ w (cid:44) ∇ w f 0 ( X ) ∇ w f 0 ( X ) T and Θ 0 (cid:44) ∇ w f 0 ( X ) T ∇ w f 0 ( X ) .
If the SVD of ∇ w f 0 ( X ) is
UDV T , then Θ w = UDU T and Θ
=
V DV T .
Additionally, we can extend the columns of U to full basis of R d (denoted with ˜ U ) and append zeros to D (denoted with ˜ D ) to write down the eigen decomposition Θ w = ˜ U
˜ D
˜ U T .
With this, we have (Θ w + λI ) − 1 = ˜
U
( ˜ D + λI ) − 1 ˜ U T .
Continuing (38) we have
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
[Equation] where ˜ Z = (cid:20) [Equation] D + λI
[Equation]
D
[Equation]
∈ R d × nk .
Denoting Z (cid:44) ( D + λI ) − 1 D and continuing,
[Equation]
Solving for outputs.
Having w ( t ) derived
[Equation]
we can write down dynamics of f t ( x ) for any x :
[Equation]
[Equation]