E STIMATING E XAMPLE D IFFICULTY USING V ARIANCE OF G RADIENTS
Anonymous authors
Paper under double-blind review
A BSTRACT
In machine learning, a question of great interest is understanding what examples are challenging for a model to classify.
Identifying atypical examples helps inform safe deployment of models, isolates examples that require further human inspection, and provides interpretability into model behavior.
In this work, we propose Variance of
Gradients ( V OG ) as a valuable and efﬁcient proxy metric for detecting outliers in the data distribution.
We provide quantitative and qualitative support that V OG is a meaningful way to rank data by difﬁculty and to surface a tractable subset of the most challenging examples for human-in-the-loop auditing.
Data points with high V OG scores are far more difﬁcult for the model to learn and over-index on corrupted or memorized examples.
I NTRODUCTION
Reasoning about model behavior is often easier when presented with a subset of data points that are relatively more difﬁcult for a trained model to learn.
This not only aids interpretability through case based reasoning (Kim et al., 2016; Caruana, 2000; Hooker et al., 2019), but can also be used as a mechanism to surface a tractable subset of atypical examples for further human auditing (Leibig et al., 2017; Zhang, 1992; Hooker et al., 2019), for active learning to inform model improvements, or to choose not to classify certain examples when the model is uncertain (Bartlett & Wegkamp, 2008;
Cortes et al., 2016).
One of the biggest bottlenecks for human auditing is the large scale size of modern datasets and the cost of annotating each feature (Veale & Binns, 2017).
Methods which automatically surface a subset of relatively more challenging examples for human inspection help prioritize limited human annotation and auditing time.
Despite the urgency of this use-case, ranking examples by difﬁculty has had limited treatment in the context of deep neural networks due to the computational cost of ranking a high dimensional feature space.
Recent work in this direction has either been limited to small scale datasets or features a computational cost which is infeasible for most practitioners (Hooker et al., 2019; Carlini et al., 2019; Koh & Liang, 2017).
In this work, we start with a simple hypothesis – examples that a model has difﬁculty learning will exhibit higher variance in gradient updates over the course of training.
On the other hand, we expect the backpropagated gradients of the samples that are relatively easier to learn will have lower variance because performance on that example does not consistently dominate the loss over the course of training.
The gradient updates for the relatively easier examples are expected to stabilize early in training and converge to a narrow range of values.
We term this class normalized ranking mechanism
Variance of Gradients V OG , and demonstrate across a variety of large scale datasets that it efﬁciently ranks the difﬁculty of both training and test examples.
V OG can be computed using either the predicted or true label, making it a valuable unsupervised auditing tool at test time when the true label is unknown.
Validating the behavior of VOG on artiﬁcial data.
To begin, we illustrate the principle and effectiveness of V OG in a contrived toy example setting.
The data was generated using two separate isotropic Gaussian clusters with a total of 500 data points.
In such a simple low dimensional problem, the most challenging examples for the model to classify are closer to the decision boundary.
In
Fig.
1a we visualize the trained decision boundary of a multiple layer perceptron (MLP) with a single hidden layer trained for 15 epochs.
V OG is computed at relative intervals for each training data point.
In Fig.
1b, we plot the ﬁnal V OG score (Sec. 2) against the distance to the trained boundary.
As expected, V OG allocates the highest scores to the most challenging examples that are closest to the decision boundary which exhibit the greatest variance in gradient updates over the course of the training process.
[Figure]
Contributions We scale this toy experiment and demonstrate consistent results across two different architectures and three datasets – Cifar-10, Cifar-100 Krizhevsky et al.
(2009) and ImageNet (Russakovsky et al., 2015).
Our contributions can be enumerated as follows:
We propose Variance of Gradienst (VOG) – a class-normalized variance gradient score for determining the relative ease of learning data samples within a given class (Sec. 2).
We show that V OG is an effective auditing tool for ranking high dimensional datasets by difﬁculty.
V OG assigns higher scores to test-set examples that are more challenging for the model to classify.
Restricting evaluation to the test-set examples with the lowest V OG greatly improves generalization performance.
(Sec. 3).
V OG identiﬁes clusters of images with clearly distinct semantic properties.
As seen in Fig. 4), Low scores feature images with far less cluttered backgrounds and more prototypical vantage points of the object.
In contrast, images with high scores over-index on images with cluttered backgrounds and atypical vantage points of the object of interest (zoomed in on part of the object, side proﬁle of the object, shot from above).
V OG effectively surfaces OOD and memorized examples We empirically show that V OG allocates higher scores to examples that require memorization (Sec. 5) and out-ofdistribution examples from curated benchmarks like ImageNet-O (Hendrycks et al., 2019a).
We use V OG to explore how learning differs at different stages of training and show that V OG rankings are sensitive to the stage of training and provide insight into the learning process in deep neural networks.
Implications of this work It is becoming increasingly important for deep neural networks (DNNs) to make decisions that are interpretable to both researchers and end-users.
In sensitive domains such as health care diagnostics (Xie et al., 2019;
Gruetzemacher et al., 2018; Badgeley et al., 2019;
Oakden-Rayner et al., 2019), self-driving cars (NHTSA, 2017) and hiring (Dastin,
; Harwell, 2019) providing tools for domain experts to audit models is of upmost importance.
Our work offers an efﬁcient method to rank the global difﬁculty of examples and automatically surface a possible subset to aid human interpretability.
V OG can be computed using checkpoints stored over the course of training and is model agnostic.
Critically, V OG can be computed using the predicted label, which makes it an unsupervised auditing tool at test time.
M ETHODOLOGY
We consider a supervised classiﬁcation problem where a DNN is trained to approximate the function
F that maps an input variable X to an output variable Y , formally F : X (cid:55)→ Y .
Without loss of generality we represent the image input as a feature vector x .
y ∈ Y is a discrete label vector
C i f a r - 10 ; p l a n e
C i f a r -; a pp l e
Lowest V OG
Highest V OG
Lowest V OG
Highest V OG
[Figure]
associated with each input x .
Each label y corresponds to one of C categories or classes.
A given input image x can be decomposed into a set of pixels { x i } Ni =1 .
For each image in the training and test set, we compute the gradient of the A ln with respect to each pixel x i .
Here, l designates the pre-softmax layer of the network and p is the index of either the true or the predicted class probability.
[Equation]
This formulation may feel familiar as it is often computed based upon the weights of a trained model and visualized as a image heatmap for interpretability purposes (Baehrens et al., 2010; Simonyan et al., 2013).
Here, we instead intend to compute the average variance of the input gradients for the same image across training to arrive at a scalar score that is a proxy measure of how challenging the example is to learn.
For x i ∈ x , we compute the input gradients S ti at different intervals across training, where t indicates the epoch number.
We end up with a set of K gradient snapshots computed at different input steps.
We then calculate the mean gradient input over all epochs in training µ i :
[Equation]
For example, if the number of epochs is 300 and the interval between each snapshot is 10 , K = 30 .
We note that V OG can be computed for both training and test sets by measuring the variance in gradients using checkpoints stored at different points in training.
Finally, we compute V OG for each data sample:
[Equation]
Here, N is the total number of pixels in a given image.
Hence, for every data sample x ∈ X we compute a scalar value indicating the variance in gradients score.
In order to account for inherent differences in variance between classes, we normalize the absolute ranking of the V OG score by class-level mean and standard deviation.
This amounts to asking: What is the variance of gradients for this image relative to all other exemplars for this class category?
2. E XPERIMENTAL S ETUP
Datasets: We evaluate our methodology on Cifar-10 and Cifar-100 Krizhevsky et al.
(2009) and
ImageNet (Russakovsky et al., 2015).
Cifar Training: We use a ResNet-18 network (He et al., 2016) for both Cifar-10 and Cifar-100.
For each dataset, we train for 350 epochs using stochastic gradient descent (SGD) and compute the input gradients for each sample every 10 epochs.
We implemented standard data augmentation by applying cropping and horizontal ﬂips of input images.
We use a base learning rate schedule of 0 .
1 and adaptively change to 0 .
01 at 150 th and 0 .
001 at 250 th training epochs.
The top-1 test-set accuracy for Cifar-10 and Cifar-100 were 89 .
57% and 66 .
86% respectively.
ImageNet Training: We use a ResNet-50 (He et al., 2015) trained on ImageNet.
The network was trained with batch normalization (Ioffe & Szegedy, 2015), weight decay, decreasing learning rate schedules, and augmented training data.
We train for 32 , 000 steps (approximately 90 epochs) on
ImageNet with a batch size of 1024 images.
We store 32 checkpoints over the course of training, but in practice observe that V OG ranking is very stable computed with as few as 3 checkpoints.
Our model achieves a top-1 accuracy of 76 .
68% and top-5 accuracy of 93 .
[Figure]
E VALUATING VOG
For all datasets considered, we compute V OG for both training and eval sets.
In this section, we evaluate the utility of V OG as an auditing tool.
We evaluate the stability of the V OG ranking, measure how discriminative it is at seperating easy examples from difﬁcult, and comment on the qualitative properties of images at either end of the V OG spectrum.
Qualitative inspection of ranking A qualitative inspection of examples with high and low V OG scores shows that there are distinct properties to the images at either end of the ranking.
We visualize 25 images ranked lowest and highest according to V OG for both the entire dataset (visualized for
ImageNet in Fig. 5) and for speciﬁc classes (visualized for ImageNet in Fig.
3 and for Cifar-10 and
Cifar-100 in Fig. 2).
We observe that ranking by V OG produces clusters with clearly distinguished semantic features.
Images with low V OG score tend to have uncluttered and often white backgrounds with the object of interest centered clearly in the frame.
Images with the highest V OG scores have cluttered backgrounds and the object of interest is not easily distinguishable from the background.
We also note that images with high V OG score tend to feature atypical vantage points of the objects such as highly zoomed frames, side proﬁles of the object or shots taken from above.
Other, the object of interest is partially occluded or there are image corruptions present such as heavy blur.
Test-set error and VOG A valuable property of an auditing tool is to be able to effectively discriminate between easy and challenging examples.
Here, we measure whether V OG is able to do so.
In
Fig.
4, we plot the test-set error of examples bucketed by V OG decile.
For this and the remainder of the experiments, we compute V OG using checkpoints stored from the ﬁrst and last 3 epochs.
Thus, at each point of the x-axis, we are computing the test-set error on the 10% of data whose V OG score falls between each decile.
Note that we plot error, so lower is better.
We show that examples at the lowest percentiles of V OG have far lower error rates.
Mis-classiﬁcation increases with an increase in V OG scores.
Our results are consistent across all datasets, yet more pronounced for the more complex datasets Cifar-100 and ImageNet.
We ascribe this to differences in underlying model complexity.
In Fig.
10 we observe that test-set accuracy on the lowest V OG scored images improves beyond baseline test-set performance.
Our models show improved generalization when restricted to low V OG images.
[Figure]
an increase in VOG scores.
Across all datasets we observe that the group of samples in the toppercentile VOG scores have the highest error rate, i.e. , contains most number of misclassiﬁed samples.
For all datasets, model generalization improves on the bottom 10th percentile relative to the entire dataset.
VOG as an unsupervised auditing tool Many auditing tools to evaluate and understand possible model bias require the presence of labels for protected attributes and underlying variables.
However, this is highly infeasible in real-world settings (Veale & Binns, 2017).
For image and language datasets, the high dimensionality of the problem makes it hard to identify a priori what are underlying variables to be aware of.
Even acquiring the labels for a limited number of attributes protected by law (gender, race) is expensive and/or may be perceived as intrusive leading to noisy or incomplete labels.
One key advantage of VOG is that we show it continues to produce a useful ranking even when the gradients are computed with respect to the predicted label.
In Fig.
5, we include the top and bottom 25 VOG
ImageNet test images using predictions.
We also computed the mean test-error for the predicted VOG distribution, and ﬁnd that it also effectively discriminates between top-10 and bottom-10 examples with 73.6% and 77.5% accuracy respectively (Fig. 11).
Stability of VOG ranking To build trust with the end-user, a key desirable property of any auditing tool is consistency in performance.
We would expect a consistent method to produce a ranking with a closely bounded distribution of scores across independently trained runs of the same model architecture and dataset.
To measure the consistency of the V OG ranking, we train 5 Cifarnetworks from random initialization using the training methodology explained in Sec. 2.1.
Empirically,
Fig.
6a shows that VOG rankings evidence a consistent distribution of test-error at each percentile given the same model and dataset.
L EVERAGING VOG TO UNDERSTAND EARLY AND LATE TRAINING DYNAMICS
Recent work has shown that there are distinct stages to training in deep neural networks (Achille et al., 2017; Jiang et al., 2020; Mangalam & Prabhu, 2019; Faghri et al., 2020).
In our second set of experiments, we explore whether rankings according to V OG are sensitive to the stage of the training process.
Hence, we compute V OG separately for two different stages of the training process,
[Figure]
[Figure]
which we term (1) the Early stage (ﬁrst three epochs), and (2) the Late stage (last three epochs).
Test-set accuracy at the Early stage is 44 .
16% and 51 .
87% for Cifar-10, Cifar-100 and
ImageNet respectively.
In the Late stage it is 89 .
86% and 76 .
69% for Cifar-10, Cifarand ImageNet respectively.
We ﬁnd that there is a noticeable visual difference between the image ranking computed for Early and Late stages of training.
As seen in Fig.
2, for some classes such as apple it appears that V OG scores also capture network color bias present during the Early training stages.
For these classes, the lowest V OG scores over-index on red colored apples.
For ImageNet, we compute the distribution of error in early and late stage and ﬁnd a remarkable ﬂipping point (Fig. 7).
During the early training stage, samples having higher VOG score tend to have a lower error rate as the gradient updates center on easy examples.
This phenomenon is reversed during the late stage of the training where most easy example have been learnt, and updates to the harder examples dominate the the computation of variance.
Hence, samples having high VOG results in higher error rate.
[Figure] [Figure]
R ELATIONSHIP BETWEEN VOG S CORES AND M EMORIZED /OOD E XAMPLES
Recent work has highlighted that deep neural networks produce output probabilities that are uncalibrated (Guo et al., 2017; Kendall & Gal, 2017; Lakshminarayanan et al., 2017; Hendrycks & Gimpel, 2016) and thus cannot be interpreted as a measure of certainty.
If VOG is a useful auditing tool, we expect it to capture model uncertainty even when this is not reﬂected in the end probabilities.
To this end, we consider V OG rankings on two tasks where the network produces highly certain predictions for incorrect or out-of-distribution inputs.
Surfacing examples that require memorization Overparameterized networks have been shown to achieve zero training error by memorizing examples (Zhang et al., 2016; Feldman, 2020).
We explore whether V OG is able to identify examples that require memorization and the rest of the dataset.
To do this, we replicate the general experiment setup of Zhang et al.
and replace 20% of all labels in the training set with random shufﬂed labels.
We re-train the model from random initialization and compute V OG scores at relative intervals across training for all examples in the training set.
Our network achieves 0% training error which would only be possible given successful memorization of the noisy examples with shufﬂed labels.
Is VOG able to discriminate between these memorized examples and the rest of the dataset?
In Fig.8 we plot the box plot distribution of V OG scores for the subset of the data with shufﬂed labels that required memorization beside correct labels.
We ﬁnd that the mean and spread of the examples with the shufﬂed labels are higher when compared to the rest of the dataset.
ImageNet-O experiments We consider ImaageNet-O
(Hendrycks et al., 2019b), an open source curated out-of-distribution (OOD) dataset designed to fool classiﬁers.
ImageNet-O consists of images that are not included in the original 1000 ImageNet classes.
These images were selected with the goal of producing high conﬁdence incorrect ImageNet-1K predictions of labels from within the training distribution.
We are interesting in understanding if V OG can correctly rank Imagenet-O examples as being atypical or out of distribution.
We would expect to observe that ImageNet-O examples would be overrepresented in top percentiles of V OG scores vs the lowest scores.
In Fig.
6b, we see that this is indeed the case.
We plot the count of ImageNet-O image in each percentile as a fraction of the total count of ImageNet-O images ( 2000 images in total).
ImageNet-O images are relatively over-represented at high levels of VOG, with 30% of all images in the top-25th percentile vs 24% in the bottom 25th percentile.
R ELATED W ORK
Our work proposes a method to rank training and test examples by estimated difﬁculty.
Given the size of modern day datasets (Hooker, 2020), this can be a powerful interpretability tool to isolate a tractable subset of examples for human-in-the-loop audting and also aid in curriculum learning (Bengio et al., 2009).
Prior work has proposed different notions of what subset merits surfacing.
Early work by (Zhang, 1992; Bien & Tibshirani, 2012; Kim et al., 2015; Kim et al., 2016) that introduced the notion of prototypes, quintessential examples in the dataset, but did not focus on deep neural networks.
Kim et al.
(2016) also requires assumptions about the statistics of the input distribution.
Gal & Ghahramani (2016) showed how we can use dropouts as a bayesian approximation method for representing model uncertainty in deep learning.
Work by Li et al.
(2017) requires modifying the architecture to preﬁx an autoencoder in order to surface a set of prototypes.
Koh & Liang (2017) proposes inﬂuence functions to identify training points most inﬂuential on a given prediction.
Unlike previous works, we propose a measure that can be extended to rank the entire dataset by estimated difﬁculty (rather than surfacing a prototypical subset).
Additionally, ranking individual samples using methods like Koh & Liang (2017) would be extremely computationally extensive.
Our method does not require modifying the architecture or making any any assumptions about the statistics of the input distribution.
In that sense, our work is complementary to recent work by Jiang et al.
(2020) which proposes a c-score to rank each example by alignment with the training instances,
Hooker et al.
(2019) which classify examples as outliers according to sensitivity to varying model capacity and Carlini et al.
(2019) which consider several different measures to isolate prototypes that could conceivably be extended to rank the entire dataset.
We note that the c-score method proposed by Jiang et al.
(2020) is considerably computationally intensive to compute than V OG as it requires training up to 20 , 000 network replications per data set.
Several of the propotype methods considered by Carlini et al.
(2019) require training ensembles of models, as does the compression sensitivity measure proposed by Hooker et al.
Our method is both different in formulation and can be leveraged using a small number of existing checkpoints saved over the course of training.
C ONCLUSION AND F UTURE W ORK
Our methodology offers one way for humans to better understand the relative difﬁculty of different examples.
One of our key ﬁndings is that V OG is far more challenging to classify for the algorithm and surfaces clusters of images with distinct visual properties.
V OG is straight-forward to compute and can take advantage of current best practices of storing multiple checkpoints over the course of training.
In practice, a domain expert may choose to compute V OG for a class of particular interest which would further reduce the computational cost.
A A PPENDIX
Toy Experiment We generate the clusters for classiﬁcation using scikit-learn library.
We used a 90-10% split for dividing the dataset into the training and testing category.
A linear Multiple Layer
Perceptron (MLP) network with a single hidden layer of 10 neurons was trained using Stochastic
Gradient Descent (SGD) optimizer for 15 epochs.
We divided the training process into three stages: (1) Early [0, 5) epochs, (2) Middle [5, 10), and (3) Late stage [10, 15).
Our trained model achieves a 0% testing error using a linear boundary (Fig. 1a).
Class Level Error Metrics and VOG
Here, we explore whether V OG is able to capture class level differences in difﬁculty.
We compute V OG scores for each image in the test-set of Cifar-10 and
Cifar-100 (both test-sets have 10 , 000 images).
In Fig.
9, we plot the average absolute V OG score for each class against the false negative rate for each class.
We ﬁnd that there is a positive, albeit weak, correlation between the two, classes with higher V OG scores have higher mis-classiﬁcation error rate.
The correlation between these metrics is 0 .
65 and 0 .
59 for Cifar-10 and Cifar-100 respectively.
Given that V OG is computed a per-example level, we ﬁnd it interesting that the aggregate average of
V OG is able to capture class level differences in difﬁculty.
[Equation]
val
[Equation]
[Figure]
[Figure] [Figure]