Defuse: Training More Robust Models through Creation and Correction of Novel Model Errors
Abstract
We typically compute aggregate statistics on held-out test data to assess the generalization of machine learning models.
However, test data is only so comprehensive, and in practice, important cases are often missed.
Thus, the performance of deployed machine learning models can be variable and untrustworthy.
Motivated by these concerns, we develop methods to generate and correct novel model errors beyond those available in the data.
We propose Defuse: a technique that trains a generative model on a classiﬁer’s training dataset and then uses the latent space to generate new samples which are no longer correctly predicted by the classiﬁer.
For instance, given a classiﬁer trained on the MNIST dataset that correctly predicts a test image, Defuse then uses this image to generate new similar images by sampling from the latent space.
Defuse then identiﬁes the images that differ from the label of the original test input.
Defuse enables efﬁcient labeling of these new images, allowing users to re-train a more robust model, thus improving overall model performance.
We evaluate the performance of Defuse on classiﬁers trained on real world datasets and ﬁnd it reveals novel sources of model errors.
Introduction
A key goal of machine learning models is generalization.
We typically measure generalization through performance on a held-out test set.
Ideally, models that score well on held-out test sets should perform the same when deployed.
Indeed, researchers track progress using leader boards that use such aggregate statistics [1, 2].
Nevertheless, it has become increasingly apparent test set accuracy alone does not fully describe the performance of machine learning models.
For instance, statistics like held out test accuracy may overestimate generalization performance [3–5].
Also, test statistics offer little insight into or remedy speciﬁc model failures [6].
Last, test data itself is often limited and may not cover all the possible deployment scenarios [7].
Because metrics on test set data often fail to describe the performance of machine learning systems fully, it is difﬁcult to verify and trust the behavior of machine learning models when deployed.
As a result, researchers have developed a variety of techniques to evaluate models.
Such methods include explanations [8–10], fairness metrics [11, 12], and data set replication [3, 13].
Natural language processing (NLP) has increasingly turned to software engineering inspired behavioral testing tools to ﬁnd errors in models [4].
Though these techniques may help ﬁnd the reasons for misclassiﬁcations (e.g., explanations), they do not ﬁnd novel situations in which the model fails.
Other routes to discover model errors are labor-intensive and may require a high amount of task-speciﬁc expertise (e.g., dataset replication and behavioral testing).
To help remedy these issues, we introduce Defuse: 2 a technique that trains a generative model on a classiﬁer’s training dataset and then uses the latent space to generate new samples which are no longer correctly predicted by the classiﬁer.
For instance, given a classiﬁer trained on the MNIST dataset that correctly predicts a test image showing digit 8, Defuse then uses this image to generate new similar images by sampling from the latent space and these new images are then mistaken by the classiﬁer for instance as digit 3.
To do this, Defuse identiﬁes regions in the latent space of the generative model associated with different types of incorrect classiﬁer predictions and learns a secondary generative model of these regions.
This process allows direct sampling of novel model misclassiﬁcations, which is useful for model understanding.
We also demonstrate that similar images are usually in close proximity within the latent space and Defuse leverages this to efﬁciently label those mis-classiﬁcations.
For example, we run Defuse on a classiﬁer trained on MNIST and provide an overview in ﬁgure 1.
Defuse works in three steps:
Identiﬁcation (ﬁrst pane in ﬁgure 1): Defuse generates new images and identiﬁes the ones that are incorrectly predicted by the classiﬁer.
By doing so, Defuse identiﬁes regions in the latent space that are associated with errors.
The red number in the upper right-hand corner of the image is the classiﬁer’s prediction.
For instance, the generated images showing digit 0 are incorrectly classiﬁed as digit 9. 2.
Distillation (second pane in ﬁgure 1): Next, the method performs the distillation step.
A generative clustering model groups together the latent codes of the images from the previous step.
For instance, Defuse groups together generated images showing digit 8 that are incorrectly classiﬁed as 3. 3.
Correction (third pane in ﬁgure 1): The previous set of clusters are then annotated by labelers.
Defuse then runs the correction step using both the annotator labeled generated images and the original training data.
After the correction step the model correctly classiﬁes the generated images (third pane in ﬁgure 1)
In our benchmarks, the model maintains its predictive performance, scoring 99.1% accuracy after tuning.
We see that Defuse serves as a general-purpose method to both discover and correct model errors.
Notation and Background
In this section, we establish notation and background.
Though it is conceivable to apply Defuse to many domains, we focus on images in this work.
Notation Let f : R N → [0 , 1] C denote a classiﬁer that accepts a data point x ∈ X , where X is a set of legitimate images.
The classiﬁer f returns the probability that x belongs to class c ∈ { 1 , ..., C } .
Next, assume f is trained on a data set D consisting of d tuples ( x, y ) containing data point x and ground truth label y using loss function L . Finally, suppose there exists an oracle o : x ∈ X → { 1 , ..., C } that outputs a label for x .
Variational Autoencoders (VAEs)
In order to ﬁnd identiﬁed classiﬁer mistakes, it is necessary to model the set of legitimate images X .
We use a VAE to create such a model.
A VAE is composed of an encoder and a decoder neural networks.
These networks are used to model the relationship between data x and latent factors z ∈ R K .
Where x is generated by some ground truth latent factors v ∈ R M , we wish to train a model such that the learned generative factors closely resemble the true factors: p ( x | v )
≈ p ( x | z ) .
In order to train such a model, we employ the β -VAE [14].
This technique produces encoder q φ ( z | x ) that maps from the data and latent codes and decoder p θ ( x | z ) that maps from codes to data.
Methods
In this section, we introduce Defuse.
We describe the three main steps in the method.
3.1 Identiﬁcation
Algorithm 1 Identiﬁcation 1: Identify: Classiﬁer f , Decoder p , Encoder q , Data point x , Label y , Beta parameters ( a , b ) 2: ψ := {} { Initialize misclassified images set } 3: µ, σ :=
q φ ( x ) { Compute encoded data point } 4: for i ∈ { 1 , ..., Q } do 5: (cid:15) :=
[ Beta ( a, b ) 1 , ..., Beta ( a, b ) M ] { Draw perturbations } 6: x decoded := p θ ( µ + (cid:15) ) { Compute decoded perturbed data points } 7: if y (cid:54) =
f ( x decoded ) then 8: ψ := ψ ∪ x decoded { Store the data point if its likely misclassified } 9: end if 10: end for 11: Return likely misclassiﬁed data points ψ
This section describes the identiﬁcation step in Defuse (ﬁrst pane in ﬁgure 1).
The aim of the identiﬁcation step is to generate many misclassiﬁed examples for a model.
To generate these examples, we note previous works observe that slight perturbations to images in the latent space of generative models which lead to different classiﬁcations than the original image tend be misclassiﬁed [15, 16].
Following this observation, we encode all the images from the training data.
We perturb the latent codes with a small amount of noise drawn from a Beta distribution.
We use a Beta distribution so that it is possible to control the shape of the applied noise.
We save instances that are classiﬁed differently from ground truth by the model f when decoded.
We denote the set of these instances across all instances in a dataset as ψ .
We provide pseudocode of the algorithm for generating identiﬁed classiﬁer mistakes for a single instance x in algorithm 1.
3.2 Distillation
Next, we describe how to group the individual mistakes into high level errors.
We denote these high level errors as misclassiﬁcation regions and deﬁne them formally.
Misclassiﬁcation regions Let z ∈ R K be the latent codes corresponding to image x ∈ X and q φ ( · ) : x → z be the encoder mapping the relationship between images and latent codes.
Deﬁnition 3.1.
Misclassiﬁcation region.
Given a constant (cid:15) > 0 , vector norm ||·|| , model f , and point z (cid:48) , a misclassiﬁcation regions is a set of images A R = { x ∈ X | (cid:15) >
|| q φ ( x ) − z (cid:48)
||
∧ o ( x ) (cid:54) =
f ( x ) } .
Because the latent space of the VAE tends to take on Gaussian form due to the prior, we can use euclidean distance to deﬁne these regions.
If we were to deﬁne misclassiﬁcation regions on the original data manifold, we may need a much more complex distance function.
Because it is likely too
Deﬁnition 3.2.
Relaxed misclassiﬁcation region.
Given a constant (cid:15) > 0 , vector norm || · || , point z (cid:48) , model f , and threshold ρ , a relaxed misclassiﬁcation regions is a set of images A f = { x ∈ X | (cid:15) >
|| q φ ( x ) − z (cid:48) ||} such that |{ x ∈
A f | o ( x ) (cid:54) =
f ( x ) }| / |A
f | > ρ .
In this work, we adopt the latter deﬁnition of misclassiﬁcation regions.
To concretize misclassiﬁcation regions and provide evidence for their existence, we continue our MNIST example from ﬁgure 1.
We plot the t-SNE embeddings of the latent codes of 10000 images from the training set andidentiﬁed classiﬁer mistakes created during the identiﬁcation step in ﬁgure 2 (details of how we generate identiﬁed classiﬁer mistakes in section 3.1).
We see that the identiﬁed classiﬁer mistakes are from similar regions in the latent space.
3.2.1 Distilling misclassiﬁcation regions
Based on our deﬁnition of misclassiﬁcation regions, we describe a general procedure for learning them.
We do so through clustering the latent codes of the identiﬁed classiﬁer mistakes ψ in order to diagnose misclassiﬁcation regions (second pane of ﬁgure 1).
We require our clustering method to (1) infer the correct number of clusters from the data, and (2) be capable of generating instances of each cluster.
We need to infer the number of clusters from the data because the number of misclassiﬁcation regions is unknown ahead of time.
Further, we must generate many instances from each cluster so that we have enough data to ﬁnetune on to correct the faulty model behavior.
Also, generating many failure instances enables model designers to see numerous examples from the misclassiﬁcation regions, which encourages understanding the model failure modes.
Though any such clustering method under this description is compatible with distillation, we use a Gaussian mixture model (GMM) with the
Dirichlet process prior.
We use the Dirichlet process because it describes the clustering problem where the number of mixtures is unknown beforehand, fulﬁlling our ﬁrst criteria [17].
Additionally, because the model is generative, we can sample new instances, satisfying our second criteria.
In practice, we use the truncated stick-breaking construction of the Dirichlet process, where K is the upper bound of the number of mixtures.
The truncated stick-breaking construction simpliﬁes inference making computation more efﬁcient [17].
The method outputs a set of clusters θ j = ( µ j , σ j , π j ) where j ∈ { 1 , ..., K } .
The parameters µ and σ describe the mean and variance of a multivariate normal distribution and π indicates the cluster weight.
To perform inference on the model, we employ expectation maximization (EM) described in [18] and use the implementation provided in [19].
Once we run EM and determine the parameter values, we throw away cluster components that are not used by the model.
We ﬁx some small (cid:15) and deﬁne the set of misclassiﬁcation regions Λ generated at the distillation step as: Λ := { ( µ j , Σ j , π j ) | π j > (cid:15) } .
3.3 Correction
This section describes the procedure for labeling the misclassiﬁcation regions and ﬁnetuning the model to ﬁx the classiﬁer errors.
Labeling First, an annotator assigns the correct label to the misclassiﬁcation regions.
For each misclassiﬁcation regions identiﬁed in Λ , we sample Q latent codes from z ∼ N ( µ j , τ · σ j ) .
Here, τ ∈ R is a hyperparameter that controls sample diversity from the misclassiﬁcation regions.
Because it could be possible for multiple ground truth classes to be present in a misclassiﬁcation region, we set this parameter tight enough such that the sampled instances are from the same class.
We reconstruct the latent codes using the decoder p θ ( x | z ) .
Next, an annotator reviews the reconstructed instances from the scenario and decides whether the scenario constitutes a model failure.
If so, the annotator assigns the correct label to all of the instances.
The correct label constitutes a single label for all of the instances generated from the scenario.
We repeat this process for each of the scenarios identiﬁed in Λ and produce a dataset of failure instances D f .
Pseudocode for the procedure is given in algorithmin appendix A.
Finetuning We ﬁnetune on the training data with an additional regularization term to ﬁx the classiﬁer performance on the misclassiﬁcation regions.
The regularization term is the cross-entropy loss between the identiﬁed misclassiﬁcation regions and the annotator label.
Where L is the crossentropy loss applied to the failure instances D f and λ is the hyperparameter for the regularization term, we optimize the following objective using gradient descent,
[Equation]
This objective encourages the model to maintain its predictive performance on the original training data while encouraging the model to predict the failure instances correctly.
The regularization term λ controls the pressure applied to the model to classify the failure instances correctly.
Dataset # Scenarios
Validation
Test
M.R.
Before Finetuning
MNIST - - 98.
SVHN93.
German Signs98.
Identiﬁed
MNIST - - 99.
Mistakes
SVHN93.
German Signs - - - -
Defuse
MNIST99.
SVHN 93.
German Signs 98.
Experiments 4.1 Setup
Datasets We evaluate Defuse on three datasets: 3 MNIST [20], the German Trafﬁc Signs dataset [21], and the Street view house numbers dataset (SVHN) [22]. MNIST consists of 60 , 000 32 Xhandwritten digits for training and 10 , 000 digits for testing. The images are labeled corresponding to the digits 0 − 9 . The German trafﬁc signs data set includes 26 , 640 training and 12 , 630 testing images of size 128 X 128 .
We randomly split the testing data in half to produce a validation and testing set.
The images are labeled from 43 different classes to indicate the type of trafﬁc signs.
The SVHN data set consists of 73 , 257 training and 26 , 032 testing images of size 32 X 32 .
The images include digits of house numbers from Google streetview with labels 0 − 9 .
We split the testing set in half to produce a validation and testing set.
Models On MNIST, we train a CNN scoring 98 .
3% test set accuracy following the architecture from [23].
On German trafﬁc signs and SVHN, we ﬁnetune a Resnet18 model pretrained on ImageNet [24].
The German signs and SVHM models score 98 .
7% and 93 .
2% test accuracy respectively.
We train a β -VAE the on the training data set to model the set of legitimate images in Defuse.
We use an
Amazon EC2 P3 instance with a single NVIDIA Tesla V100 GPU for training.
We follow similar architectures to [14].
We set the size of the latent dimension z to 10 for MNIST/SVHN and 15 for
German signs.
We provide our β -VAE architectures in appendix B.
Defuse We run Defuse on each classiﬁer.
In the identiﬁcation step, we ﬁx the parameters of the Beta distribution noise a and b to a = b = 50 . 0 for MNIST and a = b = 75 . 0 for SVHN and German signs.
We found these parameters were good choices because they produce a minimal amount of perturbation noise, making the decoded instances slightly different from the original instances.
During distillation, we set the upper bound on the number of components K to 100 .
We generally found the actual number of clusters to be much lower than this level.
Thus, this serves as an appropriate upper bound.
We also ﬁxed the weight threshold for clusters (cid:15) to 0 .
01 during distillation to remove clusters with very low weighting.
We also randomly downsample the number of identiﬁed classiﬁer mistakes to 50 , 000 to make the GMM more efﬁcient.
We sample ﬁnetuning and testing sets consisting of 256 images each from every misclassiﬁcation region for correction.
We found empirically that this number of samples is appropriate because it captures the breadth of possible images in the scenario.
We use the ﬁnetuning set as the set of failure instances D f .
We used the test set as held out data to evaluate classiﬁer performance on the misclassiﬁcation regions after correction.
During sampling, we ﬁx the sample diversity τ to 0 .
5 for MNIST and 0 .
01 for SVHN and German signs because the samples from each of the misclassiﬁcation regions appear to be in the same class using these values.
During correction, we ﬁnetune over a range of λ ’s to ﬁnd the best balance between training and
[Figure]
We assess both test set accuracy and accuracy on the test misclassiﬁcation region data ﬁnetuning over a range of λ ’s and plot the trade off.
There is an optimal λ for each classiﬁer where test set and misclassiﬁcation region accuracy are both high.
This result conﬁrms that the correction step in
Defuse adequately balances both generalization and accuracy on the misclassiﬁcation regions .
misclassiﬁcation region data. We use 3 epochs for MNIST and 5 for both SVHN and German Signs because training converged within this amount of epochs.
During ﬁnetuning, we select the model for each λ according to the highest training set accuracy for MNIST or validation set accuracy for
SVHM and German trafﬁc signs at the end of each ﬁnetuning epoch.
We select the best model overall as the highest training or validation performance over all λ ’s.
Annotator Labeling Because Defuse requires human supervision, we use Amazon Sagemaker
Ground Truth human workers to both determine whether clusters generated in the distillation step are misclassiﬁcation regions and to generate their correct label.
To determine whether clusters are misclassiﬁcation regions, we sample 10 instances from each cluster in the distillation step.
It is usually apparent the classiﬁer disagrees with many of the ground truth labels within 10 instances, and thus it is appropriate to label the cluster as a misclassiﬁcation region.
To reduce noise in the annotation process, we assign the same image to 5 different workers and take the majority annotated label as ground truth.
The workers label the images using an interface that includes a single image and the possible labels for that task.
We additionally instruct workers to select “None of the above” if the image does not belong to any class and discard these labels.
For instance, the MNIST interface includes a single image and buttons for the digits 0 − 9 along with a “None of the above” button.
We provide a screenshot of this interface in ﬁgure 15.
If more than half (i.e. setting ρ = 0 . 5 ) of worker labeled instances disagree with the classiﬁer predictions on the 10 instances, we call the cluster a misclassiﬁcation region.
We chose ρ = 0 . 5 because clusters are highly dense with incorrect predictions at this level, making them useful for both understanding model failures and worthwhile for correction.
We take the majority prediction over each of the 10 ground truth labels as the label for the misclassiﬁcation region.
As an exception, annotating the German trafﬁc signs data requires speciﬁc knowledge of trafﬁc signs.
The German trafﬁc signs data ranges across 43 different types of trafﬁc signs.
It is not reasonable to assume annotators have enough familiarity with this data and can label it accurately.
For this data set, we, the authors, reviewed the distilled clusters and determined which clusters constituted misclassiﬁcation regions.
We labeled the clusters with more than half of the instances misclassiﬁed as misclassiﬁcation regions.
Though this procedure is less rigorous, the results still provide good insight into the model bugs discovered by Defuse.
4.2 Illustrative misclassiﬁcation region examples
We demonstrate Defuse’s potential to identify critical model bugs.
We review misclassiﬁcation regions from three datasets we consider.
Defuse returns 19 misclassiﬁcation regions for MNIST, 6 for
SVHN, and 8 for German signs.
We provide samples from three misclassiﬁcation regions for each dataset in ﬁgure 3.
The misclassiﬁcation regions include numerous mislabeled examples of similar style.
For example, in MNIST, the misclassiﬁcation region in the upper left-hand corner of ﬁgureincludes a similar style of incorrectly predicted 4 ’s. The misclassiﬁcation regions generally include “corner case” images.
These images are challenging to classify and thus highly insightful from a debugging perspective.
For instance, the misclassiﬁed 6 ’s are relatively thin, making them appear like 1 ’s in some cases.
There are similar trends in SVHN and German Signs.
In SVHN, the model misclassiﬁes particular types of 5 ’s and 8 ’s. The same is true in German signs, where the model [Figure] predicts styles of 50 km/h and 30 km/h signs incorrectly.
We provide additional samples from other misclassiﬁcation regions in appendix D. These results demonstrate Defuse uncovers signiﬁcant and insightful model bugs.
4.3 Novelty of the errors
We expect Defuse to ﬁnd novel model misclassiﬁcations beyond those revealed by the available data.
Thus, it is critical to evaluate whether the errors produced by Defuse are the same as those already in the training data.
We compare the similarity of the errors proposed by Defuse (the misclassiﬁcation region data) and the misclassiﬁed training data.
We perform this analysis on MNIST.
We chooseimages from the misclassiﬁcation regions and ﬁnd the nearest neighbor in the misclassiﬁed training data according to (cid:96) 2 distance on the images.
We provide the results in ﬁgure 6.
We see that the data in the misclassiﬁcation regions reveal different types of errors than the training set.
Interestingly, though some of the images are quite similar, they are predicted differently by the model.
This result indicates the misclassiﬁcation regions reveal new model failures.
These results demonstrate Defuse can be used to reveal novel sources of model error.
4.4 Correcting misclassiﬁcation regions
After running correction, classiﬁer accuracy should improve on the misclassiﬁcation region data indicating we have corrected the bugs discovered in earlier steps.
Also, the classiﬁer accuracy on the test set should stay at a similar level or improve, indicating that the model generalization according to the test set is still strong.
We show that this is the case using Defuse.
We assess accuracy on both the misclassiﬁcation region test data and the original test set after performing correction.
We compare Defuse against ﬁnetuning only on the images generated in the identiﬁcation step that are labeled as classiﬁer mistakes by annotators.
We expect this baseline to be reasonable because related works that focus on robustness to classic adversarial attacks demonstrate the effectiveness of tuning directly on the adversarial examples [25].
We ﬁnetune on the identiﬁed classiﬁer mistakes sweeping over a range of different λ ’s and taking the best model as described in section 4.1.
We use this baseline for MNIST and SVHN but not German Signs because we do not have ground truth labels for identiﬁed classiﬁer mistakes.
We provide an overview of the models before ﬁnetuning, ﬁnetuning with the identiﬁed classiﬁer mistakes, and using Defuse in ﬁgure 4.
Defuse scores highly on the misclassiﬁcation region data after correction compared to before ﬁnetuning.
There is only a marginal improvement in the baseline.
These results indicate Defuse corrects the faulty model performance on the misclassiﬁcation regions.
Also, these results show the clustering step in Defuse is critical to its success.
We see this because ﬁnetuning on the identiﬁed classiﬁer mistakes performs worse than ﬁnetuning on the misclassiﬁcation regions.
Last, there are minor effects on test set performance during ﬁnetuning, demonstrating Defuse does not harm generalization according to the test set.
Further, we plot the relationship between test set accuracy and misclassiﬁcation region test accuracy in ﬁgure 5 varying over λ .
There is an appropriate λ for each model where test set accuracy and accuracy on the misclassiﬁcation regions are both high.
Overall, these results indicate the correction step in
Defuse is highly effective at correcting the errors discovered during identiﬁcation and distillation. Dataset [Table] [Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] 4.5 Annotator Agreement Because we rely on annotators to provide the ground truth labels for the identiﬁed classiﬁer mistakes, we investigate the agreement between the annotators during labeling.
The annotators should agree on the labels for the identiﬁed classiﬁer mistakes.
Agreement indicates we have high conﬁdence our evaluation is based on accurately labeled data.
We evaluate the annotator agreement by assessing the percent of annotators that voted for the majority label prediction for a single instance.
This metric will be high when the annotators agree and low when only a few annotators constitute the majority vote.
Further, we calculate the annotator agreement for every annotated instance.
We provide the annotator agreement on MNIST and SVHN in ﬁgure 7 broken down into misclassiﬁcation region data, non-misclassiﬁcation region data, and their combination.
Interestingly, the misclassiﬁcation region data has slightly lower annotator agreement indicating these tend to be more ambiguous examples.
Further, there is lower agreement on SVHN than MNIST, likely because this data is more complex.
All in all, there is generally high annotator agreement across all the data.
Related Work
A number of related approaches for improving classiﬁer performance use data created from generative models — mostly generative adversarial networks (GANs) [26–29].
These methods use GANs to generate instances from classes that are underrepresented in the training data to improve generalization performance.
Additional methods use generative models for semi-supervised learning [30–33].
Though these methods are similar in nature to the correction step of our work, a key difference is
Defuse focuses on summarizing and presenting high level model failures.
Also, [34] provide a system to debug data generated from a GAN when the training set may be inaccurate.
Though similar, we ultimately use a generative model to debug a classiﬁer and do not focus on the generative model itself.
Last, similar to [16], [15], [35] provide a method to generate highly conﬁdent misclassiﬁed instances.
One aspect of our work looks at improving performance on identiﬁed classiﬁer mistakes.
Thus, there are similarities between our work and methods that improve robustness to adversarial attacks.
Similar to Defuse, several techniques demonstrate that tuning on additional data helps improve classic adversarial robustness.
[36] demonstrate robustness is improved with the addition of unlabeled data during training.
Also, [37] show directly training on the adversarial examples improves robustness. [38] characterize a trade-off between robustness and accuracy in perturbation based data augmentations during adversarial training.
Because we train with on manifold data created by generative models, there is not so much of a trade-off between robustness and accuracy.
We ﬁnd that we can achieve high performance on the identiﬁed classiﬁer mistakes with minimal change to test accuracy.
Though related, our work demonstrates that robustness to naturally occurring adversarial examples show different robustness dynamics than classic adversarial examples.
Related to debugging models, [39] focus on model assertions that ﬂag failures during production.
Also, [37, 40] investigate debugging the training set for incorrectly labeled instances.
We focus on preemptively identifying model bugs and do not focus on incorrectly labeled test set instances.
Additionally, [4] propose a set of behavioral testing tools that help model designers ﬁnd bugs in NLP models.
This technique requires a high level of supervision and thus might not be appropraite in some settings.
Last, [41] provide a technique to debug neural networks through perturbing data inputs with various types of noise.
Conclusion
In this paper, we present Defuse: a method that summarizes and corrects high level model errors.
Our experimental results show that Defuse identiﬁes critical model issues for classiﬁers with real world impacts (i e. trafﬁc sign classiﬁcation) and verify our results using ground truth annotator labels.
We also demonstrate that Defuse successfully resolves these issues.
A potential direction for future research is to explore directly optimizing for the misclassiﬁcation regions, instead of identifying individual example and clustering.
Acknowledgments
This work was supported by Amazon AWS.
DS would also like to acknowledge support from the HPI Research Center in Machine Learning and Data Science at UC Irvine.
A Defuse Psuedo Code
In algorithm 3, Correct ( · ) and Label ( · ) are the steps where the annotator decides if the scenario warrants correction and the annotator label for the misclassiﬁcation region.
Algorithm
1: Identify: f, p, q, x, y, a, b 2: ψ := {} 3: µ, σ :=
q φ ( x ) 4: for i ∈ { 1 , ..., Q } do 5: (cid:15) :=
[ Beta ( a, b ) 1 , 6: ..., Beta ( a, b ) M ] 7: x decoded := p θ ( µ + (cid:15) ) 8: if y (cid:54) =
f ( x decoded ) then 9: ψ := ψ ∪ x decoded 10: end if 11: end for 12: Return ψ
Algorithm
Label Scenarios Q, Λ , p, q, τ
D f := {} for ( µ, σ, π ) ∈ Λ do
X d := {} for i ∈ { 1 , .., Q } do
X d := X d ∪ q ψ ( N ( µ, τ · σ )) end for if Correct ( X d ) then
D f :=
D f ∪ { X d , Label ( X d ) } end if end for
Return (cid:83) D f
B Training details B.1 GMM details
In all experiments, we use the implementation of Gaussian mixture model with dirichlet process prior from [[19]].
We run our experiments with the default parameters and full component covariance.
B.2 MNIST details
Model details We train a CNN on the MNIST data set using the architecture in ﬁgure 8.
We used the Adadelta optimizer with the learning rate set to 1 .
We trained for 5 epochs with a batch size of 64 .
Architecture4x4conv.,64 ReLU stride4x4 conv., 64 ReLU stride4x4 conv., 64 ReLU stride4x4 conv., 64 ReLU stride
Fully connected
[Figure]
β -VAE training details We train a β -VAE on MNIST using the architectures in ﬁgure 9 and 10.
We set β to 4 .
We trained for 800 epochs using the Adam optimizer with a learning rate of 0 .
001 , a minibatch size of 2048 , and β set to 0 .
We also applied a linear annealing schedule on the
KL-Divergence for 500 optimization steps.
We set z to have 10 dimensions.
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Figure]
Identiﬁcation We performed identiﬁcation with Q set to 500 .
We set a and b both to 50 .
We ran identiﬁcation over the entire training set.
Last, we limited the max allowable size of ψ to 100 .
Distillation We ran the distillation step setting K , the upper bound on the number of mixtures, to 100 .
We ﬁxed (cid:15) to 0 .
01 and discarded clusters with mixing proportions less than this value.
This left 44 possible scenarios.
We set τ to 0 .
5 during review.
We used Amazon Sagemaker Ground Truth to determine misclassiﬁcation regions and labels.
The labeling procedure is described in section 4.1.
This produced 19 misclassiﬁcation regions.
Correction We sampled 256 images from each of the misclassiﬁcation regions for both ﬁnetuning and testing.
We ﬁnetuned with minibatch size of 256 , the Adam optimizer, and learning rate set to 0 .
We swept over a range of correction regularization λ ’s consisting of [1 e − 10 , 1 e − 9 , 1 e − 8 , 1 e − 7 , 1 e − 6 , 1 e − 5 , 1 e − 4 , 1 e − 3 , 1 e − 2 , 1 e − 1 , 1 , 2 , 5 , 10 , 20 , 100 , 1000] and ﬁnetuned for 3 epochs on each.
B.3 German Signs Dataset Details
Dataset The data consists of 26640 training images and 12630 testing images consisting ofdifferent types of trafﬁc signs.
We randomly split the testing data in half to produce 6315 testing and validation images.
Additionally, we resize the images to 128 x 128 pixels.
f
We ﬁne-tuned the ResNet18 model for 20 epochs using Adam with the cross entropy loss, learning rate of 0 .
001 , batch size of 256 on the training data set, and assessed the validation accuracy at the end of each epoch.
We saved the model with the highest validation accuracy. β -VAE training details We trained for 800 epochs using the Adam optimizer with a learning rate of 0 .
001 , a minibatch size of 2048 , and β set to 4 .
We also applied a linear annealing schedule on the
KL-Divergence for 500 optimization steps.
We set z to have 15 dimensions.
[Figure]
Identiﬁcation We performed identiﬁcation with Q set to 100 .
We set a and b both to 75 .
Distillation We ran the distillation step setting K to 100 .
We ﬁxed (cid:15) to 0 .
01 and discarded clusters with mixing proportions less than this value.
This left 38 possible scenarios.
We set τ to 0 .
01 during review.
We determined 8 of these scenarios were particularly concerning.
[Figure]
Correction We ﬁnetuned with minibatch size of 256 , the Adam optimizer, and learning rate set to 0 .
We swept over a range of correction regularization λ ’s consisting of [1 e − 10 , 1 e − 9 , 1 e − 8 , 1 e − 7 , 1 e − 6 , 1 e − 5 , 1 e − 4 , 1 e − 3 , 1 e − 2 , 1 e − 1 , 1 , 2 , 5 , 10 , 20 , 100 , 1000] and ﬁnetuned for 5 epochs on each.
B.4 SVHN details
Dataset The data set consists of 73257 training and 26032 testing images.
We also randomly split the testing data to create a validation data set.
Thus, the ﬁnal validation and testing set correspond to 13016 images each.
Classiﬁer f
We ﬁne tuned for 10 epochs using the Adam optimizer, learning rate set to 0 .
001 , and a batch size of 2048 .
We chose the model which scored the best validation accuracy when measured at the end of each epoch. β -VAE training details We trained the β -VAE for 400 epochs using the Adam optimizer, learning rate 0 .
001 , and minibatch size of 2048 .
We set β to 4 and applied a linear annealing schedule on the
Kl-Divergence for 5000 optimization steps.
We set z to have 10 dimensions.
[Figure]
[Figure]
Identiﬁcation We set Q to 100 .
We also set the maximum size of ψ to 10 .
We set a and b to 75 .
Distillation We set K to 100 .
We ﬁxed (cid:15) to 0 .
The distillation step identiﬁed 32 plausible misclassiﬁcation regions.
The annotators deemed 6 of these to be misclassiﬁcation regions.
We set τ to 0 .
01 during review.
Correction We set the minibatch size of 2048 , the Adam optimizer, and learning rate set to 0 . 001 .
We considered a range of λ ’s:
[1 e − 10 , 1 e − 9 , 1 e − 8 , 1 e − 7 , 1 e − 6 , 1 e − 5 , 1 e − 4 , 1 e − 3 , 1 e − 2 , 1 e − 1 , 1 , 2 , 5 , 10 , 20 , 100 , 1000] .
We ﬁnetuned for 5 epochs.
B.5 t-SNE example details
We run t-SNE on 10 , 000 examples from the training data and 516 identiﬁed classiﬁer mistakes setting perplexity to 30 .
For the sake of clarity, we do not include outliers from the identiﬁed classiﬁer mistakes.
Namely, we only include identiﬁed classiﬁer mistakes with > 1% probabilitdefuse-iclr clusters.
C Annotator interface
We provide a screenshot of the annotator interface in ﬁgure 15.
D Additional experimental results D.1 Additional samples from MNIST misclassiﬁcation regions
We provide additional examples from 10 randomly selected (no cherry picking)
MNIST misclassiﬁcation regions.
We include the annotator consensus label for each misclassiﬁcation region.
D.2 Additional samples from German signs misclassiﬁcation regions
We provide samples from all of the German signs misclassiﬁcation regions.
We provide the names of the class labels in ﬁgure 26.
For each misclassiﬁcation region, we indicate our assigned class label in the caption and the classiﬁer predictions in the upper right hand corner of the image.
ClassId
SignName
Speed limit (20km/h)
Speed limit (30km/h)
Speed limit (50km/h)
Speed limit (60km/h)
Speed limit (70km/h)
Speed limit (80km/h)
End of speed limit (80km/h)
Speed limit (100km/h)
Speed limit (120km/h)
D.3 Additional samples from SVHN misclassiﬁcation regions
We provide additional samples from each of the SVHN misclassiﬁcation regions.
The digit in the upper left hand corner is the classiﬁer predicted label.
The caption includes the Ground Truth worker labels.