D EFUSE : D EBUGGING C LASSIFIERS T HROUGH D ISTILLING U NRESTRICTED A DVERSARIAL E XAMPLES
Anonymous authors
Paper under double-blind review
A BSTRACT
With the greater proliferation of machine learning models, the imperative of diagnosing and correcting bugs in models has become increasingly clear.
As a route to better discover and ﬁx model bugs, we propose failure scenarios : regions on the data manifold that are incorrectly classiﬁed by a model.
We propose an end-to-end debugging framework called Defuse to use these regions for ﬁxing faulty classiﬁer predictions.
The Defuse framework works in three steps.
First, Defuse identiﬁes many unrestricted adversarial examples—naturally occurring instances that are misclassiﬁed—using a generative model.
Next, the procedure distills the misclassiﬁed data using clustering into failure scenarios.
Last, the method corrects model behavior on the distilled scenarios through an optimization based approach.
We illustrate the utility of our framework on a variety of image data sets.
We ﬁnd that Defuse identiﬁes and resolves concerning predictions while maintaining model generalization.
I NTRODUCTION
Debugging machine learning (ML) models is a critical part of the ML development life cycle.
Uncovering bugs helps ML developers make important decisions about both development and deployment.
In practice, much of debugging uses aggregate test statistics (like those in leader board style challenges [Rajpurkar et al.
and continuous evaluation and monitoring post deployment [Liberty et al.
Simon (2019)].
However, additional issues arise with over-reliance on test statistics.
For instance, aggregate statistics like held out test accuracy are known to overestimate generalization performance [Recht et al.
Further, statistics offer little insight nor remedy for speciﬁc model failures [Ribeiro et al.
; Wu et al.
Last, reactive debugging of failures as they occur in production does little to mitigate harmful user experiences [La Fors et al.
Several techniques exist for identifying undesirable behavior in machine learning models.
These methods include explanations [Ribeiro et al.
; Slack et al.
(2020b);
Lakkaraju et al.
Lundberg & Lee (2017)], fairness metrics [Feldman et al. (2015), Slack et al.
(2020a)], data set replication [Recht et al.
Engstrom et al.
(2020)], and behavioral testing tools
[Ribeiro et al.
However, these techniques do not provide methods to remedy model bugs or require a high level of human supervision.
To enable model designers to discover and correct model bugs beyond aggregate test statistics, we analyze unrestricted adversarial examples : instances on the data manifold that are misclassiﬁed
[Song et al.
We identify model bugs through diagnosing common patterns in unrestricted adversarial examples.
In this work, we propose Defuse: a technique for debugging classifiers through distillingunrestricted adversarial examples.
Defuse works in three steps.
First, Defuse identiﬁes unrestricted adversarial examples by making small, semantically meaningful changes to input data using a variational autoencoder (VAE).
If the classiﬁer prediction deviates from the ground truth label on the altered instance, it returns the data instance as a potential model failure.
This method employs similar techniques from [Zhao et al.
Namely, small perturbations in the latent space of generative models can produce images that are misclassiﬁed.
Second, Defuse distills the changes through clustering on the unrestricted adversarial example’s latent codes.
In this way, Defuse diagnoses regions in the latent space that are problematic for the classiﬁer.
This method produces a set of clusters in the latent space where it is likely to ﬁnd misclassiﬁed data.
We call these localities failure scenarios .
An annotator reviews the failure scenarios and assigns the correct label— one label per scenario.
Third, Defuse corrects the model behavior on the discovered failure scenarios through optimization.
Because we use a generative clustering model to describe the failure scenarios, we sample many unrestricted adversarial examples and ﬁnetune to ﬁx the classiﬁer. Critically, failure scenarios are highly useful for model debugging because they reveal high level patterns in the way the model fails.
By understanding these consistent trends in model failures, model designers can more effectively understand problematic deployment scenarios for their models.
To illustrate the usefulness of failure scenarios, we run Defuse on a classiﬁer trained on MNIST and provide an overview in ﬁgure 1.
In the identiﬁcation step (ﬁrst pane in ﬁgure 1), Defuse generates unrestricted adversarial examples for the model.
The red number in the upper right hand corner of the image is the classiﬁer’s prediction.
Although the classiﬁer achieves high test set performance, we ﬁnd naturally occurring examples that are classiﬁed incorrectly.
Next, the method performs the distillation step (second pane in ﬁgure 1).
The clustering model groups together similar failures for annotator labeling.
We see that similar mistakes are grouped together.
For instance, Defuse groups together a similar style of incorrectly classiﬁed eights in the ﬁrst row of the second pane in ﬁgure 1.
Next, Defuse receives annotator labels for each of the clusters.
Last, we run the correction step using both the annotator labeled data and the original training data.
We see that the model correctly classiﬁes the images (third pane in ﬁgure 1).
Importantly, the model maintains its predictive performance, scoring 99 .
1% accuracy after tuning.
We see that Defuse enables model designers to both discover and correct naturally occurring model failures.
We provide the necessary background in Defuse ( § 2).
Next, we detail the three steps in Defuse: identiﬁcation , distillation , and correction ( § 3).
We then demonstrate the usefulness of Defuse on three image data sets: MNIST [LeCun et al.
(2010)], the German trafﬁc signs data set [Stallkamp et al.
(2011)], and the Street view house numbers data set [Netzer et al.
(2011)], and ﬁnd that Defuse discovers and resolves critical bugs in high performance classiﬁers trained on these datasets ( § 4).
N OTATION AND B ACKGROUND
In this section, we establish notation and background on unrestricted adversarial examples.
Though unrestricted adversarial examples can be found in many domains, we focus on Defuse applied to image classiﬁcation.
Unrestricted adversarial examples Let f : R N !
[0 , 1] C denote a classiﬁer that accepts a data point x 2 X , where X is the set of legitimate images.
The classiﬁer f returns the probability that x belongs to class c 2 { 1 , ..., C } .
Next, assume f is trained on a data set D consisting of d tuples ( x, y ) containing data point x and ground truth label y using loss function L .
Finally, suppose there exists an oracle o : x 2 X ! { 1 , ..., C } that outputs a label for x .
We deﬁne unrestricted adversarial examples as the set
[Equation] et
[Equation] [Equation] Variational Autoencoders (VAEs)
In order to discover unrestricted adversarial examples, it is necessary to model the set of legitimate images.
We use a VAE to create such a model.
A VAE is composed of an encoder and a decoder neural networks.
These networks are used to model the relationship between data x and latent factors z 2 R K .
Where x is generated by some ground truth latent factors v 2 R M , we wish to train a model such that the learned generative factors closely resemble the true factors: p ( x | v ) ⇡ p ( x | z ) .
In order to train such a model, we employ the  -VAE [Higgins et al.
This technique produces encoder q  ( z | x ) that maps from the data and latent codes and decoder
p ✓ ( x | z ) that maps from codes to data.
[Equation]
[-D f := {} 3: for ( µ,  , ⇡ ) 
-]
ETHODS 3. F AILURE SCENARIOS
We begin by formalizing our notion of failure scenarios .
Let z 2 R K be the latent codes corresponding to image x 2 X and q  ( · ) : x ! z be the encoder mapping the relationship between images and latent codes.
Deﬁnition 3.1.
Failure scenario.
Given a constant ✏ > 0 , vector norm || · || , and point z 0 , a failure scenario is a set of images
[Equation]
Previous works that investigate unrestricted adversarial examples look for speciﬁc instances where the oracle and the model disagree [Song et al.
; Zhao et al.
We instead look for regions in the latent space where this is the case.
Because the latent space of the VAE tends to take on Gaussian form due to the prior, we can use euclidean distance to deﬁne these regions.
If we were to deﬁne failure scenarios on the original data manifold, we may need a much more complex distance function.
Because it is likely too strict to assume the oracle and model disagree on every instance in such a region, we also introduce a relaxation.
Deﬁnition 3.2.
Relaxed failure scenario.
Given a constant ✏ > 0 , vector norm || · || , point z 0 , and threshold ⇢ , a relaxed failure scenario is a set of images
[Equation]
} such that
[Equation]
In this work, we adopt the latter deﬁnition of failure scenarios.
To concretize failure scenarios and provide evidence for their existence, we continue our MNIST example from ﬁgure 1.
We plot the t-SNE embeddings of the latent codes of 10000 images from the training set and 516 unrestricted adversarial examples created during the identiﬁcation step in ﬁgure 2 (details of how we generate unrestricted adversarial examples in section 3.2.1).
We see that the unrestricted adversarial examples are from similar regions in the latent space.
3. D EFUSE
In this section, we introduce Defuse: our procedure for identifying and correcting classiﬁer performance on failure scenarios.
First, we explain how we identity unrestricted adversarial examples using VAEs.
Next, we describe our clustering approach that distills these instances into failure scenarios .
Last, we introduce our approach to correct classiﬁer predictions on the failure scenarios.
3.2. I DENTIFYING UNRESTRICTED ADVERSARIAL EXAMPLES
This section describes the identiﬁcation step in Defuse (ﬁrst pane in ﬁgure 1).
The aim of the identiﬁcation step is to generate many unrestricted adversarial examples.
In essence, we encode all the images from the training data.
We perturb the latent codes with a small amount of noise drawn from a Beta distribution.
We save instances that are classiﬁed differently from ground truth by f when decoded.
By perturbing the latent codes with a small amount of noise, we expect the decoded instances to have small but semantically meaningful differences from the original instances.
Thus, if the classiﬁer prediction deviates on the perturbation the instance is likely misclassiﬁed.
We denote the set of unrestricted adversarial examples for a single instance . We generate unrestricted adversarial examples over each instance x 2 X producing a set of unrestricted adversarial containing the produced for each instance x .
Pseudo code of the algorithm for generating a single unrestricted adversarial example is given in algorithm 1 in appendix A.
Our technique is related to the method for generating natural adversarial examples from [Zhao et al.
(2018)] — a very similar but slightly different concept from unrestricted adversarial examples.
The authors use a similar stochastic search method in the latent space of a GAN.
They start with a small amount of noise and increase magnitude of the noise until they ﬁnd a unrestricted adversarial example.
Thus, they save only the unrestricted adversarial examples which are minimally distant from a data point.
They also save images that differ in prediction from the original decoded instance.
Because we iterate over the entire data set, it is simpler to keep the level of noise ﬁxed and sample a predetermined number of times.
In addition, we save images that differ in ground truth label from the original decoded instance because we seek to debug a classiﬁer.
Meaning, if the original instance is misclassiﬁed we wish to save this instance as a model failure.
3.2. D ISTILLING FAILURE SCENARIOS
This section describes the distillation step in defuse (second pane of ﬁgure 1).
The goal of the distillation step is to cluster the latent codes of the set of unrestricted adversarial examples in order to diagnose failure scenarios. We require our clustering method to (1) infer the correct number of clusters from the data and (2) be capable of generating instances of each cluster.
We need to infer the number of clusters from the data because the number of failure scenarios are unknown ahead of time. Further, we must be capable of generating many instances from each cluster so that we have enough data to ﬁnetune on in order to correct the faulty model behavior.
In addition, generating many failure instances enables model designers to see numerous examples from the failure scenarios, which encourages understanding of the model failure modes. Though any such clustering method that ﬁts this description could be used for distillation, we use a Gaussian mixture model (GMM) with Dirichlet process prior. We use the Dirichlet process because it nicely describes the clustering problem where the number of mixtures is unknown before hand, fulﬁlling our ﬁrst criteria [Sudderth (2006)].
Additionally, because the model is generative, we can sample new instances, which satisﬁes our second criteria.
In pratice, we use the truncated stick breaking construction of the dirchlet process, where K is the upper bound of the number of mixtures. The truncated stick breaking construction simpliﬁes inference making computation more efﬁcient [Sudderth (2006)].
The method outputs a set of clusters ✓ j = ( µ j ,  j , ⇡ j ) where j 2 { 1 , ..., K } . The parameters µ and  describe the mean and variance of a multivariate normal distribution and ⇡ indicates the cluster weight.
To perform inference on the model, we employ expectation maximization (EM) described in [Bishop (2006)] and use the implementation provided in [Pedregosa et al.
Once we run EM and determine the parameter values, we throw away cluster components that are not used by the model.
We ﬁx some small ✏ and deﬁne the set of failure scenarios ⇤ generated at the distillation step as: ⇤ :
= { ( µ j , ⌃ j , ⇡ j ) | ⇡ j > ✏ } . [Equation]
C ORRECTING FAILURE SCENARIOS
Labeling First, an annotator assigns the correct label to the failure scenarios.
For each failure scenario identiﬁed in ⇤ , we sample Q latent codes from z ⇠ N ( µ j , ⌧ ·  j ) .
Here, ⌧ 2 R is a hyperparameter that controls the diversity of samples from the failure scenario.
Because it could be possible for multiple ground truth classes to be present in a failure scenario, we set this parameter tight enough such that the sampled instances are from the same class to make labeling easier.
We reconstruct the latent codes using the decoder p ✓ ( x | z ) .
Next, an annotator reviews the reconstructed instances from the scenario and decides whether the scenario constitutes a model failure.
If so, the annotator assigns the correct label to all of the instances.
The correct label constitutes a single label for all of the instances generated from the scenario.
We repeat this process for each of the scenarios identiﬁed in ⇤ and produce a dataset of failure instances D f .
Pseudo code for the procedure is given in algorithm 2 in appendix A.
Finetuning We ﬁnetune on the training data with an additional regularization term to ﬁx the classiﬁer performance on the failure scenarios.
The regularization term is the cross entropy loss between the identiﬁed failure scenarios and the annotator label.
Where CE is the cross entropy loss applied to the failure instances D f and  is the hyperparameter for the regularization term, we optimize the following objective using gradient descent: F ( D , D f ) = L ( D ) +  · CE
( D f ) .
This objective encourages the model to maintain its predictive performance on the original training data while en- couraging the model to predict the failure instances correctly.
The regularization term  controls the pressure applied to the model to classify the failure instances correctly.
E XPERIMENTS 4. S ETUP
Datasets We evaluate Defuse on three datasets: MNIST [LeCun et al.
(2010)], the German Trafﬁc
Signs dataset [Stallkamp et al.
(2011)], and the Street view house numbers dataset (SVHN)
[Netzer et al.
MNIST consists of 60 , 000 32 X 32 handwritten digits for training and 10 , 000 digits for testing.
The images are labeled corresponding to the digits
9 .
The German trafﬁc signs data set includes 26 , 640 training and 12 , 630 testing images of size 128 X 128 .
We randomly split the testing data in half to produce a validation and testing set.
The images are labeled from 43 different classes to indicate the type of trafﬁc signs.
The SVHN data set consists of 73 , 257 training and 26 , 032 testing images of size 32 X 32 .
The images include digits of house numbers from Google streetview with labels
9 .
We split the testing set in half to produce a validation and testing set.
Models On MNIST, we train a CNN scoring 98 .
3% test set accuracy following the architecture from [Paszke et al.
On German trafﬁc signs and SVHN, we ﬁnetune a Resnet18 model pretrained on ImageNet [He et al.
The German signs and SVHM models score 98 .
7% and 93 .
2% test accuracy respectively.
We train a  -VAE on all available data from each data set to model the set of legitimate images in Defuse.
We use an Amazon EC2 P3 instance with a single NVIDIA Tesla
V100 GPU for training.
We follow similar architectures to [Higgins et al.
We set the size of the latent dimension z to 10 for MNIST/SVHN and 15 for German signs. We provide our  -VAE architectures in appendix B. Defuse In the identiﬁcation step, we ﬁx the parameters of the {+Beta distribution noise a and b to a = b = 50 . 0 for MNIST and a = b = 75 . 
+}
for SVHN and German signs. We found these parameters were good choices because they produce a very small amount of perturbation noise making the decoded instance only slightly different than the original instance. During distillation, we set the upper bound on the number of components K to 100 . We generally found the actual number of clusters to be much lower than this level.
Thus, this serves as an appropriate upper bound. We also ﬁxed the weight threshold for clusters ✏ to 0 .
01 during distillation in order to remove clusters with very low weighting.
We additionally randomly down sample the number of unrestricted adversarial examples to 50 , 000 to make inference of the GMM more efﬁcient.
For correction, we sample ﬁnetuning and testing sets consisting of 256 images each from every failure scenario.
This number of samples captures the breadth of possible images in the scenario, so it is appropriate for tuning and evaluation. We use the ﬁnetuning set as the set of failure instances D f .
We use the test set as held out data for evaluating classiﬁer performance on the failure scenarios after correction. During sampling, we ﬁx the sample diversity ⌧ to 0 .
5 for MNIST and 0 . 01 for SVHN and German signs because the samples from each of the failure scenarios appear to be in the same class using these values.
We ﬁnetune over a range of  ’s in order to ﬁnd the best balance between training and failure scenario data. We use 3 epochs for MNIST and 5 for both SVHN and German Signs because training converged within both these time frames. During ﬁnetuning, we select the model for each  according to the highest training set accuracy for MNIST or validation set accuracy for SVHM and German trafﬁc signs at the end of each ﬁnetuning epoch.
We select the best model overall as the highest training or validation performance over all  ’s. Annotator Labeling Because Defuse requires human supervision, we use Amazon Sagemaker Ground Truth to both determine whether clusters generated in the distillation step are failure scenarios and to generate their correct label.
In order to determine whether clusters are failure scenarios, we sample 10 instances from each cluster in the distillation step.
It is usually apparent the classiﬁer disagrees with many of the ground truth labels within 10 instances, and thus it is appropriate to label the cluster as a failure scenario.
For example, in ﬁgure 3 it is generally clear the classiﬁer incorrectly predicts the data within only a few examples. As such, 10 instances is a reasonable choice. To reduce noise in the annotation process, we assign the same image to 5 different workers and take the majority annotated label as ground truth. The workers label the images using an interface that includes a single image and the possible labels for that task.
We additionally instruct workers to select “None of the above” if the image does not belong to any class and discard these {+labels. For instance, the MNIST interface includes a single image and buttons for the digits 
  9 along with a Dataset # Scenarios+}
Validation
Test Failure Scenario Before Finetuning MNIST - - 98.
SVHN93.
German Signs98.
Unrestricted
MNIST - - 99.
Adversarial Examples
SVHN93. German Signs - - - - Defuse MNIST99.
SVHN 93. German Signs 98. “None of the above” button. We provide a screen shot of this interface in ﬁgure 14.
If more than half (i.e. setting ⇢ = 0 . 5 ) of worker labeled instances disagree with the classiﬁer predictions on theinstances, we call the cluster a failure scenario.
We chose ⇢ = 0 . 5 because clusters are highly dense with incorrect predictions at this level, making them useful for both understanding model failures and worthwhile for correction. We take the majority prediction over each of the 10 ground truth labels as the label for the failure scenario.
As an exception, annotating the German trafﬁc signs data requires speciﬁc knowledge of trafﬁc signs.
The German trafﬁc signs data ranges across 43 different types of trafﬁc signs.
It is not reasonable to assume annotators have enough familiarity with this data and can label it accurately.
For this data set, we, the authors, reviewed the distilled clusters and determined which clusters constituted failure scenarios.
We labeled the cluster a failure scenario if half the instances appeared to be misclassiﬁed.
4. I LLUSTRATIVE FAILURE SCENARIO EXAMPLES
We demonstrate the potential of Defuse for identifying critical model bugs.
We review failure scenarios produced in the three datasets we consider. All together, Defuse produces 19 failure scenarios for MNIST, 6 for SVHN, and 8 for German signs.
For each dataset, we provide samples from three failure scenarios in ﬁgure 3. The failure scenarios include numerous mislabeled examples.
Each failure scenario is composed of mislabeled examples of a similar style.
For example, in MNIST, the failure scenario in the upper left hand corner of ﬁgure 3 includes a similar style of 4 ’s that are generally predicted incorrectly.
The same is true for the failure scenarios in the center and right column where a certain style of 2 ’s and 6 ’s are mistaken.
The failure scenarios generally include images which seem difﬁcult to classify.
For instance, the misclassiﬁed 6 ’s are quite thin making them appear like 1 ’s in some cases.
There are similar trends in SVHN and German Signs. In SVHN, particular types of 5 ’s and 8 ’s are misclassiﬁed. The same is true in German signs where styles of 50 km/h and 30 km/h signs are predicted incorrectly.
Generally, these methods reveal important bugs in each of the classiﬁers. It is clear from the MNIST example for instance that very skinny 6 ’s are challenging for the classiﬁer to predict correctly. Further, the German signs classiﬁer has a difﬁcult time with 50 km/h signs and tends to frequently mistake them as 80 km/h. We provide further samples from other failure scenarios in appendix D. These results clearly demonstrate Defuse reveals insightful model bugs which are useful for model designers to understand. 4. C ORRECTING F AILURE S CENARIOS
We show that Defuse resolves the failure scenarios while maintaining model generalization on the test set. To perform this analysis, we assess accuracy on both the failure scenario test data and test set after correction. It is important for classiﬁer accuracy to improve on the failure scenario data in order [Figure]
to correct the bugs discovered while running Defuse. At the same time, the classiﬁer accuracy on the test set should stay at a similar level or improve indicating that model generalization according to the test set is still strong. We compare Defuse against ﬁnetuning only on the unrestricted adversarial examples labeled by annotators.
We expect this baseline to be reasonable because related works which focus on robustness to classic adversarial attacks demonstrate that tuning directly on the adversarial examples is effective [Zhang et al. We ﬁnetune on the unrestricted adversarial examples sweeping over a range of different  ’s in the same way as Defuse described in section 4.1. We use this baseline for MNIST and SVHN and not German signs because we, the authors, assigned the failure scenarios for this data set. Thus, we do not have ground truth labels for unrestricted adversarial examples.
We provide an overview of the models before ﬁnetuning, ﬁnetuning with the unrestricted adversarial examples, and using Defuse in ﬁgure 4. Defuse scores highly on the failure scenario data after correction compared to before ﬁnetuning. There is only marginal improvement ﬁnetuning on the unrestricted adversarial examples. These results indicate Defuse corrects the faulty model performance on the identiﬁed failure scenarios.
Further, we see the clustering step in Defuse is critical to its success because of the technique’s superior performance compared to ﬁnetuning on the unrestricted adversarial examples.
In addition, there are minor effects on test set performance during ﬁnetuning.
The test set accuracy increases slightly for MNIST and decreases marginally for SVHN and German
Signs for both tuning on the unrestricted adversarial examples and using Defuse.
Though the test set performance changes marginally, the increased performance on the failure scenarios demonstrates
Defuse’s capacity to correct important model errors. Further, we plot the relationship between test set accuracy and failure scenario test accuracy in ﬁgure 5. We generally see there is an appropriate  for each model where there is both high test set performance and accuracy on the failure scenarios. All in all, these results indicate Defuse serves as an effective method for correcting speciﬁc cases of faulty classiﬁer performance while maintaining model generalization.
4. A NNOTATOR A GREEMENT
Because we rely on annotators to provide the ground truth labels for the unrestricted adversarial examples, we investigate the agreement between the annotators during labeling. It is important for the annotators to agree on the labels for the unrestricted adversarial examples so that we can have high conﬁdence our evaluation is based on accurately labeled data. We evaluate the annotator agreement through assessing the percent of annotators that voted for the majority label prediction in an unrestricted adversarial example across all the annotated examples.
This metric will be high when the annotators are in consensus and low when only a few annotators constitute the majority vote.
We provide the annotator agreement on MNIST and SVHN in ﬁgure 6 broken down into failure scenario data, non-failure scenario data, and their combination.
Interestingly, the failure scenario data has slightly lower annotator agreement indicating these tend to be more ambiguous examples.
Further, there is lower agreement on SVHN than MNIST, likely because this data is more complex.
All in all, there is generally high annotator agreement across all the data. [Figure]  R ELATED W ORK A number of related approaches for improving classiﬁer performance use data created from generative models — mostly generative adversarial networks (GANs) [Sandfort et al.
Milz et al.
; Antoniou et al.
These methods use GANs to generate instances from classes that are underrepresented in the training data to improve generalization performance.
Additional methods use generative models for semi-supervised learning [Kingma et al.
; Varma et al.
Kumar et al.
; Dumoulin et al.
Though these methods are similar in nature to the correction step of our work, a key difference is Defuse focuses on summarizing and presenting high level model failures.
Also, [Varma et al.
(2017)] provide a system to debug data generated from a GAN when the training set may be inaccurate.
Though similar, we ultimately use a generative model to debug a classiﬁer and do not focus on the generative model itself.
Last, similar to [Song et al.
, Zhao et al.
(2018)], [Booth et al.
(2020)] provide a method to generate highly conﬁdent misclassiﬁed instances.
Related to debugging models, [Kang et al.
(2018)] focus on model assertions that ﬂag failures during production.
Also, [Zhang et al.
(2018)] investigate debugging the training set for incorrectly labeled instances.
We focus on preemptively identifying model bugs and do not focus on incorrectly labeled test set instances.
Additionally, [Ribeiro et al.
(2020)] propose a set of behavioral testing tools that help model designers ﬁnd bugs in NLP models.
This technique requires a high level of supervision and thus might not be appropraite in some settings.
Last, [Odena et al.
(2019)] provide a technique to debug neural networks through perturbing data inputs with various types of noise.
By leveraging unrestricted adversarial examples, we distill high level patterns in critical and naturally occurring model bugs.
This technique requires minimal human supervision while presenting important types of model errors to designers.
C ONCLUSION
In this paper, we present Defuse: a method that generates and aggregates unrestricted adversarial examples to debug classiﬁers.
Though unrestricted adversarial examples have been proposed in previous works, we harness such examples for the purpose of debugging classiﬁers.
We accomplish this task through identifying failure scenarios: regions in the latent space of a VAE with many unrestricted adversarial examples.
On a variety of data sets, we ﬁnd that samples from failure scenarios are useful in a number of ways.
First, failure scenarios are informative for understanding the ways certain models fail.
Second, the generative aspect of failure scenarios is very useful for correcting failure scenarios.
In our experimental results, we show that these failure scenarios include critical model issues for classiﬁers with real world impacts — i.e. trafﬁc sign classiﬁcation — and verify our results using ground truth annotator labels.
We demonstrate that Defuse successfully resolves these issues.
Although Defuse identiﬁes important errors in classiﬁers, the technique requires a minimal level of human supervision.
Namely, the failure scenarios must be reviewed before correction.
In the future, it will be crucial to investigate automatic ways of reviewing failure scenarios. B T RAINING DETAILS B. GMM DETAILS In all experiments, we use the implementation of Gaussian mixture model with dirichlet process prior from [Pedregosa et al.
We run our experiments with the default parameters and full component covariance . B. MNIST DETAILS
Model details We train a CNN on the MNIST data set using the architecture in ﬁgure 7.
We used the Adadelta optimizer with the learning rate set to 1 . We trained for 5 epochs with a batch size of 64 . [Figure]  -VAE training details We train a  -VAE on MNIST using the architectures in ﬁgure 8 and 9.
We set  to 4 .
We trained for 800 epochs using the Adam optimizer with a learning rate of 0 .
001 , a minibatch size of 2048 , and  set to 0 .
We also applied a linear annealing schedule on the
KL-Divergence for 500 optimization steps.
We set z to have 10 dimensions.
Identiﬁcation We performed identiﬁcation with Q set to 500 .
We set a and b both to 50 .
We ran identiﬁcation over the entire training set.
Last, we limited the max allowable size of to 100 .
Distillation We ran the distillation step setting K , the upper bound on the number of mixtures, to 100 .
We ﬁxed ✏ to 0 .
01 and discarded clusters with mixing proportions less than this value.
This left 44 possible scenarios.
We set ⌧ to 0 .
5 during review.
We used Amazon Sagemaker Ground Truth
[Figure]
[Figure]
to determine failure scenarios and labels.
The labeling procedure is described in section 4.1.
This produced 19 failure scenarios.
Correction We sampled 256 images from each of the failure scenarios for both ﬁnetuning and testing.
We ﬁnetuned with minibatch size of 256 , the Adam optimizer, and learning rate set to 0 .
We swept over a range of correction regularization  ’s consisting of [1 e  10 , 1 e  9 , 1 e  8 , 1 e  7 , 1 e  6 , 1 e  5 , 1 e  4 , 1 e  3 , 1 e  2 , 1 e  1 , 1 , 2 , 5 , 10 , 20 , 100 , 1000] and ﬁnetuned for 3 epochs on each.
B. G ERMAN S IGNS D ATASET D ETAILS
Dataset The data consists of 26640 training images and 12630 testing images consisting ofdifferent types of trafﬁc signs.
We randomly split the testing data in half to produce 6315 testing and validation images.
Additionally, we resize the images to 128 x 128 pixels.
Classiﬁer f
We ﬁne-tuned the ResNet18 model for 20 epochs using Adam with the cross entropy loss, learning rate of 0 .
001 , batch size of 256 on the training data set, and assessed the validation accuracy at the end of each epoch.
We saved the model with the highest validation accuracy.
-VAE training details We trained for 800 epochs using the Adam optimizer with a learning rate of 0 .
001 , a minibatch size of 2048 , and  set to 4 .
We also applied a linear annealing schedule on the KL-Divergence for 500 optimization steps.
We set z to have 15 dimensions.
[Figure]
Identiﬁcation We performed identiﬁcation with Q set to 100 .
We set a and b both to 75 .
[Figure]
Distillation We ran the distillation step setting K to 100 . We ﬁxed ✏ to 0 . 01 and discarded clusters with mixing proportions less than this value. This left 38 possible scenarios.
We set ⌧ to 0 .
01 during review.
We determined 8 of these scenarios were particularly concerning.
Correction We ﬁnetuned with minibatch size of 256 , the Adam optimizer, and learning rate set to 0 .
We swept over a range of correction regularization  ’s consisting of [1 e  10 , 1 e  9 , 1 e  8 , 1 e  7 , 1 e  6 , 1 e  5 , 1 e  4 , 1 e  3 , 1 e  2 , 1 e  1 , 1 , 2 , 5 , 10 , 20 , 100 , 1000] and ﬁnetuned for 5 epochs on each.
B. SVHN DETAILS
Dataset The data set consists of 73257 training and 26032 testing images.
We also randomly split the testing data to create a validation data set.
Thus, the ﬁnal validation and testing set correspond to 13016 images each.
Classiﬁer f
We ﬁne tuned for 10 epochs using the Adam optimizer, learning rate set to 0 .
001 , and a batch size of 2048 .
We chose the model which scored the best validation accuracy when measured at the end of each epoch.
-VAE training details We trained the  -VAE for 400 epochs using the Adam optimizer, learning rate 0 .
001 , and minibatch size of 2048 .
We set  to 4 and applied a linear annealing schedule on the
Kl-Divergence for 5000 optimization steps.
We set z to have 10 dimensions.
[Figure]
[Equation]
64 ReLU
[Figure]
[Figure]
Identiﬁcation We set Q to 100 .
We also set the maximum size of to 10 .
We set a and b to 75 .
Distillation We set K to 100 .
We ﬁxed ✏ to 0 . The distillation step identiﬁed 32 plausible failure scenarios. The annotators deemed 6 of these to be failure scenarios. We set ⌧ to 0 .
01 during review.
Correction We set the minibatch size of 2048 , the Adam optimizer, and learning rate set to 0 . 001 .
We considered a range of  ’s:
[1 e  10 , 1 e  9 , 1 e  8 , 1 e  7 , 1 e  6 , 1 e  5 , 1 e  4 , 1 e  3 , 1 e  2 , 1 e  1 , 1 , 2 , 5 , 10 , 20 , 100 , 1000] .
We ﬁnetuned for 5 epochs.
B. T -SNE EXAMPLE DETAILS
We run t-SNE on 10 , 000 examples from the training data and 516 unrestricted adversarial examples setting perplexity to 30 .
For the sake of clarity, we do not include outliers from the unrestricted adversarial examples.
Namely, we only include unrestricted adversarial examples with > 1% probability of being in one of the MNIST failure scenario clusters.
C A NNOTATOR INTERFACE
We provide a screenshot of the annotator interface in ﬁgure 14.
D A DDITIONAL EXPERIMENTAL RESULTS D. A DDITIONAL SAMPLES FROM MNIST FAILURE SCENARIOS
We provide additional examples from 10 randomly selected (no cherry picking)
MNIST failure scenarios.
We include the annotator consensus label for each failure scenario.
D. A DDITIONAL SAMPLES FROM G ERMAN SIGNS FAILURE SCENARIOS
We provide samples from all of the German signs failure scenarios.
We provide the names of the class labels in ﬁgure 25.
For each failure scenario, we indicate our assigned class label in the caption and the classiﬁer predictions in the upper right hand corner of the image.
ClassId
SignName
Speed limit (20km/h)
Speed limit (30km/h)
Speed limit (50km/h)
Speed limit (60km/h)
Speed limit (70km/h)
Speed limit (80km/h)
End of speed limit (80km/h)
Speed limit (100km/h)
Speed limit (120km/h)
No passing
No passing for vehicles
3.5 metric tons
Right way at next intersection
Priority road
Yield
Stop
No vehicles
Vehicles tons prohibited
No caution
Road signals
Pedestrians crossing
Bicycles
Roundabout mandatory
End no passing
End tons
D. A DDITIONAL SAMPLES FROM SVHN FAILURE SCENARIOS
We provide additional samples from each of the SVHN failure scenarios.
The digit in the upper left hand corner is the classiﬁer predicted label.
The caption includes the Ground Truth worker labels.