Original: P ROTOTYPICAL E XAMPLES IN D EEP L EARNING : M ETRICS , C HARACTERISTICS , AND U TILITY
Revised: P ROTOTYPICAL E XAMPLES IN D EEP L EARNING : M ETRICS , C HARACTERISTICS , AND U TILITY

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Machine learning (ML) research has investigated prototypes examples that are representative of the behavior to be learned.
Revised: Machine learning ML) research has investigated prototypes examples that are representative of the behavior to be learned

Original: We systematically evaluate ﬁve methods for identifying prototypes  both ones previously introduced as well as new ones we propose, ﬁnding all of them to provide meaningful but different interpre- tations
Revised: We systematically evaluate ﬁve meth ods for identifying prototypes , both ones previously introduced as well as new ones we propose, ﬁnding all of them to provide meaningful but different interpretations.

Original: Through a human study, we conﬁrm that all ﬁve metrics are well matched to human intuition.
Revised: Through a human study, we conﬁrm that all ﬁve metrics are well matched to human intuition.

Original: Examining cases where the metrics disagree offers an infor mative perspective on the properties of data and algorithms used in learning with implications for data corpus construction efﬁciency adversarial robustness in- terpretability, and other ML aspects.
Revised: Examining cases where the metrics disagree offers an informative perspective on the properties of data and algorithms used in learning with implications for data -corpus construction, efﬁciency, adversarial robustness in- terpretability, and other ML aspects

Original: In particular
Revised: In particular

Original: , we conﬁrm that the “train on hard” curriculum approach can improve accuracy on many datasets and tasks, but that it is strictly worse when there are many mislabeled or ambiguous examples.
Revised: , we conﬁrm that the “train on hard” curriculum approach can improve accuracy on many datasets and tasks, but that it is strictly worse when there are many mislabeled or ambiguous examples.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: When reasoning about ML tasks, it is natural to look for a set of training or test examples that is somehow prototypical —i.e., that is representative of the desired learned behavior.
Revised: When reasoning about ML tasks, it is natural to look for a set of training or test examples that is somehow prototypical —i.e., that is representative of the desired learned behavior.

Original: Although such prototypical examples have been central to several research efforts, e.g., in interpretability (Bien &
Revised: Although such prototypical examples have been central to several research efforts, e.g., in interpretability (Bien &

Original: Tibshirani, 2011) and curriculum learning (Bengio et al., 2009), neither are their characteristics are well deﬁned, nor are the metrics for identifying them generally agreed upon.
Revised: Tibshirani, 2011) and curriculum learning (Bengio et al., 2009), no generally-agreed-upon deﬁnition seems to exist for prototypes, or their characteristics.

Original: For modern deeplearning models, whose behavior is often inscrutable, even the very existence and usefulness of prototypical examples has seemed uncertain until the recent work of Stock & Cisse (2017).
Revised: For modern deep-learning models, whose behavior is often inscrutable, even the very existence and usefulness of prototypical examples has seemed uncertain until the recent work of Stock & Cisse (2017).

Original: Inspired by that work we (1) identify a set of desirable properties for prototypicality deﬁnitions; (2) systematically explore different metrics used in prior work, as well as new metrics we develop, for identifying prototypical examples in both training and test data; (3) study the characteristics of those metrics’ prototypes and their complement set—the outliers —using both quantitative measures and a qualitative human study; and, (4) evaluate the usefulness of prototypes for machine-learning purposes such as reducing sample complexity, or improving adversarial robustness and interpretability.
Revised: Inspired by that work we (1) identify a set of desirable properties for prototypicality deﬁnitions; (2) systematically explore different metrics used in prior work, as well as new metrics we develop, for identifying prototypical examples in both training and test data; (3) study the characteristics of those metrics’ prototypes and their complement set—the outliers —using both quantitative measures and a qualitative human study; and, (4) evaluate the usefulness of prototypes for machine-learning purposes such as reducing sample complexity or improving adversarial robustness and interpretability.

Original: Our prototypicality metrics are based on adversarial robustness, retraining stability, ensemble agreement, and differentially-private learning.
Revised: Our prototypicality metrics are based on adversarial robustness, retraining stability, ensemble agreement, and differentially-private learning.

Original: As an independent result, we show that predictive stability under retraining strongly correlates with adversarial distance, and may be used as an approximation.
Revised: As an independent result, we show that predictive stability under retraining strongly correlates with adversarial distance, and may be used as an approximation.

Original: Unequivocally, we ﬁnd that distinct sets of prototypical and outlier examples exist for the datasets we consider: MNIST (LeCun et al., 2010),
Revised: Unequivocally, we ﬁnd that distinct sets of prototypical and outlier examples exist for the datasets we consider: MNIST (LeCun et al., 2010),

Original: Fashion-MNIST
Revised: Fashion-MNIST

Original: (Xiao et al., 2017), CIFAR-10 (Krizhevsky & Hinton, 2009), and ImageNet (Russakovsky et al., 2015).
Revised: (Xiao et al., 2017), CIFAR-10 (Krizhevsky & Hinton, 2009), and ImageNet (Russakovsky et al., 2015).

Original: Between all of our metrics, as well as human evaluators, there is overall agreement on the examples that are prototypes and those that are outliers.
Revised: Between all of our metrics, as well as human evaluators, there is overall agreement on the examples that are prototypes and those that are outliers.

Original: Furthermore, the differences between metrics constitute informative exceptions, e.g., identifying uncommon submodes in the data as well as spurious, ambiguous, or misleading examples.
Revised: Furthermore, the differences between metrics constitute informative exceptions, e.g., identifying uncommon submodes in the data as well as spurious, ambiguous, or misleading examples.

Original: Usefully, there are advantages to training models using only prototypical examples: the models learn much faster, their accuracy loss is not great and occurs almost entirely on outlier test examples, and the models are both easier to interpret and more adversarially robust.
Revised: Usefully, there are advantages to training models using only prototypical examples: the models learn much faster, their accuracy loss is not great and occurs almost entirely on outlier test examples, and the models are both easier to interpret and more adversarially robust.

Original: Conversely, at the same sample complexity, signiﬁcantly higher overall accuracy can be achieved by training models exclusively on outliers—once erroneous and misleading examples have been eliminated from the dataset.
Revised: Conversely, at the same sample complexity, signiﬁcantly higher overall accuracy can be achieved by training models exclusively on outliers—once erroneous and misleading examples have been eliminated from the dataset.

Original:  D EFINING AND I DENTIFYING P ROTOTYPES
Revised:  D EFINING AND I DENTIFYING P ROTOTYPES

Original: In designing a metric to identify prototypes, many approaches may seem intuitive.
Revised: In designing a metric to identify prototypes, many approaches may seem intuitive.

Original: Through our empirical work, we have identiﬁed the following properties as desirable for any prototypicality metric:
Revised: We identify the following properties as desirable for any prototypicality metric:

Original: • Independent of the learning task: A prototypicality metric should be applicable to all types of machine-learning tasks, whether they be unsupervised or supervised approaches,  classiﬁcation tasks or generative tasks, etc.
Revised: • Independent of the learning task: A prototypicality metric should be applicable to all types of machine-learning tasks, whether they are based upon unsupervised or supervised approaches, or whether they constitute classiﬁcation tasks or generative tasks, etc.

Original: • Independent of the modeling approach: A metric should identify the overall same prototypical examples regardless of the machine-learning paradigm, model architecture, capacity, or hyperparameters that are used to learn the task.
Revised: • Independent of the modeling approach: A metric should identify the overall same prototypical examples regardless of the machine-learning paradigm, model architecture, capacity, or hyperparameters that are used to learn the task.

Original: • Aligned with human intuition: At least for tasks on which humans do well, prototypical examples identiﬁed by a metric should strongly overlap with those identiﬁed by humans.
Revised: • Aligned with human intuition: At least for tasks on which humans do well, prototypical examples identiﬁed by a metric should strongly overlap with those identiﬁed by humans.

Original: • Covers all apparent data modes: A metric should provide a balanced view of all modes of prototypical examples (e.g., even when multiple disparate modes have a single output classiﬁcation label).
Revised: • Covers all apparent data modes: A metric should provide a balanced view of all modes of prototypical examples (e.g., even when multiple disparate modes have a single output classiﬁcation label).

Original: In the presence of imbalance in the frequency of data modes, the metric should provide coverage while reﬂecting frequency: a mode supported by only a handful of examples need not be as prototypical as other modes in the same class.
Revised: In the presence of imbalance in the frequency of data modes, the metric should provide coverage while reﬂecting frequency: a mode supported by only a handful of examples need not be as prototypical as other modes in the same class.

Original: • Provides a stable ranking: A metric should not only identify examples, but also rank them in terms of prototypicality, and this rank should be stable (i
Revised: • Provides a stable ranking: A metric should not only identify examples, but also rank them in terms of prototypicality, and this rank should be stable (i

Original: e., have low variance) even though the metric is likely computed by a randomized procedure.
Revised: e., have low variance) even though the metric is likely computed by a randomized procedure.

Original: • Applies to both training and test data: A metric should allow both training examples and test examples to be ranked in terms of prototypicality.
Revised: • Applies to both training and test data: A metric should allow both training examples and test examples to be ranked in terms of prototypicality.

Original: • Predicts test accuracy: Models that are trained only on prototypical training examples should still achieve good accuracy on the prototypical test examples—at least when the training and test datasets are balanced and the models use machine-learning methods known to perform well overall.
Revised: • Predicts test accuracy: Models that are trained only on prototypical training examples should still achieve good accuracy on the prototypical test examples—at least when the training and test datasets are balanced and the models use machine-learning methods known to perform well overall.

Original: Other cases need not be so predictive (e.g., if training is done only on outliers); we explore this experimentally in Section 3.
Revised: Other cases need not be so predictive (e.g., if training is done only on outliers); we explore this experimentally in Section 3 and Section 4.

Original: Although metrics for prototypicality should generally satisfy the above properties, they need not do so perfectly.
Revised: Although metrics for prototypicality should generally satisfy the above properties, they need not do so perfectly.

Original: In fact, our experimental results show that the differences between metrics can be highly informative.
Revised: In fact, our experimental results show that the differences between metrics can be highly informative.

Original: In particular, as described in Section 3.3, such differences can improve interpretability and provide insights into model behavior that are not achievable by a single, overall accuracy number, e.g., by giving explanations of failures by example (Caruana et al., 1999).
Revised: In particular, as described in Section 3.3, such differences can improve interpretability and provide insights into model behavior that are not achievable by a single, overall accuracy number, e.g., by giving explanations of failures by example (Caruana et al., 1999).

Original: 2. M ETRICS FOR P ROTOTYPICALITY
Revised: 2. M ETRICS FOR P ROTOTYPICALITY

Original: A number of metrics for identifying prototypes might satisfy the desirable properties we identiﬁed above, possibly even the early methods based on concept similarity introduced by Zhang (1992).
Revised: A number of metrics for identifying prototypes might satisfy the desirable properties we identiﬁed above, possibly even the early methods based on concept similarity introduced by Zhang (1992).

Original: In addition to the work presented below, a complete survey of related work is found in the Appendix.
Revised: In addition to the work presented below, a complete survey of related work is found in the Appendix.

Original: For example, consider two strawmen prototypicality metrics based on time-to-learn and gradient magnitude.
Revised: To start with, consider two strawmen prototypicality metrics based on either learning order or gradient magnitude.

Original: A model may be expected to learn prototypes early in training because they are presumably more common and somehow “simpler” than all the edge-case outliers.
Revised: A model may be expected to learn prototypes early in training because they are presumably more common and somehow “simpler” than all the edge-case outliers.

Original: Conversely, near the end of training, one might expect the relative magnitude of the gradient (cid:107)∇ θ (cid:96) ( f θ ( x ))
Revised: Conversely, near the end of training, one might expect the relative magnitude of the gradient (cid:107)∇ θ (cid:96) ( f θ ( x ))

Original: (cid:107) to be small for prototypical examples x , as each such example should have very little to teach the model.
Revised: (cid:107) to be small for prototypical examples x , as each such example should have very little to teach the model.

Original: From this, two metrics can be deﬁned, which we evaluated in experiments that averaged hundreds of model training runs to minimize random-initialization and stochastic-learning effects.
Revised: From this, two metrics can be deﬁned, which we evaluated in experiments that averaged hundreds of model training runs to minimize random-initialization and stochastic-learning effects.

Original: Unfortunately, both of these metrics exhibited both very high variance as well as apparently-low signal, deﬁning prototypes sets that appeared random upon inspection and that did not satisfy our stated desirable property of predicting test accuracy.
Revised: Unfortunately, both of these metrics exhibited both very high variance as well as apparently-low signal, deﬁning prototypes sets that appeared random upon inspection and that did not satisfy our stated desirable property of predicting test accuracy.

Original: Thus, we do not present results for these two metrics.
Revised: Thus, we do not present results for these two metrics.

Original: Instead, we deﬁne and apply the following ﬁve metrics, each of which ranks examples by their relative ordinal number (i.e., position in the sorted order of the measured value):
Revised: Instead, we deﬁne and apply the following ﬁve metrics, each of which ranks examples by their relative ordinal number (i.e., position in the sorted order of the measured value):

Original: Adversarial Robustness ( adv ): Prototypical examples should be more adversarially robust.
Revised: Adversarial Robustness ( adv ): Prototypical examples should be more adversarially robust.

Original: As a measure of prototypicality, the distance to the decision boundary measured by an adversarialexample attack was recently proposed and utilized by Stock & Cisse (2017).
Revised: As a measure of prototypicality, the distance to the decision boundary measured by an adversarialexample attack was recently proposed and utilized by Stock & Cisse (2017).

Original: Speciﬁcally, for an example x , the measure ﬁnds the perturbation δ with minimal (cid:107) δ (cid:107) such that the original x and the adversarial example x + δ are classiﬁed differently (Biggio et al., 2013; Szegedy et al., 2013).
Revised: Speciﬁcally, for an example x , the measure ﬁnds the perturbation δ with minimal (cid:107) δ (cid:107) such that the original x and the adversarial example x + δ are classiﬁed differently (Biggio et al., 2013; Szegedy et al., 2013).

Original: To compare prototypicality, the work of Stock & Cisse (2017) that inspired our current work used a simple and efﬁcient (cid:96) ∞ -based adversarial-example attack based on an iterative gradient descent introduced by Kurakin et al.
Revised: To compare prototypicality, the work of Stock & Cisse (2017) that inspired our current work used a simple and efﬁcient (cid:96) ∞ -based adversarial-example attack based on an iterative gradient descent introduced by Kurakin et al.

Original: That attack procedure computes gradients to ﬁnd directions that will increase the model’s loss on the input within an (cid:96) ∞ -norm ball.
Revised: That attack procedure computes gradients to ﬁnd directions that will increase the model’s loss on the input within an (cid:96) ∞ -norm ball.

Original: They deﬁne prototypicality as the number of gradient descent iterations necessary to change the class of the perturbed input.
Revised: They deﬁne prototypicality as the number of gradient descent iterations necessary to change the class of the perturbed input.

Original: Instead, the adv metric ranks by the (cid:96) 2 norm (or faster, less accurate (cid:96) ∞ norm) of the minimal-found adversarial perturbation (Carlini & Wagner, 2017).
Revised: Instead, the adv metric ranks by the (cid:96) 2 norm (or faster, less accurate (cid:96) ∞ norm) of the minimal-found adversarial perturbation (Carlini & Wagner, 2017).

Original: This is generally more accurate at measuring the distance to the decision boundary, but comes at a performance cost (it is on average 10-100 × slower).
Revised: This is generally more accurate at measuring the distance to the decision boundary, but comes at a performance cost (it is on average 10-100 × slower).

Original: Holdout Retraining ( ret ): The intuition beind our ret metric is that a model should treat a prototypical example the same regardless of whether, or when, it was used in the training process.
Revised: Holdout Retraining ( ret ): The intuition beind our ret metric is that a model should treat a prototypical example the same regardless of whether, or when, it was used in the training process.

Original: Assume we are given a training dataset X , a disjoint holdout dataset ¯ X , and an example x ∈ X for which to assess prototypicality.
Revised: Assume we are given a training dataset X , a disjoint holdout dataset ¯ X , and an example x ∈ X for which to assess prototypicality.

Original: To begin, we train a model f ( · ) on the training data X to obtain model weights θ .
Revised: To begin, we train a model f ( · ) on the training data X to obtain model weights θ .

Original: We train this model just as how we would typically do—i.e., with the same learning rate schedule, hyper-parameter settings, etc.
Revised: We train this model just as how we would typically do—i.e., with the same learning rate schedule, hyper-parameter settings, etc.

Original: Then, we ﬁne-tune the weights of this ﬁrst model f θ ( · ) on the held-out training data ¯ X to obtain new weights ¯ θ .
Revised: Then, we ﬁne-tune the weights of this ﬁrst model f θ ( · ) on the held-out training data ¯ X to obtain new weights ¯ θ .

Original: To perform this ﬁne-tuning, we use a smaller learning rate and train until the training loss stops decreasing.
Revised: To perform this ﬁne-tuning, we use a smaller learning rate and train until the training loss stops decreasing.

Original: (We have found it is important to obtain ¯ θ by ﬁne-tuning θ as opposed to training from scratch; otherwise, the randomness of training leads to unstable rankings that yield specious results.)
Revised: (We have found it is important to obtain ¯ θ by ﬁne-tuning θ as opposed to training from scratch; otherwise, the randomness of training leads to unstable rankings that yield specious results.)

Original: Finally, given these two models, we compute the prototypicality of x as the difference (cid:107) f θ ( x ) − f ¯ θ ( x ) (cid:107) .
Revised: Finally, given these two models, we compute the prototypicality of x as the difference (cid:107) f θ ( x ) − f ¯ θ ( x ) (cid:107) .

Original: The exact choice of metric (cid:107)·(cid:107) is not important; the results in this paper use the symmetric KL-divergence.
Revised: The exact choice of metric (cid:107)·(cid:107) is not important; the results in this paper use the symmetric KL-divergence.

Original: While this metric is similar to the one considered in Ren et al.
Revised: While this metric is similar to the one considered in Ren et al.

Original: , it differs in important ways: notably, our holdout retraining metric is conceptually simpler, more stable numerically, and more computationally efﬁcient (because it does not require a backward pass to estimate gradients in addition to the forward pass needed to compare model outputs).
Revised: , it differs in important ways: notably, our holdout retraining metric is conceptually simpler, more stable numerically, and more computationally efﬁcient (because it does not require a backward pass to estimate gradients in addition to the forward pass needed to compare model outputs).

Original: Because our metric is only meaningful for data used to train the initial model, in order to measure the prototypicality of arbitrary test points, we actually train on the test data and perform holdout retraining on the original training data.
Revised: Because our metric is only meaningful for data used to train the initial model, in order to measure the prototypicality of arbitrary test points, we actually train on the test data and perform holdout retraining on the original training data.

Original: Ensemble Agreement ( agr ): Prototypical examples should be easy for many types of models to learn.
Revised: Ensemble Agreement ( agr ): Prototypical examples should be easy for many types of models to learn.

Original: We train multiple models of varying capacity on different subsets of the training data (see
Revised: We train multiple models of varying capacity on different subsets of the training data (see

Original: Appendix B).
Revised: Appendix C).

Original: The agr metric ranks examples’ prototypicality based on the agreement within this ensemble, as measured by the symmetric KL-divergence between the models’ output.
Revised: The agr metric ranks examples’ prototypicality based on the agreement within this ensemble, as measured by the symmetric KL-divergence between the models’ output.

Original: Concretely, we train many models f
Revised: Concretely, we train many models f

Original: θ i ( · ) and, for each example x , evaluate the model predictions, and then compute 1 N 2 (cid:80) Ni =1 (cid:80) Nj =1 JS-Divergence ( f θ i ( x ) , f θ j ( x )) to rank the example’s prototypicality.
Revised: θ i ( · ) and, for each example x , evaluate the model predictions, and then compute 1 N 2 (cid:80) Ni =1 (cid:80) Nj =1 JS-Divergence ( f θ i ( x ) , f θ j ( x )) to rank the example’s prototypicality.

Original: Model Conﬁdence ( conf ): We expect models to be conﬁdent on prototypical examples.
Revised: Model Conﬁdence ( conf ): We expect models to be conﬁdent on prototypical examples.

Original: Based on an ensemble of models like that used by the agr metric, the conf metric ranks examples by the mean conﬁdence in the models’ predictions, i.e., ranking each example x by 1 N (cid:80) Ni =1 max f θ i ( x ) .
Revised: Based on an ensemble of models like that used by the agr metric, the conf metric ranks examples by the mean conﬁdence in the models’ predictions, i.e., ranking each example x by 1 N (cid:80) Ni =1 max f θ i ( x ) .

Original: Privacy-preserving Training ( priv ): We can expect prototypical, well-represented modes of examples to be classiﬁed properly by models (e.g., deep neural networks) even when trained with guarantees of differential privacy (Abadi et al., 2016; Papernot et al., 2016).
Revised: Privacy-preserving Training ( priv ): We can expect prototypical, well-represented modes of examples to be classiﬁed properly by models (e.g., deep neural networks) even when trained with guarantees of differential privacy (Abadi et al., 2016; Papernot et al., 2016).

Original: However, such privacypreserving models should exhibit signiﬁcantly reduced accuracy on any rare or exceptional examples, because differentially-private learning attenuates gradients and introduces noise to prevent the details about any speciﬁc training examples from being memorized.
Revised: However, such privacypreserving models should exhibit signiﬁcantly reduced accuracy on any rare or exceptional examples, because differentially-private learning attenuates gradients and introduces noise to prevent the details about any speciﬁc training examples from being memorized.

Original: Outliers are disproportionally likely to be impacted by this attenuation and added noise, whereas the common signal found across many prototypes must have been preserved in models trained to reasonable accuracy.
Revised: Outliers are disproportionally likely to be impacted by this attenuation and added noise, whereas the common signal found across many prototypes must have been preserved in models trained to reasonable accuracy.

Original: Our priv metric is based on training an ensemble of models with increasingly greater ε privacy (i.e., more attenuation and noise) using ε -differentially-private stochastic gradient descent (Abadi et al., 2016).
Revised: Our priv metric is based on training an ensemble of models with increasingly greater ε privacy (i.e., more attenuation and noise) using ε -differentially-private stochastic gradient descent (Abadi et al., 2016).

Original: Our metric then ranks the prototypicality of an example based on the maximum ε privacy at which the example is correctly classiﬁed in a reliable manner (which we take as being also classiﬁed correctly in 90% of less-private models).
Revised: Our metric then ranks the prototypicality of an example based on the maximum ε privacy at which the example is correctly classiﬁed in a reliable manner (which we take as being also classiﬁed correctly in 90% of less-private models).

Original: This ranking embodies the intuition that the more tolerant an example is to noise and attenuation during learning, the more prototypical it is.
Revised: This ranking embodies the intuition that the more tolerant an example is to noise and attenuation during learning, the more prototypical it is.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  E VALUATING THE F IVE D IFFERENT P ROTOTYPICALITY M ETRICS
Revised:  E VALUATING THE F IVE D IFFERENT P ROTOTYPICALITY M ETRICS

Original: As the ﬁrst step in an evaluation, it is natural to consider how the above ﬁve metrics satisfy the properties we identiﬁed earlier—in Section 2—as being desirable for all prototypicality metrics.
Revised: As the ﬁrst step in an evaluation, it is natural to consider how the above ﬁve metrics satisfy the properties we identiﬁed earlier—in Section 2—as being desirable for all prototypicality metrics.

Original: In general, all our ﬁve metrics satisfy these desirable properties.
Revised: In general, all our ﬁve metrics satisfy these desirable properties.

Original: All ﬁve metrics can be applied to both training and test data to induce a ranking and, empirically, we ﬁnd this ranking is stable when computed multiple times.
Revised: All ﬁve metrics can be applied to both training and test data to induce a ranking and, empirically, we ﬁnd this ranking is stable when computed multiple times.

Original: Furthermore, overall, each of the ﬁve metrics exhibits good coverage, with a few informative exceptions, and provides a view of all distinct, prototypical data modes that is proportionally balanced—even in the presence of data skew—in particular across output class labels.
Revised: Furthermore, overall, each of the ﬁve metrics exhibits good coverage, with a few informative exceptions, and provides a view of all distinct, prototypical data modes that is proportionally balanced—even in the presence of data skew—in particular across output class labels.

Original: Notably, none of the metrics fails by ranking some class of labeled examples (e.g., the easiest-tolearn class) as being strictly more prototypical than all examples with other labels; even within each class, any substantial fraction of the most prototypical examples exhibits good modal coverage.
Revised: Notably, none of the metrics fails by ranking some class of labeled examples (e.g., the easiest-tolearn class) as being strictly more prototypical than all examples with other labels; even within each class, any substantial fraction of the most prototypical examples exhibits good modal coverage.

Original: Our metrics are widely applicable, as they are not speciﬁc to any learning task or model (some, like ret and priv might be applicable even to unsupervised learning), and experimentally we have conﬁrmed that the metrics give overall the same results despite large changes in hyperparameters or even the model architecture.
Revised: Our metrics are widely applicable, as they are not speciﬁc to any learning task or model (some, like ret and priv might be applicable even to unsupervised learning), and experimentally we have conﬁrmed that the metrics give overall the same results despite large changes in hyperparameters or even the model architecture.

Original: Finally, as described further in Section 4, all of our metrics provide strong predictive accuracy: training on prototypes gives good test performance on prototypes.
Revised: Finally, as described further in Section 4, all of our metrics provide strong predictive accuracy: training on prototypes gives good test performance on prototypes.

Original: (Experimental results supporting the above observations can be seen in Appendix G.)
Revised: (Experimental results supporting the above observations can be seen in the Appendices.)

Original: 3. Q UANTITATIVE E VALUATION OF C ORRELATION C OEFFICIENTS
Revised: 3. Q UANTITATIVE E VALUATION OF C ORRELATION C OEFFICIENTS

Original: Figure 1 shows the correlation coefﬁcients computed pairwise between each of our metrics for all our datasets, as well as on three metrics for ImageNet 1 (the tables are symmetric across the diagonal).
Revised: Figure 1 shows the correlation coefﬁcients computed pairwise between each of our metrics for all our datasets, as well as on three metrics for ImageNet 1 (the tables are symmetric across the diagonal).

Original: The metrics are overall strongly correlated, and the differences in correlation are informative.
Revised: The metrics are overall strongly correlated, and the differences in correlation are informative.

Original: Unsurprisingly, since they measure very similar properties, the agr (ensemble agreement) and the conf (model conﬁdence) show the highest correlation, However, somewhat unexpectedly, we ﬁnd that the adv (adversarial robustness) correlates very strongly with ret (retraining distance).
Revised: Unsurprisingly, since they measure very similar properties, the agr (ensemble agreement) and the conf (model conﬁdence) show the highest correlation, However, somewhat unexpectedly, we ﬁnd that the adv (adversarial robustness) correlates very strongly with ret (retraining distance).

Original: This is presumably because these two metrics both measure the distance to the model’s decision boundary—even though adv measures this distance by perturbing each example while ret measures by the effect of perturbing the model boundaries themselves on the evaluation of each example.
Revised: This is presumably because these two metrics both measure the distance to a model’s decision boundary— even though adv measures this distance by perturbing each example while ret measures how the evaluation of each example is affected when models’ decision boundaries themselves are perturbed.

Original: (This strong correlation between adv and ret is a new result that may be of independent interest and some signiﬁcance.
Revised: (This strong correlation between adv and ret is a new result that may be of independent interest and some signiﬁcance.

Original: Measurement of adversarial distance is a useful and highly-utalized technique, but it is undeﬁned or ill-deﬁned on many learning tasks and its computation is difﬁcult, expensive, and hard to calibrate.
Revised: Measurement of adversarial distance is a useful and highly-utalized technique, but it is undeﬁned or ill-deﬁned on many learning tasks and its computation is difﬁcult, expensive, and hard to calibrate.

Original: On the other hand, given any holdout dataset and any measure of divergence, the ret metric we deﬁne in Section 2.1 should be easily computable for any ML model or task.)
Revised: On the other hand, given any holdout dataset and any measure of divergence, the ret metric we deﬁne in Section 2.1 should be easily computable for any ML model or task.)

Original: 3. Q UALITATIVE E VALUATION BY I NFORMAL I NSPECTION AND BY A H UMAN S TUDY
Revised: 3. Q UALITATIVE E VALUATION BY I NFORMAL I NSPECTION AND BY A H UMAN S TUDY

Original: It remains to be established that our ﬁve metrics rank examples in a manner that corresponds to human intuition.
Revised: It remains to be established that our ﬁve metrics rank examples in a manner that corresponds to human intuition.

Original: For this, we perform a subjective visual inspection of how the different metrics rank the example training and test data for different machine-learning tasks.
Revised: For this, we perform a subjective visual inspection of how the different metrics rank the example training and test data for different machine-learning tasks.

Original: We establish that there is a clear, intuitive difference between the prototype and outlier extremes of the ranking.
Revised: We establish that there is a clear, intuitive difference between the prototype and outlier extremes of the ranking.

Original: Informally,
Revised: Informally,

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: P i c k
Revised: P i c k

Original: B e s t
Revised: B e s t

Original: [Table]
Revised: [Table]

Original: best one rank in the 90 th percentile on the agr metric).
Revised: best one rank in the 90 th percentile on the agr metric).

Original: Figure 2 and the ﬁgures in Appendix B conﬁrm that there is an obviously apparent difference between  outliers and prototypes in the MNIST, Fashion-MNIST, and CIFAR-10 training examples, and the ImageNet validation examples—although between datasets and classes, the ﬁve metrics differ in how clearly this difference can be seen.
Revised: Figure 2 and the ﬁgures in Appendix C conﬁrm that there is an obviously apparent difference between at least the extreme outliers and prototypes in the MNIST, Fashion-MNIST, and CIFARtraining examples, and the ImageNet validation examples—although between datasets and classes, the ﬁve metrics differ in how clearly this difference can be seen.

Original: To validate and quantify how our metrics correlate with human perception, we performed an online human study using Amazon’s Mechanical Turk service.
Revised: To validate and quantify how our metrics correlate with human perception, we performed an online human study using Amazon’s Mechanical Turk service.

Original: For each output class in the training and test data of MNIST, Fashion-MNIST, and CIFAR-10, the study had human evaluators choose the image that was most or least representative of the class from amongst 9 randomly-selected images.
Revised: For each output class in the training and test data of MNIST, Fashion-MNIST, and CIFAR-10, the study had human evaluators choose the image that was most or least representative of the class from amongst 9 randomly-selected images.

Original: In the study, over 100,000 images were assessed by over 400 different human evaluators.
Revised: In the study, over 100,000 images were assessed by over 400 different human evaluators.

Original: Concretely, in this study (as shown in Appendix D), each human evaluator saw a 3x3 grid ofrandom images and was asked to pick the worst image—or the best image—and this was repeated multiple times.
Revised: Concretely, in this study (as shown in Appendix E), each human evaluator saw a 3x3 grid ofrandom images and was asked to pick the worst image—or the best image—and this was repeated multiple times.

Original: Evaluators exclusively picked either best or worst images and were only shown random images from one output class under a heading with the label name of that class; thus one person would pick only the best MNIST digits “7” while another picked only the worst CIFAR10 “cars.”
Revised: Evaluators exclusively picked either best or worst images and were only shown random images from one output class under a heading with the label name of that class; thus one person would pick only the best MNIST digits “7” while another picked only the worst CIFAR10 “cars.”

Original: (As dictated by good study design, we inserted “Gold Standard” questions with known answers to catch workers answering randomly or incorrectly, eliminating the few such workers from our data.)
Revised: (As dictated by good study design, we inserted “Gold Standard” questions with known answers to catch workers answering randomly or incorrectly, eliminating the few such workers from our data.)

Original: For all datasets, picking non-representative images proved to be the easier task: in a side study where 50 evaluators were shown the same identical 3x3 grids, agreement was 80% on the worst image but only 27% on the best image (random choice would give 11% agreement).
Revised: For all datasets, picking non-representative images proved to be the easier task: in a side study where 50 evaluators were shown the same identical 3x3 grids, agreement was 80% on the worst image but only 27% on the best image (random choice would give 11% agreement).

Original: The results of our human study are presented in Table 1.
Revised: The results of our human study are presented in Table 1.

Original: One of the takeaways is that the assessment of human evaluators is strongly correlated with the one of our metrics: humans mostly picked lowprototypicality images as the worst examples and examples with signiﬁcantly higher-prototypicality as being the best.
Revised: One of the takeaways is that the assessment of human evaluators is strongly correlated with each one of our metrics: humans mostly picked low-prototypicality images as the worst examples and examples with signiﬁcantly higherprototypicality as being the best.

Original: Somewhat surprisingly, there are some large, notable differences between metrics and datasets in their correspondence to human perception—e.g., for Pick Worst, the lowest percentile split of the priv metric does poorly (at 8%) whereas the next does extremely well (at 29%)—suggesting that further investigation is warranted.
Revised: Somewhat surprisingly, there are some large, notable differences between metrics and datasets in their correspondence to human perception—e.g., for Pick Worst, the lowest percentile split of the priv metric does poorly (at 8%) whereas the next does extremely well (at 29%)—suggesting that further investigation is warranted.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 3. C OMPARING P ROTOTYPICALITY M ETRICS AND THEIR C HARACTERISTICS
Revised: 3. C OMPARING P ROTOTYPICALITY M ETRICS AND THEIR C HARACTERISTICS

Original: Because our ﬁve metrics for prototypicality are not perfectly correlated, there are likely to be many examples that are highly prototypical under one metric but not under another.
Revised: Because our ﬁve metrics for prototypicality are not perfectly correlated, there are likely to be many examples that are highly prototypical under one metric but not under another.

Original: To quantify the number and types of those differences we can try looking at their visual correlation in a scatter plot; doing so can be informative, as can be seen in Figure 3(a) where the easily-learned, yet fragile, examples of class “1” in MNIST models have high conﬁdence but low adversarial robustness.
Revised: To quantify the number and types of those differences we can try looking at their visual correlation in a scatter plot; doing so can be informative, as can be seen in Figure 3(a) where the easily-learned, yet fragile, examples of class “1” in MNIST models have high conﬁdence but low adversarial robustness.

Original: To further quantify, we can also compute the Jaccard distance to assess the relative size of the intersections between different sets of prototypes and outliers; the rest of Figure 3 shows the results of doing so at the 25% threshold for Fashion-MNIST.
Revised: To further quantify, we can also compute the Jaccard distance to assess the relative size of the intersections between different sets of prototypes and outliers; the rest of Figure 3 shows the results of doing so at the 25% threshold for Fashion-MNIST.

Original: The results show substantial disagreement between metrics.
Revised: The results show substantial disagreement between metrics.

Original: To understand disagreements, we can consider examples that are prototypical in one metric but outliers in others, ﬁrst combining the union of adv and ret prototypes into a single boundary metric, and the union of adv and ret prototypes into an ensemble metric, because of their high correlation.
Revised: To understand disagreements, we can consider examples that are prototypical in one metric but outliers in others, ﬁrst combining the union of adv and ret prototypes into a single boundary metric, and the union of adv and ret prototypes into an ensemble metric, because of their high correlation.

Original: Memorized exceptions:
Revised: Memorized exceptions:

Original: Recalling the unusual dress-looking “shirt” of Figure 2, and how it seemed to have been memorized with high conﬁdence, we can intersect the top 25% prototypical ensemble images with the bottom-half outliers in both the boundary and priv metrics.
Revised: Recalling the unusual dress-looking “shirt” of Figure 2, and how it seemed to have been memorized with high conﬁdence, we can intersect the top 25% prototypical ensemble images with the bottom-half outliers in both the boundary and priv metrics.

Original: For the Fashion-MNIST “shirt” class, this set—visually shown in Figure 4 on the right—includes not only the dress-looking example but a number of other atypical “shirt” images, including some looking like shorts.
Revised: For the Fashion-MNIST “shirt” class, this set—visually shown in Figure 4 on the right—includes not only the dress-looking example but a number of other atypical “shirt” images, including some looking like shorts.

Original: Also apparent in the set are a number of T-shirt-like and pullover-like images, which are misleading, given the other output classes of
Revised: Also apparent in the set are a number of T-shirt-like and pullover-like images, which are misleading, given the other output classes of

Original: Fashion-MNIST.
Revised: Fashion-MNIST.

Original: For these sets, which are likely to include spurious, erroneously-labeled, and inherently ambiguous examples, we use the name memorized exceptions because they must be memorized as exceptions for models to have been able to reach very high conﬁdence during training.
Revised: For these sets, which are likely to include spurious, erroneously-labeled, and inherently ambiguous examples, we use the name memorized exceptions because they must be memorized as exceptions for models to have been able to reach very high conﬁdence during training.

Original: Similarly, Figure 5(a) shows a large (green) cluster of highly ambiguous boot-like sneakers, which appear indistinguishable from a cluster of memorized exceptions in the Fashion-MNIST “ankle boot” class (see Appendix B).
Revised: Similarly, Figure 5(a) shows a large (green) cluster of highly ambiguous boot-like sneakers, which appear indistinguishable from a cluster of memorized exceptions in the Fashion-MNIST “ankle boot” class (see Appendix C).

Original: Uncommon submodes: On the other hand, the priv metric is based on differentially-private learning which ensures that no small group of examples can possibly be memorized during training: the privacy stems from adding noise and attenuating gradients in a manner that will mask the signal from rare examples during training.
Revised: Uncommon submodes: On the other hand, the priv metric is based on differentially-private learning which ensures that no small group of examples can possibly be memorized: the privacy stems from adding noise and attenuating gradients in a manner that will mask the signal from rare examples during training.

Original: This suggests that
Revised: This suggests that

Original: we can ﬁnd uncommon submodes of the examples in learning tasks by intersecting the bottom-most outlier examples on the priv metric with the union of top prototypes in the boundary and ensamble metrics.
Revised: we can ﬁnd uncommon submodes of the examples in learning tasks by intersecting the bottom-most outlier examples on the priv metric with the union of top prototypes in the boundary and ensamble metrics.

Original: Figure 5(b) shows uncommon submodes discovered in MNIST using the 25% lowest outliers on priv and top 50% prototypes on other metrics.
Revised: Figure 5(b) shows uncommon submodes discovered in MNIST using the 25% lowest outliers on priv and top 50% prototypes on other metrics.

Original: Notably, all of the “serif 1s” in the entire MNIST training set are found as a submode.
Revised: Notably, all of the “serif 1s” in the entire MNIST training set are found as a submode.

Original: Canonical prototypes: Finally, we can simply consider the intersection of the sets of all the topmost prototypical examples in all of our metrics.
Revised: Canonical prototypes: Finally, we can simply consider the intersection of the sets of all the topmost prototypical examples in all of our metrics.

Original: The differences between our metrics should ensure that this intersection is free of spurious or misleading examples; yet, our experiments and human study suggest the set will provide good coverage.
Revised: The differences between our metrics should ensure that this intersection is free of spurious or misleading examples; yet, our experiments and human study suggest the set will provide good coverage.

Original: Hence, we call this set canonical prototypes .
Revised: Hence, we call this set canonical prototypes .

Original: Figure 5(c) shows the airplanes that are canonical prototypes in CIFAR-10.
Revised: Figure 5(c) shows the airplanes that are canonical prototypes in CIFAR-10.

Original: [Figure]
Revised: [Figure]

Original: To further aid interpretability in Figures 4 and 5, we perform a combination of dimensionality reduction and clustering.
Revised: To further aid interpretability in Figures 4 and 5, we perform a combination of dimensionality reduction and clustering.

Original: Concretely, we apply t-SNE (Maaten & Hinton, 2008) on the pixel space (for MNIST and Fashion-MNIST) or ResNetv2 feature space (for CIFAR10) to project the example sets into two dimensions.
Revised: Concretely, we apply t-SNE (Maaten & Hinton, 2008) on the pixel space (for MNIST and Fashion-MNIST) or ResNetv2 feature space (for CIFAR10) to project the example sets into two dimensions.

Original: We then cluster this two-dimensional data using HDBSCAN (Campello et al., 2013), a hierarchical and density-based clustering algorithm which does not try to assign all points to clusters—which not only can improve clusters but also identify spurious data.
Revised: We then cluster this two-dimensional data using HDBSCAN (Campello et al., 2013), a hierarchical and density-based clustering algorithm which does not try to assign all points to clusters—which not only can improve clusters but also identify spurious data.

Original: We believe that other types of data projection and clustering could also be usefully applied to our metrics, and offer signiﬁcant insight into ML datasets.
Revised: We believe that other types of data projection and clustering could also be usefully applied to our metrics, and offer signiﬁcant insight into ML datasets.

Original: (See Appendix F for this section’s ﬁgures shown larger.)
Revised: (See Appendix G for this section’s ﬁgures shown larger.)

Original:  U TILIZING P ROTOTYPES TO I MPROVE A SPECTS OF M ACHINE L EARNING
Revised:  U TILIZING P ROTOTYPES TO I MPROVE A SPECTS OF M ACHINE L EARNING

Original: By usingour prototype metrics we can improve models’ sample complexity, accuracy, or robustness.
Revised: By using prototype metrics, we can improve models’ sample complexity, accuracy, or robustness.

Original: We perform two experiments on the three datasets to investigate whether it is better to train on outliers or on prototypes—exploring the “train on hard data” vs. “train on easy data” question of curriculum learning (Ren et al., 2018), which is discussed in Appendix A. To begin, we order all training data according to its prototypicality as measured by our adv metric.
Revised: We perform two experiments on the three datasets to investigate whether it is better to train on outliers or on prototypes—exploring the “train on hard data” vs. “train on easy data” question of curriculum learning (Ren et al., 2018), which is discussed in Appendix A. To begin, we order all training data according to its prototypicality as measured by our adv metric.

Original: First, we experiment with training on splits of 5 , 000 training examples (approximately 10% of the training data) chosen by taking the k -th most prototypical example to the ( k +5000)-th most prototypical.
Revised: First, we experiment with training on splits of 5 , 000 training examples (approximately 10% of the training data) chosen by taking the k -th most prototypical example to the ( k +5000)-th most prototypical.

Original: As shown in Figure 6, we ﬁnd that the split that yields the most accurate model varies substantially across the datasets and tasks.
Revised: As shown in Figure 6, we ﬁnd that the split that yields the most accurate model varies substantially across the datasets and tasks.

Original: On MNIST, training on the least prototypical examples gives the highest accuracy; conversely, on CIFAR-10, training on nearly-the-most prototypical examples gives the highest accuracy.
Revised: On MNIST, training on the least prototypical examples gives the highest accuracy; conversely, on CIFAR-10, training on nearly-the-most prototypical examples gives the highest accuracy.

Original: We conjecture this is due to the dataset complexity: because nearly all of MNIST is very easy, it makes sense to train on the hardest, most outlier examples.
Revised: We conjecture this is due to the dataset complexity: because nearly all of MNIST is very easy, it makes sense to train on the hardest, most outlier examples.

Original: However, because CIFAR-10 is very difﬁcult, training on very prototypical examples is better.
Revised: However, because CIFAR-10 is very difﬁcult, training on very prototypical examples is better.

Original: Notably, many of the CIFAR-10 and Fashion-MNIST outliers appear to be inherently misleading or ambiguous examples, and several are simply erroneously labeled.
Revised: Notably, many of the CIFAR-10 and Fashion-MNIST outliers appear to be inherently misleading or ambiguous examples, and several are simply erroneously labeled.

Original: We ﬁnd that about 10% of the ﬁrst 5,000 outliers meet our deﬁnition of memorized exceptions.
Revised: We ﬁnd that about 10% of the ﬁrst 5,000 outliers meet our deﬁnition of memorized exceptions.

Original: Also, we ﬁnd that inserting 10% label noise causes model accuracy to decrease by about 10%, regardless of the split trained on— i.e., that to achieve high accuracy on small training data erroneous and misleading outliers must be removed—and explaining the low accuracy shown on the left in the graph of Figures 6(b) and 6(c).
Revised: Also, we ﬁnd that inserting 10% label noise causes model accuracy to decrease by about 10%, regardless of the split trained on— i.e., that to achieve high accuracy on small training data erroneous and misleading outliers must be removed—and explaining the low accuracy shown on the left in the graph of Figures 6(b) and 6(c).

Original: The prior experiment assumes the amount of data is ﬁxed, and we must choose which percentile of data to use.
Revised: The prior experiment assumes the amount of data is ﬁxed, and we must choose which percentile of data to use.

Original: Now, we examine what the best strategy is to apply if the amount of training data is not ﬁxed, and ask: is it better to train on the k -most or k -least prototypical examples?
Revised: Now, we examine what the best strategy is to apply if the amount of training data is not ﬁxed, and ask: is it better to train on the k -most or k -least prototypical examples?

Original: Again, we ﬁnd the answer depends on the dataset.
Revised: Again, we ﬁnd the answer depends on the dataset.

Original: On MNIST, training on the k -least prototypical examples is always better.
Revised: On MNIST, training on the k -least prototypical examples is always better.

Original: However, on Fashion-MNIST and CIFAR-10, training on the prototypical examples is better when k is small, but as soon as we begin to collect more than roughly 10 , 000 examples for Fashion-MNIST or 20 , 000 for CIFAR-10, training on the outliers begins to give more accurate models.
Revised: However, on Fashion-MNIST and CIFAR-10, training on the prototypical examples is better when k is small, but as soon as we begin to collect more than roughly 10 , 000 examples for Fashion-MNIST or 20 , 000 for CIFAR-10, training on the outliers begins to give more accurate models.

Original: However, we ﬁnd that training only on the most prototypical examples  gives extremely high  accuracy on theother prototypical examples.
Revised: However, we ﬁnd that training only on the most prototypical examples found in the training data gives extremely high test accuracy on the prototypical examples found in the test data.

Original: [Figure]
Revised: [Figure]

Revised: On MNIST (a) training on the outliers gives strictly higher accuracy.

Revised: However, on Fashion-MNIST (b) and CIFAR-10 (c), given only a limited amount of training data, prototype-training is better; but if more data is available outlier-training becomes superior.

Revised: At least for some datasets, heavy data augmentation and regularization techniques like dropout can partially recover the utility loss that results from training only with more protoypical examples; the dotted line in subﬁgure (a) shows the result of one such experiment for

Revised: MNIST.

Revised: The second row of subﬁgures (d), (e), and (f) plots the model accuracy resulting from training on a fractional slice of prototypicality-ranked training examples, with the fraction increased from small percentages (on the left) to the entire training data (extreme right).

Revised: One solid line (blue) shows the accuracy resulting from training on training data fractions that are becoming increasingly more prototypical (i.e., the left-most part of the blue line results from training on the most outlier fraction of training examples); conversely, the other solid line (yellow) shows accuracy from training on an increasing, less-and-less prototypical fraction of training data, from left to right.

Revised: Finally, the dotted line (yellow) in subﬁgures (d), (e), and (f) shows the models’ test accuracy on only those examples in the test data that have been deemed prototypical; as can be seen, even training only on a small fraction of the most prototypical training examples can sufﬁce to achieve high accuracy on prototypical test examples.

Original: 4. T RAINING ON PROTOTYPES GIVES SIMPLER DECISION BOUNDARIES
Revised: 4. T RAINING ON PROTOTYPES GIVES SIMPLER DECISION BOUNDARIES

Original: While training exclusively on prototypes often gives inferior accuracy compared to training on the outliers, the former has the beneﬁt of obtaining models with simpler decision boundaries.
Revised: While training exclusively on prototypes often gives inferior accuracy compared to training on the outliers, the former has the beneﬁt of obtaining models with simpler decision boundaries.

Original: Thus, it is natural to ask whether training on prototypes gives models that are more robust to adversarial examples.
Revised: Thus, it is natural to ask whether training on prototypes gives models that are more robust to adversarial examples.

Original: In fact, prior work has found that discarding outliers from the training data can help with both classifying and detecting adversarial examples (Liu et al., 2018).
Revised: In fact, prior work has found that discarding outliers from the training data can help with both classifying and detecting adversarial examples (Liu et al., 2018).

Original: To show that simpler boundaries can lead to more robustness, we train models on ﬁxed-sized subsets of the data where we vary the prototypicality of the 5 , 000 training points included in the subset.
Revised: To show that simpler boundaries can lead to more robustness, we train models on ﬁxed-sized subsets of the data where we vary the prototypicality of the 5 , 000 training points included in the subset.

Original: For each model, we then compute the mean (cid:96) ∞ adversarial distance needed to ﬁnd adversarial examples.
Revised: For each model, we then compute the mean (cid:96) ∞ adversarial distance needed to ﬁnd adversarial examples.

Original: As shown in Figure 10 of Appendix E, the Fashion-MNIST and CIFAR-10 models that are trained on prototypical examples are more robust to adversarial examples than those trained on a slice of training data that is mostly made up of outliers.
Revised: As shown in Figure 10 of Appendix F, the Fashion-MNIST and CIFAR-10 models that are trained on prototypical examples are more robust to adversarial examples than those trained on a slice of training data that is mostly made up of outliers.

Original: However, these models trained on a slice of 5 ,prototypical examples remain comparably robust to a baseline model trained on the entire data.
Revised: However, these models trained on a slice of 5 ,prototypical examples remain comparably robust to a baseline model trained on the entire data.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper explores prototypes: starting with the properties we would like them to satisfy, then evaluating metrics for computing them, and discussing how we can utilize them during training.
Revised: This paper explores prototypes: starting with the properties we would like them to satisfy, then evaluating metrics for computing them, and discussing how we can utilize them during training.

Original: The ﬁve metrics we study all are highly correlated, and capture human intuition behind what is meant by “prototypical”.
Revised: The ﬁve metrics we study all are highly correlated, and capture human intuition behind what is meant by “prototypical”.

Original: When the metrics disagree on the prototypicality of an example, we can often learn something interesting about that example (e.g., that it is from a rare submode of a class).
Revised: When the metrics disagree on the prototypicality of an example, we can often learn something interesting about that example (e.g., that it is from a rare submode of a class).

Original: Further, we explore the many reasons to utalize prototypes: we ﬁnd that models trained on prototypes often have simpler decision boundaries and are thus more adversarially robust.
Revised: Further, we explore the many reasons to utalize prototypes: we ﬁnd that models trained on prototypes often have simpler decision boundaries and are thus more adversarially robust.

Original: However, training only on prototypes often yields inferior accuracy compared to training on outliers.
Revised: However, training only on prototypes often yields inferior accuracy compared to training on outliers.

Original: We believe that further exploring metrics for identifying prototypes and developing methods for using them during training is an important area of future work, and hope that our analysis will be useful towards that end goal.
Revised: We believe that further exploring metrics for identifying prototypes and developing methods for using them during training is an important area of future work, and hope that our analysis will be useful towards that end goal.

Original: A PPENDIX A R ELATED W ORK
Revised: A PPENDIX A R ELATED W ORK

Original: Prototypes.
Revised: Prototypes.

Original: At least since the work of Zhang (1992) which was based on intra- and inter-concept similarity, prototypes have been examined using several metrics derived from the intuitive notion that one could ﬁnd “quintessential observations that best represent clusters in a dataset” (Kim et al., 2014).
Revised: At least since the work of Zhang (1992) which was based on intra- and inter-concept similarity, prototypes have been examined using several metrics derived from the intuitive notion that one could ﬁnd “quintessential observations that best represent clusters in a dataset” (Kim et al., 2014).

Original: Several more formal variants of this deﬁnition were proposed in the literature—along with corresponding techniques for ﬁnding prototypes.
Revised: Several more formal variants of this deﬁnition were proposed in the literature—along with corresponding techniques for ﬁnding prototypes.

Original: Kim et al.
Revised: Kim et al.

Original: (2016) select prototypes according to their maximum mean discrepancy with the data, which assumes the existence of an appropriate kernel for the data of interest.
Revised: (2016) select prototypes according to their maximum mean discrepancy with the data, which assumes the existence of an appropriate kernel for the data of interest.

Original: Li et al.
Revised: Li et al.

Original: (2017) circumvent this limitation by prepending classiﬁers with an autoencoder projecting the input data on a manifold of reduced dimensionality.
Revised: (2017) circumvent this limitation by prepending classiﬁers with an autoencoder projecting the input data on a manifold of reduced dimensionality.

Original: A prototype layer, which serves as the classiﬁer’s input, is then trained to minimize the distance between inputs and a set of prototypes on this manifold.
Revised: A prototype layer, which serves as the classiﬁer’s input, is then trained to minimize the distance between inputs and a set of prototypes on this manifold.

Original: While this method improves interpretability by ensuring that prototypes are central to the classiﬁer’s logic, it does require that one modify the model’s architecture.
Revised: While this method improves interpretability by ensuring that prototypes are central to the classiﬁer’s logic, it does require that one modify the model’s architecture.

Original: Instead, metrics considered in our manuscript all operate on existing architectures.
Revised: Instead, metrics considered in our manuscript all operate on existing architectures.

Original: Stock &
Revised: Stock &

Original: Cisse (2017) proposed to use distance to the boundary—approximately measured with an adversarial example algorithm—as a proxy for prototypicality.
Revised: Cisse (2017) proposed to use distance to the boundary—approximately measured with an adversarial example algorithm—as a proxy for prototypicality.

Original: Other interpretability approaches.
Revised: Other interpretability approaches.

Original: Prototypes enable interpretability because they provide a subset of examples that summarize the original dataset and best explain a particular decision made at test time (Bien & Tibshirani, 2011).
Revised: Prototypes enable interpretability because they provide a subset of examples that summarize the original dataset and best explain a particular decision made at test time (Bien & Tibshirani, 2011).

Original: Other approaches like saliency maps instead synthetize new inputs to visualize what a neural network has learned.
Revised: Other approaches like saliency maps instead synthetize new inputs to visualize what a neural network has learned.

Original: This is typically done by gradient descent with respect to the input space (Zeiler & Fergus, 2014; Simonyan et al., 2013).
Revised: This is typically done by gradient descent with respect to the input space (Zeiler & Fergus, 2014; Simonyan et al., 2013).

Original: Because they rely on model gradients, saliency maps can be fragile and only locally applicable (Fong & Vedaldi, 2017).
Revised: Because they rely on model gradients, saliency maps can be fragile and only locally applicable (Fong & Vedaldi, 2017).

Original: Beyond interpretability, prototypes are also motivated by additional use cases, some of which we discussed in Section 4.
Revised: Beyond interpretability, prototypes are also motivated by additional use cases, some of which we discussed in Section 4.

Original: Next, we review related work in two of these applications: namely, curriculum learning and reducing sample complexity.
Revised: Next, we review related work in two of these applications: namely, curriculum learning and reducing sample complexity.

Original: Curriculum learning.
Revised: Curriculum learning.

Original: Based on the observation that the order in which training data is presented to the model can improve performance (e.g., convergence) of optimization during learning and circumvent limitations of the dataset (e.g., data imbalance or noisy labels), curriculum learning seeks to ﬁnd the best order in which to analyze training data (Bengio et al., 2009).
Revised: Based on the observation that the order in which training data is presented to the model can improve performance (e.g., convergence) of optimization during learning and circumvent limitations of the dataset (e.g., data imbalance or noisy labels), curriculum learning seeks to ﬁnd the best order in which to analyze training data (Bengio et al., 2009).

Original: This ﬁrst effort further hypothesizes that easy-to-classify samples should be presented early in training while complex samples gradually inserted as learning progresses.
Revised: This ﬁrst effort further hypothesizes that easy-to-classify samples should be presented early in training while complex samples gradually inserted as learning progresses.

Original: While Bengio et al.
Revised: While Bengio et al.

Original: (2009) assumed the existence of hard-coded curriculum labels in the dataset, Chin & Liang (2017) sample an order for the training set by assigning each point a sampling probability proportional to its leverage score—the distance between the point and a linear model ﬁtted to the whole data.
Revised: (2009) assumed the existence of hard-coded curriculum labels in the dataset, Chin & Liang (2017) sample an order for the training set by assigning each point a sampling probability proportional to its leverage score—the distance between the point and a linear model ﬁtted to the whole data.

Original: Instead, we use metrics that also apply to data that cannot be modelled linearly.
Revised: Instead, we use metrics that also apply to data that cannot be modelled linearly.

Original: The curriculum may also be generated online during training, so as to take into account progress made by the learner (Kumar et al., 2010).
Revised: The curriculum may also be generated online during training, so as to take into account progress made by the learner (Kumar et al., 2010).

Original: For instance, Katharopoulos & Fleuret (2017) train an auxiliary LSTM model to predict the loss of training samples, which they use to sample a subset of training points analyzed by the learner at each training iteration.
Revised: For instance, Katharopoulos & Fleuret (2017) train an auxiliary LSTM model to predict the loss of training samples, which they use to sample a subset of training points analyzed by the learner at each training iteration.

Original: Similarly, Jiang et al.
Revised: Similarly, Jiang et al.

Original: (2017) have an auxiliary model predict the curriculum.
Revised: (2017) have an auxiliary model predict the curriculum.

Original: This auxiliary model is trained using the learner’s current feature representation of a smaller holdout set of data for which ground-truth curriculum is known.
Revised: This auxiliary model is trained using the learner’s current feature representation of a smaller holdout set of data for which ground-truth curriculum is known.

Original: However, as reported in our experiments, training on easy samples is beneﬁcial when the dataset is noisy, whereas training on hard examples is on the contrary more effective when data is clean.
Revised: However, as reported in our experiments, training on easy samples is beneﬁcial when the dataset is noisy, whereas training on hard examples is on the contrary more effective when data is clean.

Original: These observations oppose self-paced learning (Kumar et al., 2010) with hard example mining (Shrivastava et al., 2016).
Revised: These observations oppose self-paced learning (Kumar et al., 2010) with hard example mining (Shrivastava et al., 2016).

Original: Several strategies have been proposed to perform better in both settings.
Revised: Several strategies have been proposed to perform better in both settings.

Original: Assuming the existence of a holdout set as well, Ren et al.
Revised: Assuming the existence of a holdout set as well, Ren et al.

Original: (2018) assign a weight to each training example that characterizes the alignment of both the logits and gradients of the learner on training and heldout data.
Revised: (2018) assign a weight to each training example that characterizes the alignment of both the logits and gradients of the learner on training and heldout data.

Original: Chang et al.
Revised: Chang et al.

Original: (2017) propose to train on points with high prediction variance or whose average prediction is close from the decision threshold.
Revised: (2017) propose to train on points with high prediction variance or whose average prediction is close from the decision threshold.

Original: Both the variance and average are estimated by analyzing a sliding window of the history of prediction probabilities throughout training epochs.
Revised: Both the variance and average are estimated by analyzing a sliding window of the history of prediction probabilities throughout training epochs.

Original: Sample complexity.
Revised: Sample complexity.

Original: Prototypes of a given task share some intuition with the notion of coresets (Agarwal et al., 2005; Huggins et al., 2016; Bachem et al., 2017; Tolochinsky & Feldman, 2018) because both prototypes and coresets describe the dataset in a more compact way—by returning a (potentially weighted) subset of the original dataset.
Revised: Prototypes of a given task share some intuition with the notion of coresets (Agarwal et al., 2005; Huggins et al., 2016; Bachem et al., 2017; Tolochinsky & Feldman, 2018) because both prototypes and coresets describe the dataset in a more compact way—by returning a (potentially weighted) subset of the original dataset.

Original: For instance, clustering algorithms may rely on both prototypes (Biehl et al., 2016) or coresets (Biehl et al., 2016) to cope with the high dimensionality of a task.
Revised: For instance, clustering algorithms may rely on both prototypes (Biehl et al., 2016) or coresets (Biehl et al., 2016) to cope with the high dimensionality of a task.

Original: However, prototypes and coresets differ in essential ways.
Revised: However, prototypes and coresets differ in essential ways.

Original: In particular, coresets are deﬁned according to a metric of interest (e.g., the loss that one would like to minimize during training) whereas prototypes are independent of any machine-learning aspects as indicated in our list of desirable properties for prototypicality metrics from Section 2.
Revised: In particular, coresets are deﬁned according to a metric of interest (e.g., the loss that one would like to minimize during training) whereas prototypes are independent of any machine-learning aspects as indicated in our list of desirable properties for prototypicality metrics from Section 2.

Original: Taking a different approach, Wang et al.
Revised: Taking a different approach, Wang et al.

Original: (2018) apply inﬂuence functions (Koh & Liang, 2017) to discard training inputs that do not affect learning.
Revised: (2018) apply inﬂuence functions (Koh & Liang, 2017) to discard training inputs that do not affect learning.

Original: Conversely,  we found in our experiments that removing individual training examples did not have a measurable impact on model performance.
Revised: Conversely, for MNIST, we found in our experiments that removing individual training examples did not have a measurable impact on the predictions of individual test examples. Speciﬁcally, we trained many models to 100% training accuracy where we left one training example out for each model. There was no statistically signiﬁcant difference between the models predictions on each individual test example

Original: B F IGURES OF OUTLIERS AND PROTOTYPES
Revised: B F IGURES OF OUTLIERS AND PROTOTYPES

Original: The following are training examples from MNIST, FashionMNIST, and CIFAR10 that are identiﬁed as most outlier (left of the red bar) or prototypical (right of the green bar).
Revised: The following are training examples from MNIST, FashionMNIST, and CIFAR10 that are identiﬁed as most outlier (left of the red bar) or prototypical (right of the green bar).

Original: Images are presented in groups by class.
Revised: Images are presented in groups by class.

Original: Each row in these groups corresponds to one of the ﬁve metrics in Section 2.1.
Revised: Each row in these groups corresponds to one of the ﬁve metrics in Section 2.1.

Original: B. MNIST EXTREME OUTLIERS AND PROTOTYPES
Revised: B. MNIST EXTREME OUTLIERS AND PROTOTYPES

Original: All MNIST results were obtained with a CNN made up of two convolutional layers (each with kernel size of 5x5 and followed by a 2x2 max-pooling layer) and a fully-connected layer of 256 units.
Revised: All MNIST results were obtained with a CNN made up of two convolutional layers (each with kernel size of 5x5 and followed by a 2x2 max-pooling layer) and a fully-connected layer of 256 units.

Original: It was trained with Adam at a learning rate of 10 − 3 with a 10 − 3 decay.
Revised: It was trained with Adam at a learning rate of 10 − 3 with a 10 − 3 decay.

Original: When an ensemble of models was needed (e.g., for the agr metric), these were obtained by using different random initializations.
Revised: When an ensemble of models was needed (e.g., for the agr metric), these were obtained by using different random initializations.

Original: adv ret agr conf priv adv ret agr conf priv adv ret agr conf priv adv ret agr conf priv
Revised: adv ret agr conf priv adv ret agr conf priv adv ret agr conf priv adv ret agr conf priv

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: B. F ASHION -MNIST EXTREME OUTLIERS AND PROTOTYPES
Revised: B. F ASHION -MNIST EXTREME OUTLIERS AND PROTOTYPES

Original: The Fashion-MNIST model architecture is identical to the one used for MNIST.
Revised: The Fashion-MNIST model architecture is identical to the one used for MNIST.

Original: It was also trained with the same optimizer and hyper-parameters
Revised: It was also trained with the same optimizer and hyper-parameters

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Revised: adv ret agr conf priv adv ret agr conf priv

Original: B. CIFAR EXTREME OUTLIERS AND PROTOTYPES
Revised: B. CIFAR EXTREME OUTLIERS AND PROTOTYPES

Original: All CIFAR results were obtained with a ResNetv2 trained  with the Adam optimizer  at an initial learning rate of 10 − 3 eventually decaying down to5 . 10 − 6 .
Revised: All CIFAR results were obtained with a ResNetv2 trained on batches of 32 points with the Adam optimizer for 100 epochs at an initial learning rate of 10 − 3 decayed down to 10 − 4 after 80 epochs.

Original:  When an ensemble of models was needed (e.g., for the agr metric), these were obtained by using different random initializations. 
Revised: We adapted the following data augmentation and training script: https://raw.githubusercontent.com/keras-team/keras/master/ examples/cifar10_resnet.py When an ensemble of models was needed (e.g., for the agr metric), these were obtained by using different random initializations. adv ret agr conf priv adv ret agr conf priv

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: B. I MAGE N ET EXTREME OUTLIERS AND PROTOTYPES
Revised: B. I MAGE N ET EXTREME OUTLIERS AND PROTOTYPES

Original: The following pre-trained ImageNet models were used: DenseNet121, DenseNet169, DenseNet
Revised: The following pre-trained ImageNet models were used: DenseNet121, DenseNet169, DenseNet

Original: InceptionV3, InceptionResNetV2, Large NASNet, Mobile NASNet, ResNet50, VGG16, VGG19, and Xception.
Revised: InceptionV3, InceptionResNetV2, Large NASNet, Mobile NASNet, ResNet50, VGG16, VGG19, and Xception.

Revised: They are all found in the Keras library: https://keras.io/applications .

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: C P ROTOTYPE ACCURACY WHEN TRAINING ON PROTOTYPES
Revised: C P ROTOTYPE ACCURACY WHEN TRAINING ON PROTOTYPES

Original: The three matrices that follow respectively report the accuracy of MNIST, Fashion-MNIST and
Revised: The three matrices that follow respectively report the accuracy of MNIST, Fashion-MNIST and

Original: CIFAR-10 models learned on training examples with varying degrees of prototypicality and evaluated on test examples also with varying degrees of prototypicality.
Revised: CIFAR-10 models learned on training examples with varying degrees of prototypicality and evaluated on test examples also with varying degrees of prototypicality.

Original: Speciﬁcally, the model used to compute cell ( i, j ) of a matrix is learned on training data that is ranked in the i th percentile of adv prototypicality.
Revised: Speciﬁcally, the model used to compute cell ( i, j ) of a matrix is learned on training data that is ranked in the i th percentile of adv prototypicality.

Original: The model is then evaluated on the test examples whose adv prototypicality falls under the j th prototypicality percentile.
Revised: The model is then evaluated on the test examples whose adv prototypicality falls under the j th prototypicality percentile.

Original: For all datasets, these matrices show that performing well on prototypes is possible even when the model is trained on outliers.
Revised: For all datasets, these matrices show that performing well on prototypes is possible even when the model is trained on outliers.

Original: For MNIST, this shows again that training on outliers provides better performance across the range of test data (from outliers to prototypes).
Revised: For MNIST, this shows again that training on outliers provides better performance across the range of test data (from outliers to prototypes).

Original: For Fashion-MNIST and CIFAR-10, this best performance is achieved by training on examples that are neither prototypical nor outliers.
Revised: For Fashion-MNIST and CIFAR-10, this best performance is achieved by training on examples that are neither prototypical nor outliers.

Original: Prototypicality Percentile of Test Data
Revised: Prototypicality Percentile of Test Data

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: P r o t o t y p
Revised: P r o t o t y p

Original: i c a li t y P e r c e n t il e o f T r a i n
Revised: i c a li t y P e r c e n t il e o f T r a i n

Original: D a t a
Revised: D a t a

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: P r o t o t y p
Revised: P r o t o t y p

Original: i c a li t y P e r c e n t il e o f T r a i n
Revised: i c a li t y P e r c e n t il e o f T r a i n

Original: D a t a
Revised: D a t a

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: D
Revised: D

Original: H UMAN S TUDY
Revised: UMAN S TUDY

Original: E XAMPLE
Revised: XAMPLE

Original: We presented Mechanical Turk taskers with the following webpage, asking them to select the worst image of the nine in the grid.
Revised: We presented Mechanical Turk taskers with the following webpage, asking them to select the worst image of the nine in the grid.

Original: E A DVERSARIAL ROBUSTNESS OF MODEL TRAINED ON PROTOTYPES
Revised: E A DVERSARIAL ROBUSTNESS OF MODEL TRAINED ON PROTOTYPES

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: R EVEALING AND CLUSTERING INTERESTING EXAMPLES AND SUBMODES
Revised: R EVEALING AND CLUSTERING INTERESTING EXAMPLES AND SUBMODES

Original: [Table]
Revised: [Table]

Original: l a
Revised: l a

Original: [Table]
Revised: [Table]

Original: L a
Revised: L a

Original: [Table]
Revised: [Table]

Original: l
Revised: l

Original: [Table]
Revised: [Table]

Original: Distribution of MNIST classes for metric adv 0% 10% 20% 30% 40% Prototypicality Percentile
Revised: Distribution of MNIST classes for metric adv 0% 10% 20% 30% 40% Prototypicality Percentile

Original: C l a ss L a b e l 3.4% 3.1% 6% 6.9% 6.3% 6.4% 3.6% 6.9% 9.7% 7.7% 3.7% 3.2% 6% 6.3% 6.8% 6.5% 4.9% 7.1% 8.3% 7.1% 5.1% 2.2% 5.1% 7.8% 7.4%
Revised: C l a ss L a b e l 3.4% 3.1% 6% 6.9% 6.3% 6.4% 3.6% 6.9% 9.7% 7.7% 3.7% 3.2% 6% 6.3% 6.8% 6.5% 4.9% 7.1% 8.3% 7.1% 5.1% 2.2% 5.1% 7.8% 7.4%

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Percentile
Revised: Percentile

Original: C l a ss L a b e l 3.4% 3%
Revised: C l a ss L a b e l 3.4% 3%

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Distribution of MNIST classes for metric priv 0% 10% 20% 30% 40% Prototypicality Percentile
Revised: Distribution of MNIST classes for metric priv 0% 10% 20% 30% 40% Prototypicality Percentile

Original: C l a ss L a b e l 4.1% 3.7% 6.2% 5.4% 5.8% 5.5% 4.8% 7.7% 5.8% 11% 5.5% 8.9% 5.5% 4.5% 4.3% 4.
Revised: C l a ss L a b e l 4.1% 3.7% 6.2% 5.4% 5.8% 5.5% 4.8% 7.7% 5.8% 11% 5.5% 8.9% 5.5% 4.5% 4.3% 4.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: shirt/top
Revised: shirt/top

Original: Trouser
Revised: Trouser

Original: Pullover
Revised: Pullover

Original: Dress
Revised: Dress

Original: Coat
Revised: Coat

Original: Sandal
Revised: Sandal

Original: Shirt
Revised: Shirt

Original: Sneaker
Revised: Sneaker

Original: Bag
Revised: Bag

Original: Ankle boot
Revised: Ankle boot

Original: C l a ss L a b e l 10.2% 0.5% 9.5% 5.3% 11.6% 1% 13.9% 4.2% 0.4% 3.5% 8.7% 0.7% 9.8%
Revised: C l a ss L a b e l 10.2% 0.5% 9.5% 5.3% 11.6% 1% 13.9% 4.2% 0.4% 3.5% 8.7% 0.7% 9.8%

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Distribution of Fashion-MNIST classes for metric adv
Revised: Distribution of Fashion-MNIST classes for metric adv

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Percentile
Revised: Percentile

Original: [Figure]
Revised: [Figure]

Original: shirt/top
Revised: shirt/top

Original: Trouser
Revised: Trouser

Original: Pullover
Revised: Pullover

Original: Dress
Revised: Dress

Original: Coat
Revised: Coat

Original: Sandal
Revised: Sandal

Original: Shirt
Revised: Shirt

Original: Sneaker
Revised: Sneaker

Original: Bag
Revised: Bag

Original: Ankle boot
Revised: Ankle boot

Original: C l
Revised: C l

Original: [Figure]
Revised: [Figure]

Original: ss L a b e l
Revised: ss L a b e l

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: Distribution of
Revised: Distribution of

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: % 20% 30% 40% Prototypicality Percentile
Revised: % 20% 30% 40% Prototypicality Percentile

Original: [Figure]
Revised: [Figure]

Original: shirt/top
Revised: shirt/top

Original: Trouser
Revised: Trouser

Original: Pullover
Revised: Pullover

Original: Dress
Revised: Dress

Original: Coat
Revised: Coat

Original: Sandal
Revised: Sandal

Original: Shirt
Revised: Shirt

Original: Sneaker
Revised: Sneaker

Original: Bag
Revised: Bag

Original: Ankle boot
Revised: Ankle boot

Original: C l a ss L a b e l
Revised: C l a ss L a b e l

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Distribution of Fashion
Revised: Distribution of Fashion

Original: [Figure]
Revised: [Figure]

Original: classes for metric agr 0% 10% 20% 30%
Revised: classes for metric agr 0% 10% 20% 30%

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Percentile
Revised: Percentile

Original: [Figure]
Revised: [Figure]

Original: /top
Revised: /top

Original: Trouser
Revised: Trouser

Original: Pullover
Revised: Pullover

Original: Dress
Revised: Dress

Original: Coat
Revised: Coat

Original: [Figure]
Revised: [Figure]

Original: Shirt
Revised: Shirt

Original: [Figure]
Revised: [Figure]

Original: Bag
Revised: Bag

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Airplane
Revised: Airplane

Original: Automobile
Revised: Automobile

Original: Bird
Revised: Bird

Original: Cat
Revised: Cat

Original: Deer
Revised: Deer

Original: Dog
Revised: Dog

Original: Frog
Revised: Frog

Original: Horse
Revised: Horse

Original: Ship
Revised: Ship

Original: Truck
Revised: Truck

Original: C l a ss L a b e l 3.3% 2.8% 5.5% 12% 5.1% 9.6% 2.3% 4.4% 2.3% 2.8%
Revised: C l a ss L a b e l 3.3% 2.8% 5.5% 12% 5.1% 9.6% 2.3% 4.4% 2.3% 2.8%

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Distribution of CIFAR-10 classes for metric adv 0% 10% 20% 30% 40% Prototypicality Percentile
Revised: Distribution of CIFAR-10 classes for metric adv 0% 10% 20% 30% 40% Prototypicality Percentile

Original: Airplane
Revised: Airplane

Original: Automobile
Revised: Automobile

Original: Bird
Revised: Bird

Original: Cat
Revised: Cat

Original: Deer
Revised: Deer

Original: Dog
Revised: Dog

Original: Frog
Revised: Frog

Original: Horse
Revised: Horse

Original: Ship
Revised: Ship

Original: Truck
Revised: Truck

Original: C l a ss L a b e l 4.2% 1.6% 7.6% 9.6% 6.3% 7.5% 3% 5.1% 2.9% 2.2% 4.4% 1.9% 8.1% 8.5% 7.3% 6.9% 3% 4.4% 3.2% 2.3% 5% 2.2% 7.3% 8.1% 6.9% 6.8% 3% 4.5% 3.5% 2.6% 5% 2.8% 7.3% 7.1% 6.5% 6.4% 3.5% 4.5% 4% 2.9% 5.6% 3.6% 5.8% 6.1% 6.3% 5.6% 3.9% 4.7% 4.5% 3.9% 5.6% 4.8% 5.1% 4.5% 6.1% 5.2% 4.4% 4.5% 5.2% 4.7% 6% 6.2% 4% 3.3% 4.5% 4.5% 4.8% 4.5% 6.1% 6% 6% 8.2% 2.6% 2% 3.8% 3.7% 5.1% 4.6% 7% 7% 5.4% 8.8% 1.7% 0.7% 1.8% 2.3% 6.
Revised: C l a ss L a b e l 4.2% 1.6% 7.6% 9.6% 6.3% 7.5% 3% 5.1% 2.9% 2.2% 4.4% 1.9% 8.1% 8.5% 7.3% 6.9% 3% 4.4% 3.2% 2.3% 5% 2.2% 7.3% 8.1% 6.9% 6.8% 3% 4.5% 3.5% 2.6% 5% 2.8% 7.3% 7.1% 6.5% 6.4% 3.5% 4.5% 4% 2.9% 5.6% 3.6% 5.8% 6.1% 6.3% 5.6% 3.9% 4.7% 4.5% 3.9% 5.6% 4.8% 5.1% 4.5% 6.1% 5.2% 4.4% 4.5% 5.2% 4.7% 6% 6.2% 4% 3.3% 4.5% 4.5% 4.8% 4.5% 6.1% 6% 6% 8.2% 2.6% 2% 3.8% 3.7% 5.1% 4.6% 7% 7% 5.4% 8.8% 1.7% 0.7% 1.8% 2.3% 6.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Airplane
Revised: Airplane

Original: Automobile
Revised: Automobile

Original: Bird
Revised: Bird

Original: Cat
Revised: Cat

Original: Deer
Revised: Deer

Original: Dog
Revised: Dog

Original: Frog
Revised: Frog

Original: Horse
Revised: Horse

Original: Ship
Revised: Ship

Original: Truck
Revised: Truck

Original: C l a ss L a b e l
Revised: C l a ss L a b e l

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Distribution of CIFAR-10 classes for metric agr 0% 10% 20% 30% 40% Prototypicality Percentile
Revised: Distribution of CIFAR-10 classes for metric agr 0% 10% 20% 30% 40% Prototypicality Percentile

Original: Airplane
Revised: Airplane

Original: Automobile
Revised: Automobile

Original: Bird
Revised: Bird

Original: Cat
Revised: Cat

Original: Deer
Revised: Deer

Original: Dog
Revised: Dog

Original: Frog
Revised: Frog

Original: Horse
Revised: Horse

Original: Ship
Revised: Ship

Original: Truck
Revised: Truck

Original: C l a ss L a b e l 3.8% 1.
Revised: C l a ss L a b e l 3.8% 1.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Percentile
Revised: Percentile

Original: Airplane
Revised: Airplane

Original: Automobile
Revised: Automobile

Original: Bird
Revised: Bird

Original: Cat
Revised: Cat

Original: Deer
Revised: Deer

Original: Dog
Revised: Dog

Original: Frog
Revised: Frog

Original: Horse
Revised: Horse

Original: Ship
Revised: Ship

Original: Truck
Revised: Truck

Original: C l a ss L a b e l 3.5% 1.8% 6.6% 11.1% 6.8% 9% 2% 3.5% 3.1% 2.6% 5.7% 2.2% 6.
Revised: C l a ss L a b e l 3.5% 1.8% 6.6% 11.1% 6.8% 9% 2% 3.5% 3.1% 2.6% 5.7% 2.2% 6.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: H A LL MEMORIZED EXCEPTIONS FOR ALL F ASHION -MNIST CLASSES
Revised: H A LL MEMORIZED EXCEPTIONS FOR ALL F ASHION -MNIST CLASSES

Original: Below are all the memorized exceptions, as deﬁned in the body of the paper, for all Fashion-MNIST output classes:
Revised: Below are all the memorized exceptions, as deﬁned in the body of the paper, for all Fashion-MNIST output classes:

Original: • Tshirt/top • Trouser • Pullover • Dress • Coat • Sandal • Shirt • Sneaker • Bag • Ankle boot
Revised: • Tshirt/top • Trouser • Pullover • Dress • Coat • Sandal • Shirt • Sneaker • Bag • Ankle boot

Original: / fashion.html ﬁle:///usr/local/google/home/ncarlini/Desktop/fashion.html 1/ 
Revised: / fashion.html ﬁle:///usr/local/google/home/ncarlini/Desktop/fashion.html 1/ [Table] [Table] Maine

Revised: [Table]

Revised: Homarus americanus

Revised: Index 124 crayfish, crawfish, crawdad, crawdaddy

Revised: Index 126 isopod

Revised: Index 161 basset, basset hound

Revised: Index 166 Walker hound, Walker foxhound

Revised: Index 167 English foxhound

Revised: Index 170 Irish wolfhound

Revised: Index 179 Staffordshire bullterrier, Staffordshire bull terrier

Revised: Index 180 American Staffordshire terrier, Staffordshire terrier, American pit bull terrier, pit bull terrier

Revised: Index 196 miniature schnauzer

Revised: Index 197 giant schnauzer

Revised: Index 198 standard schnauzer

Revised: Index 206 curly-coated retriever

Revised: Index 214 Gordon setter

Revised: Index 223 schipperke

Revised: Index 238 Greater Swiss Mountain dog

Revised: Index 248 Eskimo dog, husky

Revised: Index 250 Siberian husky

Revised: Index 264 Cardigan, Cardigan Welsh corgi

Revised: Index 265 toy poodle

Revised: Index 266 miniature poodle

Revised: Index 270 white wolf, Arctic wolf, Canis lupus tundrarum

Revised: Index 278 kit fox, Vulpes macrotis

Revised: Index 290 jaguar, panther, Panthera onca, Felis onca

Revised: Index 304 leaf beetle, chrysomelid

Revised: Index 311 grasshopper, hopper

Revised: Index 312 cricket

Revised: Index 319 dragonfly, darning needle, devil's darning needle, sewing needle, snake feeder, snake doctor, mosquito hawk, skeeter hawk

Revised: Index 320 damselfly

Revised: Index 334 porcupine, hedgehog

Revised: Index 341 hog, pig, grunter, squealer, Sus scrofa

Revised: Index 342 wild boar, boar, Sus scrofa

Revised: Index 345 ox

Revised: Index 348 ram, tup

Revised: Index 349 bighorn, bighorn sheep, cimarron, Rocky Mountain bighorn, Rocky Mountain sheep, Ovis canadensis

Revised: Index 359 black-footed ferret, ferret, Mustela nigripes

Revised: Index 380 titi, titi monkey

Revised: Index 383 Madagascar cat, ring-tailed lemur, Lemur catta

Revised: Index 386 African elephant, Loxodonta africana

Revised: Index 390 eel

Revised: Index 399 abaya

Revised: Index 400 academic gown, academic robe, judge's robe

Revised: Index 409 analog clock

Revised: Index 413 assault rifle, assault gun

Revised: Index 417 balloon

Revised: Index 419 Band Aid

Revised: Index 423 barber chair

Revised: Index 424 barbershop

Revised: Index 429 baseball

Revised: Index 434 bath towel

Revised: Index 435 bathtub, bathing tub, bath, tub

Revised: Index 440 beer bottle

Revised: Index 461 breastplate, aegis, egis

Revised: Index 465 bulletproof vest

Revised: Index 479 car wheel

Revised: Index 484 catamaran

Revised: Index 505 coffeepot

Revised: Index 516 cradle

Revised: Index 524 cuirass

Revised: Index 538 dome

Revised: Index 541 drum, membranophone, tympan

Revised: Index 550 espresso maker

Revised: Index 579 grand piano, grand

Revised: Index 583 guillotine

Revised: Index 591 handkerchief, hankie, hanky, hankey

Revised: Index 595 harvester, reaper

Revised: Index 604 hourglass

Revised: Index 619 lampshade, lamp shade

Revised: Index 620 laptop, laptop computer

Revised: Index 636 mailbag, postbag

Revised: Index 638 maillot

Revised: Index 639 maillot, tank suit

Revised: Index 643 mask

Revised: Index 647 measuring cup

Revised: Index 657 missile

Revised: Index 665 moped

Revised: Index 667 mortarboard

Revised: Index 668 mosque

Revised: Index 670 motor scooter, scooter

Revised: Index 678 neck brace

Revised: Index 681 notebook, notebook computer

Revised: Index 700 paper towel

Revised: Index 739 potter's wheel

Revised: Index 744 projectile, missile

Revised: Index 748 purse

Revised: Index 764 rifle

Revised: Index 804 soap dispenser

Revised: Index 808 sombrero

Revised: Index 810 space bar

Revised: Index 817 sports car, sport car

Revised: Index 827 stove

Revised: Index 830 stretcher

Revised: Index 836 sunglass

Revised: Index 837 sunglasses, dark glasses, shades

Revised: Index 841 sweatshirt

Revised: Index 842 swimming trunks, bathing trunks

Revised: Index 846 table lamp

Revised: Index 847 tank, army tank, armored combat vehicle, armoured combat vehicle

Revised: Index 876 tub, vat

Revised: Index 878 typewriter keyboard

Revised: Index 892 wall clock

Revised: Index 903 wig

Revised: Index 907 wine bottle

Revised: Index 914 yawl

Revised: Index 925 consomme

Revised: Index 928 ice cream, icecream

Revised: Index 939 zucchini, courgette

Revised: Index 954 banana

Revised: Index 960 chocolate sauce, chocolate syrup

Revised: Index 961 dough

Revised: Index 962 meat loaf, meatloaf

Revised: Index 966 red wine

Revised: Index 981 ballplayer, baseball player

Revised: Index 987 corn