Original: I AM G OING MAD: M AXIMUM D ISCREPANCY C OMPETITION FOR C OMPARING C LASSIFIERS A DAPTIVELYAnonymous authors
Revised: I A M G OING MAD: M AXIMUM D ISCREPANCY C OMPETITION FOR C OMPARING C LASSIFIERS A DAPTIVELY

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The learning of hierarchical representations for image classiﬁcation has experienced an impressive series of successes due in part to the availability of largescale labeled data for training.
Revised: The learning of hierarchical representations for image classiﬁcation has experienced an impressive series of successes due in part to the availability of largescale labeled data for training.

Original: On the other hand, the trained classiﬁers have traditionally been evaluated on a handful of test images, which are deemed to be extremely sparsely distributed in the space of all natural images.
Revised: On the other hand, the trained classiﬁers have traditionally been evaluated on small and ﬁxed sets of test images, which are deemed to be extremely sparsely distributed in the space of all natural images.

Original: It is thus questionable whether recent performance improvements on the excessively re-used test sets generalize to real-world natural images with much richer content variations.
Revised: It is thus questionable whether recent performance improvements on the excessively re-used test sets generalize to real-world natural images with much richer content variations.

Original: In addition, studies on adversarial learning show that it is effortless to construct adversarial examples that fool nearly all image classiﬁers, adding more complications to relative performance comparison of existing models. This work presents an efﬁcient  framework for comparing image classiﬁers, which we name the MAximum Discrepancy (MAD) competition.
Revised: Inspired by efﬁcient stimulus selection for testing perceptual models in psychophysical and physiological studies, we present an alternative framework for comparing image classiﬁers, which we name the MAximum Discrepancy (MAD) competition.

Original: Rather than comparing image classiﬁers on ﬁxed test sets, we adaptively sample a  test set from an arbitrarily large corpus of unlabeled images so as to maximize the discrepancies between the classiﬁers, measured by the distance over WordNet hierarchy.
Revised: Rather than comparing image classiﬁers using ﬁxed test images, we adaptively sample a small test set from an arbitrarily large corpus of unlabeled images so as to maximize the discrepancies between the classiﬁers, measured by the distance over WordNet hierarchy.

Original: Human labeling on the resultingsmall and model-dependent image sets reveals the relative performance of the competing classiﬁers and provides useful insights on potential ways to improve them.
Revised: Human labeling on the resulting model-dependent image sets reveals the relative performance of the competing classiﬁers, and provides useful insights on potential ways to improve them.

Original: We report the MAD competition results of eleven ImageNet classiﬁers while noting that the framework is readily extensible and cost-effective to add future classiﬁers into the competition. 
Revised: We report the MAD competition results of eleven ImageNet classiﬁers while noting that the framework is readily extensible and costeffective to add future classiﬁers into the competition. Codes can be found at https: github .com/TAMU-VITA/MAD .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Large-scale human-labeled image datasets such as ImageNet (Deng et al., 2009) have greatly contributed to the rapid progress of research in image classiﬁcation.
Revised: Large-scale human-labeled image datasets such as ImageNet (Deng et al., 2009) have greatly contributed to the rapid progress of research in image classiﬁcation.

Original: In recent years, considerable effort has been put to designing novel network architectures (He et al., 2016;
Revised: In recent years, considerable effort has been put into designing novel network architectures (He et al., 2016;

Original: Hu et al., 2018) and advanced optimization algorithms (Kingma & Ba, 2015) to improve the training of image classiﬁers based on deep neural networks (DNNs), while little attention has been paid to comprehensive and fair evaluation/comparison of their model performance.
Revised: Hu et al., 2018) and advanced optimization algorithms (Kingma & Ba, 2015) to improve the training of image classiﬁers based on deep neural networks (DNNs), while little attention has been paid to comprehensive and fair evaluation/comparison of their model performance.

Original: Conventional model evaluation methodology for image classiﬁcation generally follows a three-step approach (Burnham & Anderson, 2003).
Revised: Conventional model evaluation methodology for image classiﬁcation generally follows a three-step approach (Burnham & Anderson, 2003).

Original: First, pre-select a number of images from the space of all possible natural images ( i .
Revised: First, pre-select a number of images from the space of all possible natural images ( i .

Original: e ., natural image manifold) to form the test set.
Revised: e ., natural image manifold) to form the test set.

Original: Second, collect the human label for each image in the test set to identify its ground-truth category.
Revised: Second, collect the human label for each image in the test set to identify its ground-truth category.

Original: Third, rank the competing classiﬁers according to their goodness of ﬁt ( e .
Revised: Third, rank the competing classiﬁers according to their goodness of ﬁt ( e .

Original: g ., accuracy) on the test set; the one with the best goodness of ﬁt is declared the winner.
Revised: g ., accuracy) on the test set; the one with the best result is declared the winner.

Original: A signiﬁcant problem with this methodology is the apparent contradiction between the enormous size and high dimensionality of natural image manifold and the limited scale of affordable testing ( i .
Revised: A signiﬁcant problem with this methodology is the apparent contradiction between the enormous size and high dimensionality of natural image manifold and the limited scale of affordable testing ( i .

Original: e ., human labeling, or verifying predicted labels, which is expensive and time consuming).
Revised: e ., human labeling, or verifying predicted labels, which is expensive and time consuming).

Original: As a result, a typical “large-scale” test set for image classiﬁcation allows foronly tens of thousands of natural  images to be examined, which are deemed to be extremely sparsely distributed in natural image manifold.
Revised: As a result, a typical “large-scale” test set for image classiﬁcation allows for tens of thousands of natural [Figure] images to be examined, which are deemed to be extremely sparsely distributed in natural image manifold.

Original: Model comparison based on a limited number of samples assume that they are sufﬁciently representative of the whole population, an assumption that has been proven to be doubtful in image classiﬁcation.
Revised: Model comparison based on a limited number of samples assumes that they are sufﬁciently representative of the whole population, an assumption that has been proven to be doubtful in image classiﬁcation.

Original: Speciﬁcally, Recht et al.
Revised: Speciﬁcally, Recht et al.

Original: (2019) found that a minute natural distribution shift leads to a large drop in accuracy for a broad range of image classiﬁers on both CIFAR-10 (Krizhevsky, 2009) and ImageNet (Deng et al., 2009), suggesting that the current test sets may  far less sufﬁce to represent hard natural images encountered in the real world.
Revised: (2019) found that a minute natural distribution shift leads to a large drop in accuracy for a broad range of image classiﬁers on both CIFAR-10 (Krizhevsky, 2009) and ImageNet (Deng et al., 2009), suggesting that the current test sets may be far from sufﬁcient to represent hard natural images encountered in the real world.

Original: Another problem with the conventional model comparison methodology is that the test sets are pre-selected and therefore ﬁxed.
Revised: Another problem with the conventional model comparison methodology is that the test sets are pre-selected and therefore ﬁxed.

Original: This leaves[Figure] open the door of adapting classiﬁers to the test images, deliberately or unintentionally, via extensive hyperparameter tuning, raising the risk of overﬁtting.
Revised: This leaves the door open for adapting classiﬁers to the test images, deliberately or unintentionally, via extensive hyperparameter tuning, raising the risk of overﬁtting.

Original: As a result, it is never guaranteed that image classiﬁers with highly competitive performance on such small and ﬁxed test sets can generalize to real-world natural images with much richer content variations.
Revised: As a result, it is never guaranteed that image classiﬁers with highly competitive performance on such small and ﬁxed test sets can generalize to real-world natural images with much richer content variations.

Original: In addition, recent studies on adversarial learning (Goodfellow et al., 2015; Madry et al., 2018) indicate that adversarial images produced to mislead a speciﬁc image classiﬁer have strong transferability to fool other classiﬁers even if their design philosophies differ substantially, which further complicates the relative performance comparison of existing classiﬁers.

Original: In order to reliably measure the progress in image classiﬁcation and to fairly test the generalizability of existing classiﬁers in a natural setting, we believe  a much larger test set in the order of millions or even billions must be used.
Revised: In order to reliably measure the progress in image classiﬁcation and to fairly test the generalizability of existing classiﬁers in a natural setting, we believe that it is necessary to compare the classiﬁers on a much larger image collection in the order of millions or even billions.

Original: Apparently, the main challenge here is how to exploit such a large-scale test set under the constraint ofthe very limited budget for human labeling, knowing that collecting ground-truth labels for all images is extremely difﬁcult, if not impossible.
Revised: Apparently, the main challenge here is how to exploit such a large-scale test set under the constraint of very limited budgets for human labeling, knowing that collecting ground-truth labels for all images is extremely difﬁcult, if not impossible.

Original: In this work, we propose an efﬁcient and practical methodology, namely the MAximum Discrepancy (MAD) competition, to meet this challenge.
Revised: In this work, we propose an efﬁcient and practical methodology, namely the MAximum Discrepancy (MAD) competition, to meet this challenge.

Revised: Inspired by Wang & Simoncelli (2008) and Ma et al.

Original: Instead of trying to prove an image classiﬁer to be correct using a small and ﬁxed test set, MAD starts with a large-scale unlabeled image set and attempts to falsify a classiﬁer by ﬁnding a set of images, whose predictions are in strong disagreements with the rest competing classiﬁers (See Figure 1).
Revised: (2019), instead of trying to prove an image classiﬁer to be correct using a small and ﬁxed test set, MAD starts with a large-scale unlabeled image set, and attempts to falsify a classiﬁer by ﬁnding a set of images, whose predictions are in strong disagreement with the rest competing classiﬁers (see Figure 1).

Original: A classiﬁer that is harder to be falsiﬁed  is considered better.
Revised: A classiﬁer that is harder to be falsiﬁed in MAD is considered better.

Original: The initial image set for MAD to explore can be made arbitrarily large provided that the cost of computational prediction for all competing classiﬁers is cheap.
Revised: The initial image set for MAD to explore can be made arbitrarily large provided that the cost of computational prediction for all competing classiﬁers is cheap.

Original: To quantify the discrepancy between two classiﬁers on one image, we propose a weighted distance over WordNet hierarchy (Miller, 1998), which is more semantically aligned with human cognition compared with traditional binary judgment (agree vs. disagree).
Revised: To quantify the discrepancy between two classiﬁers on one image, we propose a weighted distance over WordNet hierarchy (Miller, 1998), which is more semantically aligned with human cognition compared with traditional binary judgment (agree vs. disagree).

Original: The set of model-dependent images selected by MAD are the most informative in discriminating the competing classiﬁers.
Revised: The set of model-dependent images selected by MAD are the most informative in discriminating the competing classiﬁers.

Original: Subjective experiments on the MAD test set reveal the relative strengths and weaknesses among the classiﬁers and identify the training techniques and architecture choices that improve the generalizability to natural image manifold.
Revised: Subjective experiments on the MAD test set reveal the relative strengths and weaknesses among the classiﬁers, and identify the training techniques and architecture choices that improve the generalizability to natural image manifold.

Original: Moreover, careful inspection of the selected images may suggest potential ways to improve a classiﬁer or to combine aspects of multiple classiﬁers.
Revised: This suggests potential ways to improve a classiﬁer or to combine aspects of multiple classiﬁers.

Original: We apply the MAD competition to compare eleven ImageNet classiﬁers and ﬁnd that MAD veriﬁes the relative improvements achieved by recent DNN-based methods, with a minimal subjective testing budget.
Revised: We apply the MAD competition to compare eleven ImageNet classiﬁers, and ﬁnd that MAD veriﬁes the relative improvements achieved by recent DNN-based methods, with a minimal subjective testing budget.

Original: MAD is readily extensible, allowing future classiﬁers to be added into  competition with little additional cost.
Revised: MAD is readily extensible, allowing future classiﬁers to be added into the competition with little additional cost.

Original: Moreover, the application scope of MAD is far beyond image classiﬁcation.

Original: It can be extended to many other research ﬁelds that expect discrete-valued outputs, and is especially useful when the sample space is enormous and the ground-truth measurement is expensive.

Original:  T HE MAD C OMPETITION M ETHODOLOGY
Revised:  T HE MAD C OMPETITION M ETHODOLOGY

Original: The general problem of model comparison in image classiﬁcation may be formulated as follows.
Revised: The general problem of model comparison in image classiﬁcation may be formulated as follows.

Original: We work with the natural image manifold X , upon which we deﬁne a class label f
Revised: We work with the natural image manifold X , upon which we deﬁne a class label f

Original: ( x ) ∈ Y for all natural images x ∈ X , where Y = { 1 , 2 , . .
Revised: ( x ) ∈ Y for every x ∈ X , where Y = { 1 , 2 , . .

Original: . , c } indicates which of c categories the object in x belongs to.
Revised: , c } and c is the number of categories.

Original: We assume a subjective assessment environment, in which a human subject can identify the category membership for any natural image x among all possible categories.
Revised: We assume a subjective assessment environment, in which a human subject can identify the category membership for any natural image x among all possible categories.

Original: A group of image classiﬁers F = { f i } mi =1 are also assumed, each of which takes a natural image x as input and makes a prediction of f ( x ) , collectively denoted by { f i ( x ) } mi =1 .
Revised: A group of image classiﬁers F = { f i } m i =1 are also assumed, each of which takes a natural image x as input and makes a prediction of f ( x ) , collectively denoted by { f i ( x ) } mi =1 .

Original: The goal is to compare the relative performance of m classiﬁers under very limited resource for subjective testing.
Revised: The goal is to compare the relative performance of m classiﬁers under very limited resource for subjective testing.

Original: A conventional model comparison method for image classiﬁcation ﬁrst samples a natural image set D = { x k } nk =1 ⊂ X .
Revised: The conventional model comparison method for image classiﬁcation ﬁrst samples a natural image set D = { x k } nk =1 ⊂ X .

Original: For each image x k ∈ D , we ask human annotators to provide the ground-truth label f ( x k ) ∈ Y .
Revised: For each image x k ∈ D , we ask human annotators to provide the groundtruth label f ( x k ) ∈ Y .

Original: Since human labeling isan expensive and time consuming process and DNNbased classiﬁers are hungry for labeled data in the training stage (Krizhevsky et al., 2012), n is typically small in the order of tens of thousands (Russakovsky et al., 2015).
Revised: Since human labeling is expensive and time consuming, and DNN-based classiﬁers are hungry for labeled data in the training stage (Krizhevsky et al., 2012), n is typically small in the order of tens of thousands (Russakovsky et al., 2015).

Original: The predictions of the classiﬁers are compared with the human labels by computing the empirical classiﬁcation accuracy
Revised: The predictions of the classiﬁers are compared against the human labels by computing the empirical classiﬁcation accuracy

Original: [Equation]
Revised: [Equation]

Original: The classiﬁer f i with a higher classiﬁcation accuracy is said to outperform f j with a lower accuracy.
Revised: The classiﬁer f i with a higher classiﬁcation accuracy is said to outperform f j with a lower accuracy.

Original: As an alternative, the proposed MAD competition methodology aims to falsify a classiﬁer in the most efﬁcient way with the help of other competing classiﬁers.
Revised: As an alternative, the proposed MAD competition methodology aims to falsify a classiﬁer in the most efﬁcient way with the help of other competing classiﬁers.

Original: A classiﬁer that is more likely to be falsiﬁed is considered worse.
Revised: A classiﬁer that is more likely to be falsiﬁed is considered worse.

Original: 2. T HE MAD C OMPETITION P ROCEDURE
Revised: 2. T HE MAD C OMPETITION P ROCEDURE

Original: The MAD competition methodology starts by sampling an image set D
Revised: The MAD competition methodology starts by sampling an image set D

Original: = { x k } nk =1 from the natural image manifold X .
Revised: = { x k } nk =1 from the natural image manifold X .

Original: Since the number of images selected by MAD for subjective testing is independent of the size of D , we may make n  arbitrarily large such that D provides dense coverage of ( i .
Revised: Since the number of images selected by MAD for subjective testing is independent of the size of D , we may choose n to be arbitrarily large such that D provides dense coverage of ( i .

Original: e ., sufﬁciently represents) X .
Revised: e ., sufﬁciently represents) X .

Original: MAD relies on a distance measure to quantify the degree of discrepancy between the predictions of any two classiﬁers.
Revised: MAD relies on a distance measure to quantify the degree of discrepancy between the predictions of any two classiﬁers.

Original: The most straightforward measure is the 0-1 loss:
Revised: The most straightforward measure is the 0-1 loss:

Original: [Equation]
Revised: [Equation]

Original: Unfortunately, it ignores the semantic relations between class labels, which may be crucial in distinguishing two classiﬁers, especially when they share similar design philosophies ( e .
Revised: Unfortunately, it ignores the semantic relations between class labels, which may be crucial in distinguishing two classiﬁers, especially when they share similar design philosophies ( e .

Original: g ., using DNNs as backbones) and is trained on the same image set ( e .
Revised: g ., using DNNs as backbones) and are trained on the same image set ( e .

Original: g ., ImageNet).
Revised: g ., ImageNet).

Original: For example, misclassifying a “chihuahua” as a dog of other species is clearly more acceptable compared to misclassifying it as a “watermelon”.
Revised: For example, misclassifying a “chihuahua” as a dog of other species is clearly more acceptable compared with misclassifying it as a “watermelon”.

Original: We propose to leverage the semantic hierarchies in WordNet (Miller, 1998) to measure the distance between two (predicted) class labels.
Revised: We propose to leverage the semantic hierarchy in WordNet (Miller, 1998) to measure the distance between two (predicted) class labels.

Original: Speciﬁcally, we model WordNet as a weighted undirected graph G ( V, E ) 1 .
Revised: Speciﬁcally, we model WordNet as a weighted undirected graph G ( V, E ) 1 .

Original: Each edge e = ( u, v ) ∈
Revised: Each edge e = ( u, v ) ∈

Original: E connects a parent node u of a more general level ( e .
Revised: E connects a parent node u of a more general level ( e .

Original: g ., canine) to its child node v of a more speciﬁc level ( e .
Revised: g ., canine) to its child node v of a more speciﬁc level ( e .

Original: g ., dog), for u, v ∈ V .
Revised: g ., dog), for u, v ∈ V .

Original: A nonnegative weight w ( e ) is assigned to each edge e
Revised: A nonnegative weight w ( e ) is assigned to each edge e

Original: = ( u, v ) to encode the semantic similarity between u and v .
Revised: = ( u, v ) to encode the semantic distance [Figure] [Figure] [Figure]

Original:  A larger w ( e ) indicates that u and v are semantically more dissimilar.
Revised: between u and v . A larger w ( e ) indicates that u and v are semantically more dissimilar.

Original: We measure the distance between two labels as the sum of the weights assigned to the edges along the shortest path P connecting them
Revised: We measure the distance between two labels as the sum of the weights assigned to the edges along the shortest path P connecting them

Original: [Equation]
Revised: [Equation]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: Eq.
Revised: Eq.

Original: (3) reduces to the standard graph hop distance between two vertices by setting w ( e ) = 1 .
Revised: (3) reduces to the standard graph hop distance between two vertices by setting w ( e ) = 1 .

Original: Here we design w ( e ) to be inversely proportional to the tree depth level l of the parent node ( e .
Revised: We design w ( e ) to be inversely proportional to the tree depth level l of the parent node ( e .

Original: g ., w ( e ) ∝ 2 − l ).
Revised: g ., w ( e ) ∝ 2 − l ).

Original: In other words, we prefer the shortest paths to traverse the root node (or nodes with smaller l ) as a way of encouraging f i ( x ) and f j ( x ) to differ in a more general level ( e .
Revised: In other words, we prefer the shortest paths to traverse the root node (or nodes with smaller l ) as a way of encouraging f i ( x ) and f j ( x ) to differ in a more general level ( e .

Original: g ., vehicle rather than watercraft).
Revised: g ., vehicle rather than watercraft).

Original: Figure 2 shows the semantic advantages of our choice of weights compared to the equal weight.
Revised: Figure 2 shows the semantic advantages of our choice of weighting compared with the equal weighting.

Original: With the distance measure at hand, the optimal image in terms of discriminating f i and f j can be obtained by maximizing the discrepancies between the two classiﬁers on D
Revised: With the distance measure at hand, the optimal image in terms of discriminating f i and f j can be obtained by maximizing the discrepancy between the two classiﬁers on D

Original: [Equation]
Revised: [Equation]

Original: The queried image label f ( x (cid:63) ) leads to three possible outcomes (see Figure 1):
Revised: The queried image label f ( x (cid:63) ) leads to three possible outcomes (see Figure 1):

Original: • Case I .
Revised: • Case I .

Original: Both classiﬁers make correct predictions.
Revised: Both classiﬁers make correct predictions.

Original: Although theoretically impossible based on the general problem formulation, it is not uncommon in practice that a natural image may contain multiple distinct objects ( e .
Revised: Although theoretically impossible based on the general problem formulation, it is not uncommon in practice that a natural image may contain multiple distinct objects ( e .

Original: g ., guacamole and mortar).
Revised: g ., guacamole and mortar).

Original: In this case, f i and f j successfully recognize different objects in x (cid:63) , indicating that both classiﬁers tend to perform at a high level and cannot be differentiated.
Revised: In this case, f i and f j successfully recognize different objects in x (cid:63) , indicating that both classiﬁers tend to perform at a high level.

Original: By restricting D tocontain only  natural images with a single salient object, we may reduce the possibility of this outcome.
Revised: By restricting D to only contain natural images with a single salient object, we may reduce the possibility of this outcome.

Original: • Case II .
Revised: • Case II .

Original: f i (or f j ) makes correct prediction, while f j (or f i ) makes incorrect prediction.
Revised: f i (or f j ) makes correct prediction, while f j (or f i ) makes incorrect prediction.

Original: In this case, MAD automatically identiﬁes a strong failure case to falsify one classiﬁer, not the other; a clear winner is obtained.
Revised: In this case, MAD automatically identiﬁes a strong failure case to falsify one classiﬁer, not the other; a clear winner is obtained.

Original: The selected image x (cid:63) provides the strongest evidence in ranking the relative performance between the two classiﬁers.
Revised: The selected image x (cid:63) provides the strongest evidence in differentiating the two classiﬁers as well as ranking their relative performance.

Original: • Case III .
Revised: • Case III .

Original: Both classiﬁers make incorrect predictions in a multiclass image classiﬁcation problem ( i .
Revised: Both classiﬁers make incorrect predictions in a multiclass image classiﬁcation problem ( i .

Original: e ., c ≥ 3 ).
Revised: e ., c ≥ 3 ).

Original: Although both classiﬁers make mistakes, they differ substantially during inference, which in turn provides a strong indication of their respective weaknessesand suggests potential ways to combine them into a single better classiﬁer 2 .
Revised: Although both classiﬁers make mistakes, they differ substantially during inference, which in turn provides a strong indication of their respective weaknesses 2 .

Original: However, in this case, x (cid:63) contributes little to performance comparison between the two classiﬁers.
Revised: Depending on the subjective experimental setting, x (cid:63) may be used to rank the classiﬁers based on Eq.

Original: To obtain a reliable performance comparison between f i and f j in practice, we choose top- k images in D with k largest distances computed by Eq. (3) to form the test subset S { i,j } . MAD runs this game among all (cid:0) m 2 (cid:1) distinct pairs of classiﬁers, resulting in the ﬁnal MAD test set S =
Revised: (3) if the full label f

Original: (cid:83) S { i,j } .
Revised: ( x (cid:63) ) has been collected.

Original: The number of natural images in S is at most m ( m − 1) k/ 2 , which is independent of the size n of
Revised: As will be clear later, due to the difﬁculty in subjective testing, we collect a partial label: “ x (cid:63) does not contain f i ( x (cid:63) ) nor f j ( x (cid:63) ) ”.

Original: D .
Revised: In this case, only Eq.

Original: As a result, applying MAD to a larger image set has no impact on the cost of human labeling.
Revised: (1) can be applied, and x (cid:63) contributes less to performance comparison between the two classiﬁers.

Original: In scenarios where the cost of computational prediction can be ignored, MAD encourages to expand D to cover as many “free” natural images as possible. We now describe our subjective assessment environment for collecting human labels.
Revised: Algorithm

Original: Given an image x ∈ S , which is associated with two classiﬁers  f i and f j , we pick two binary questions forhuman annotators: “Does x contain an f i ( x ) ?” and “Does x contain an f j ( x ) ?”.
Revised: Input: An unlabeled image set D , a group of image classiﬁers F = { f i } mi =1 to be ranked, a distance measure d w deﬁned over WordNet hierarchy Output: A global ranking vector r ∈ R m 1 S ← ∅ , B ← I 2 for i ← 1 to m do

Original: When both answers are no (corresponding to Case III), we cease to source the ground-truth label of x because f  ( x ) is uninformative in distinguishing f i and f j . Nevertheless, it is difﬁcult for humans to select one among c classes, especially when c is large and the ontology of classes is complex. 5 for  ←  m do  j ←  do
Revised: Compute classiﬁer predictions { f i ( x ) , x ∈ D} 4 end 5 for i ← 1 to m do for j ← i + 1 to m do

Original: Compute the distances 
Revised: Compute the distances using

Original:  { d w ( f i ( x ) , f j ( x )) , x ∈ D}
Revised: Eq. (3) { d w ( f i ( x ) , f j ( x )) , x ∈ D}

Original: Select top k images with k largest distances to form S 
Revised: Select top- k images with k largest distances to form S { i,j }

Original: S ← S (cid:83) S { i, } end 11 end 12 Source human labels for S 13 Compute the pairwise accuracy matrix A with a ij =
Revised: S ← S (cid:83) S { i,j } end 11 end 12 Source human labels for S 13 Compute the pairwise accuracy matrix A with a ij =

Original: Acc( f i ; S { i,j } ) using Eq.
Revised: Acc( f i ; S { i,j } ) using Eq.

Original: (1) 14 Compute the pairwise dominance matrix B with b ij = a ij /a ji 15 Compute the global ranking vector r using 
Revised: (1) 14 Compute the pairwise dominance matrix B with b ij = a ij /a ji 15 Compute the global ranking vector r using Eq.

Original: Eq.
Revised: (5) into the MAD competition Input: An unlabeled image set D , the pairwise dominance matrix B ∈ R m × m for F = { f i } mi =1 , a new classiﬁer f m +1 to be ranked, distance measure d w

Original: 2 Compute
Revised: Output: A global ranking vector r ∈ R m + 1 S ← ∅ , B (cid:48) ← (cid:20) B 0 T (cid:21) ∈

Original:  3 for  m do 
Revised: R ( m +1) × ( m +1) 2 Compute the predictions of the new image classiﬁer { f m +1 ( x ) , x ∈ D} 3 for i ← 1 to m do Compute the distances using Eq. (3) { d w ( f i ( x ) , f m +1 ( x )) , x ∈ D}

Original: Compute distances to form S { i,m +1 }
Revised: Select top- k images with k largest distances to form S { i,m +1 }

Original: S = S (cid:83) S { i,m +1 } 7 end 8 Source human labels for S 9 for i ← 1 to m do a i,m +1 =
Revised: S = S (cid:83) S { i,m +1 } 7 end 8 Source human labels for S 9 for i ← 1 to m do a i,m +1 =

Original: Acc( f i ; S { i,m +1 } ) a m +

Revised: Acc( f i ; S { i,m +1 } ) a m +

Original: i
Revised: i

Original: =
Revised: =

Original: Acc( f m +1 ; S { i,m +1 } ) 11 end 12 Update the pairwise dominance matrix B (cid:48) with b (cid:48) i,m +

Revised: Acc( f m +1 ; S { i,m +1 } ) 11 end 12 Update the pairwise dominance matrix B (cid:48) with b (cid:48) i,m +

Original: = 1 /b (cid:48) m +1 ,i = a i,m +1 /a m
Revised: = 1 /b (cid:48) m +1 ,i = a i,m +1 /a m

Original: +1 ,i 13 Compute
Revised: +1 ,i 13 Compute

Original: the global ranking vector r ∈
Revised: the global ranking vector r ∈

Original: R m +1 using Eq.
Revised: R m +1 using Eq.

Original: After subjective testing, we ﬁrst compare the classiﬁers in  pairs and aggregate the pairwise statistics into a global ranking.
Revised: In practice, to obtain reliable performance comparison between f i and f j , we choose top- k images in D with k largest distances computed by Eq. (3) to form the test subset S { i,j } . MAD runs this game among all (cid:0) m 2 (cid:1) distinct pairs of classiﬁers, resulting in the ﬁnal MAD test set S = (cid:83) S { i,j } .

Revised: The number of natural images in S is at most m ( m − 1) k/ 2 , which is independent of the size n of

Revised: D .

Revised: In other words, applying MAD to a larger image set has no impact on the cost of human labeling.

Revised: In scenarios where the cost of computational prediction can be ignored, MAD encourages to expand

Revised: D to cover as many “free” natural images as possible.

Revised: We now describe our subjective assessment environment for collecting human labels.

Revised: Given an image x ∈ S , which is associated with two classiﬁers f i and f j , we pick two binary questions for human annotators: “Does x contain an f i ( x ) ?”

Revised: and “Does x contain an f j ( x ) ?”.

Revised: When both answers are no (corresponding to Case III), we stop querying the ground-truth label of x because it is difﬁcult for humans to select one among c classes, especially when c is large and the ontology of classes is complex.

Revised: After subjective testing, we ﬁrst compare the classiﬁers in pairs and aggregate the pairwise statistics into a global ranking.

Original: Speciﬁcally, we compute the empirical classiﬁcation accuracies of f i and f j on S { i,j } using Eq.
Revised: Speciﬁcally, we compute the empirical classiﬁcation accuracies of f i and f j on S { i,j } using Eq.

Original: (1), denoted by a ij and a ji , respectively.
Revised: (1), denoted by a ij and a ji , respectively.

Original: When k is small, Laplace smoothing is employed to smooth the estimation.
Revised: When k is small, Laplace smoothing is employed to smooth the estimation.

Original: Note that a ij + a ji may be greater than one because of Case I.
Revised: Note that a ij + a ji may be greater than one because of Case I.

Original: The pairwise accuracy statistics of all classiﬁers form a matrix A , from which we compute another matrix B with b ij = a ij /a ji  indicating the pairwise dominance of f i over f j .
Revised: The pairwise accuracy statistics of all classiﬁers form a matrix A , from which we compute another matrix B with b ij = a ij /a ji , indicating the pairwise dominance of f i over f j .

Original: We aggregate  pairwise comparison results into a global ranking r ∈ R m using Perron rank (Saaty & Vargas, 1984):
Revised: We aggregate the pairwise comparison results into a global ranking r ∈ R m using Perron rank (Saaty & Vargas, 1984):

Original: [Equation]
Revised: [Equation]

Original: where 1 is an m -dimensional vector of all ones.
Revised: where 1 is an m -dimensional vector of all ones.

Original: The limit of Eq.
Revised: The limit of Eq.

Original: (5) is the normalized principal eigenvector of B corresponding to the largest eigenvalue,
Revised: (5) is the normalized principal eigenvector of B corresponding to the largest eigenvalue,

Original: where r i > 0 for i = 1 , .
Revised: where r i > 0 for i = 1 , .

Original: m and (cid:80) i r i = 1 .
Revised: m and (cid:80) i r i = 1 .

Original: The larger r i is, the better f i performs in the MAD competition.
Revised: The larger r i is, the better f i performs in the MAD competition.

Original: Other ranking aggregation methods such as HodgeRank (Jiang et al., 2011) may also be applied.
Revised: Other ranking aggregation methods such as HodgeRank (Jiang et al., 2011) may also be applied.

Original: We summarize the workﬂow of the MAD competition in Algorithm 1.
Revised: We summarize the workﬂow of the MAD competition in Algorithm 1.

Original: Finally, it is straightforward and cost-effective to add the m + 1 -th classiﬁer into the current MAD competition.
Revised: Finally, it is straightforward and cost-effective to add the m + 1 -th classiﬁer into the current MAD competition.

Original: No change is necessary for the sampled S and the associated subjective testing.
Revised: No change is needed for the sampled S and the associated subjective testing.

Original: Theonly additional work is to select a total of mk new images from D for human labeling.
Revised: The additional work is to select a total of mk new images from D for human labeling.

Original: We then enlarge B by one along its row and column, and insert the pairwise comparison statistics between f m +1 and the previous m classiﬁers.
Revised: We then enlarge B by one along its row and column, and insert the pairwise comparison statistics between f m +1 and the previous m classiﬁers.

Original: An updated global ranking vector r ∈ R m +1 can be computed using Eq.
Revised: An updated global ranking vector r ∈ R m +1 can be computed using Eq.

Original: We summarize the procedure of adding a new classiﬁer in Algorithm 2.
Revised: We summarize the procedure of adding a new classiﬁer in Algorithm 2.

Original:  A PPLICATION TO I MAGE N ET C LASSIFIERS
Revised:  A PPLICATION TO I MAGE N ET C LASSIFIERS

Original: In this section, we apply the proposed MAD competition methodology to comparing ImageNet classiﬁers.
Revised: In this section, we apply the proposed MAD competition methodology to comparing ImageNet classiﬁers.

Original: We focus on ImageNet (Deng et al., 2009) not only because it is one of the ﬁrst large-scale  datasets in image classiﬁcation, but also because the improvements on ImageNet seem to plateau, which sets up an ideal platform for MAD to distinguish the newly proposed image classiﬁers ﬁner.
Revised: We focus on ImageNet (Deng et al., 2009) for two reasons. First, it is one of the ﬁrst large-scale and widely used datasets in image classiﬁcation. Second, the improvements on ImageNet seem to plateau, which provides an ideal platform for MAD to distinguish the newly proposed image classiﬁers ﬁner.

Original: 3. E XPERIMENTAL S ETUPS 
Revised: 3. E XPERIMENTAL S ETUPS Constructing

Original: Constructing D
Revised: D

Original: Inspired by (Hendrycks et al., 2019), we focus on the same 200 out of 1 ,classes to avoid rare and abstract classes, and classes that have changed much since 2012.
Revised: Inspired by Hendrycks et al. (2019), we focus on the same 200 out of 1 ,classes to avoid rare and abstract ones, and classes that have changed much since 2012.

Original: For each class, we crawl a large number of images from Flickr, resulting in a total of n = 168 , 000 natural images.
Revised: For each class, we crawl a large number of images from Flickr, resulting in a total of n = 168 , 000 natural images.

Original: Although MAD allows us to arbitrarily increase n with essentially no cost, we choose the size of D to beonly approximately three times larger than the ImageNet validation set to provide a relatively easy environment for probing the generalizability of the classiﬁers.
Revised: Although MAD allows us to arbitrarily increase n with essentially no cost, we choose the size of D to be approximately three times larger than the ImageNet validation set to provide a relatively easy environment for probing the generalizability of the classiﬁers.

Original: As will be cleared in Section 3.2, the current setting of n is sufﬁcient to discriminate the competing classiﬁers.
Revised: As will be clear in Section 3.2, the current setting of n is sufﬁcient to discriminate the competing classiﬁers.

Original: To guarantee the content independence between ImageNet and our test set, we collect images that have been uploaded after 2013.
Revised: To guarantee the content independence between ImageNet and D , we collect images that have been uploaded after 2013.

Original: It is worth noting that no data cleaning ( e . g ., inappropriate content and near-duplicate removal) is necessary at this stage since we only need to ensure the selected subset S for human labeling are eligible.
Revised: It is worth noting that no data cleaning ( e . g ., inappropriate content and nearduplicate removal) is necessary at this stage since we only need to ensure the selected subset S for human labeling is eligible.

Original: Competing Algorithms We select eleven representative ImageNet classiﬁers for benchmarking: VGG16BN (Simonyan & Zisserman, 2014) with batch normalization (Ioffe & Szegedy, 2015), ResNet34, ResNet101 (He et al., 2016), WRN101-2 (Zagoruyko & Komodakis, 2016),
Revised: Competing Classiﬁers We select eleven representative ImageNet classiﬁers for benchmarking: VGG16BN (Simonyan & Zisserman, 2014) with batch normalization (Ioffe & Szegedy, 2015), ResNet34, ResNet101 (He et al., 2016), WRN101-2 (Zagoruyko & Komodakis, 2016),

Original: ResNeXt101-32 × 4 (Xie et al., 2017), SE-ResNet-101, SENet154 (Hu et al., 2018), NASNet-ALarge (Zoph et al., 2018), PNASNet-5-Large (Liu et al., 2018), EfﬁcientNet-B7 (Tan & Le, 2019), and WSL-ResNeXt101-32 × 48 (Mahajan et al., 2018).
Revised: ResNeXt101-32 × 4 (Xie et al., 2017), SE-ResNet-101, SENet154 (Hu et al., 2018), NASNet-ALarge (Zoph et al., 2018), PNASNet-5-Large (Liu et al., 2018), EfﬁcientNet-B7 (Tan & Le, 2019), and WSL-ResNeXt101-32 × 48 (Mahajan et al., 2018).

Original: Since VGG16BN and ResNet34 have nearly identical accuracies(both top- 1 and top- 5 ) on  ImageNet validation set, it is particularly interesting to see which method generalizes better in natural image manifold.
Revised: Since VGG16BN and ResNet34 have nearly identical accuracies on the ImageNet validation set, it is particularly interesting to see which method generalizes better to natural image manifold.

Original: We compare ResNet34 with ResNet101 to see the inﬂuence of DNN depth on generalizability.
Revised: We compare ResNet34 with ResNet101 to see the inﬂuence of DNN depth.

Original: WRN101-2, ResNeXt101-32 × 4, SE-ResNet-101 are different improved versions over ResNet-101.
Revised: WRN101-2, ResNeXt101-32 × 4, SE-ResNet-101 are different improved versions over ResNet-101.

Original: We also include two state-ofthe-art classiﬁers on ImageNet validation set: WSL-ResNeXt101-32 × 48 and EfﬁcientNet-B7.
Revised: We also include two state-of-the-art classiﬁers: WSL-ResNeXt10132 × 48 and EfﬁcientNet-B7.

Original: The former leverages the power of weakly supervised pre-training on Instagram data, while the latter makes use of compound scaling method.
Revised: The former leverages the power of weakly supervised pre-training on Instagram data, while the latter makes use of compound scaling.

Original: We use publicly available code repositories for all DNNbased models, whose top- 1 accuracies on  ImageNet validation set are listed in Table 1 for reference.
Revised: We use publicly available code repositories for all DNN-based models, whose top- 1 accuracies on the ImageNet validation set are listed in Table 1 for reference.

Original: Constructing S
Revised: Constructing S

Original: When constructing S using the maximum discrepancy principle, we add another constraint based on prediction conﬁdence.
Revised: When constructing S using the maximum discrepancy principle, we add another constraint based on prediction conﬁdence.

Original: Speciﬁcally, a candidate image x associated with f i and f j is ﬁltered out if it does not satisfy min( p i ( x ) , p j ( x )) ≥ T , where p i ( x ) is the conﬁdence score ( i .
Revised: Speciﬁcally, a candidate image x associated with f i and f j is ﬁltered out if it does not satisfy min( p i ( x ) , p j ( x )) ≥ T , where p i ( x ) is the conﬁdence score ( i .

Original: e ., probability produced by the last softmax layer) of f i ( x ) and T is a predeﬁned threshold set to 0 .
Revised: e ., probability produced by the last softmax layer) of f i ( x ) and T is a predeﬁned threshold set to 0 .

Original: We include the conﬁdence constraint for two main reasons.
Revised: We include the conﬁdence constraint for two main reasons.

Original: First, if f i misclassiﬁes x with low conﬁdence, it is highly likely that x is near the decision boundary and thus contains less information on improving the decision rules of f i .
Revised: First, if f i misclassiﬁes x with low conﬁdence, it is highly likely that x is near the decision boundary and thus contains less information on improving the decision rules of f i .

Original: Second, some images in D do not necessarily fall into the 1 , 000 classes in ImageNet, which are bound to be misclassiﬁed (a problem closely related to outof-distribution detection).
Revised: Second, some images in D do not necessarily fall into the 1 , 000 classes in ImageNet, which are bound to be misclassiﬁed (a problem closely related to outof-distribution detection).

Original: If they are misclassiﬁed by f i with high conﬁdence, we consider them as hard counterexamples of f i .
Revised: If they are misclassiﬁed by f i with high conﬁdence, we consider them as hard counterexamples of f i .

Original: To encourage class diversity in S , we remain a maximum of three images  with the same predicted label by f i .
Revised: To encourage class diversity in S , we retain a maximum of three images [Figure] with the same predicted label by f i .

Original: In addition, we exclude images that are non-natural.
Revised: In addition, we exclude images that are non-natural.

Original: Figure 3 visually compares representative “manhole cover” images in S and  ImageNet validation set (see more in Figure 6).
Revised: Figurevisually compares representative “manhole cover” images in S and the ImageNet validation set (see more in Figure 6).

Original: Collecting Human Labels As described in  2.1, given an image x ∈ S , human annotators need to answer two binary questions.
Revised: Collecting Human Labels As described in Section 2.1, given an image x ∈ S , human annotators need to answer two binary questions.

Original: In our subjective experiment, we choose k = 30 and invite ﬁve volunteer graduate students, who are experts in computer vision to label a total of 11 × 10 × 30 / 2 = 1 , 650 images.
Revised: In our subjective experiments we choose k = 30 and invite ﬁve volunteer graduate students, who are experts in computer vision, to label a total of 11 × 10 × 30 / 2 = 1 , 650 images.

Original: If more than three of them ﬁnd difﬁculty in labeling x (associated with f i and f j ), it is discarded and replaced by x (cid:48)
Revised: If more than three of them ﬁnd difﬁculty in labeling x (associated with f i and f j ), it is discarded and replaced by x (cid:48)

Original: ∈ D with the k + 1 -th largest distance d w ( f i ( x (cid:48) ) , f j ( x (cid:48) )) .
Revised: ∈ D with the k + 1th largest distance d w ( f i ( x (cid:48) ) , f j ( x (cid:48) )) .

Original: Majority vote is adopted to decide the ﬁnal label when disagreement occurs.
Revised: Majority vote is adopted to decide the ﬁnal label when disagreement occurs.

Original: After subjective testing, we ﬁnd that 53 .
Revised: After subjective testing, we ﬁnd that 53 .

Original: 5% of annotated images belong to Case II, which form the cornerstone of the subsequent data analysis.
Revised: 5% of annotated images belong to Case II, which form the cornerstone of the subsequent data analysis.

Original: Besides, 32 . 9% and 13 .
Revised: Besides, 32 . 9% and 13 .

Original: 6% images pertain to Case I and Case III, respectively.
Revised: 6% images pertain to Case I and Case III, respectively.

Original: 3. E XPERIMENTAL R ESULTS
Revised: 3. E XPERIMENTAL R ESULTS

Original: Pairwise Ranking Results Figure 4 shows the pairwise accuracy matrix A in the current MAD competition, where a larger value of an entry (a brighter color) indicates a higher accuracy in the subset selected together by the corresponding row and column models.
Revised: Pairwise Ranking Results Figure 3 shows the pairwise accuracy matrix A in the current MAD competition, where a larger value of an entry (a brighter color) indicates a higher accuracy in the subset selected together by the corresponding row and column models.

Original: An interesting phenomenon[Figure] we ﬁnd is that when two classiﬁers f i and f j perform at a similar level on S { i,j } ( i  e ., | Acc( f i ) −
Revised: An interesting phenomenon we observe is that when two classiﬁers f i and f j perform at a similar level on S { i,j } ( i . e ., | Acc( f i ) −

Original: Acc( f j ) | is small), max(Acc( f i ) , Acc( f j )) is also small.
Revised: Acc( f j ) | is small), max(Acc( f i ) , Acc( f j )) is also small.

Original: That is, more images on which they both make incorrect but different predictions (Case III) have been selected compared to images falling into Case I. Taking a closer look at images in S { i,j } , we may reveal the respective model biases of f i  and f j .
Revised: That is, more images on which they both make incorrect but different predictions (Case III) have been selected compared with images falling into Case I. Taking a closer look at images in S { i,j } , we may reveal the respective model biases of f i [Table] and f j .

Original: For example, we ﬁnd that WSL-ResNeXt101-32 × 48 tend to focus on foreground objects while EfﬁcientNet-B7 attends more to background objects (See Figure 8).
Revised: For example, we ﬁnd that WSL-ResNeXt101-32 × 48 tends to focus on foreground objects, while EfﬁcientNet-B7 attends more to background objects (see Figure 7).

Original: We also observe several common failure modes of the competing classiﬁers through pairwise comparison, e.
Revised: We also ﬁnd several common failure modes of the competing classiﬁers through pairwise comparison, e

Original:  g ., excessive reliance on relation inference (see Figure 9), bias towards low-level visual features (see Figure 10), and difﬁculty in recognizing rare instantiations of objects (see Figures 3 and 6).
Revised: [Table] g ., excessive reliance on relation inference (see Figure 8), bias towards low-level visual features (see Figure 9), and difﬁculty in recognizing rare instantiations of objects (see Figures 4 and 6).

Original: Global Ranking Results We present the global ranking results by MAD in Table 1, where we ﬁnd that MAD tracks the steady progress in image classiﬁcation, as veriﬁed by a reasonable Spearman rank-order correlation coefﬁcient (SRCC) of 0 .
Revised: Global Ranking Results We present the global ranking results by MAD in Table 1, where we ﬁnd that MAD tracks the steady progress in image classiﬁcation, as veriﬁed by a reasonable Spearman rank-order correlation coefﬁcient (SRCC) of 0 .

Original: 89 between the accuracy rank on  ImageNet validation set and the MAD rank on our test set D .
Revised: 89 between the accuracy rank on the ImageNet validation set and the MAD rank on our test set D .

Original: Moreover, by looking at the differences between the two rankings, we obtain a number of interesting ﬁndings.
Revised: Moreover, by looking at the differences between the two rankings, we obtain a number of interesting ﬁndings.

Original: First, VGG16BN outperforms not only ResNet34 but also ResNet101, suggesting that under similar computation budgets, VGG-like networks may exhibit better generalizability to hard samples than networks with residual connections.
Revised: First, VGG16BN outperforms not only ResNet34 but also ResNet101, suggesting that under similar computation budgets, VGG-like networks may exhibit better generalizability to hard samples than networks with residual connections.

Original: Second, both networks equipped with the squeezeand-extraction mechanism, i .
Revised: Second, both networks equipped with the squeeze-and-extraction mechanism, i .

Original: e . SE-ResNetand SENet154, move up by two places in the MAD ranking.
Revised: e ., SE-ResNet-101 and SENet154, move up by two places in the MAD ranking.

Original: This indicates that explicitly modeling dependencies between channel-wise feature maps seem quite beneﬁcial to image classiﬁcation.
Revised: This indicates that explicitly modeling dependencies between channel-wise feature maps seems quite beneﬁcial to image classiﬁcation.

Original: Third, for the two models that[Table] exploit neural architecture search, NASNet-A-Large is still ranked high by MAD; interestingly, the rank of PNASNet-5-Large drops a lot.
Revised: Third, for the two models that exploit neural architecture search, NASNet-A-Large is still ranked high by MAD; interestingly, the rank of PNASNet-5-Large drops a lot.

Original: That implies MAD may prefer the global search strategy used in NASNet-A-Large to the progressive cell-wise search strategy adopted in PNASNet-5-Large, although the former is slighter inferior in ImageNet top-1 accuracy.
Revised: This implies MAD may prefer the global search strategy used in NASNet-A-Large to the progressive cell-wise search strategy adopted in PNASNet-5-Large, although the former is slightly inferior in ImageNet top- 1 accuracy.

Original: Last but not least, the topperformers, WSL-ResNeXt101-32 × 48 and EfﬁcientNet-B7, are still the best in MAD competition (irrespective of their relative rankings), verifying the effectiveness of large-scale hashtag data pretraining and compound scalingmethod in the context of image classiﬁcation.k rankings, k = { 1 , 2 , . .
Revised: Last but not least, the topperformers, WSL-ResNeXt101-32 × 48 and EfﬁcientNet-B7, are still the best in MAD competition (irrespective of their relative rankings), verifying the effectiveness of large-scale hashtag data pretraining and compound scaling in the context of image classiﬁcation.

Original: Ablation Study We analyze the key hyperparameter k in MAD, i .
Revised: Ablation Study We analyze the key hyperparameter k in MAD, i .

Original: e ., the number of images in S { i,j } selected for subjective testing.
Revised: e ., the number of images in S { i,j } selected for subjective testing.

Original: We calculate the SRCC values between the top30 ranking (as reference) and other top- k rankings with k = { 1 , 2 , . .
Revised: We calculate the SRCC values between the top- 30 ranking (as reference) and other top- k rankings with k = { 1 , 2 , . .

Original: As shown in Figure 5, the ranking results are fairly stable ( SRCC > 0 . 90 )
Revised: As shown in Figure 5, the ranking results are fairly stable ( SRCC > 0 . 90 )

Original: when k > 15 .
Revised: when k > 15 .

Original: This supports our choice of k = 30 since the ﬁnal global ranking already seems to enter a stable plateau.
Revised: This supports our choice of k = 30 since the ﬁnal global ranking already seems to enter a stable plateau.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We have presented a new methodology for comparing image classiﬁcation models.
Revised: We have presented a new methodology, MAD competition, for comparing image classiﬁcation models.

Original: MAD effectively mitigates the conﬂict between the prohibitively large natural image manifold that we have to evaluate against and the expensive human labeling effort that we aim to minimize.
Revised: MAD effectively mitigates the conﬂict between the prohibitively large natural image manifold that we have to evaluate against and the expensive human labeling effort that we aim to minimize.

Original: Much of our endeavor has been dedicated to selecting natural images that are optimal in term of distinguishing or falsifying classiﬁers.
Revised: Much of our endeavor has been dedicated to selecting natural images that are optimal in terms of distinguishing or falsifying classiﬁers.

Original: MAD requires explicit speciﬁcation of image classiﬁers to be compared, and provides an effective means of exposing the respective ﬂaws of competing classiﬁers.
Revised: MAD requires explicit speciﬁcation of image classiﬁers to be compared, and provides an effective means of exposing the respective ﬂaws of competing classiﬁers.

Original: It also directly contributes to model interpretability and helps us analyze the models’ focus and bias when making predictions.
Revised: It also directly contributes to model interpretability and helps us analyze the models’ focus and bias when making predictions.

Original: We have demonstrated the effectiveness of MAD competition on ImageNet classiﬁers, and concluded a number of interesting observations, which were not apparently drawn from the (often quite close) accuracy numbers on thesmall and ﬁxed ImageNet validation set. 
Revised: We have demonstrated the effectiveness of MAD competition in comparing ImageNet classiﬁers, and concluded a number of interesting observations, which were not apparently drawn from the (often quite close) accuracy numbers on the ImageNet validation set. The application scope of MAD is far beyond image classiﬁcation.

Original: MAD is widely applicable to computational models that produce discrete-valued outputs, and is particularly useful when the sample space is large and the ground-truth label being predicted is expensive to measure.
Revised: It can be applied to computational models that produce discrete-valued outputs, and is particularly useful when the sample space is large and the ground-truth label being predicted is expensive to measure.

Original: Examples include medical and hyperspectral image classiﬁcation (Filipovych & Davatzikos, 2011;
Revised: Examples include medical and hyperspectral image classiﬁcation (Filipovych & Davatzikos, 2011;

Original: Wang et al., 2014), where signiﬁcation domain expertise is crucial to obtain correct labels.
Revised: Wang et al., 2014), where signiﬁcation domain expertise is crucial to obtain correct labels.

Original: MAD can also be applied towards spotting rare but fatal failures in high-cost and failure-sensitive applications, e .
Revised: MAD can also be used to spot rare but fatal failures in high-cost and failure-sensitive applications, e .

Original: g .,
Revised: g .,

Original: comparing perception systems of autonomous cars (Chen et al., 2015) in unconstrained real-world weathers, lighting conditions, and road scenes.
Revised: comparing perception systems of autonomous cars (Chen et al., 2015) in unconstrained real-world weathers, lighting conditions, and road scenes.

Original: In addition, by restricting the test set  to some domain of interest, MAD allows comparison of classiﬁers in more speciﬁc applications, e .
Revised: In addition, by restricting the test set D to some domain of interest, MAD allows comparison of classiﬁers in more speciﬁc applications, e .

Original: g ., ﬁne-grained image recognition (Xiao et al., 2015).
Revised: g ., ﬁne-grained image recognition.

Original: Wealso feel it important to note thecurrent limitations of  MAD.
Revised: We feel it important to note the limitations of the current MAD.

Original: First, MAD cannot prove a model’s predictions to be correct and therefore it should be viewed as complementary to, rather than a replacement for, the conventional accuracy comparison for image classiﬁcation.
Revised: First, MAD aims at relatively comparing models, and cannot give an absolute performance measure. Second, as an “error spotting” mechanism, MAD implicitly assumes that models in the competition are reasonably good ( e . g ., ImageNet classiﬁers); otherwise, the selected counterexamples may be less meaningful.

Original: Second, although the distance in Eq.
Revised: Third , although the distance in Eq.

Original: (3) is sufﬁcient to distinguish multiple classiﬁers in the current experimental setting, it does not yet fully reﬂect human cognition of image label semantics.
Revised: (3) is sufﬁcient to distinguish multiple classiﬁers in the current experimental setting, it does not yet fully reﬂect human cognition of image label semantics.

Original: Third, the computation of the conﬁdence  used to select images is not perfectly grounded.
Revised: Fourth, the conﬁdence computation used to select images is not perfectly grounded.

Original: How to marry the MAD competition with Bayesian probability theory to model uncertainties during image selection is an interesting direction for future research.
Revised: How to marry the MAD competition with Bayesian probability theory to model uncertainties during image selection is an interesting direction for future research.

Revised: Due to the above issues, MAD should be viewed as complementary to, rather than a replacement for, the conventional accuracy comparison for image classiﬁcation.

Original: Our method arises as a natural combination of concepts drawn from two separate lines of research.
Revised: Our method arises as a natural combination of concepts drawn from two separate lines of research.

Original: The ﬁrst explores the idea of model falsiﬁcation asa model comparison.
Revised: The ﬁrst explores the idea of model falsiﬁcation as model comparison.

Original: Wang & Simoncelli (2008) introduced the maximum differentiation competition for comparing computational models of continuous perceptual quantities, which was further extended by (Ma et al., 2019).
Revised: Wang & Simoncelli (2008) introduced the maximum differentiation competition for comparing computational models of continuous perceptual quantities, which was further extended by Ma et al.

Original: Berardino et al.
Revised: Berardino et al.

Original: (2017) developed a computational method for comparing hierarchical image representations in terms of their ability to explain perceptual sensitivity in humans.
Revised: (2017) developed a computational method for comparing hierarchical image representations in terms of their ability to explain perceptual sensitivity in humans.

Original: MAD, on the other hand, is tailored to applications with discrete model responses and relies on a semantic distance measure to compute model discrepancy.
Revised: MAD, on the other hand, is tailored to applications with discrete model responses and relies on a semantic distance measure to compute model discrepancy.

Original: The second endeavour arises from machine learning literature on generating adversarial examples (Szegedy et al., 2013; Goodfellow et al., 2015; Madry et al., 2018) and evaluating image classiﬁers on new test sets (Geirhos et al., 2019; Recht et al., 2019; Hendrycks &
Revised: The second endeavour arises from machine learning literature on generating adversarial examples (Szegedy et al., 2013; Goodfellow et al., 2015; Madry et al., 2018) and evaluating image classiﬁers on new test sets (Geirhos et al., 2019; Recht et al., 2019; Hendrycks &

Original: Dietterich, 2019; Hendrycks et al., 2019).
Revised: Dietterich, 2019; Hendrycks et al., 2019).

Original: The images selected by MAD can be seen as a form of natural adversarial examples as each of them is able to fool at least one classiﬁer (when Case I is eliminated).
Revised: The images selected by MAD can be seen as a form of natural adversarial examples as each of them is able to fool at least one classiﬁer (when Case I is eliminated).

Original: Unlike adversarial images with inherent transferability to mislead most classiﬁers,
Revised: Unlike adversarial images with inherent transferability to mislead most classiﬁers,

Original: MAD-selected images emphasize on their discriminability of the competing models.
Revised: MAD-selected images emphasize on their discriminability of the competing models.

Original: Different from recently created test sets, the MAD-selectedtest set is adapted to the competing classiﬁers with the goal of minimizing human labeling effort.
Revised: Different from recently created test sets, the MAD-selected set is adapted to the competing classiﬁers with the goal of minimizing human labeling effort.

Original:  A PPENDIX
Revised: In addition, MAD can also be linked to the popular technique of differential testing (McKeeman, 1998) in software engineering. A M ORE V ISUALIZATION R ESULTS Due to the page limit

Original: [Figure]
Revised: [Figure]

Revised: we put some additional ﬁgures here.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]