Original: I NCREASING THE C OVERAGE AND B ALANCE OF R OBUSTNESS B ENCHMARKS BY U SING N ONO VERLAPPING C ORRUPTIONS
Revised: I NCREASING THE C OVERAGE AND B ALANCE OF R OBUSTNESS B ENCHMARKS BY U SING N ONO VERLAPPING C ORRUPTIONS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Neural Networks are sensitive to various corruptions that usually occur in realworld applications such as blurs, noises, low-lighting conditions, etc.
Revised: Neural Networks are sensitive to various corruptions that usually occur in realworld applications such as blurs, noises, low-lighting conditions, etc.

Original: To estimate the robustness of neural networks to these common corruptions, we generally use a group of modeled corruptions gathered into a benchmark.
Revised: To estimate the robustness of neural networks to these common corruptions, we generally use a group of modeled corruptions gathered into a benchmark.

Original: We argue that corruption benchmarks often have a poor coverage: being robust to them only imply being robust to a narrow range of corruptions.
Revised: We argue that corruption benchmarks often have a poor coverage: being robust to them only imply being robust to a narrow range of corruptions.

Original: They are also often unbalanced: they give too much importance to some corruptions compared to others.
Revised: They are also often unbalanced: they give too much importance to some corruptions compared to others.

Original: In this paper, we propose to build corruption benchmarks with only non-overlapping corruptions, to improve their coverage and their balance.
Revised: In this paper, we propose to build corruption benchmarks with only non-overlapping corruptions, to improve their coverage and their balance.

Original: Two corruptions overlap when the robustnesses of neural networks to these corruptions are correlated.
Revised: Two corruptions overlap when the robustnesses of neural networks to these corruptions are correlated.

Original: We propose the ﬁrst metric to measure the overlapping between two corruptions.
Revised: We propose the ﬁrst metric to measure the overlapping between two corruptions.

Original: We provide an algorithm that uses this metric to build benchmarks of Non-Overlapping
Revised: We provide an algorithm that uses this metric to build benchmarks of Non-Overlapping

Original: Corruptions.
Revised: Corruptions.

Original: Using this algorithm, we build from ImageNet a new corruption benchmark called ImageNet-NOC.
Revised: Using this algorithm, we build from ImageNet a new corruption benchmark called ImageNet-NOC.

Original: We show that ImageNet-NOC is balanced and covers several kinds of corruptions that are not covered by ImageNet-C.
Revised: We show that ImageNet-NOC is balanced and covers several kinds of corruptions that are not covered by ImageNet-C.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Neural Networks perform poorly when they deal with images that are drawn from a different distribution than their training samples.
Revised: Neural Networks perform poorly when they deal with images that are drawn from a different distribution than their training samples.

Original: Indeed, neural networks are sensitive to adversarial examples (Szegedy et al., 2014), background changes (Xiao et al., 2020), and common corruptions (Hendrycks & Dietterich, 2019).
Revised: Indeed, neural networks are sensitive to adversarial examples (Szegedy et al., 2014), background changes (Xiao et al., 2020), and common corruptions (Hendrycks & Dietterich, 2019).

Original: Common corruptions are perturbations that change the appearance of images without changing their semantic content.
Revised: Common corruptions are perturbations that change the appearance of images without changing their semantic content.

Original: For instance, neural networks are sensitive to noises (Koziarski & Cyganek, 2017), blurs (Vasiljevic et al., 2016) or lighting condition variations (Temel et al., 2017).
Revised: For instance, neural networks are sensitive to noises (Koziarski & Cyganek, 2017), blurs (Vasiljevic et al., 2016) or lighting condition variations (Temel et al., 2017).

Original: Contrary to adversarial examples (Szegedy et al., 2014), common corruptions are not artiﬁcial perturbations especially crafted to fool neural networks.
Revised: Contrary to adversarial examples (Szegedy et al., 2014), common corruptions are not artiﬁcial perturbations especially crafted to fool neural networks.

Original: They naturally appear in industrial applications without any human interfering, and can signiﬁcantly reduce the performances of neural networks.
Revised: They naturally appear in industrial applications without any human interfering, and can signiﬁcantly reduce the performances of neural networks.

Original: A neural network is robust to a corruption c , when its performances on samples corrupted with c are close to its performances on clean samples.
Revised: A neural network is robust to a corruption c , when its performances on samples corrupted with c are close to its performances on clean samples.

Original: Some methods have been recently proposed to make neural networks more robust to common corruptions (Geirhos et al., 2019;
Revised: Some methods have been recently proposed to make neural networks more robust to common corruptions (Geirhos et al., 2019;

Original: Hendrycks* et al., 2020;
Revised: Hendrycks* et al., 2020;

Original: Rusak et al., 2020).
Revised: Rusak et al., 2020).

Original: To determine whether these approaches are effective, it is required to have a method to measure the neural network robustness to common corruptions.
Revised: To determine whether these approaches are effective, it is required to have a method to measure the neural network robustness to common corruptions.

Original: The most commonly used method consists in evaluating the performances of neural networks on images distorted by various kinds of common corruptions: (Hendrycks & Dietterich, 2019; Karahan et al., 2016; Geirhos et al., 2019;
Revised: The most commonly used method consists in evaluating the performances of neural networks on images distorted by various kinds of common corruptions: (Hendrycks & Dietterich, 2019; Karahan et al., 2016; Geirhos et al., 2019;

Original: Temel et al., 2017).
Revised: Temel et al., 2017).

Original: In this study, we call the group of perturbations used to make the robustness estimation a corruption benchmark.
Revised: In this study, we call the group of perturbations used to make the robustness estimation a corruption benchmark.

Original: We also use this term to refer to a set of test images that have been corrupted with these various corruptions.
Revised: We also use this term to refer to a set of test images that have been corrupted with these various corruptions.

Original: We identify two important factors that should be taken into account when building a corruption benchmark: the balance and the coverage .
Revised: We identify two important factors that should be taken into account when building a corruption benchmark: the balance and the coverage .

Original: In this paper, we consider that a corruption c is covered by a benchmark, when increasing the robustness of a network to all the corruptions of this benchmark, also increases the robustness of the network to c .
Revised: In this paper, we consider that a corruption c is covered by a benchmark, when increasing the robustness of a network to all the corruptions of this benchmark, also increases the robustness of the network to c .

Original: For instance, a benchmark that contains a camera shake blur corruption covers the defocus blur corruption, because the robustnesses towards these two corruptions are correlated (Vasiljevic et al., 2016).
Revised: For instance, a benchmark that contains a camera shake blur corruption covers the defocus blur corruption, because the robustnesses towards these two corruptions are correlated (Vasiljevic et al., 2016).

Original: The coverage of a benchmark is deﬁned as the number of corruptions covered by this benchmark.
Revised: The coverage of a benchmark is deﬁned as the number of corruptions covered by this benchmark.

Original: The more a benchmark covers a wide range of common corruptions, the more it gives a complete view of the robustness of a neural network.
Revised: The more a benchmark covers a wide range of common corruptions, the more it gives a complete view of the robustness of a neural network.

Original: At the same time, we consider a benchmark as balanced when it gives the same importance to the robustness to every corruption it contains.
Revised: At the same time, we consider a benchmark as balanced when it gives the same importance to the robustness to every corruption it contains.

Original: For instance, according to a balanced benchmark, being robust to noises is as important as being robust to brightness variations.
Revised: For instance, according to a balanced benchmark, being robust to noises is as important as being robust to brightness variations.

Original: We argue that most of the existing corruption benchmarks are unbalanced: they give too much importance to the robustness to some corruptions compared to others.
Revised: We argue that most of the existing corruption benchmarks are unbalanced: they give too much importance to the robustness to some corruptions compared to others.

Original: The coverage and balance of corruption benchmarks are related to the notion of corruption overlappings .
Revised: The coverage and balance of corruption benchmarks are related to the notion of corruption overlappings .

Original: We say that two corruptions overlap when the robustnesses of neural networks towards these corruptions are correlated.
Revised: We say that two corruptions overlap when the robustnesses of neural networks towards these corruptions are correlated.

Original: The contribution of this paper is fourfold:
Revised: The contribution of this paper is fourfold:

Original: We propose the ﬁrst method to estimate to what extent two corruptions overlap.
Revised: We propose the ﬁrst method to estimate to what extent two corruptions overlap.

Original: We show that building corruption benchmarks with non-overlapping corruptions make them more balanced and able to cover a wider range of corruptions.
Revised: We show that building corruption benchmarks with non-overlapping corruptions make them more balanced and able to cover a wider range of corruptions.

Original: We propose a method to build benchmarks that contain only non-overlapping corruptions.
Revised: We propose a method to build benchmarks that contain only non-overlapping corruptions.

Original: We use this method to build from ImageNet, a benchmark of Non-Overlapping Corruptions called ImagNet-NOC, to estimate the robustness of image classiﬁers to common corruptions.
Revised: We use this method to build from ImageNet, a benchmark of Non-Overlapping Corruptions called ImagNet-NOC, to estimate the robustness of image classiﬁers to common corruptions.

Original: We show that ImagNet-NOC is balanced and covers corruptions that are not covered by ImageNet-C: a reference corruption benchmark (Hendrycks
Revised: We show that ImagNet-NOC is balanced and covers corruptions that are not covered by ImageNet-C: a reference corruption benchmark (Hendrycks

Original: Dietterich
Revised: Dietterich

Original:  B ACKGROUND AND R ELATED W ORKS 2. E STIMATING THE R OBUSTNESS OF N ETWORKS WITH OUT - OF - DISTRIBUTION S AMPLES
Revised:  B ACKGROUND AND R ELATED W ORKS 2. E STIMATING THE R OBUSTNESS OF N ETWORKS WITH OUT - OF - DISTRIBUTION S AMPLES

Original: Studying the performances of neural networks on samples that lie outside training distributions, is a widely studied domain.
Revised: Studying the performances of neural networks on samples that lie outside training distributions, is a widely studied domain.

Original: Being able to understand out-of-distribution (o.o.d) samples is essential to guarantee that neural networks are reliable in real-world applications.
Revised: Being able to understand out-of-distribution (o.o.d) samples is essential to guarantee that neural networks are reliable in real-world applications.

Original: Several benchmarks and methods have been proposed to study this ﬁeld.
Revised: Several benchmarks and methods have been proposed to study this ﬁeld.

Original: For instance, ImageNet-A (Dan Hendrycks & Song, 2019) is a simple benchmark for ImageNet classiﬁers that contains samples drawn from a different source than the one used to build ImageNet.
Revised: For instance, ImageNet-A (Dan Hendrycks & Song, 2019) is a simple benchmark for ImageNet classiﬁers that contains samples drawn from a different source than the one used to build ImageNet.

Original: Adversarial examples, are samples that have been slightly modiﬁed to fool neural networks (Szegedy et al., 2014).
Revised: Adversarial examples, are samples that have been slightly modiﬁed to fool neural networks (Szegedy et al., 2014).

Original: Making sure that models are robust to these kinds of o.o.d samples is essential in terms of security.
Revised: Making sure that models are robust to these kinds of o.o.d samples is essential in terms of security.

Original: Artistic renditions (Hendrycks et al., 2020) or sketches (Haohan et al., 2019), can also be useful to determine if neural networks understand the abstract concepts we want them to learn.
Revised: Artistic renditions (Hendrycks et al., 2020) or sketches (Haohan et al., 2019), can also be useful to determine if neural networks understand the abstract concepts we want them to learn.

Original: Methods to study how classiﬁers are affected by background changes have also been recently proposed (Beery et al., 2018; Xiao et al., 2020).
Revised: Methods to study how classiﬁers are affected by background changes have also been recently proposed (Beery et al., 2018; Xiao et al., 2020).

Original: Another important aspect of the robustness of neural networks to o.o.d samples, is the robustness to common corruptions.
Revised: Another important aspect of the robustness of neural networks to o.o.d samples, is the robustness to common corruptions.

Original: This aspect of the robustness is generally estimated by gathering several commonly encountered corruptions, and by testing the performances of neural networks on images corrupted with these corruptions.
Revised: This aspect of the robustness is generally estimated by gathering several commonly encountered corruptions, and by testing the performances of neural networks on images corrupted with these corruptions.

Original: Diverse selections of common corruptions have been proposed to make a robustness estimation (Karahan et al., 2016; Laugros et al., 2019; Geirhos et al., 2019).
Revised: Diverse selections of common corruptions have been proposed to make a robustness estimation (Karahan et al., 2016; Laugros et al., 2019; Geirhos et al., 2019).

Original: In particular, ImageNet-C is a popular benchmark used to measure the robustness of ImageNet classiﬁers (Hendrycks & Dietterich, 2019).
Revised: In particular, ImageNet-C is a popular benchmark used to measure the robustness of ImageNet classiﬁers (Hendrycks & Dietterich, 2019).

Original: Different common corruption benchmarks have also been proposed in the context of object detection (Michaelis et al., 2019), scene classiﬁcation (Tadros et al., 2019) or, eye-tracking (Che et al., 2020).
Revised: Different common corruption benchmarks have also been proposed in the context of object detection (Michaelis et al., 2019), scene classiﬁcation (Tadros et al., 2019) or, eye-tracking (Che et al., 2020).

Original: It is worth noting that some transformations that are in between adversarial attacks and common corruptions have been recently proposed to measure the robustness of image classiﬁers (Kang et al., 2019; Dunn et al., 2019;
Revised: It is worth noting that some transformations that are in between adversarial attacks and common corruptions have been recently proposed to measure the robustness of image classiﬁers (Kang et al., 2019; Dunn et al., 2019;

Original: Liu et al., 2019).
Revised: Liu et al., 2019).

Original: 2. C ORRUPTION O VERLAPPINGS IN B ENCHMARKS
Revised: 2. C ORRUPTION O VERLAPPINGS IN B ENCHMARKS

Original: It has been noticed that ﬁne-tuning a model with camera shake blur helps it to deal with defocus blur and conversely (Vasiljevic et al., 2016).
Revised: It has been noticed that ﬁne-tuning a model with camera shake blur helps it to deal with defocus blur and conversely (Vasiljevic et al., 2016).

Original: The robustnesses to diverse kinds of noises have also been shown to be closely related (Laugros et al., 2019).
Revised: The robustnesses to diverse kinds of noises have also been shown to be closely related (Laugros et al., 2019).

Original: Even for two corruptions that do not look similar to the human eye, increasing the robustness of a model to one of these corruptions, can imply increasing the robustness to the other corruption (Kang et al., 2019).
Revised: Even for two corruptions that do not look similar to the human eye, increasing the robustness of a model to one of these corruptions, can imply increasing the robustness to the other corruption (Kang et al., 2019).

Original: In general, it has been shown that the robustnesses to the corruptions that distort the high-frequency content of images are correlated (Yin et al., 2019).
Revised: In general, it has been shown that the robustnesses to the corruptions that distort the high-frequency content of images are correlated (Yin et al., 2019).

Original: In the context of adversarial examples, it is known that the robustness towards one adversarial attack can be correlated with the robustness to another attack (Tramer &
Revised: In the context of adversarial examples, it is known that the robustness towards one adversarial attack can be correlated with the robustness to another attack (Tramer &

Original: Boneh, 2019).
Revised: Boneh, 2019).

Original: So, it is generally recommended to evaluate the adversarial robustness with attacks that are clearly different from each other (Carlini et al., 2019).
Revised: So, it is generally recommended to evaluate the adversarial robustness with attacks that are clearly different from each other (Carlini et al., 2019).

Original: The experiments carried out in this paper suggest that this recommendation should also be followed in the context of common corruption robustness estimation.
Revised: The experiments carried out in this paper suggest that this recommendation should also be followed in the context of common corruption robustness estimation.

Original:  C ORRUPTION O VERLAPPING 3. T HE C ORRUPTION O VERLAPPING S CORE
Revised:  C ORRUPTION O VERLAPPING 3. T HE C ORRUPTION O VERLAPPING S CORE

Original: We consider that two corruptions overlap when the robustness to one of these corruptions is correlated with the robustness to the other corruption.
Revised: We consider that two corruptions overlap when the robustness to one of these corruptions is correlated with the robustness to the other corruption.

Original: In this section, we propose a methodology to estimate to what extent two corruptions overlap.
Revised: In this section, we propose a methodology to estimate to what extent two corruptions overlap.

Original: The Robustness Score.
Revised: The Robustness Score.

Original: To determine whether two corruptions overlap, we ﬁrst need to introduce a metric called the robustness score.
Revised: To determine whether two corruptions overlap, we ﬁrst need to introduce a metric called the robustness score.

Original: This score gives an estimation of the robustness of a model m to a corruption c .
Revised: This score gives an estimation of the robustness of a model m to a corruption c .

Original: It is computed with the following formula: R mc =
Revised: It is computed with the following formula: R mc =

Original: A c A clean .
Revised: A c A clean .

Original: A clean is the accuracy of m on an uncorrupted test set and A c is the accuracy of m on the same test set corrupted with c .
Revised: A clean is the accuracy of m on an uncorrupted test set and A c is the accuracy of m on the same test set corrupted with c .

Original: The higher R mc is, the more robust m is.
Revised: The higher R mc is, the more robust m is.

Original: Please note that using this metric requires to monitor A clean and make sure it is relatively high.
Revised: Please note that using this metric requires to monitor A clean and make sure it is relatively high.

Original: Otherwise, an untrained model for which A c equals A clean , would be considered as robust for example.
Revised: Otherwise, an untrained model for which A c equals A clean , would be considered as robust for example.

Original: In this study, this metric is used only in the methodology we propose to estimate the overlapping between two corruptions.
Revised: In this study, this metric is used only in the methodology we propose to estimate the overlapping between two corruptions.

Original: The Corruption Overlapping Score.
Revised: The Corruption Overlapping Score.

Original: We consider two neural networks m 1 and m 2 and two corruptions
Revised: We consider two neural networks m 1 and m 2 and two corruptions

Original: c 1 and c 2 .
Revised: c 1 and c 2 .

Original: m 1 and m 2 are identical, and trained with exactly the same settings except that their training sets are respectively augmented with the corruptions c 1 and c 2 .
Revised: m 1 and m 2 are identical, and trained with exactly the same settings except that their training sets are respectively augmented with the corruptions c 1 and c 2 .

Original: A standard model is trained the same way but only with non-corrupted samples.
Revised: A standard model is trained the same way but only with non-corrupted samples.

Original: We propose a method to measure to what extent c 1 and c 2 overlap.
Revised: We propose a method to measure to what extent c 1 and c 2 overlap.

Original: The idea of the method is to see if a data augmentation with cmakes a model more robust to c 2 and conversely.
Revised: The idea of the method is to see if a data augmentation with cmakes a model more robust to c 2 and conversely.

Original: To determine this, m 1 , m 2 , and a test set are used to compute the following expression:
Revised: To determine this, m 1 , m 2 , and a test set are used to compute the following expression:

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst term of (1) measures whether a model that ﬁts exactly c 2 is more robust to c 1 than the standard model.
Revised: The ﬁrst term of (1) measures whether a model that ﬁts exactly c 2 is more robust to c 1 than the standard model.

Original: Symmetrically, the second term measures whether a model that ﬁts exactly c 1 is more robust than the standard model to c 2 .
Revised: Symmetrically, the second term measures whether a model that ﬁts exactly c 1 is more robust than the standard model to c 2 .

Original: The more making a model ﬁt c 1 implies being more robust to c 2 and reciprocally, and the more we can suppose that the robustnesses to c 1 and c 2 are correlated in practice.
Revised: The more making a model ﬁt c 1 implies being more robust to c 2 and reciprocally, and the more we can suppose that the robustnesses to c 1 and c 2 are correlated in practice.

Original: In other words, the expression (1) gives an estimation of the overlapping between c 1 and c 2 .
Revised: In other words, the expression (1) gives an estimation of the overlapping between c 1 and c 2 .

Original: To be more convenient, we would like to build a corruption overlapping score equal to 1 when c 1 = c 2 , and equal to 0 when the robustnesses to c 1 and c 2 are not correlated at all.
Revised: To be more convenient, we would like to build a corruption overlapping score equal to 1 when c 1 = c 2 , and equal to 0 when the robustnesses to c 1 and c 2 are not correlated at all.

Original: We propose a new expression that respects both conditions:
Revised: We propose a new expression that respects both conditions:

Original: [Equation]
Revised: [Equation]

Original: The expression (2) is a normalized version of (1).
Revised: The expression (2) is a normalized version of (1).

Original: It measures the overlapping between two corruptions while respecting the conditions mentioned above.
Revised: It measures the overlapping between two corruptions while respecting the conditions mentioned above.

Original: Indeed, if a data augmentation with c 1 does not increase the robustness to c 2 at all and conversely, then the ratios in (2) are null or negative, so the whole overlapping score is maximized to zero.
Revised: Indeed, if a data augmentation with c 1 does not increase the robustness to c 2 at all and conversely, then the ratios in (2) are null or negative, so the whole overlapping score is maximized to zero.

Original: In other words, when c 1 and c 2 do not overlap at all, the overlapping score is equal to 0.
Revised: In other words, when c 1 and c 2 do not overlap at all, the overlapping score is equal to 0.

Original: Besides, when c 1 = c 2 , R m 1 c 2 =
Revised: Besides, when c 1 = c 2 , R m 1 c 2 =

Original: R m 2 c 2 and R m 2 c 1 =
Revised: R m 2 c 2 and R m 2 c 1 =

Original: R m 1 c 1 , so both ratios of (2) are equal to 1.
Revised: R m 1 c 1 , so both ratios of (2) are equal to 1.

Original: Then, O c 1 ,c 2 = 1 when c 1 and c 2 completely overlap.
Revised: Then, O c 1 ,c 2 = 1 when c 1 and c 2 completely overlap.

Original: How to compute an overlapping score.
Revised: How to compute an overlapping score.

Original: To get the overlapping score between c 1 and c 2 , we follow the method illustrated in Figure 1.
Revised: To get the overlapping score between c 1 and c 2 , we follow the method illustrated in Figure 1.

Original: This method has six steps, and requires to have a training set, a test set and three untrained models that share the same architecture ( m 1 , m 2 and standard ).
Revised: This method has six steps, and requires to have a training set, a test set and three untrained models that share the same architecture ( m 1 , m 2 and standard ).

Original: The step (1), consists in using the corruptions c 1 and c 2 to get two training sets, each corrupted with one corruption.
Revised: The step (1), consists in using the corruptions c 1 and c 2 to get two training sets, each corrupted with one corruption.

Original: Then, the obtained corrupted sets are used to train the models m 1 and m 2 in step (2).
Revised: Then, the obtained corrupted sets are used to train the models m 1 and m 2 in step (2).

Original: The standard model is also trained during this step but only with non-corrupted samples.
Revised: The standard model is also trained during this step but only with non-corrupted samples.

Original: In step (3), similarly to step (1), we use c 1 and c 2 to get two corrupted versions of the test set.
Revised: In step (3), similarly to step (1), we use c 1 and c 2 to get two corrupted versions of the test set.

Original: The accuracies of the three models on the three test sets are computed in step (4).
Revised: The accuracies of the three models on the three test sets are computed in step (4).

Original: The scores obtained are used in step (5), to get the robustness scores of each model for the corruptions c 1 and c 2 .
Revised: The scores obtained are used in step (5), to get the robustness scores of each model for the corruptions c 1 and c 2 .

Original: The results obtained are used to compute the overlapping score between c 1 and c 2 in step (6).
Revised: The results obtained are used to compute the overlapping score between c 1 and c 2 in step (6).

Original: 3. C ORRUPTION O VERLAPPING AND C OVERAGE OF B ENCHMARKS
Revised: 3. C ORRUPTION O VERLAPPING AND C OVERAGE OF B ENCHMARKS

Original: With our deﬁnition, a corruption c is covered by a benchmark, when increasing the robustness of a network to all the corruptions of this benchmark, also increases the robustness of the network to c .
Revised: With our deﬁnition, a corruption c is covered by a benchmark, when increasing the robustness of a network to all the corruptions of this benchmark, also increases the robustness of the network to c .

Original: The more a benchmark covers a wide range of corruptions, the more being robust to this benchmark provides a strong guarantee about the robustness of a neural network.
Revised: The more a benchmark covers a wide range of corruptions, the more being robust to this benchmark provides a strong guarantee about the robustness of a neural network.

Original: Then, a benchmark should cover as much common corruptions as possible.
Revised: Then, a benchmark should cover as much common corruptions as possible.

Original: To illustrate the notion of coverage, let us consider bench1 , a benchmark that contains three corruptions of ImageNet-C: Gaussian noise, shot noise and impulse noise (Hendrycks & Dietterich, 2019).
Revised: To illustrate the notion of coverage, let us consider bench1 , a benchmark that contains three corruptions of ImageNet-C: Gaussian noise, shot noise and impulse noise (Hendrycks & Dietterich, 2019).

Original: We also consider bench2 , that contains the Gaussian noise, brightness and elastic corruptions of ImageNet-C. Intuitively, being robust to bench1 implies being robust only to noises while being robust to bench2 implies being robust to a wider range of corruptions.
Revised: We also consider bench2 , that contains the Gaussian noise, brightness and elastic corruptions of ImageNet-C. Intuitively, being robust to bench1 implies being robust only to noises while being robust to bench2 implies being robust to a wider range of corruptions.

Original: Then, we can suppose that bench1 has a lower coverage than bench2 .
Revised: Then, we can suppose that bench1 has a lower coverage than bench2 .

Original: When we compute the overlapping scores of these benchmarks, we observe that the overlapping between the corruptions of bench1 are close to 1, while they are close to 0 in bench2 (see Figure 3).
Revised: When we compute the overlapping scores of these benchmarks, we observe that the overlapping between the corruptions of bench1 are close to 1, while they are close to 0 in bench2 (see Figure 3).

Original: The corruptions of bench1 clearly overlap while the ones of bench2 do not.
Revised: The corruptions of bench1 clearly overlap while the ones of bench2 do not.

Original: We argue that the overlappings in benchmarks tend to reduce their coverage.
Revised: We argue that the overlappings in benchmarks tend to reduce their coverage.

Original: Indeed, the more two corruptions c 1 and c 2 overlap, the more it is likely that a corruption covered by c 1 is also covered by c 2 and conversely.
Revised: Indeed, the more two corruptions c 1 and c 2 overlap, the more it is likely that a corruption covered by c 1 is also covered by c 2 and conversely.

Original: So, when two corruptions overlap, their range of covered corruptions overlap too.
Revised: So, when two corruptions overlap, their range of covered corruptions overlap too.

Original: By reducing the overlappings in a benchmark, we separate the ranges of corruptions covered by the corruptions of this benchmark, which results in increasing its coverage.
Revised: By reducing the overlappings in a benchmark, we separate the ranges of corruptions covered by the corruptions of this benchmark, which results in increasing its coverage.

Original: In Section 5.2, we show that we can cover the ﬁfteen corruptions of ImageNet-C with only eight non-overlapping corruptions.
Revised: In Section 5.2, we show that we can cover the ﬁfteen corruptions of ImageNet-C with only eight non-overlapping corruptions.

Original: 3. C ORRUPTION O VERLAPPING AND B ALANCE OF B ENCHMARKS
Revised: 3. C ORRUPTION O VERLAPPING AND B ALANCE OF B ENCHMARKS

Original: We consider that a benchmark is balanced, when it gives the same importance to the robustness to every corruption it contains.
Revised: We consider that a benchmark is balanced, when it gives the same importance to the robustness to every corruption it contains.

Original: For instance in Section 5.3, we show that the ImageNet-C benchmark gives more importance to the blur corruptions than to the corruptions that affect the brightness of images.
Revised: For instance in Section 5.3, we show that the ImageNet-C benchmark gives more importance to the blur corruptions than to the corruptions that affect the brightness of images.

Original: Yet, in a real-world applications, we think that being robust to different kinds of blurs is not more valuable than being robust to lighting condition variations.
Revised: Yet, in a real-world applications, we think that being robust to different kinds of blurs is not more valuable than being robust to lighting condition variations.

Original: Being unbalanced is in general not a desirable property, it makes benchmarks give biased estimations of neural network robustness.
Revised: Being unbalanced is in general not a desirable property, it makes benchmarks give biased estimations of neural network robustness.

Original: Overlappings between the corruptions of a benchmark can make it unbalanced.
Revised: Overlappings between the corruptions of a benchmark can make it unbalanced.

Original: Let us consider three corruptions c 1 , c 2 , and c 3 with c 1 and c 2 that completely overlap, and c 1 and c 2 that do not overlap at all with c 3 .
Revised: Let us consider three corruptions c 1 , c 2 , and c 3 with c 1 and c 2 that completely overlap, and c 1 and c 2 that do not overlap at all with c 3 .

Original: A model robust to c 3 , is robust to one third of the corruptions of the benchmark.
Revised: A model robust to c 3 , is robust to one third of the corruptions of the benchmark.

Original: But a model robust to c 1 is also robust to c 2 , because c 1 and c 2 overlap.
Revised: But a model robust to c 1 is also robust to c 2 , because c 1 and c 2 overlap.

Original: So being robust to c 1 or c 2 implies being robust to two third of the corruptions of the benchmark.
Revised: So being robust to c 1 or c 2 implies being robust to two third of the corruptions of the benchmark.

Original: Then, this benchmark rewards more the robustness to c 1 or c 2 than the robustness to c 3 : it is unbalanced.
Revised: Then, this benchmark rewards more the robustness to c 1 or c 2 than the robustness to c 3 : it is unbalanced.

Original: In general, if one corruption contributes more to the total overlapping of a benchmark than another corruption of this benchmark, then the benchmark is unbalanced.
Revised: In general, if one corruption contributes more to the total overlapping of a benchmark than another corruption of this benchmark, then the benchmark is unbalanced.

Original: In Section 5.3, we show that a benchmark built with non-overlapping corruptions is more balanced than ImageNet-C.
Revised: In Section 5.3, we show that a benchmark built with non-overlapping corruptions is more balanced than ImageNet-C.

Original:  C ONSTRUCTION OF A N ON -O VERLAPPING C ORRUPTION B ENCHMARKS : I MAGE N ET -NOC
Revised:  C ONSTRUCTION OF A N ON -O VERLAPPING C ORRUPTION B ENCHMARKS : I MAGE N ET -NOC

Original: Experimental Set-up.
Revised: Experimental Set-up.

Original: For every training of this study, we use the following parameters.
Revised: For every training of this study, we use the following parameters.

Original: The used optimizer is SGD with a momentum of 0.9.
Revised: The used optimizer is SGD with a momentum of 0.9.

Original: The used cost function is a cross-entropy function, with a weight decay set to 10 − 4 .
Revised: The used cost function is a cross-entropy function, with a weight decay set to 10 − 4 .

Original: Models are trained for 40 epochs with a batch size of 256.
Revised: Models are trained for 40 epochs with a batch size of 256.

Original: The initial learning rate is set to 0.1 and is divided by 10 at epoch 20 and 30.
Revised: The initial learning rate is set to 0.1 and is divided by 10 at epoch 20 and 30.

Original: In all the experiments we use
Revised: In all the experiments we use

Original: ImageNet-100: a subset of ImageNet that contains every tenth ImageNet class by WordNetID order (Deng et al., 2009).
Revised: ImageNet-100: a subset of ImageNet that contains every tenth ImageNet class by WordNetID order (Deng et al., 2009).

Original: All images are resized to the 224x224 format, and randomly horizontally ﬂipped with a probability of 0.5 during trainings.
Revised: All images are resized to the 224x224 format, and randomly horizontally ﬂipped with a probability of 0.5 during trainings.

Original: When we use a data augmentation with a corruption in a training, half of the images of each training batch are transformed with the corruption, while the other half is not corrupted.
Revised: When we use a data augmentation with a corruption in a training, half of the images of each training batch are transformed with the corruption, while the other half is not corrupted.

Original: We present Algorithm 1: a general method to build benchmarks that do not contain any overlapping corruption.
Revised: We present Algorithm 1: a general method to build benchmarks that do not contain any overlapping corruption.

Original: We argue that this method helps to build balanced benchmarks that have a large coverage.
Revised: We argue that this method helps to build balanced benchmarks that have a large coverage.

Original: a corruptions
Revised: a corruptions

Original: A
Revised: A

Original: Require
Revised: Require

Original: : the maximum overlapping score allowed in the benchmark (0) n ← 2 .
Revised: : the maximum overlapping score allowed in the benchmark (0) n ← 2 .

Original: n is the current number of corruptions in the benchmark.
Revised: n is the current number of corruptions in the benchmark.

Original: It is initialized to 2.
Revised: It is initialized to 2.

Original: (1) Use the train set , the test set and the chosen network architecture to apply the methodology presented in Section 3.1, to get all the overlapping scores between the corruptions of S. (2) Pick the largest subsets of S with overlapping scores under the overlapping threshold .
Revised: (1) Use the train set , the test set and the chosen network architecture to apply the methodology presented in Section 3.1, to get all the overlapping scores between the corruptions of S. (2) Pick the largest subsets of S with overlapping scores under the overlapping threshold .

Original: (3) Among the retained subsets, select the one with the lowest mean overlapping score to form the benchmark.
Revised: (3) Among the retained subsets, select the one with the lowest mean overlapping score to form the benchmark.

Original: We want to use this algorithm to build a new benchmark that measures the robustness of image classiﬁers to common corruptions.
Revised: We want to use this algorithm to build a new benchmark that measures the robustness of image classiﬁers to common corruptions.

Original: Algorithm 1 requires to gather a group of candidate corruptions called S .
Revised: Algorithm 1 requires to gather a group of candidate corruptions called S .

Original: The larger S , the more combinations of non-overlapping corruptions can be found in S , the larger the benchmarks built by the algorithm.
Revised: The larger S , the more combinations of non-overlapping corruptions can be found in S , the larger the benchmarks built by the algorithm.

Original: Then, we recommend to use a large intial set of corruptions to increase the coverage of the built benchmarks.
Revised: Then, we recommend to use a large intial set of corruptions to increase the coverage of the built benchmarks.

Original: For this study, we implemented two dozens of image corruptions (illustrated in Figure 2) to constitute S .
Revised: For this study, we implemented two dozens of image corruptions (illustrated in Figure 2) to constitute S .

Original: All these corruptions are associated with a severity range.
Revised: All these corruptions are associated with a severity range.

Original: A value is randomly chosen from the severity range of the considered corruption each time an image is corrupted.
Revised: A value is randomly chosen from the severity range of the considered corruption each time an image is corrupted.

Original: The higher this value is, the more the aspect of the corrupted image changes.
Revised: The higher this value is, the more the aspect of the corrupted image changes.

Original: More information about the modeled common corruptions can be found in Appendix A.
Revised: More information about the modeled common corruptions can be found in Appendix A.

Original: We apply Algorithm 1, using this set of corruptions, the ImageNet train set and test set , and the ResNet-18 architecture ; with different values of the overlapping threshold .
Revised: We apply Algorithm 1, using this set of corruptions, the ImageNet train set and test set , and the ResNet-18 architecture ; with different values of the overlapping threshold .

Original: The corruption benchmarks obtained for different values of the threshold are shown in Appendix B.
Revised: The corruption benchmarks obtained for different values of the threshold are shown in Appendix B.

Original: The higher the overlapping threshold is, the more the number of corruptions included in the constructed benchmarks increases, and so does the coverage of the constructed benchmarks.
Revised: The higher the overlapping threshold is, the more the number of corruptions included in the constructed benchmarks increases, and so does the coverage of the constructed benchmarks.

Original: However, the coverage gain due to the increase of the threshold is reduced by overlappings, because overlapping corruptions tend to cover the same kind of corruptions.
Revised: However, the coverage gain due to the increase of the threshold is reduced by overlappings, because overlapping corruptions tend to cover the same kind of corruptions.

Original: Besides, as explained in Section 3.3, the more there are overlappings in a benchmark, the more it is likely to be unbalanced.
Revised: Besides, as explained in Section 3.3, the more there are overlappings in a benchmark, the more it is likely to be unbalanced.

Original: All in all, selecting the overlapping threshold, determines when the coverage gain does not worth the balance loss.
Revised: All in all, selecting the overlapping threshold, determines when the coverage gain does not worth the balance loss.

Original: Choosing this value depends on the application case and the kind of robustness estimation we want to make.
Revised: Choosing this value depends on the application case and the kind of robustness estimation we want to make.

Original: Each benchmark obtained with Algorithm 1 that contains n corruptions, are the n -corruptions benchmarks with the lowest mean overlapping as possible.
Revised: Each benchmark obtained with Algorithm 1 that contains n corruptions, are the n -corruption benchmarks with the lowest mean overlapping as possible.

Original: As explained in Sections 3.2 and 3.3, we expect that benchmarks that are optimal in terms of overlapping will have a good balance and coverage.
Revised: As explained in Sections 3.2 and 3.3, we expect that benchmarks that are optimal in terms of overlapping will have a good balance and coverage.

Original: We propose to study the set of eight corruptions obtained with overlapping threshold = 0 .
Revised: We propose to study the set of eight corruptions obtained with overlapping threshold = 0 .

Original: 1 , which is: rain, quantization, shear, brightness, hue, vertical artifacts, blur and border .
Revised: 1 , which is: rain, quantization, shear, brightness, hue, vertical artifacts, blur and border .

Original: The overlapping scores between these corruptions are displayed in the right lower square in Figure 3.
Revised: The overlapping scores between these corruptions are displayed in the right lower square in Figure 3.

Original: Eight corrupted ImageNet validation sets that are corrupted each  with one of the eight corruptions, are gathered to form ImageNet-NOC .
Revised: Eight corrupted ImageNet validation sets, each corrupted with one of the eight corruptions, are gathered to form ImageNet-NOC .

Original: We run the algorithm a second time using a DenseNet-121 architecture instead of the ResNet-18 one.
Revised: We run the algorithm a second time using a DenseNet-121 architecture instead of the ResNet-18 one.

Original: The overlapping scores obtained by running the step (1) of Algorithm 1 are displayed in Appendix A.
Revised: The overlapping scores obtained by running the step (1) of Algorithm 1 are displayed in Appendix A.

Original: The benchmark obtained with overlapping threshold = 0 .
Revised: The benchmark obtained with overlapping threshold = 0 .

Original: 1 is rain, Gaussian, shear, brightness, hue, vertical artifacts, elastic .
Revised: 1 is rain, Gaussian, shear, brightness, hue, vertical artifacts, elastic .

Original: This benchmark shares six corruptions with ImageNet-NOC, and the overlapping score computed with Gaussian noise and quantization equals 0.6 and the one computed with elastic and blur equals 0.26.
Revised: This benchmark shares six corruptions with ImageNet-NOC, and the overlapping score computed with Gaussian noise and quantization equals 0.6 and the one computed with elastic and blur equals 0.26.

Original: So, the only two corruptions that are not shared by the two benchmarks appear to be correlated in terms of robustness.
Revised: So, the only two corruptions that are not shared by the two benchmarks appear to be correlated in terms of robustness.

Original: So, using a DenseNet-121 architecture makes the algorithm build a benchmark that is fairly similar to the one obtained using ResNet-18.
Revised: So, using a DenseNet-121 architecture makes the algorithm build a benchmark that is fairly similar to the one obtained using ResNet-18.

Original: Running the Algorithm 1 requires to complete one training for each corruption in S .
Revised: Running the Algorithm 1 requires to complete one training for each corruption in S .

Original: It took one week with a single GPU Nvidia Tesla V100 to get all the overlapping scores of Figure 3.
Revised: It took one week with a single GPU Nvidia Tesla V100 to get all the overlapping scores of Figure 3.

Original: While this computational cost is high, we think that the process could be accelerated by ﬁne-tuning models for a few epochs instead of training them from scratch.
Revised: While this computational cost is high, we think that the process could be accelerated by ﬁne-tuning models for a few epochs instead of training them from scratch.

Original: Further investigations should be conducted to determine to what extent this alternative would modify the obtained results.
Revised: Further investigations should be conducted to determine to what extent this alternative would modify the obtained results.

Original: How to Use ImageNet-NOC .
Revised: How to Use ImageNet-NOC .

Original: We recommend to use the CE metric (Hendrycks & Dietterich, 2019) to measure the robustness of an image classiﬁer to an ImageNet-NOC corruption.
Revised: We recommend to use the CE metric (Hendrycks & Dietterich, 2019) to measure the robustness of an image classiﬁer to an ImageNet-NOC corruption.

Original: The mean CE score computed with a benchmark is called mCE.
Revised: The mean CE score computed with a benchmark is called mCE.

Original: Using the mCE metric avoids several pitfalls when measuring the robustness of neural networks.
Revised: Using the mCE metric avoids several pitfalls while measuring the robustness of neural networks.

Original: To compute a CE score, it is required to get the error rate of a pretrained AlexNet model, on the ImageNet validation set corrupted with the considered corruption.
Revised: To compute a CE score, it is required to get the error rate of a pretrained AlexNet model, on the ImageNet validation set corrupted with the considered corruption.

Original: The error rates of the torchvision pretrained AlexNet computed with the corruptions introduced in this paper are displayed in Appendix A.
Revised: The error rates of the torchvision pretrained AlexNet computed with the corruptions introduced in this paper are displayed in Appendix A.

Original: We provide the ImageNet-NOC
Revised: We provide the ImageNet-NOC

Original: CE scores of some traditionally used ImageNet classiﬁers in Appendix C. More details about how to use ImageNet-NOC can be found by visiting [Link available upon acceptance].
Revised: CE scores of some traditionally used ImageNet classiﬁers in Appendix C. More details about how to use ImageNet-NOC can be found by visiting [Link available upon acceptance].

Original:  C OMPARISON BETWEEN I MAGE N ET -NOC AND I MAGE N ET -C 5. C ORRUPTION O VERLAPPINGS IN I MAGE N ET -C AND I MAGE N ET -NOC
Revised:  C OMPARISON BETWEEN I MAGE N ET -NOC AND I MAGE N ET -C 5. C ORRUPTION O VERLAPPINGS IN I MAGE N ET -C AND I MAGE N ET -NOC

Original: ImageNet-C is a benchmark commonly used to measure the robustness of ImageNet classiﬁers to common corruptions (Hendrycks & Dietterich, 2019).
Revised: ImageNet-C is a benchmark commonly used to measure the robustness of ImageNet classiﬁers to common corruptions (Hendrycks & Dietterich, 2019).

Original: It is built on ﬁfteen common corruptions called Gaussian noise, shot noise, impulse noise, defocus blur, glass blur, motion blur, zoom blur, snow, frost, fog, brightness, contrast, elastic, pixelate, and jpeg compression.
Revised: It is built on ﬁfteen common corruptions called Gaussian noise, shot noise, impulse noise, defocus blur, glass blur, motion blur, zoom blur, snow, frost, fog, brightness, contrast, elastic, pixelate, and jpeg compression.

Original: Each corruption is associated with ﬁve severity levels that determine to what extend the corrupted images are distorted.
Revised: Each corruption is associated with ﬁve severity levels that determine to what extend the corrupted images are distorted.

Original: Please note that in general, the benchmark corruptions should not be used during a training of a model.
Revised: Please note that in general, the benchmark corruptions should not be used during a training of a model.

Original: Indeed, corruption benchmarks are built to estimate the robustness to unforeseen corruptions.
Revised: Indeed, corruption benchmarks are built to estimate the robustness to unforeseen corruptions.

Original: In this study, we use the ImageNet-C and ImageNet-NOC corruptions during some training phases only because we analyze the benchmarks themselves.
Revised: In this study, we use the ImageNet-C and ImageNet-NOC corruptions during some training phases only because we analyze the benchmarks themselves.

Original: Using the ResNet-18 model, the ImageNet training and validation sets, we apply the method illustrated in Figure 1 to get the overlapping score of every couple of ImageNet-C corruptions.
Revised: Using the ResNet-18 model, the ImageNet training and validation sets, we apply the method illustrated in Figure 1 to get the overlapping score of every couple of ImageNet-C corruptions.

Original: The corruption severity is randomly selected for each image corrupted in the process.
Revised: The corruption severity is randomly selected for each image corrupted in the process.

Original: The obtained scores are displayed in the upper left square of Figure 3.
Revised: The obtained scores are displayed in the upper left square of Figure 3.

Original: We observe that all the corruptions that damage the textures in images (blurs, noises, pixelate and jpeg compression) signiﬁcantly overlap.
Revised: We observe that all the corruptions that damage the textures in images (blurs, noises, pixelate and jpeg compression) signiﬁcantly overlap.

Original: This result is consistent with the experiments carried out by Yin et al.
Revised: This result is consistent with the experiments carried out by Yin et al.

Original: (2019): they argue that the robustnesses of neural networks to corruptions that alter high-frequency information of images are correlated.
Revised: (2019): they argue that the robustnesses of neural networks to corruptions that alter high-frequency information of images are correlated.

Original: Concerning the corruptions that alter low-frequency information of images, the overlappings are less pronounced.
Revised: Concerning the corruptions that alter low-frequency information of images, the overlappings are less pronounced.

Original: But we do observe some signiﬁcant overlappings.
Revised: But we do observe some signiﬁcant overlappings.

Original: There is a clear overlapping between fog and contrast or between snow and frost.
Revised: There is a clear overlapping between fog and contrast or between snow and frost.

Original: As explained in Section 3.2 and 3.3, all these overlappings suggest that ImageNet-C is unbalanced and has a poor coverage.
Revised: As explained in Section 3.2 and 3.3, all these overlappings suggest that ImageNet-C is unbalanced and has a poor coverage.

Original: Figure 3 reveals that ImageNet-NOC contains far less overalpping corruptions than ImageNet-C.
Revised: Figure 3 reveals that ImageNet-NOC contains far less overalpping corruptions than ImageNet-C.

Original: We compute again the overlapping scores of ImageNet-C and ImageNet-NOC with the DenseNet121 (Huang et al., 2017) and WideResNet-50-2 (Zagoruyko & Komodakis, 2016) architectures.
Revised: We compute again the overlapping scores of ImageNet-C and ImageNet-NOC with the DenseNet121 (Huang et al., 2017) and WideResNet-50-2 (Zagoruyko & Komodakis, 2016) architectures.

Original: The aspect of the overlapping arrays obtained with these two architectures is the same as the one obtained with ResNet-18 (see Appendix D).
Revised: The aspect of the overlapping arrays obtained with these two architectures is the same as the one obtained with ResNet-18 (see Appendix D).

Original: For traditionally used image classiﬁers, it appears that overlapping scores do not vary much with the architecture of the model used to compute them.
Revised: For traditionally used image classiﬁers, it appears that overlapping scores do not vary much with the architecture of the model used to compute them.

Original: 5. C OVERAGE OF I MAGE N ET -NOC AND I MAGE N ET -C
Revised: 5. C OVERAGE OF I MAGE N ET -NOC AND I MAGE N ET -C

Original: In the lower left square of Figure 3, are displayed all the overlapping scores computed with one
Revised: In the lower left square of Figure 3, are displayed all the overlapping scores computed with one

Original: ImageNet-C corruption and one ImageNet-NOC corruption.
Revised: ImageNet-C corruption and one ImageNet-NOC corruption.

Original: We observe that for every ImageNet-C corruption c 1 , there is always at least one ImageNet-NOC corruption c 2 , for which the overlapping score computed with c 1 and c 2 is higher than 0.3.
Revised: We observe that for every ImageNet-C corruption c 1 , there is always at least one ImageNet-NOC corruption c 2 , for which the overlapping score computed with c 1 and c 2 is higher than 0.3.

Original: On the other hand, two ImageNet-NOC corruptions ( hue and border ) do not overlap at all with any of the ImageNet-C corruptions.
Revised: On the other hand, two ImageNet-NOC corruptions ( hue and border ) do not overlap at all with any of the ImageNet-C corruptions.

Original: Then, increasing the robustness to all the ImageNet-NOC corruptions should imply being more robust to all the ImageNet-C corruptions, but being robust to ImageNet-C may not imply being robust to some
Revised: Then, increasing the robustness to all the ImageNet-NOC corruptions should imply being more robust to all the ImageNet-C corruptions, but being robust to ImageNet-C may not imply being robust to some

Original: ImageNet-NOC corruptions.
Revised: ImageNet-NOC corruptions.

Original: To conﬁrm this, we train two ResNet-18 called m INOC and m IC .
Revised: To conﬁrm this, we train two ResNet-18 called m INOC and m IC .

Original: A data augmentation procedure with all the ImageNet-C corruptions is used to train m IC .
Revised: A data augmentation procedure with all the ImageNet-C corruptions is used to train m IC .

Original: Each corrupted image of this training is modiﬁed by one randomly selected corruption of ImageNet-C, with a randomly chosen severity.
Revised: Each corrupted image of this training is modiﬁed by one randomly selected corruption of ImageNet-C, with a randomly chosen severity.

Original: Similarly, m INOC is trained with a data augmentation procedure with all the ImageNet-NOC corruptions.
Revised: Similarly, m INOC is trained with a data augmentation procedure with all the ImageNet-NOC corruptions.

Original: After the trainings, we measure the robustness of m INOC to every corruption of ImageNetC by computing its CE scores.
Revised: After the trainings, we measure the robustness of m INOC to every corruption of ImageNetC by computing its CE scores.

Original: We also measure the CE scores of m IC towards the ImageNet-NOC
Revised: We also measure the CE scores of m IC towards the ImageNet-NOC

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: corruptions.
Revised: corruptions.

Original: We compare the obtained scores of both models with the ones of the standard model in
Revised: We compare the obtained scores of both models with the ones of the standard model in

Original: Table 1.
Revised: Table 1.

Original: The ﬁrst column of this table contains the error rates on non-corrupted ImageNet samples.
Revised: The ﬁrst column of this table contains the error rates on non-corrupted ImageNet samples.

Original: We observe in Table 1 that m INOC is more robust than the standard model to all the ImageNetC corruptions.
Revised: We observe in Table 1 that m INOC is more robust than the standard model to all the ImageNetC corruptions.

Original: However, m IC is not robust to the hue and border corruptions of ImageNet-NOC.
Revised: However, m IC is not robust to the hue and border corruptions of ImageNet-NOC.

Original: These results appear to conﬁrm the hypothesis made by studying the overlapping scores in Figure 3: ImageNet-C does not cover some of the ImageNet-NOC corruptions while ImageNet-NOC covers the ImageNet-C corruptions.
Revised: These results appear to conﬁrm the hypothesis made by studying the overlapping scores in Figure 3: ImageNet-C does not cover some of the ImageNet-NOC corruptions while ImageNet-NOC covers the ImageNet-C corruptions.

Original: As argued in Section 3.2, it seems that using non-overlapping corruptions helps to build benchmarks that have a larger coverage.
Revised: As argued in Section 3.2, it seems that using non-overlapping corruptions helps to build benchmarks that have a larger coverage.

Original: 5. B ALANCE OF I MAGE N ET -NOC AND I MAGE N ET -C
Revised: 5. B ALANCE OF I MAGE N ET -NOC AND I MAGE N ET -C

Original: We carry out an experiment to compare the balance of ImageNet-NOC and
Revised: We carry out an experiment to compare the balance of ImageNet-NOC and

Original: ImageNet-C. We train one ResNet-18 for each corruption of ImageNet-NOC and
Revised: ImageNet-C. We train one ResNet-18 for each corruption of ImageNet-NOC and

Original: ImageNet-C.
Revised: ImageNet-C.

Original: So, ﬁfteen models are trained using a data augmentation procedure with one corruption of ImageNet-C, and eight others are trained with one corruption of ImageNet-NOC.
Revised: So, ﬁfteen models are trained using a data augmentation procedure with one corruption of ImageNet-C, and eight others are trained with one corruption of ImageNet-NOC.

Original: Then, we estimate the robustness of the ﬁrst ﬁfteen ResNet-18 on ImageNet-C by computing their mCE scores.
Revised: Then, we estimate the robustness of the ﬁrst ﬁfteen ResNet-18 on ImageNet-C by computing their mCE scores.

Original: We also get the mCE scores of the eight remaining ResNet-18 on ImageNet-NOC.
Revised: We also get the mCE scores of the eight remaining ResNet-18 on ImageNet-NOC.

Original: The obtained scores are displayed in Tables 2.a, 2.b.
Revised: The obtained scores are displayed in Tables 2.a, 2.b.

Original: The CE scores computed to get the mCE scores can be found in Appendix E.
Revised: The CE scores computed to get the mCE scores can be found in Appendix E.

Original: The mCE scores obtained in Table 2.a are very different from each other.
Revised: The mCE scores obtained in Table 2.a are very different from each other.

Original: For instance, the mCE score of the model trained with defocus blur , is much lower than the one trained with brightness .
Revised: For instance, the mCE score of the model trained with defocus blur , is much lower than the one trained with brightness .

Original: Then, according to the robustness estimation made with ImageNet-C, one of the models is much more robust than the other one.
Revised: Then, according to the robustness estimation made with ImageNet-C, one of the models is much more robust than the other one.

Original: In other words, the ImageNet-C benchmark gives more importance to the robustness to defocus blur than to the robustness to brightness : ImageNet-C is unbalanced.
Revised: In other words, the ImageNet-C benchmark gives more importance to the robustness to defocus blur than to the robustness to brightness : ImageNet-C is unbalanced.

Original: We observe far less variations in the mCE scores in Table 2.b than in Table 2.a.
Revised: We observe far less variations in the mCE scores in Table 2.b than in Table 2.a.

Original: More precisely, the difference between the lowest mCE and the highest mCE in Table 2.a and 2.b are respectively 40 and 11.
Revised: More precisely, the difference between the lowest mCE and the highest mCE in Table 2.a and 2.b are respectively 40 and 11.

Original: And the standard deviation of the mCE scores in these tables are respectively 12.1 and 3.7.
Revised: And the standard deviation of the mCE scores in these tables are respectively 12.1 and 3.7.

Original: Then, the importance given by ImageNet-C to the robustness of its corruptions, varies a lot with the considered corruption.
Revised: Then, the importance given by ImageNet-C to the robustness of its corruptions, varies a lot with the considered corruption.

Original: This variation is way less important for our benchmark.
Revised: This variation is way less important for our benchmark.

Original: Then,
Revised: Then,

Original: ImageNet-NOC is signiﬁcantly more balanced than ImageNet-C.
Revised: ImageNet-NOC is signiﬁcantly more balanced than ImageNet-C.

Original: 5. R OBUSTNESS E STIMATIONS U SING I MAGE N ET -NOC AND I MAGE N ET -C
Revised: 5. R OBUSTNESS E STIMATIONS U SING I MAGE N ET -NOC AND I MAGE N ET -C

Original: The experiments carried out in Section 5.2 and 5.3, suggest that ImageNet-NOC has a better balance and coverage than ImageNet-C.
Revised: The experiments carried out in Section 5.2 and 5.3, suggest that ImageNet-NOC has a better balance and coverage than ImageNet-C.

Original: To determine whether using ImageNet-NOC instead of ImageNetC makes a difference in practice, we propose to compare the performances of various models on the two benchmarks.
Revised: To determine whether using ImageNet-NOC instead of ImageNetC makes a difference in practice, we propose to compare the performances of various models on the two benchmarks.

Original: First, we measure the mCE scores of several pretrained torchvision classiﬁers on both ImageNet-C and ImageNet-NOC.
Revised: First, we measure the mCE scores of several pretrained torchvision classiﬁers on both ImageNet-C and ImageNet-NOC.

Original: The results are displayed in Table 3 and the details of the computed CE scores can be found in Appendix C.
Revised: The results are displayed in Table 3 and the details of the computed CE scores can be found in Appendix C.

Original: We observe that some models are considered relatively robust to ImageNet-NOC but not robust to ImageNet-C.
Revised: We observe that some models are considered relatively robust to ImageNet-NOC but not robust to ImageNet-C.

Original: For instance, the VGG-19,
Revised: For instance, the VGG-19,

Original: ResNet-50, and WideResNet-50-2 are much more robust to ImageNet-NOC than the AlexNet model, but they are less robust than AlexNet to ImageNet-C.
Revised: ResNet-50, and WideResNet-50-2 are much more robust to ImageNet-NOC than the AlexNet model, but they are less robust than AlexNet to ImageNet-C.

Original: [Table]
Revised: [Table]

Original: The second experiment is carried out using four ResNet-50 that have been shown to be robust to
Revised: The second experiment is carried out using four ResNet-50 that have been shown to be robust to

Original: Imagenet-C. These models are called SIN+IN (Geirhos et al., 2019), ANT 3x3 (Rusak et al., 2020),
Revised: ImageNet-C. These models are called SIN+IN (Geirhos et al., 2019), ANT 3x3 (Rusak et al., 2020),

Original: Augmix (Hendrycks* et al., 2020) and DeepAugment (Hendrycks et al., 2020).
Revised: Augmix (Hendrycks* et al., 2020) and DeepAugment (Hendrycks et al., 2020).

Original: We measure the robustness of these models to ImageNet-NOC by computing their CE scores.
Revised: We measure the robustness of these models to ImageNet-NOC by computing their CE scores.

Original: Then, we compare these scores with the ones obtained with the torchvision pretrained ResNet-50 in Table 4.
Revised: Then, we compare these scores with the ones obtained with the torchvision pretrained ResNet-50 in Table 4.

Original: We observe that the SIN+IN , ANT 3x3 , and DeepAugment models are not robust to border .
Revised: We observe that the SIN+IN , ANT 3x3 , and DeepAugment models are not robust to border .

Original: Interestingly, we show in Section 5.2 that this corruption is not covered by ImageNet-C.
Revised: Interestingly, we show in Section 5.2 that this corruption is not covered by ImageNet-C.

Original: This result conﬁrms that ImageNetNOC can reveal a low robustness of models to corruptions that are not covered by ImageNet-C.
Revised: This result conﬁrms that ImageNetNOC can reveal a low robustness of models to corruptions that are not covered by ImageNet-C.

Original: The ImageNet-NOC and ImageNet-C mCE scores of the ﬁve considered models are compared in the two last columns of Table 4.
Revised: The ImageNet-NOC and ImageNet-C mCE scores of the ﬁve considered models are compared in the two last columns of Table 4.

Original: We observe that the robustness ranking established with ImageNet-C is different from the one established with ImageNet-NOC.
Revised: We observe that the robustness ranking established with ImageNet-C is different from the one established with ImageNet-NOC.

Original: For instance, ANT 3x3 is very robust to
Revised: For instance, ANT 3x3 is very robust to

Original: ImageNet-C but not robust to ImageNet-NOC.
Revised: ImageNet-C but not robust to ImageNet-NOC.

Original: We think this is a direct consequence of the lack of balance of ImageNet-C.
Revised: We think this is a direct consequence of the lack of balance of ImageNet-C.

Original: Indeed, we provide evidence in Section 5.3 that ImageNet-C gives a lot of importance to the noise robustness, and ANT 3x3 has been shown to be particularly robust to noises (Rusak et al., 2020).
Revised: Indeed, we provide evidence in Section 5.3 that ImageNet-C gives a lot of importance to the noise robustness, and ANT 3x3 has been shown to be particularly robust to noises (Rusak et al., 2020).

Original: So ANT 3x3 is considered as very robust to ImageNet-C, but not to ImageNetNOC which is more balanced.
Revised: So ANT 3x3 is considered as very robust to ImageNet-C, but not to ImageNetNOC which is more balanced.

Original: We note that Augmix obtains a relatively low ImageNet-NOC mCE compared to the ImageNet-C one.
Revised: We note that Augmix obtains a relatively low ImageNet-NOC mCE compared to the ImageNet-C one.

Original: This result should be considered cautiously because shears and quantizations are used by the Augmix data augmentation procedure, and these corruptions overlap with some of the ImageNet-NOC corruptions.
Revised: This result should be considered cautiously because shears and quantizations are used by the Augmix data augmentation procedure, and these corruptions overlap with some of the ImageNet-NOC corruptions.

Original: This is a reason why the Augmix model obtains low shear and quantization CE scores.
Revised: This is a reason why the Augmix model obtains low shear and quantization CE scores.

Original: The experiments carried out in this section show that the robustness estimations made with
Revised: The experiments carried out in this section show that the robustness estimations made with

Original: ImageNet-NOC are different from the ones made with ImageNet-C.
Revised: ImageNet-NOC are different from the ones made with ImageNet-C.

Original: We think that ImageNet-NOC should be preferred to ImageNet-C because of its coverage and balance.
Revised: We think that ImageNet-NOC should be preferred to ImageNet-C because of its coverage and balance.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We proposed a metric called the corruption overlapping score, that measures to what extend the robustnesses towards two corruptions are correlated.
Revised: We proposed a metric called the corruption overlapping score, that measures to what extend the robustnesses towards two corruptions are correlated.

Original: We showed that the overlappings between the corruptions of a benchmark can reduce its coverage and make it unbalanced.
Revised: We showed that the overlappings between the corruptions of a benchmark can reduce its coverage and make it unbalanced.

Original: We provided a benchmark of Non-Overlapping Corruptions called ImageNet-NOC to measure the robustness of image classiﬁers.
Revised: We provided a benchmark of Non-Overlapping Corruptions called ImageNet-NOC to measure the robustness of image classiﬁers.

Original: We showed that ImageNet-NOC is balanced and covers several kinds of common corruptions that are not covered by ImageNet-C. ImageNet-NOC is built thanks to the method we proposed to construct non-overlapping corruption benchmarks.
Revised: We showed that ImageNet-NOC is balanced and covers several kinds of common corruptions that are not covered by ImageNet-C. ImageNet-NOC is built thanks to the method we proposed to construct non-overlapping corruption benchmarks.

Original: This method can be easily adapted to other computer vision tasks.
Revised: This method can be easily adapted to other computer vision tasks.

Original: We hope it will be used to build other non-overlapping corruption benchmarks, that will help to make better estimations of the robustness of neural networks.
Revised: We hope it will be used to build other non-overlapping corruption benchmarks, that will help to make better estimations of the robustness of neural networks.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: A P RESENTATION OF THE M ODELED C OMMON C ORRUPTIONS
Revised: A P RESENTATION OF THE M ODELED C OMMON C ORRUPTIONS

Original: The common corruptions gathered in Section 4 are implemented with computationally cheap image transformations that can be easily added in an image pipeline.
Revised: The common corruptions gathered in Section 4 are implemented with computationally cheap image transformations that can be easily added in an image pipeline.

Original: We provide in Figures 5 and 6, information about how the common corruptions used in this study are modeled.
Revised: We provide in Figures 5 and 6, information about how the common corruptions used in this study are modeled.

Original: These corruptions are implemented for images that have pixel values in [0-1].
Revised: These corruptions are implemented for images that have pixel values in [0-1].

Original: The Severity Range column of these arrays precises how the severity of each corruption is set.
Revised: The Severity Range column of these arrays precises how the severity of each corruption is set.

Original: The lower bound of each severity range is selected to get a robustness score of 0.95 with the standard
Revised: The lower bound of each severity range is selected to get a robustness score of 0.95 with the standard

Original: ResNet-18 model, tested on the ImageNet validation set that has been altered with the considered corruption.
Revised: ResNet-18 model, tested on the ImageNet validation set that has been altered with the considered corruption.

Original: The upper bound is selected to get a robustness score of 0.5 in the same conditions.
Revised: The upper bound is selected to get a robustness score of 0.5 in the same conditions.

Original: The upper bounds of the hue and gray scale corruptions are different because these corruptions are not harmful enough to reach a robustness score of 0.5.
Revised: The upper bounds of the hue and gray scale corruptions are different because these corruptions are not harmful enough to reach a robustness score of 0.5.

Original: They respectively reach a robustness score of 0.63 and 0.70 for the standard ResNet-18.
Revised: They respectively reach a robustness score of 0.63 and 0.70 for the standard ResNet-18.

Original: The last column of the Figures 5 and 6 corresponds to the error rate of the torchvision pretrained AlexNet on the ImageNet validation set corrupted with the corruption indicated in the ﬁrst column.
Revised: The last column of the Figures 5 and 6 corresponds to the error rate of the torchvision pretrained AlexNet on the ImageNet validation set corrupted with the corruption indicated in the ﬁrst column.

Original: In Section 4, by completing the step (1) of Algorithm 1, we compute the overlapping scores between the corruptions gathered in Figure 2.
Revised: In Section 4, by completing the step (1) of Algorithm 1, we compute the overlapping scores between the corruptions gathered in Figure 2.

Original: These overlapping scores are displayed in Figure 4.
Revised: These overlapping scores are displayed in Figure 4.

Original: [Figure]
Revised: [Figure]

Original: ENCHMARKS O BTAINED FOR D IFFERENT V
Revised: ENCHMARKS O BTAINED FOR D IFFERENT V

Original: OF THE O VERLAPPING T
Revised: OF THE O VERLAPPING T

Original: C P ERFORMANCES OF V ARIOUS M ODELS ON I MAGE N ET -NOC AND I MAGE N ET -C
Revised: C P ERFORMANCES OF V ARIOUS M ODELS ON I MAGE N ET -NOC AND I MAGE N ET -C

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: D O VERLAPPINGS O BTAINED FOR T HREE D IFFERENT A RCHITECTURES
Revised: D O VERLAPPINGS O BTAINED FOR T HREE D IFFERENT A RCHITECTURES

Original: E P ERFORMANCES OF M ODELS T RAINED WITH A D ATA A UGMENTATION WITH O NE C ORRUPTION ON I MAGE N ET -C AND I MAGE N ET -NOC
Revised: E P ERFORMANCES OF M ODELS T RAINED WITH A D ATA A UGMENTATION WITH O NE C ORRUPTION ON I MAGE N ET -C AND I MAGE N ET -NOC

Original: We provide in Table 7 and 8, the CE scores computed and averaged to get the mCE scores of Tables 2.a and 2.b.
Revised: We provide in Table 7 and 8, the CE scores computed and averaged to get the mCE scores of Tables 2.a and 2.b.

Original: The ﬁrst column of these arrays contains the error rate on the non-corrupted ImageNet validation set.
Revised: The ﬁrst column of these arrays contains the error rate on the non-corrupted ImageNet validation set.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: 83 Frost 0.
Revised: 83 Frost 0.

Original: 79 Fog 0.
Revised: 79 Fog 0.

Original: 86 Bright 0.
Revised: 86 Bright 0.

Original: 89 Contr 0.
Revised: 89 Contr 0.

Original: 78 Elastic 0.
Revised: 78 Elastic 0.

Original: 92 Pixel 0.
Revised: 92 Pixel 0.

Original: 85 Jpeg 0.
Revised: 85 Jpeg 0.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 83 Rain 0.
Revised: 83 Rain 0.

Original: 83 Border 0.
Revised: 83 Border 0.

Original: [Table] [Table]
Revised: [Table] [Table]