Original: I MPROVING R OBUSTNESS W ITHOUT S ACRIFICING A CCURACY WITH P ATCH G AUSSIAN A UGMENTATION
Revised: I MPROVING R OBUSTNESS W ITHOUT S ACRIFICING A CCURACY WITH P ATCH G AUSSIAN A UGMENTATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deploying machine learning systems in the real world requires both high accuracy on clean data and robustness to naturally occurring corruptions
Revised: Deploying machine learning systems in the real world requires both high accuracy on clean data and robustness to naturally occurring corruptions

Original: While architec
Revised: While architec

Original: - tural advances have led to improved accuracy building robust models remains challenging involving major changes in training procedure and datasets.
Revised: - tural advances have led to improved accuracy building robust models remains challenging involving major changes in training procedure and datasets.

Original: Prior work has argued that there is an inherent trade-off between robustness and accuracy as exempliﬁed by standard data augmentation techniques such as Cutout
Revised: Prior work has argued that there is an inherent trade-off between robustness and accuracy as exempliﬁed by standard data augmentation techniques such as Cutout

Original: , which improves clean accuracy but not robustness, and additive Gaussian noise, which improves robustness but hurts accuracy
Revised: , which improves clean accuracy but not robustness, and additive Gaussian noise, which improves robustness but hurts accuracy

Original: We introduce
Revised: We introduce

Original: Patch
Revised: Patch

Original: Gaussian , a simple augmentation scheme that adds noise to randomly selected patches in an input image.
Revised: Gaussian , a simple augmentation scheme that adds noise to randomly selected patches in an input image.

Original: Models trained with Patch Gaussian achieve state of the art on the CIFAR-10 and ImageNet Common Corruptions benchmarks while also maintaining accuracy on clean data.
Revised: Models trained with Patch Gaussian achieve state of the art on the CIFAR-10 and ImageNet Common Corruptions benchmarks while also maintaining accuracy on clean data.

Original: We ﬁnd that this augmentation leads to reduced sensitivity to high frequency noise (similar to Gaussian ) while retaining the ability to take advantage of relevant high frequency information in the image (similar to
Revised: We ﬁnd that this augmentation leads to reduced sensitivity to high frequency noise (similar to Gaussian ) while retaining the ability to take advantage of relevant high frequency information in the image (similar to

Original: Cutout ).
Revised: Cutout ).

Original: We show it can be used in conjunction with other regularization methods and data augmentation policies such as AutoAugment.
Revised: We show it can be used in conjunction with other regularization methods and data augmentation policies such as AutoAugment.

Original: Finally, we ﬁnd that the idea of restricting perturbations to patches can also be useful in the context of adversarial learning, yielding models without the loss in accuracy that is found with unconstrained adversarial training
Revised: Finally, we ﬁnd that the idea of restricting perturbations to patches can also be useful in the context of adversarial learning, yielding models without the loss in accuracy that is found with unconstrained adversarial training

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: [Figure]
Revised: [Figure]

Original: the accuracy robustness tradeoff observed in other augmentation strate
Revised: the accuracy robustness tradeoff observed in other augmentation strate

Original: Larger σ of Patch Gaussian ( → ) improves mean corruption error (mCE) and maintains clean accuracy, whereas larger σ of Gaussian ( → ) and patch size of Cutout ( → ) hurt accuracy or robustness.
Revised: Larger σ of Patch Gaussian ( → ) improves mean corruption error (mCE) and maintains clean accuracy, whereas larger σ of Gaussian ( → ) and patch size of Cutout ( → ) hurt accuracy or robustness.

Original: More robust and accurate models are down and to the right.
Revised: More robust and accurate models are down and to the right.

Original: Modern deep neural networks can achieve impressive performance at classifying images in curated datasets (Karpathy, 2011; Krizhevsky et al., 2012;
Revised: Modern deep neural networks can achieve impressive performance at classifying images in curated datasets (Karpathy, 2011; Krizhevsky et al., 2012;

Original: Tan & Le, 2019).
Revised: Tan & Le, 2019).

Original: Yet, they lack robustness to various forms of distribution shift that typically occur in real-world settings.
Revised: Yet, they lack robustness to various forms of distribution shift that typically occur in real-world settings.

Original: For example, neural networks are sensitive to small translations and changes in scale (Azulay & Weiss, 2018), blurring and additive noise (Dodge & Karam, 2017), small objects placed in images (Rosenfeld et al., 2018), and even different images from a distribution similar to the training set (Recht et al., 2019; 2018).
Revised: For example, neural networks are sensitive to small translations and changes in scale (Azulay & Weiss, 2018), blurring and additive noise (Dodge & Karam, 2017), small objects placed in images (Rosenfeld et al., 2018), and even different images from a distribution similar to the training set (Recht et al., 2019; 2018).

Original: For models to be useful in the real world, they need to be both accurate on a high-quality held-out set of images, which we refer to as “clean accuracy,” and robust on corrupted images, which we refer to as “robustness.”
Revised: For models to be useful in the real world, they need to be both accurate on a high-quality held-out set of images, which we refer to as “clean accuracy,” and robust on corrupted images, which we refer to as “robustness.”

Original: Most of the literature in machine learning has focused on architectural changes (Simonyan & Zisserman, 2015;
Revised: Most of the literature in machine learning has focused on architectural changes (Simonyan & Zisserman, 2015;

Original: Szegedy et al., 2015;
Revised: Szegedy et al., 2015;

Original: He et al., 2016; Zoph & Le, 2017; Szegedy et al., 2017; Han et al., 2017; Zoph et al., 2017; Hu et al., 2017; Liu et al., 2018) to improve clean accuracy but interest has recently shifted toward robustness as well.
Revised: He et al., 2016; Zoph & Le, 2017; Szegedy et al., 2017; Han et al., 2017; Zoph et al., 2017; Hu et al., 2017; Liu et al., 2018) to improve clean accuracy but interest has recently shifted toward robustness as well.

Original: Research in neural network robustness has tried to quantify the problem by establishing benchmarks that directly measure it (Hendrycks & Dietterich, 2018;
Revised: Research in neural network robustness has tried to quantify the problem by establishing benchmarks that directly measure it (Hendrycks & Dietterich, 2018;

Original: Gu et al., 2019) and comparing the performance of humans and neural networks (Geirhos et al., 2018b; Elsayed et al., 2018).
Revised: Gu et al., 2019) and comparing the performance of humans and neural networks (Geirhos et al., 2018b; Elsayed et al., 2018).

Original: Others have tried to understand robustness by highlighting systemic failure modes of current methods.
Revised: Others have tried to understand robustness by highlighting systemic failure modes of current methods.

Original: For instance, networks exhibit excessive invariance to visual features (Jacobsen et al., 2018), texture bias (Geirhos et al., 2018a), sensitivity to worst-case (adversarial) perturbations (Goodfellow et al., 2014), and a propensity to rely on non-robust, but highly predictive features for classiﬁcation (Doersch et al., 2015;
Revised: For instance, networks exhibit excessive invariance to visual features (Jacobsen et al., 2018), texture bias (Geirhos et al., 2018a), sensitivity to worst-case (adversarial) perturbations (Goodfellow et al., 2014), and a propensity to rely on non-robust, but highly predictive features for classiﬁcation (Doersch et al., 2015;

Original: Ilyas et al., 2019).
Revised: Ilyas et al., 2019).

Original: Of particular relevance, Ford et al.
Revised: Of particular relevance, Ford et al.

Original: (2019) has established connections between popular notions of adversarial robustness and some measures of distribution shift considered here.
Revised: (2019) has established connections between popular notions of adversarial robustness and some measures of distribution shift considered here.

Original: Another line of work has attempted to increase model robustness performance, either by projecting out superﬁcial statistics (Wang et al., 2019), via architectural improvements (Cubuk et al., 2017), pretraining schemes (Hendrycks et al., 2019), or with the use of data augmentations.
Revised: Another line of work has attempted to increase model robustness performance, either by projecting out superﬁcial statistics (Wang et al., 2019), via architectural improvements (Cubuk et al., 2017), pretraining schemes (Hendrycks et al., 2019), or with the use of data augmentations.

Original: Data augmentation increases the size and diversity of the training set, and provides a simple way to learn invariances that are challenging to encode architecturally (Cubuk et al., 2017).
Revised: Data augmentation increases the size and diversity of the training set, and provides a simple way to learn invariances that are challenging to encode architecturally (Cubuk et al., 2017).

Original: Recent work in this area includes learning better transformations (DeVries & Taylor, 2017; Zhang et al., 2017; Zhong et al., 2017), inferring combinations of them (Cubuk et al., 2018), unsupervised methods (Xie et al., 2019), theory of data augmentation (Dao et al., 2018), and applications for one-shot learning (Asano et al., 2019).
Revised: Recent work in this area includes learning better transformations (DeVries & Taylor, 2017; Zhang et al., 2017; Zhong et al., 2017), inferring combinations of them (Cubuk et al., 2018), unsupervised methods (Xie et al., 2019), theory of data augmentation (Dao et al., 2018), and applications for one-shot learning (Asano et al., 2019).

Original: Despite these advances, individual data augmentation methods that improve robustness do so at the expense of reduced clean accuracy.
Revised: Despite these advances, individual data augmentation methods that improve robustness do so at the expense of reduced clean accuracy.

Original: Further, achieving robustness on par with the human visual system is thought to require major changes in training procedures and datasets: the current state of the art in robustness benchmarks involves creating a custom dataset with styled-transferred images before training (Geirhos et al., 2018a), and still incurs a signiﬁcant drop in clean accuracy.
Revised: Further, achieving robustness on par with the human visual system is thought to require major changes in training procedures and datasets: the current state of the art in robustness benchmarks involves creating a custom dataset with styled-transferred images before training (Geirhos et al., 2018a), and still incurs a signiﬁcant drop in clean accuracy.

Original: The ubiquity of reported robustness/accuracy trade-offs in the literature have even led to the hypothesis that these trade-offs may be inevitable (Tsipras et al., 2018).
Revised: The ubiquity of reported robustness/accuracy trade-offs in the literature have even led to the hypothesis that these trade-offs may be inevitable (Tsipras et al., 2018).

Original: Because of this, many recent works focus on improving either one or the other (Madry et al., 2017; Geirhos et al., 2018a).
Revised: Because of this, many recent works focus on improving either one or the other (Madry et al., 2017; Geirhos et al., 2018a).

Original: In this work we propose a simple data augmentation method that overcomes this trade-off, achieving improved robustness while maintaining clean accuracy.
Revised: In this work we propose a simple data augmentation method that overcomes this trade-off, achieving improved robustness while maintaining clean accuracy.

Original: Our contributions are as follows:
Revised: Our contributions are as follows:

Original: • We characterize a trade-off between robustness and accuracy in standard data augmentations Cutout and Gaussian (Section 2.1).
Revised: • We characterize a trade-off between robustness and accuracy in standard data augmentations Cutout and Gaussian (Section 2.1).

Original: • We describe a simple data augmentation method (which we term Patch Gaussian ) that allows us to interpolate between the two augmentations above (Section 3.1).
Revised: • We describe a simple data augmentation method (which we term Patch Gaussian ) that allows us to interpolate between the two augmentations above (Section 3.1).

Original: Despite its simplicity, Patch Gaussian achieves a new state of the art in the Common Corruptions robustness benchmark (Hendrycks & Dietterich, 2018), while maintaining clean accuracy, indicating current methods have not reached this fundamental trade-off (Section 4.1).
Revised: Despite its simplicity, Patch Gaussian achieves a new state of the art in the Common Corruptions robustness benchmark (Hendrycks & Dietterich, 2018), while maintaining clean accuracy, indicating current methods have not reached this fundamental trade-off (Section 4.1).

Original: • We demonstrate that Patch Gaussian can be combined with other regularization strategies (Section 4.2) and data augmentation policies (Section 4.3).
Revised: • We demonstrate that Patch Gaussian can be combined with other regularization strategies (Section 4.2) and data augmentation policies (Section 4.3).

Original: • We perform a frequency-based analysis (Yin et al., 2019) of models trained with Patch Gaussian and ﬁnd that they can better leverage high-frequency information in lower layers, while not being too sensitive to them at later ones (Section 5.1).
Revised: • We perform a frequency-based analysis (Yin et al., 2019) of models trained with Patch Gaussian and ﬁnd that they can better leverage high-frequency information in lower layers, while not being too sensitive to them at later ones (Section 5.1).

Original: • We show a similar method can be used in adversarial training, suggesting under-explored questions about training distributions’ effect on out-of-distribution robustness (Section 5.2).
Revised: • We show a similar method can be used in adversarial training, suggesting under-explored questions about training distributions’ effect on out-of-distribution robustness (Section 5.2).

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: We start by considering two data augmentations: Cutout (DeVries & Taylor, 2017) and Gaussian (Grandvalet & Canu, 1997).
Revised: We start by considering two data augmentations: Cutout (DeVries & Taylor, 2017) and Gaussian (Grandvalet & Canu, 1997).

Original: The former sets a random patch of the input image to a constant (mean pixel in the dataset) in order to improve clean accuracy.
Revised: The former sets a random patch of the input image to a constant (mean pixel in the dataset) in order to improve clean accuracy.

Original: The latter adds independent Gaussian noise to each pixel of the input image, which directly increases robustness to Gaussian noise
Revised: The latter adds independent Gaussian noise to each pixel of the input image, which directly increases robustness to Gaussian noise

Original: . 2. C UTOUT AND G AUSSIAN EXHIBIT A TRADE - OFF BETWEEN ACCURACY AND ROBUSTNESS
Revised: . 2. C UTOUT AND G AUSSIAN EXHIBIT A TRADE - OFF BETWEEN ACCURACY AND ROBUSTNESS

Original: We compare the effectiveness of Gaussian and Cutout data augmentation for accuracy and robustness by measuring the performance of models trained with each on clean as well as corrupted data.
Revised: We compare the effectiveness of Gaussian and Cutout data augmentation for accuracy and robustness by measuring the performance of models trained with each on clean as well as corrupted data.

Original: Here, robustness is deﬁned as average accuracy of the model, when tested on data corrupted by various σ ( 0 . 1 , 0 . 2 , 0 . 3 , 0 . 5 , 0 . 8 , 1 . 0 ) of Gaussian noise, relative to the clean accuracy:
Revised: Here, robustness is deﬁned as average accuracy of the model, when tested on data corrupted by various σ ( 0 . 1 , 0 . 2 , 0 . 3 , 0 . 5 , 0 . 8 , 1 . 0 ) of Gaussian noise, relative to the clean accuracy:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Fig.
Revised: Fig.

Original: 2 highlights an apparent trade-off in using these methods.
Revised: 2 highlights an apparent trade-off in using these methods.

Original: In accordance to previous work (DeVries & Taylor, 2017), Cutout improves accuracy on clean test data.
Revised: In accordance to previous work (DeVries & Taylor, 2017), Cutout improves accuracy on clean test data.

Original: Despite this, we ﬁnd it does not lead to increased robustness.
Revised: Despite this, we ﬁnd it does not lead to increased robustness.

Original: Conversely, training with higher σ of Gaussian can lead to increased robustness to Gaussian noise, but also leads to decreased accuracy on clean data.
Revised: Conversely, training with higher σ of Gaussian can lead to increased robustness to Gaussian noise, but also leads to decreased accuracy on clean data.

Original: Therefore, any robustness gains are offset by poor overall performance: a model with a perfect Relative Robustness of 0 , but whose clean accuracy dropped to 50% will be wrong half the time, even on clean data.
Revised: Therefore, any robustness gains are offset by poor overall performance: a model with a perfect Relative Robustness of 0 , but whose clean accuracy dropped to 50% will be wrong half the time, even on clean data.

Original: [Figure]
Revised: [Figure]

Original: At ﬁrst glance, these results seem to reinforce the ﬁndings of previous work (Tsipras et al., 2018), indicating that robustness comes at the cost of generalization, which would offset any beneﬁts of improved robustness.
Revised: At ﬁrst glance, these results seem to reinforce the ﬁndings of previous work (Tsipras et al., 2018), indicating that robustness comes at the cost of generalization, which would offset any beneﬁts of improved robustness.

Original: In the following sections, we will explore whether there exists augmentation strategies that do not exhibit this limitation.
Revised: In the following sections, we will explore whether there exists augmentation strategies that do not exhibit this limitation.

Original:  M ETHOD
Revised:  M ETHOD

Original: Each of the two methods seen so far achieves one half of our stated goal: either improving robustness or slightly improving/maintaining clean test accuracy, but never both.
Revised: Each of the two methods seen so far achieves one half of our stated goal: either improving robustness or slightly improving/maintaining clean test accuracy, but never both.

Original: To explore whether this observed trade-off is fundamental, we introduce Patch Gaussian , a technique that combines the noise robustness of Gaussian with the slightly improved clean accuracy of Cutout .
Revised: To explore whether this observed trade-off is fundamental, we introduce Patch Gaussian , a technique that combines the noise robustness of Gaussian with the slightly improved clean accuracy of Cutout .

Original: Our method is intentionally simple but, as we’ll see, it’s powerful enough to overcome the limitations described and beats complex training schemes designed to provide robustness.
Revised: Our method is intentionally simple but, as we’ll see, it’s powerful enough to overcome the limitations described and beats complex training schemes designed to provide robustness.

Original: 3. P ATCH G AUSSIAN
Revised: 3. P ATCH G AUSSIAN

Original: Patch Gaussian works by adding a W × W
Revised: Patch Gaussian works by adding a W × W

Original: Figure 3)
Revised: Figure 3)

Original: As with Cutout , the center of the patch is sampled to be within the image.
Revised: As with Cutout , the center of the patch is sampled to be within the image.

Original: By varying the size of this patch and the maximum standard deviation of noise sampled σ max , we can interpolate between Gaussian (which applies additive Gaussian noise to the whole image) and an approximation of Cutout (which removes all information inside the patch).
Revised: By varying the size of this patch and the maximum standard deviation of noise sampled σ max , we can interpolate between Gaussian (which applies additive Gaussian noise to the whole image) and an approximation of Cutout (which removes all information inside the patch).

Original: See Fig.
Revised: See Fig.

Original: 9 for more examples.
Revised: 9 for more examples.

Original: 3. H YPER - PARAMETER SELECTION
Revised: 3. H YPER - PARAMETER SELECTION

Original: Our goal is to learn models that achieve both good clean accuracy and improved robustness to corruptions.
Revised: Our goal is to learn models that achieve both good clean accuracy and improved robustness to corruptions.

Original: Prior work has optimized for one or the other but, as noted before, to meaningfully improve robustness to other distributions, a method can’t incur a signiﬁcant drop in clean accuracy.
Revised: Prior work has optimized for one or the other but, as noted before, to meaningfully improve robustness to other distributions, a method can’t incur a signiﬁcant drop in clean accuracy.

Original: Therefore, when selecting hyper-parameters, we focus on identifying the models that are most robust while still achieving a minimum accuracy ( Z ) on the clean test data.
Revised: Therefore, when selecting hyper-parameters, we focus on identifying the models that are most robust while still achieving a minimum accuracy ( Z ) on the clean test data.

Original: Values of Z are selected to incur negligible decrease in clean accuracy.
Revised: Values of Z are selected to incur negligible decrease in clean accuracy.

Original: As such, they vary per dataset and model, and can be found in the Appendix (Table 5).
Revised: As such, they vary per dataset and model, and can be found in the Appendix (Table 5).

Original: If no model has clean accuracy ≥ Z , we report the model with highest clean accuracy, unless otherwise speciﬁed.
Revised: If no model has clean accuracy ≥ Z , we report the model with highest clean accuracy, unless otherwise speciﬁed.

Original: We ﬁnd that patch sizes around 25 on CIFAR ( ≤ 250 on ImageNet, i.e.: uniformly sampled with maximum value 250) with σ ≤ 1 . 0 generally perform the best.
Revised: We ﬁnd that patch sizes around 25 on CIFAR ( ≤ 250 on ImageNet, i.e.: uniformly sampled with maximum value 250) with σ ≤ 1 . 0 generally perform the best.

Original: A complete list of selected hyperparameters for all augmentations can be found in Table 5.
Revised: A complete list of selected hyperparameters for all augmentations can be found in Table 5.

Original: We are interested in out-of-distribution robustness, and report performance of selected models on
Revised: We are interested in out-of-distribution robustness, and report performance of selected models on

Original: Common Corruption (Hendrycks & Dietterich, 2018).
Revised: Common Corruption (Hendrycks & Dietterich, 2018).

Original: However, when selecting hyper-parameters, we use Relative Gaussian Robustness as a stand-in for “robustness.”
Revised: However, when selecting hyper-parameters, we use Relative Gaussian Robustness as a stand-in for “robustness.”

Original: Ford et al.
Revised: Ford et al.

Original: (2019) indicates that this metric is correlated with performance on Common Corruptions, so selected models should be generally robust beyond Gaussian corruptions.
Revised: (2019) indicates that this metric is correlated with performance on Common Corruptions, so selected models should be generally robust beyond Gaussian corruptions.

Original: By picking models based on robustness to Gaussian noise, we ensure that our selection process does not overﬁt to the Common Corruptions benchmark.
Revised: By picking models based on robustness to Gaussian noise, we ensure that our selection process does not overﬁt to the Common Corruptions benchmark.

Original:  R ESULTS
Revised:  R ESULTS

Original: Models trained with Patch Gaussian overcome the observed trade-off and gain robustness to
Revised: Models trained with Patch Gaussian overcome the observed trade-off and gain robustness to

Original: Gaussian noise while maintaining clean accuracy (Fig. 1).
Revised: Gaussian noise while maintaining clean accuracy (Fig. 1).

Original: Because Gaussian robustness is only used for hyper-parameter selection, we omit these results, but refer the curious reader to Appendix Fig.
Revised: Because Gaussian robustness is only used for hyper-parameter selection, we omit these results, but refer the curious reader to Appendix Fig.

Original: Instead, we report how this Gaussian robustness translates into better Common Corruption robustness, which is in line with reports of the correlation between the two (Ford et al., 2019).
Revised: Instead, we report how this Gaussian robustness translates into better Common Corruption robustness, which is in line with reports of the correlation between the two (Ford et al., 2019).

Original: In doing so, we establish a new state of the art in the Common Corruptions benchmark (Section 4.1), despite the simplicity of our method when compared with the previous best (Geirhos et al., 2018a).
Revised: In doing so, we establish a new state of the art in the Common Corruptions benchmark (Section 4.1), despite the simplicity of our method when compared with the previous best (Geirhos et al., 2018a).

Original: We then show that Patch Gaussian can be used in complement to other common regularization strategies (Section 4.2) and data augmentation policies (Cubuk et al., 2018)
Revised: We then show that Patch Gaussian can be used in complement to other common regularization strategies (Section 4.2) and data augmentation policies (Cubuk et al., 2018)

Original: (Section 4.3).
Revised: (Section 4.3).

Original: 4. T RAINING WITH P ATCH G AUSSIAN IMPROVES C OMMON C ORRUPTION ROBUSTNESS
Revised: 4. T RAINING WITH P ATCH G AUSSIAN IMPROVES C OMMON C ORRUPTION ROBUSTNESS

Original: In this section, we look at how our augmentations impact robustness to corruptions beyond Gaussian noise.
Revised: In this section, we look at how our augmentations impact robustness to corruptions beyond Gaussian noise.

Original: Rather than focusing on adversarial examples that are worst-case bounded perturbations, we focus on a more general set of corruptions (Gilmer et al., 2018) that models are likely to encounter in real-world settings: the Common Corruptions benchmark (Hendrycks & Dietterich, 2018).
Revised: Rather than focusing on adversarial examples that are worst-case bounded perturbations, we focus on a more general set of corruptions (Gilmer et al., 2018) that models are likely to encounter in real-world settings: the Common Corruptions benchmark (Hendrycks & Dietterich, 2018).

Original: This benchmark, also referred to as CIFAR-C and ImageNet-C, is composed of images transformed with 15 corruptions, at 5 severities each.
Revised: This benchmark, also referred to as CIFAR-C and ImageNet-C, is composed of images transformed with 15 corruptions, at 5 severities each.

Original: Each is designed to model transformations commonly found in real-world settings, such as brightness, different weather conditions, and different kinds of noise.
Revised: Each is designed to model transformations commonly found in real-world settings, such as brightness, different weather conditions, and different kinds of noise.

Original: Table 1 shows that Patch Gaussian achieves state of the art on both of these benchmarks in terms of mean Corruption Error (mCE).
Revised: Table 1 shows that Patch Gaussian achieves state of the art on both of these benchmarks in terms of mean Corruption Error (mCE).

Original: A “Corruption Error” is a model’s average error over 5 severities of a given corruption, normalized by the same average of a baseline model.
Revised: A “Corruption Error” is a model’s average error over 5 severities of a given corruption, normalized by the same average of a baseline model.

Original: However, ImageNet-C was released in compressed JPEG format (ECMA International, 2009), which alters the corruptions applied to the raw pixels.
Revised: However, ImageNet-C was released in compressed JPEG format (ECMA International, 2009), which alters the corruptions applied to the raw pixels.

Original: Therefore, we report results on the benchmark as-released (“Original mCE”) 1 as well as a version of 12 corruptions without the extra compression (“mCE”).
Revised: Therefore, we report results on the benchmark as-released (“Original mCE”) 1 as well as a version of 12 corruptions without the extra compression (“mCE”).

Original: Additionally, because Patch Gaussian is a noise-based augmentation, we wanted to verify whether its gains on this benchmark were solely due to improved performance on noise-based corruptions (Gaussian
Revised: Additionally, because Patch Gaussian is a noise-based augmentation, we wanted to verify whether its gains on this benchmark were solely due to improved performance on noise-based corruptions (Gaussian

Original: Noise, Shot Noise, and Impulse Noise).
Revised: Noise, Shot Noise, and Impulse Noise).

Original: To do this, we also measure the models’ average performance on all other corruptions, reported as “Original mCE (-noise)”, and “mCE (-noise)”.
Revised: To do this, we also measure the models’ average performance on all other corruptions, reported as “Original mCE (-noise)”, and “mCE (-noise)”.

Original: The models used to normalize Corruption Errors are the “Baselines” trained with only ﬂip and crop data augmentation.
Revised: The models used to normalize Corruption Errors are the “Baselines” trained with only ﬂip and crop data augmentation.

Original: The one exception is Original mCE ImageNet, where we use the AlexNet baseline to be directly comparable with previous work (Hendrycks & Dietterich, 2018; Geirhos et al., 2018a).
Revised: The one exception is Original mCE ImageNet, where we use the AlexNet baseline to be directly comparable with previous work (Hendrycks & Dietterich, 2018; Geirhos et al., 2018a).

Original: On CIFAR, we compare with an adversarially-trained model (Madry et al., 2017).
Revised: On CIFAR, we compare with an adversarially-trained model (Madry et al., 2017).

Original: On ImageNet, we compare with a model trained with Random Erasing (Zhong et al., 2017), as well as a shape-biased model “SIN+IN ftIN” (Geirhos et al., 2018a).
Revised: On ImageNet, we compare with a model trained with Random Erasing (Zhong et al., 2017), as well as a shape-biased model “SIN+IN ftIN” (Geirhos et al., 2018a).

Original: Finally, previous work (Yin et al., 2019) has found that augmentation diversity is a key component of robustness gains.
Revised: Finally, previous work (Yin et al., 2019) has found that augmentation diversity is a key component of robustness gains.

Original: To conﬁrm that Patch Gaussian ’s gains aren’t simply a result of using multiple augmentations, we also report results on training with
Revised: To conﬁrm that Patch Gaussian ’s gains aren’t simply a result of using multiple augmentations, we also report results on training with

Original: Cutout and Gaussian applied in sequence (“Cutout & Gaussian” and “Gaussian & Cutout”), as well as to 50% of batches (“Gaussian or Cutout”).
Revised: Cutout and Gaussian applied in sequence (“Cutout & Gaussian” and “Gaussian & Cutout”), as well as to 50% of batches (“Gaussian or Cutout”).

Original: We observe that Patch Gaussian outperforms all models, even on corruptions like fog where
Revised: We observe that Patch Gaussian outperforms all models, even on corruptions like fog where

Original: Gaussian hurts performance (Ford et al., 2019).
Revised: Gaussian hurts performance (Ford et al., 2019).

Original: Scores for each corruption can be found in the
Revised: Scores for each corruption can be found in the

Original: [Table]
Revised: [Table]

Original: Appendix (Tables 6&7).
Revised: Appendix (Tables 6&7).

Original: We observe small gains on low capacity models (ResNet-50 & Wide ResNet), but ﬁnd large robustness improvements with higher capacity ones (ResNet-200 & Shake 112).
Revised: We observe small gains on low capacity models (ResNet-50 & Wide ResNet), but ﬁnd large robustness improvements with higher capacity ones (ResNet-200 & Shake 112).

Original: These results are surprising: achieving robustness on par with the human visual system is thought to require major changes in training procedures and datasets.
Revised: These results are surprising: achieving robustness on par with the human visual system is thought to require major changes in training procedures and datasets.

Original: Training shape-biased models (Geirhos et al., 2018a) involves creating a custom dataset of style-transferred images, which is a computationallyexpensive operation.
Revised: Training shape-biased models (Geirhos et al., 2018a) involves creating a custom dataset of style-transferred images, which is a computationallyexpensive operation.

Original: Even with these, the most robust model reported SIN+IN displays a signiﬁcant drop in clean accuracy.
Revised: Even with these, the most robust model reported SIN+IN displays a signiﬁcant drop in clean accuracy.

Original: Because of this, our main comparison is with SIN+IN ftIN, which is ﬁne-tuned on ImageNet.
Revised: Because of this, our main comparison is with SIN+IN ftIN, which is ﬁne-tuned on ImageNet.

Original: A comparison with SIN+IN can be found in Appendix Table 8.
Revised: A comparison with SIN+IN can be found in Appendix Table 8.

Original: In sum, despite its simplicity, Patch Gaussian achieves a substantial decrease in mCE relative to other models, indicating that current methods have not reached the theoretical trade-off (Tsipras et al., 2018), and that complex training schemes (Geirhos et al., 2018a) are not needed for robustness.
Revised: In sum, despite its simplicity, Patch Gaussian achieves a substantial decrease in mCE relative to other models, indicating that current methods have not reached the theoretical trade-off (Tsipras et al., 2018), and that complex training schemes (Geirhos et al., 2018a) are not needed for robustness.

Original: 4. P ATCH G AUSSIAN CAN BE COMBINED WITH OTHER REGULARIZATION STRATEGIES
Revised: 4. P ATCH G AUSSIAN CAN BE COMBINED WITH OTHER REGULARIZATION STRATEGIES

Original: Since Patch Gaussian has a regularization effect on the models trained above, we compare it with other regularization methods: larger weight decay, label smoothing, and dropblock (Table 2).
Revised: Since Patch Gaussian has a regularization effect on the models trained above, we compare it with other regularization methods: larger weight decay, label smoothing, and dropblock (Table 2).

Original: We ﬁnd that while label smoothing improves clean accuracy, it weakens the robustness in all corruption metrics we have considered.
Revised: We ﬁnd that while label smoothing improves clean accuracy, it weakens the robustness in all corruption metrics we have considered.

Original: This agrees with the theoretical prediction from Cubuk et al.
Revised: This agrees with the theoretical prediction from Cubuk et al.

Original: (2017), which argued that increasing the conﬁdence of models would improve robustness, whereas label smoothing reduces the conﬁdence of predictions.
Revised: (2017), which argued that increasing the conﬁdence of models would improve robustness, whereas label smoothing reduces the conﬁdence of predictions.

Original: We ﬁnd that increasing the weight decay from the default value used in all models does not improve clean accuracy or robustness.
Revised: We ﬁnd that increasing the weight decay from the default value used in all models does not improve clean accuracy or robustness.

Original: Here, we focus on analyzing the interaction of different regularization methods with Patch
Revised: Here, we focus on analyzing the interaction of different regularization methods with Patch

Original: Gaussian .
Revised: Gaussian .

Original: Previous work indicates that improvements on the clean accuracy appear after training with Dropblock for 270 epochs (Ghiasi et al., 2018), but we did not ﬁnd that training for 270 epochs changed our analysis.
Revised: Previous work indicates that improvements on the clean accuracy appear after training with Dropblock for 270 epochs (Ghiasi et al., 2018), but we did not ﬁnd that training for 270 epochs changed our analysis.

Original: Thus, we present models trained at 90 epochs for direct comparison with other results.
Revised: Thus, we present models trained at 90 epochs for direct comparison with other results.

Original: Due to the shorter training time, Dropblock does not improve clean accuracy, yet it does make the model more robust (relative to baseline) according to all corruption metrics we consider.
Revised: Due to the shorter training time, Dropblock does not improve clean accuracy, yet it does make the model more robust (relative to baseline) according to all corruption metrics we consider.

Original: We ﬁnd that using label smoothing in addition to Patch Gaussian has a mixed effect, it improves clean accuracy while slightly improving robustness metrics except for the Original mCE.
Revised: We ﬁnd that using label smoothing in addition to Patch Gaussian has a mixed effect, it improves clean accuracy while slightly improving robustness metrics except for the Original mCE.

Original: Combining
Revised: Combining

Original: Dropblock with Patch Gaussian reduces the clean accuracy relative to the Patch Gaussian -only model, as Dropblock seems to be a strong regularizer when used for 90 epochs.
Revised: Dropblock with Patch Gaussian reduces the clean accuracy relative to the Patch Gaussian -only model, as Dropblock seems to be a strong regularizer when used for 90 epochs.

Original: However, using
Revised: However, using

Original: [Table]
Revised: [Table]

Original: Dropblock and Patch Gaussian together leads to the best robustness performance.
Revised: Dropblock and Patch Gaussian together leads to the best robustness performance.

Original: These results indicate that Patch Gaussian can be used in conjunction with existing regularization strategies.
Revised: These results indicate that Patch Gaussian can be used in conjunction with existing regularization strategies.

Original: 4. P ATCH G AUSSIAN CAN BE COMBINED WITH A UTO A UGMENT FOR IMPROVED RESULTS
Revised: 4. P ATCH G AUSSIAN CAN BE COMBINED WITH A UTO A UGMENT FOR IMPROVED RESULTS

Original: Knowing that Patch Gaussian can be combined with other regularizers, it’s natural to ask whether it can also be combined with other data augmentation policies.
Revised: Knowing that Patch Gaussian can be combined with other regularizers, it’s natural to ask whether it can also be combined with other data augmentation policies.

Original: Previous work has found that varied augmentation policies have a large positive impact on model robustness (Yin et al., 2019).
Revised: Previous work has found that varied augmentation policies have a large positive impact on model robustness (Yin et al., 2019).

Original: In this section, we verify that Patch Gaussian can be added to these policies for further gains
Revised: In this section, we verify that Patch Gaussian can be added to these policies for further gains

Original: Table 3 highlights models trained with AutoAugment (Cubuk et al., 2018).
Revised: Table 3 highlights models trained with AutoAugment (Cubuk et al., 2018).

Original: For fair comparison of mCE scores, we train all models with the best AutoAugment policy, but without contrast and
Revised: For fair comparison of mCE scores, we train all models with the best AutoAugment policy, but without contrast and

Original: Inception color pre-processing, as those are present in the Common Corruptions benchmark.
Revised: Inception color pre-processing, as those are present in the Common Corruptions benchmark.

Original: This process is imperfect since AutoAugment has many operations, some of which could be correlated with corruptions.
Revised: This process is imperfect since AutoAugment has many operations, some of which could be correlated with corruptions.

Original: Regardless, we ﬁnd that Patch Gaussian improves robustness over AutoAugment.
Revised: Regardless, we ﬁnd that Patch Gaussian improves robustness over AutoAugment.

Original: Because AutoAugment leads to state of the art accuracies, we are interested in seeing how far it can be combined with Patch Gaussian to improve results.
Revised: Because AutoAugment leads to state of the art accuracies, we are interested in seeing how far it can be combined with Patch Gaussian to improve results.

Original: Therefore, and unlike previous experiments, models are trained for 180 epochs to yield best results possible.
Revised: Therefore, and unlike previous experiments, models are trained for 180 epochs to yield best results possible.

Original: [Table]
Revised: [Table]

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: In an attempt to understand Patch Gaussian ’s performance, we perform a frequency-based analysis of models trained with various augmentations using the method introduced in Yin et al.
Revised: In an attempt to understand Patch Gaussian ’s performance, we perform a frequency-based analysis of models trained with various augmentations using the method introduced in Yin et al.

Original: First, we perturb each image in the dataset with noise sampled at each orientation and frequency in
Revised: First, we perturb each image in the dataset with noise sampled at each orientation and frequency in

Original: Fourier space.
Revised: Fourier space.

Original: Then, we measure changes in the network activations and test error when evaluated with these Fourier-noise-corrupted images: we measure the change in (cid:96) 2 norm of the tensor directly after the ﬁrst convolution, as well as the absolute test error.
Revised: Then, we measure changes in the network activations and test error when evaluated with these Fourier-noise-corrupted images: we measure the change in (cid:96) 2 norm of the tensor directly after the ﬁrst convolution, as well as the absolute test error.

Original: This procedure yields a heatmap, which indicates model sensitivity to different frequency and orientation perturbations in the Fourier domain.
Revised: This procedure yields a heatmap, which indicates model sensitivity to different frequency and orientation perturbations in the Fourier domain.

Original: Each image in Fig 4 shows ﬁrst layer (or test error) sensitivity as a function of frequency and orientation of the sampled noise, with the middle of the image containing the lowest frequencies, and the edges of the image containing highest frequencies.
Revised: Each image in Fig 4 shows ﬁrst layer (or test error) sensitivity as a function of frequency and orientation of the sampled noise, with the middle of the image containing the lowest frequencies, and the edges of the image containing highest frequencies.

Original: For CIFAR-10 models, we present this analysis for the entire Fourier domain, with noise sampled with norm 4 .
Revised: For CIFAR-10 models, we present this analysis for the entire Fourier domain, with noise sampled with norm 4 .

Original: For ImageNet, we focus our analysis on lower frequencies that are more visually salient add noise with norm 15 . 7 .
Revised: For ImageNet, we focus our analysis on lower frequencies that are more visually salient add noise with norm 15 . 7 .

Original: Note that for Cutout and Gaussian , we chose larger patch sizes and σ s than those selected with the method in Section 3.2 in order to highlight the effect of these augmentations on sensitivity.
Revised: Note that for Cutout and Gaussian , we chose larger patch sizes and σ s than those selected with the method in Section 3.2 in order to highlight the effect of these augmentations on sensitivity.

Original: Heatmaps of other models can be found in the Appendix (Figure 11).
Revised: Heatmaps of other models can be found in the Appendix (Figure 11).

Original: 5. F REQUENCY - BASED ANALYSIS OF MODELS TRAINED WITH P ATCH G AUSSIAN
Revised: 5. F REQUENCY - BASED ANALYSIS OF MODELS TRAINED WITH P ATCH G AUSSIAN

Original: We conﬁrm ﬁndings by Yin et al.
Revised: We conﬁrm ﬁndings by Yin et al.

Original: that Gaussian encourages the model to learn a low-pass ﬁlter of the inputs.
Revised: that Gaussian encourages the model to learn a low-pass ﬁlter of the inputs.

Original: Models trained with this augmentation, then, have low test error sensitivity at high frequencies, which could help robustness.
Revised: Models trained with this augmentation, then, have low test error sensitivity at high frequencies, which could help robustness.

Original: However, valuable high-frequency information (Brendel &
Revised: However, valuable high-frequency information (Brendel &

Original: Bethge, 2019) is being thrown out at low layers, which could explain the lower test accuracy.
Revised: Bethge, 2019) is being thrown out at low layers, which could explain the lower test accuracy.

Original: We further ﬁnd that Cutout encourages the use of high-frequency information, which could help explain its improved generalization performance.
Revised: We further ﬁnd that Cutout encourages the use of high-frequency information, which could help explain its improved generalization performance.

Original: Yet, it does not encourage lower test error sensitivity, which explains why it doesn’t improve robustness either.
Revised: Yet, it does not encourage lower test error sensitivity, which explains why it doesn’t improve robustness either.

Original: Patch Gaussian , on the other hand, seems to allow high-frequency information through at lower layers, but still encourages relatively lower test error sensitivity at high frequencies.
Revised: Patch Gaussian , on the other hand, seems to allow high-frequency information through at lower layers, but still encourages relatively lower test error sensitivity at high frequencies.

Original: Indeed, when we measure accuracy on images ﬁltered with a high-pass ﬁlter, we see that Patch Gaussian models can maintain accuracy in a similar way to the baseline and to Cutout , where Gaussian fails to.
Revised: Indeed, when we measure accuracy on images ﬁltered with a high-pass ﬁlter, we see that Patch Gaussian models can maintain accuracy in a similar way to the baseline and to Cutout , where Gaussian fails to.

Original: See
Revised: See

Original: Figure 4 for full results.
Revised: Figure 4 for full results.

Original: [Figure]
Revised: [Figure]

Original: Heatmaps depict model sensitivity to various sinusoidal gratings.
Revised: Heatmaps depict model sensitivity to various sinusoidal gratings.

Original: Cutout encourages the use of high frequencies in earlier layers, but its test error remains too sensitive to them.
Revised: Cutout encourages the use of high frequencies in earlier layers, but its test error remains too sensitive to them.

Original: learns low-pass of features, which increases robustness at later layers, but makes lower layers too invariant to highfrequency information (thus hurting accuracy).
Revised: learns low-pass of features, which increases robustness at later layers, but makes lower layers too invariant to highfrequency information (thus hurting accuracy).

Original: Patch allows high frequencies to be used in lower
Revised: Patch allows high frequencies to be used in lower

Original: , and its remains relatively robust to them.
Revised: , and its remains relatively robust to them.

Original: This can also be seen by the presence of high in the of
Revised: This can also be seen by the presence of high in the of

Original: (or lack thereof, in the of right
Revised: (or lack thereof, in the of right

Original: Indeed,
Revised: Indeed,

Original: Baseline when presented with only the high frequency information of images, fails to effectively utilize information (see Appendix Fig.
Revised: Baseline when presented with only the high frequency information of images, fails to effectively utilize information (see Appendix Fig.

Original: 12 for details) of reduced predictions to high frequencies in input occurs across all
Revised: 12 for details) of reduced predictions to high frequencies in input occurs across all

Original: Understanding the impact of data distributions and noise on representations has been well-studied in neuroscience (Barlow et al., 1961; Simoncelli & Olshausen, 2001; Karklin & Simoncelli, 2011).
Revised: Understanding the impact of data distributions and noise on representations has been well-studied in neuroscience (Barlow et al., 1961; Simoncelli & Olshausen, 2001; Karklin & Simoncelli, 2011).

Original: The data augmentations that we propose here alter the distribution of inputs that the network sees, and thus are expected to alter the kinds of representations that are learned.
Revised: The data augmentations that we propose here alter the distribution of inputs that the network sees, and thus are expected to alter the kinds of representations that are learned.

Original: Prior work on efﬁcient coding (Karklin & Simoncelli, 2011) and autoencoders (Poole et al., 2014) has shown how ﬁlter properties change with noise in the unsupervised setting, resulting in lower-frequency ﬁlters with Gaussian , as we observe in Fig. 4.
Revised: Prior work on efﬁcient coding (Karklin & Simoncelli, 2011) and autoencoders (Poole et al., 2014) has shown how ﬁlter properties change with noise in the unsupervised setting, resulting in lower-frequency ﬁlters with Gaussian , as we observe in Fig. 4.

Original: Consistent with prior work on natural image statistics (Torralba & Oliva, 2003), we ﬁnd that networks are least sensitive to low frequency noise where spectral density is largest.
Revised: Consistent with prior work on natural image statistics (Torralba & Oliva, 2003), we ﬁnd that networks are least sensitive to low frequency noise where spectral density is largest.

Original: Performance drops at higher frequencies when the amount of noise we add grows relative to typical spectral density observed at these frequencies.
Revised: Performance drops at higher frequencies when the amount of noise we add grows relative to typical spectral density observed at these frequencies.

Original: In future work, we hope to better understand the relationship between naturally occurring properties of images and sensitivity, and investigate whether training with more naturalistic noise can yield similar gains in corruption robustness.
Revised: In future work, we hope to better understand the relationship between naturally occurring properties of images and sensitivity, and investigate whether training with more naturalistic noise can yield similar gains in corruption robustness.

Original: 5. P ATCHING ADVERSARIAL TRAINING ALSO LEADS TO IMPROVED ROBUSTNESS
Revised: 5. P ATCHING ADVERSARIAL TRAINING ALSO LEADS TO IMPROVED ROBUSTNESS

Original: Our results indicate that patching a transformation can prevent overﬁtting to that particular transformation and maintain clean accuracy.
Revised: Our results indicate that patching a transformation can prevent overﬁtting to that particular transformation and maintain clean accuracy.

Original: To further conﬁrm this, we train a model with adversarial training applied only to a patch of the training input.
Revised: To further conﬁrm this, we train a model with adversarial training applied only to a patch of the training input.

Original: Adversarial training is a method of achieving robustness to worst-case perturbations.
Revised: Adversarial training is a method of achieving robustness to worst-case perturbations.

Original: Models trained in this setting notoriously exhibit decreased clean accuracy, so it is a good candidate to verify whether our robustness gains come from patching.
Revised: Models trained in this setting notoriously exhibit decreased clean accuracy, so it is a good candidate to verify whether our robustness gains come from patching.

Original: We train our models with PGD, in a setting equivalent to Madry et al.
Revised: We train our models with PGD, in a setting equivalent to Madry et al.

Original: For Patch PGD , the adversarial perturbation is calculated on the whole image for all steps, and patched after the fact.
Revised: For Patch PGD , the adversarial perturbation is calculated on the whole image for all steps, and patched after the fact.

Original: We also tried calculating PGD on a patch only and found similar results.
Revised: We also tried calculating PGD on a patch only and found similar results.

Original: We select hyper-parameters based on PGD performance on validation set, while maintaining accuracy above 90%.
Revised: We select hyper-parameters based on PGD performance on validation set, while maintaining accuracy above 90%.

Original: However, in this section we are not interested in improving adversarial robustness performance, but on seeing its effect on robustness to Common Corruptions, to evaluate out-of-distribution (o.o.d.) robustness.
Revised: However, in this section we are not interested in improving adversarial robustness performance, but on seeing its effect on robustness to Common Corruptions, to evaluate out-of-distribution (o.o.d.) robustness.

Original: We leave an analysis of the effect of patching on adversarial robustness to future work.
Revised: We leave an analysis of the effect of patching on adversarial robustness to future work.

Original: Indeed, Table 4 shows that training with Patch PGD obtains similar PGD accuracy to training with
Revised: Indeed, Table 4 shows that training with Patch PGD obtains similar PGD accuracy to training with

Original: PGD , but maintains most of the clean accuracy of the baseline model.
Revised: PGD , but maintains most of the clean accuracy of the baseline model.

Original: Surprisingly, Patch PGD also improves robustness to unseen Common Corruptions, when compared to the baseline without adversarial training, indicating that patching is a generally powerful tool.
Revised: Surprisingly, Patch PGD also improves robustness to unseen Common Corruptions, when compared to the baseline without adversarial training, indicating that patching is a generally powerful tool.

Original: This also suggests there are unexplored questions regarding the training distribution and how that translates into i.i.d and o.o.d generalization.
Revised: This also suggests there are unexplored questions regarding the training distribution and how that translates into i.i.d and o.o.d generalization.

Original: We hope to explore these in future work.
Revised: We hope to explore these in future work.

Original: [Table]
Revised: [Table]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we introduced a simple data augmentation operation, Patch Gaussian , which improves robustness to common corruptions without incurring a drop in clean accuracy.
Revised: In this work, we introduced a simple data augmentation operation, Patch Gaussian , which improves robustness to common corruptions without incurring a drop in clean accuracy.

Original: For models that are large relative to the dataset size (like ResNet-200 on ImageNet and all models on CIFAR-10),
Revised: For models that are large relative to the dataset size (like ResNet-200 on ImageNet and all models on CIFAR-10),

Original: Patch Gaussian improves clean accuracy and robustness concurrently.
Revised: Patch Gaussian improves clean accuracy and robustness concurrently.

Original: We showed that Patch
Revised: We showed that Patch

Original: Gaussian achieves this by interpolating between two standard data augmentation operations Cutout and Gaussian .
Revised: Gaussian achieves this by interpolating between two standard data augmentation operations Cutout and Gaussian .

Original: Finally, we analyzed the sensitivity to noise in different frequencies of models trained with Cutout and Gaussian , and showed that Patch Gaussian combines their strengths without inheriting their weaknesses.
Revised: Finally, we analyzed the sensitivity to noise in different frequencies of models trained with Cutout and Gaussian , and showed that Patch Gaussian combines their strengths without inheriting their weaknesses.

Original: Our method is much simpler than previous state of the art, and can be used in conjunction with other regularization and data augmentation strategies, indicating it is generally useful.
Revised: Our method is much simpler than previous state of the art, and can be used in conjunction with other regularization and data augmentation strategies, indicating it is generally useful.

Original: We end by showing that applying perturbations in patches can be a powerful method to vary training distributions in the adversarial setting.
Revised: We end by showing that applying perturbations in patches can be a powerful method to vary training distributions in the adversarial setting.

Original: Our results indicate current methods have not reached a fundamental robustness/accuracy trade-off, and that future work is needed to understand the effect of training distributions in o.o.d.
Revised: Our results indicate current methods have not reached a fundamental robustness/accuracy trade-off, and that future work is needed to understand the effect of training distributions in o.o.d.

Original: robustness.
Revised: robustness.

Original: M ODELS , D ATASETS , & I MPLEMENTATION D ETAILS
Revised: M ODELS , D ATASETS , & I MPLEMENTATION D ETAILS

Original: We run our experiments on CIFAR-10 (Krizhevsky & Hinton, 2009) and ImageNet (Deng et al., 2009) datasets.
Revised: We run our experiments on CIFAR-10 (Krizhevsky & Hinton, 2009) and ImageNet (Deng et al., 2009) datasets.

Original: On CIFAR-10, we use the Wide-ResNet-28-10 model (Zagoruyko & Komodakis, 2016), as well as the Shake-shake-112 model (Gastaldi, 2017), trained for 200 epochs and 600 epochs respectively.
Revised: On CIFAR-10, we use the Wide-ResNet-28-10 model (Zagoruyko & Komodakis, 2016), as well as the Shake-shake-112 model (Gastaldi, 2017), trained for 200 epochs and 600 epochs respectively.

Original: The Wide-ResNet model uses a initial learning rate of 0.1 with a cosine decay schedule.
Revised: The Wide-ResNet model uses a initial learning rate of 0.1 with a cosine decay schedule.

Original: Weight decay is set to be 5 × 10 − 4 and batch size is 128.
Revised: Weight decay is set to be 5 × 10 − 4 and batch size is 128.

Original: We train all models, including the baseline, with standard data augmentation of horizontal ﬂips and pad-and-crop.
Revised: We train all models, including the baseline, with standard data augmentation of horizontal ﬂips and pad-and-crop.

Original: Our code uses the same hyper parameters as Cubuk et al.
Revised: Our code uses the same hyper parameters as Cubuk et al.

Original: On ImageNet, we use the ResNet-50 and Resnet-200 models (He et al., 2016), trained for 90 epochs.
Revised: On ImageNet, we use the ResNet-50 and Resnet-200 models (He et al., 2016), trained for 90 epochs.

Original: We use a weight decay rate of 1 × 10 − 4 , global batch size of 512 and learning rate of 0.2.
Revised: We use a weight decay rate of 1 × 10 − 4 , global batch size of 512 and learning rate of 0.2.

Original: The learning rate is decayed by 10 at epochs 30, 60, and 80.
Revised: The learning rate is decayed by 10 at epochs 30, 60, and 80.

Original: We use standard data augmentation of horizontal ﬂips and crops.
Revised: We use standard data augmentation of horizontal ﬂips and crops.

Original: All CIFAR-10 and ImageNet experiments use the listed hyper-parameters above, unless speciﬁed otherwise.
Revised: All CIFAR-10 and ImageNet experiments use the listed hyper-parameters above, unless speciﬁed otherwise.

Original: To apply Gaussian , we uniformly sample a standard deviation σ from 0 up to some maximum value σ max , and add i.i.d.
Revised: To apply Gaussian , we uniformly sample a standard deviation σ from 0 up to some maximum value σ max , and add i.i.d.

Original: noise sampled from N (0 , σ 2 ) to each pixel.
Revised: noise sampled from N (0 , σ 2 ) to each pixel.

Original: To apply Cutout , we use a ﬁxed patch size W , and randomly set a square region with size W × W to the constant mean of each RGB channel in the dataset.
Revised: To apply Cutout , we use a ﬁxed patch size W , and randomly set a square region with size W × W to the constant mean of each RGB channel in the dataset.

Original: As in DeVries & Taylor (2017), the patch location is randomly sampled and can lie outside of the 32 × 32 CIFAR-10 (or 224 × 224 ImageNet) image but its center is constrained to lie within it.
Revised: As in DeVries & Taylor (2017), the patch location is randomly sampled and can lie outside of the 32 × 32 CIFAR-10 (or 224 × 224 ImageNet) image but its center is constrained to lie within it.

Original: Patch sizes and σ max are selected based on the method in Section 3.2.
Revised: Patch sizes and σ max are selected based on the method in Section 3.2.

Original: All image transformations, including Patch Gaussian , are performed on images with unnormalized pixel values in [0 , 1] range.
Revised: All image transformations, including Patch Gaussian , are performed on images with unnormalized pixel values in [0 , 1] range.

Original: For all images, standard random ﬂipping and cropping are applied immediately after any augmentations mentioned on CIFAR-10 ( before , on ImageNet).
Revised: For all images, standard random ﬂipping and cropping are applied immediately after any augmentations mentioned on CIFAR-10 ( before , on ImageNet).

Original: After noisebased augmentations, images are clipped to the [0 , 1] range.
Revised: After noisebased augmentations, images are clipped to the [0 , 1] range.

Original: Augmentation hyper-parameters are selected based on the method in Sec. 3.2 and shown in Tab. 5.
Revised: Augmentation hyper-parameters are selected based on the method in Sec. 3.2 and shown in Tab. 5.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Fig.
Revised: Fig.

Original: 8 shows an implementation of Patch Gaussian , highlighting its simplicity.
Revised: 8 shows an implementation of Patch Gaussian , highlighting its simplicity.

Original: Fig.
Revised: Fig.

Original: 9 shows the effect of applying Patch Gaussian at various settings to an ImageNet image.
Revised: 9 shows the effect of applying Patch Gaussian at various settings to an ImageNet image.

Original: Figs.
Revised: Figs.

Original: 6 andshow full Original Corruption Errors and Corruption Errors for ImageNet models.
Revised: 6 andshow full Original Corruption Errors and Corruption Errors for ImageNet models.

Original: Fig.
Revised: Fig.

Original: 8 shows a comparison of Patch Gaussian with SIN+IN.
Revised: 8 shows a comparison of Patch Gaussian with SIN+IN.

Original: def _get_patch_mask(patch_size): # randomly sample location in the image x = tf.random.uniform([], minval=0, maxval=224, dtype=tf.int32)
Revised: def _get_patch_mask(patch_size): # randomly sample location in the image x = tf.random.uniform([], minval=0, maxval=224, dtype=tf.int32)

Original: y
Revised: y

Original: = tf.random.uniform([], minval=0, maxval=224, dtype=tf.int32) x, y = tf.cast(x, tf.float32), tf.cast(y, tf.float32) # compute where the patch will start and end startx, starty = x - tf.floor(patch_size/2), y - tf.floor(patch_size/2) endx, endy = x + tf.ceil(patch_size/2), y + tf.ceil(patch_size/2) startx, starty = tf.maximum(startx, 0), tf.maximum(starty, 0) endx, endy = tf.minimum(endx, 224), tf.minimum(endy, 224)
Revised: = tf.random.uniform([], minval=0, maxval=224, dtype=tf.int32) x, y = tf.cast(x, tf.float32), tf.cast(y, tf.float32) # compute where the patch will start and end startx, starty = x - tf.floor(patch_size/2), y - tf.floor(patch_size/2) endx, endy = x + tf.ceil(patch_size/2), y + tf.ceil(patch_size/2) startx, starty = tf.maximum(startx, 0), tf.maximum(starty, 0) endx, endy = tf.minimum(endx, 224), tf.minimum(endy, 224)

Original: # now let’s convert these into how much we need to pad the patch lower_pad, upper_pad = 224 - endy, starty left_pad, right_pad = startx, 224 - endx padding_dims =
Revised: # now let’s convert these into how much we need to pad the patch lower_pad, upper_pad = 224 - endy, starty left_pad, right_pad = startx, 224 - endx padding_dims =

Original: [[upper_pad, lower_pad], [left_pad, right_pad]] # create mask mask =
Revised: [[upper_pad, lower_pad], [left_pad, right_pad]] # create mask mask =

Original: tf.pad(tf.zeros([endy - starty, endx - startx]), padding_dims, constant_values=1) mask =
Revised: tf.pad(tf.zeros([endy - starty, endx - startx]), padding_dims, constant_values=1) mask =

Original: tf.expand_dims(mask, -1) mask =
Revised: tf.expand_dims(mask, -1) mask =

Original: tf.tile(mask, [1, 1, 3]) return tf.equal(mask, 0) def patch_gaussian(image, patch_size, max_scale, sample_up_to): """Returns image with Patch Gaussian applied."
Revised: tf.tile(mask, [1, 1, 3]) return tf.equal(mask, 0) def patch_gaussian(image, patch_size, max_scale, sample_up_to): """Returns image with Patch Gaussian applied."

Original: "" if sample_up_to: patch_size=tf.random.uniform([], 1, patch_size, tf.int32) # otherwise, patch_size is fixed.
Revised: "" if sample_up_to: patch_size=tf.random.uniform([], 1, patch_size, tf.int32) # otherwise, patch_size is fixed.

Original: # make image (which is [0, 255]) be [0, 1] image = image / 255.
Revised: # make image (which is [0, 255]) be [0, 1] image = image / 255.

Original: # uniformly sample scale from 0 to given scale scale = max_scale * tf.random.uniform([], minval=0, maxval=1) # apply gaussian to copy of image.
Revised: # uniformly sample scale from 0 to given scale scale = max_scale * tf.random.uniform([], minval=0, maxval=1) # apply gaussian to copy of image.

Original: Will be used to replace patch in image gaussian =
Revised: Will be used to replace patch in image gaussian =

Original: tf.random.normal(stddev=scale, shape=image.shape)
Revised: tf.random.normal(stddev=scale, shape=image.shape)

Original: image_plus_gaussian
Revised: image_plus_gaussian

Original: = tf.clip_by_value(image
Revised: = tf.clip_by_value(image

Original: + gaussian, 0, 1) # create mask and apply patch image =
Revised: + gaussian, 0, 1) # create mask and apply patch image =

Original: tf.where(_get_patch_mask(patch_size), image_plus_gaussian,image)
Revised: tf.where(_get_patch_mask(patch_size), image_plus_gaussian,image)

Original: # scale back to [0, 255] return image *
Revised: # scale back to [0, 255] return image *

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: P ATCH G AUSSIAN IMPROVES PERFORMANCE IN OBJECT DETECTION
Revised: P ATCH G AUSSIAN IMPROVES PERFORMANCE IN OBJECT DETECTION

Original: Since Patch Gaussian can be combined with both regularization strategies as well as data augmentation policies, we want to see if it is generally useful beyond classiﬁcation tasks.
Revised: Since Patch Gaussian can be combined with both regularization strategies as well as data augmentation policies, we want to see if it is generally useful beyond classiﬁcation tasks.

Original: We train a RetinaNet detector (Lin et al., 2017) with ResNet-50 backbone (He et al., 2016) on the COCO dataset (Lin et al., 2014).
Revised: We train a RetinaNet detector (Lin et al., 2017) with ResNet-50 backbone (He et al., 2016) on the COCO dataset (Lin et al., 2014).

Original: Images for both baseline and Patch Gaussian models are horizontally ﬂipped half of the time, after being resized to 640 × 640 .
Revised: Images for both baseline and Patch Gaussian models are horizontally ﬂipped half of the time, after being resized to 640 × 640 .

Original: We train both models for 150 epochs using a learning rate of 0.08 and a weight decay of 1 × 10 − 4 .
Revised: We train both models for 150 epochs using a learning rate of 0.08 and a weight decay of 1 × 10 − 4 .

Original: The focal loss parameters are set to be α = 0 .
Revised: The focal loss parameters are set to be α = 0 .

Original: 25 and γ = 1 . 5 .
Revised: 25 and γ = 1 . 5 .

Original: Despite being designed for classiﬁcation, Patch Gaussian improves detection performance according to all metrics when tested on the clean COCO validation set (Table 9).
Revised: Despite being designed for classiﬁcation, Patch Gaussian improves detection performance according to all metrics when tested on the clean COCO validation set (Table 9).

Original: On the primary COCO metric mean average precision (mAP), the model trained with Patch Gaussian achieves a 1% higher accuracy over the baseline, whereas the model trained with Gaussian suffers a 2.9% loss
Revised: On the primary COCO metric mean average precision (mAP), the model trained with Patch Gaussian achieves a 1% higher accuracy over the baseline, whereas the model trained with Gaussian suffers a 2.9% loss

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 11.6 Gaussian ( σ ≤ 1 . 0 ) 9.
Revised: 11.6 Gaussian ( σ ≤ 1 . 0 ) 9.

Original: 26.1 Patch Gaussian ( W = 200 ,σ ≤ 1 . 0 ) 10.
Revised: 26.1 Patch Gaussian ( W = 200 ,σ ≤ 1 . 0 ) 10.

Original: Next, we evaluate these models on the validation set corrupted by i.i.d.
Revised: Next, we evaluate these models on the validation set corrupted by i.i.d.

Original: Gaussian noise, with σ = 0 .
Revised: Gaussian noise, with σ = 0 .

Original: We ﬁnd that model trained with Gaussian and Patch Gaussian achieve the highest mAP of 26.1% on the corrupted data, whereas the baseline achieves 11.6%.
Revised: We ﬁnd that model trained with Gaussian and Patch Gaussian achieve the highest mAP of 26.1% on the corrupted data, whereas the baseline achieves 11.6%.

Original: It is interesting to note that Patch
Revised: It is interesting to note that Patch

Original: Gaussian model achieves a better result on the harder metrics of small object detection and stricter intersection over union (IoU) thresholds, whereas the Gaussian model achieves a better result on the easier tasks of large object detection and less strict IOU threshold metric.
Revised: Gaussian model achieves a better result on the harder metrics of small object detection and stricter intersection over union (IoU) thresholds, whereas the Gaussian model achieves a better result on the easier tasks of large object detection and less strict IOU threshold metric.

Original: Overall, as was observed for the classiﬁcation tasks, training object detection models with Patch
Revised: Overall, as was observed for the classiﬁcation tasks, training object detection models with Patch

Original: Gaussian leads to signiﬁcantly more robust models without sacriﬁcing clean accuracy.
Revised: Gaussian leads to signiﬁcantly more robust models without sacriﬁcing clean accuracy.

Original: F OURIER ANALYSIS
Revised: F OURIER ANALYSIS

Original: Fig.
Revised: Fig.

Original: 10 shows a fourier analysis of selected models reported.
Revised: 10 shows a fourier analysis of selected models reported.

Original: Fig.
Revised: Fig.

Original: 11 shows complete ﬁlters for
Revised: 11 shows complete ﬁlters for

Original: ResNet-50 models.
Revised: ResNet-50 models.

Original: Fig.
Revised: Fig.

Original: 12 shows high-pass ﬁlters used in high-pass experiment in Fig.
Revised: 12 shows high-pass ﬁlters used in high-pass experiment in Fig.

Original: [Figure]
Revised: [Figure]

Original: Figure 4 for
Revised: Figure 4 for

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: N OTE ON PGD C OMMON C ORRUPTION ROBUSTNESS
Revised: N OTE ON PGD C OMMON C ORRUPTION ROBUSTNESS

Original: Ford et al.
Revised: Ford et al.

Original: (2019) reports that PGD training helps with corruption robustness.
Revised: (2019) reports that PGD training helps with corruption robustness.

Original: However, they fail to report mCE values for their models.
Revised: However, they fail to report mCE values for their models.

Original: We ﬁnd that, indeed, PGD helps with some corruptions, and when all corruption severities’ errors are averaged, it mostly maintains performance (23.8% error, compared to baseline error of 23.51%).
Revised: We ﬁnd that, indeed, PGD helps with some corruptions, and when all corruption severities’ errors are averaged, it mostly maintains performance (23.8% error, compared to baseline error of 23.51%).

Original: However, as table 4 shows, when we properly calculate mCE by normalizing with a baseline model, PGD displays much worse robustness, while Patch PGD improves performance.
Revised: However, as table 4 shows, when we properly calculate mCE by normalizing with a baseline model, PGD displays much worse robustness, while Patch PGD improves performance.

Revised: A BLATION STUDY OF FREQUENCY - BASED ANALYSIS

Revised: Figure 13 shows the frequency-based analysis (Yin et al., 2019) for models with different hyperparameters of Patch Gaussian .

Original: {+First, for hyper-parameters W = 16 , σ = 1 . 
+}
Revised: {+First, for hyper-parameters W = 16 , σ = 1 . 
+}

Revised: (center), the reader will note that these are very similar to the frequency sensitivity reported in Figure 4.

Revised: The main difference being that the smaller patch size (16 vs 25 in Figure 4) makes the model slightly more sensitive to high frequencies.

Revised: This makes sense since smaller patch size moves the model further away from a Gaussian -trained one.

Revised: When we make the scale smaller ( W = 16 , σ = 0 . 3 , left), less information is corrupted in the patch, which moves the model farther from the one trained with Cutout (and therefore closer to a Gaussiantrained one).

Revised: This can be seen in the increased invariance to high frequencies at the ﬁrst layer, which is reﬂected in invariance at test error as well.

Revised: If we, instead, make the scale larger( W = 16 , σ = 2 . 0 , right), we move the model closer to the one trained with Cutout .

Revised: Notice the higher intensity red in the ﬁrst layer plot, indicating higher sensitivity to high-frequency features.

Revised: We also see this sensitivity reﬂected in the test error, which matches the behavior for Cutout -trained models.

Revised: [Figure]

Revised: W =

Revised: [Figure]

Revised: σ =

Revised: [Figure]

Revised: [Figure]

Revised: W = 16 , σ = 1 .

Revised: Patch Gaussian

Revised: W = 16 , σ = 2 .1 s t

Revised: L a y e r

Revised: F ou r i e r

Revised: S e n s iti v it y

Revised: T e s t

Revised: E rr o r

Revised: F ou r i e r

Revised: S e n s iti v it y