Robustmix: Improving Robustness by Regularizing the Frequency Bias of Deep Nets
Abstract
Deep networks have achieved impressive results on a range of well curated benchmark datasets.
Surprisingly, their performance remains sensitive to perturbations that have little effect on human performance.
We propose a novel extension of Mixup called Robustmix that regularizes networks to classify based on lower frequency spatial features.
We show that this type of regularization improves robustness on a range of benchmarks such as Imagenet-C and Stylized Imagenet.
It adds little computational overhead and furthermore does not require a priori knowledge of a large set of image transformations.
We ﬁnd that this approach further complements recent advances in model architecture and data augmentation attaining a state-of-the-art mean corruption error (mCE) of 44.8 with an EfﬁcientNet-B8 model and RandAugment, which is a reduction of 16 mCE compared to the baseline.
Introduction
Deep neural networks have achieved state-of-the-art accuracy across a range of benchmark tasks such as image segmentation (Ren et al., 2015) and speech recognition (Hannun et al., 2014).
However, their performance can suffer greatly in the presence of small data corruptions (Szegedy et al., 2014; Goodfellow et al., 2014; Moosavi-Dezfooli et al., 2017; Athalye et al., 2018;
Hendrycks & Dietterich, 2018).
Hendrycks & Dietterich (2018) show that the accuracy of a standard model on Imagenet can drop from 76% to 20% when evaluated on images corrupted with small visual transformations.
Secondly, Szegedy et al.
(2014) show the existence of adversarial perturbations which are imperceptible to humans but have a disproportionate effect on the predictions of a network.
Numerous proposals were made to improve the robustness of deep networks, including Hendrycks et al.
(2019) which requires a priori knowledge of the visual transformations in the test domain.
Others, such as Geirhos et al.
(2018) use a deep network to generate transformations which comes with signiﬁcant computation cost.
This paper proposes a new technique to improve the robustness of deep networks by regularizing their frequency bias.
Its advantages are that: (1) it does not require knowledge of a large set of priori transformations, (2) it is inexpensive and (3) it doesn’t have many hyper-parameters.
The key idea is to bias the network to rely more on lower spatial frequencies to make predictions.
[Figure]
Related Work
The proposed approach can be seen as a generalization of Mixup (Zhang et al., 2018), a data augmentation method that regularizes models by training them on linear interpolations of two input examples and their respective labels.
Zhang et al.
(2018) show that Mixup improves the accuracy of networks and can also improve the robustness of the network.
In the past years, several versions of Mixup were proposed with application in Fairness (Chuang & Mroueh, 2021), 3D reconstruction (Cheng et al., 2022), semisupervised learning (Beckham et al., 2019), as well as robustness ((Mai et al., 2021;
Yun et al., 2019;
Faramarzi et al., 2020;
Kim et al., 2020; Verma et al., 2019)).
The novel version we propose here is frequency-based and does not include additional learnable parameters.
Augmix (Hendrycks et al., 2019) is a data augmentation technique to improve robustness by training on a mix of known image transformations.
It adds little computational overhead, but requires knowledge of a diverse set of domain speciﬁc transformations.
Hendrycks et al.
(2019) mixes a set of 9 different augmentations to reach 68 .
4 mCE on Imagenet.
In contrast, the proposed method does not rely on speciﬁc image augmentations but on the more general principle that natural images are a kind of signal where most of the energy is concentrated in the lower frequencies.
The idea of frequency ﬁltering is popular in Deep learning frameworks and has numerous applications including unsupervised domain adaptation (Yang & Soatto (2020)) and adversarial perturbation attacks (Guo et al.
; Li et al.
Unlike the latter papers which focus on measuring the accuracy of a model after an adversarial attack, we focus on common (noise) corruptions by measuring mCE as robustness assessment.
Zhang (2019) uses low pass ﬁlters directly inside the model to improve the frequency response of the network.
Wang et al.
(2019) uses a differentiable neural network to extract textual information from images without modelling the lower-frequency.
Our method also makes use of low-pass ﬁltering but does not completely remove high frequency features.
Additionally, we only use frequency ﬁltering during training and therefore no computational overhead is incurred during evaluation.
Method
Motivation Wang et al.
(2020) suggest that convolutional networks trade robustness for accuracy in their use of high frequency image features.
Such features can be perturbed in ways that change the prediction of the model even though humans cannot perceive the change.
This can lead models to make puzzling mistakes such as with adversarial examples.
Our aim is to increase robustness while retaining accuracy by regularizing how high frequency information is used by the model.
Robustmix We propose to regularize the sensitivity of the model to each frequency band by extending
Mixup’s linear interpolations with a new type of band interpolation.
The key insight is that we can condition the sensitivity to each band using images that mix the frequency bands of two different images.
Suppose that we mix the lower frequency band of an image of a boathouse with the high frequency band of an image of a dog.
We can encourage sensitivity to the lower band by training the model to predict dog for this mixed image.
However,this approach is too simplistic because it completely disregards the impact of the image in the high band.
Indeed, the ablation study at section
A.1 shows us that it is not sufﬁcient.
Instead, we interpolate the label of such mixed images according to an estimate of the importance of each frequency band.
We propose to use the relative amount of energy in each band as an estimate of the importance.
Thus the sensitivity of the model to high frequency features will be proportional to their energy contribution in natural images.
And as we can see in Figure 2a, most of the spectral energy in natural images is concentrated in the lower end of the spectrum.
This should limit the ability of high frequency perturbations to unilaterally change the prediction.
Furthermore, we use linear interpolations of images like in mixup within each band instead of raw images.
This closely reﬂects the more common case where the features in the bands are merely corrupted instead of entirely swapped.
It also has the beneﬁt of encouraging linearity inside the same frequency band.
Speciﬁcally, the mixing formula for Robustmix is given by
[Equation]
where λ L , λ H ∼ Beta( α, α ) , α is the Mixup coefﬁcient hyper-parameter, and Low ( · , c ) , High ( · , c ) are a low pass and high pass ﬁlter respectively with a uniformly sampled cutoff frequency c ∈ [0 , 1] .
And λ c is the coefﬁcient that determines how much weight is given to the lower frequency band.
It is given by the relative amount of energy in the lower frequency band for natural images
[Equation]
Results 4.1 Datasets and Metrics
The results presented in this paper rely on the mCE measurement on ImageNet-C, the clean accuracies on ImageNet and Stylized-ImageNet (SIN) as well as the shape bias on SIN.
These measurements are found in a range of paper studying robustness (Hendrycks & Dietterich, 2018;
Hendrycks et al., 2019; Geirhos et al., 2018; Laugros et al., 2020).
ImageNet.
We evaluate the common classiﬁcation accuracy on ImageNet (Deng et al., 2009) which will be referred to as clean accuracy.
We use the standard Resnet preprocessing resulting in images of size 224x
(He et al., 2015).
The standard models, without any additional data augmentation process, will be qualiﬁed as the baseline.
ImageNet-C.
This dataset is made of 15 types of corruption drawn from four main categories: noise, blur, weather and digital (Hendrycks & Dietterich, 2018).
These corruptions are applied to the validation images of ImageNet at 5 different intensities or levels of severity.
Following (Hendrycks &
Dietterich, 2018), we evaluate the robustness of our method by reporting its mean corruption error (mCE) normalized with respect to AlexNet errors.
Stylized-ImageNet.
Stylized-ImageNet (SIN) is constructed from ImageNet by replacing the texture in the original image using style transfer, such that the texture gives a misleading cue about the image label (Geirhos et al., 2018).
The 1000 classes from ImageNet are reduced to 16 shape categories, for instance all labels for dog species are grouped under one dog label, same for chair, car, etc.
There are 1280 generated cue conﬂict images (80 per category).
With SIN, we evaluate the classiﬁcation accuracy (SIN accuracy) and measure the model’s shape bias.
4.2 Experimental Setup
We chose to do evaluations on residual nets (ResNet-50 and ResNet-152) and EfﬁcientNets (EfﬁcientNet-B0, EfﬁcientNet-B1, EfﬁcientNet-B5 and EfﬁcientNet-B8).
Experiments were run [-on8x
-]{+on 8x
+}
TPUv3 instances for the the bigger EfﬁcientNets (EfﬁcientNet-B5 and EfﬁcientNet-B8); and the other experiments were run on 4x4 TPUv3 slices.
For the Resnet models, we use the same standard training setup outlined in Goyal et al.
However, we use cosine learning rate Loshchilov &
Hutter (2016) with a single cycle for Resnets that are trained for 600 epochs.
4.3 Robustness Results
Imagenet-C
In Table 1, we can see that Robustmix consistently improves robustness to the considered transformations, with a 15 point decrease in mCE over the baseline for ResNet-50.
Robustmix gives the best trade-off between accuracy and robustness over mixup.
This can be better observed in Figure 3.
While it is not directly comparable to ViT-L/16 due to its use of 300 × more data, we see that
Efﬁcientnet-B8 with Robustmix and RandAugment has better robustness at 44 . 8 mCE.
It is also competitive with DeepAugment (Hendrycks et al., 2020) which requires training additional specialized image-to-image models on tasks such as super-resolution to produce augmented images.
By comparison, our approach does not rely on extra data or extra trained models.
[Table]
[Table]
(Hendrycks et al., 2020)
[Table]
(Dosovitskiy et al.
[Table]
[Table]
Stylized-ImageNet.
We conﬁrm that our method indeed increases both accuracy on Stylized ImageNet and the shape bias as shown in table 2.
For ResNet-50, Robustmix almost doubles the shape bias from baseline (from 19 to 37) and improves it by 63% over Mixup; while relative improvements on SIN accuracy are of 72% and 33% respectively over baseline and Mixup.
The same observation for EfﬁcientNet-B5 wich improves shape bias by near 50% and SIN accuracy by near 60 % over the baseline
[Table]
Conclusion
Our method offers a promising new research direction for robustness with a number of open challenges.
We have used a standard DCT based low-pass ﬁlter on images and L2 energy metric to determine the contribution of each label.
This leaves many alternatives to be explored, such as: different data
A Appendix A.1 Ablation Study
In order to measure the effect of Robustmix we apply some simpliﬁcations both with respect to the image mixing as well as the labelling.
The results are compiled in table 3.
It can be noticed from the ﬁrst two lines that ablating the energy weighting results in a signiﬁcant drop in mCE even though there is a slight accuracy improvement.
However, keeping the energy weighting but not applying the inband mixups is largely detrimental both to accuracy and robustness.
These results show that
Robustmix achieves a better combination of mCE and accuracy than these ablations
[Table]
A.2 Analysis and Discussion
Low frequency bias In order to quantify the degree to which models rely on lower frequencies, we measure how much accuracy drops as we remove higher frequency information with a low-pass ﬁlter.
Figure 2b shows that Robustmix is comparatively more robust to the removal of high frequencies.
This indicates that models trained with Robustmix rely signiﬁcantly less on these high-frequency features to make accurate predictions.
1.0 Low-pass cut-off 0.
E n e g y
1.0 Frequency Cut-offs 0.
A cc u r a c y
RobustmixMixupBaseline 0.
0.6 0.8 1.0 Frequency Cut-offs 0.
A cc u r a c y
RobustmixMixupBaseline
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
A.3 Extras
Implementation The frequency separation is implemented using a Discrete Cosine Transform (DCT) to avoid the complex multiplication required by a Discrete Fourier Transform.
Additionally, we must apply the DCT transform over the x and y dimension separately.
Thus, 6 DCT matrix multiplications are required which results in 0 .
2 GFLOPs per image.
In contrast, just the forward pass of ResNet requires 3 .
87 GFLOPs (Hasanpour et al., 2016).
In our implementation of Robustmix, we reorder commutative operations (low pass and mixing) in order to compute the DCT only a single time per minibatch.
The pseudocode is provided in Algorithm 1, where reverse is a function that reverses the rows of its input matrix.
[Table]
Choice of hyperparameter α
In our cross-validation of α , we found small values less than 0 .
2 perform poorly both on accuracy and mCE.
Values of α such that 0 .
2 ≤ α ≤ 0 .
5 not only give the best accuracies and mCEs but also the best trade-off of mCE versus accuracy as bigger values of α have shown giving good values for accuracy but do not do as well on mCE.
In our experiments, we found that we typically achieve good results with a frequency cutoff c sampled between [0 , 1] as described in Algorithm 1.
However,
For ResNet-50 trained with a training budget that is too limited (200 instead of 600 epochs) and its smaller versions (ResNet-18 and ResNet-34), it can be beneﬁcial to ﬁx a minimum c ≥ τ for the cutoff by sampling in the interval [ τ, 1] .
The minimum cutoff determines the range at which band mixing will occur.
[Figure]
[Table]
Hendrycks
[Table]