F OURIER S ENSITIVITY AND R EGULARIZATION OF C OMPUTER V ISION
Anonymous authors
Paper under double-blind review
A BSTRACT
Deep neural networks for computer vision are highly accurate but their out-ofdistribution generalization performance remains a major challenge,
so understanding their failure modes is an important step towards improving their robustness.
Intriguingly, recent work has shown that deep neural networks latch on to the Fourier statistics of training data, show increased sensitivity to certain Fourierbasis directions in the input and are not robust when Fourier-statistics shift unfavorably at test time.
Understanding and modifying this Fourier-sensitivity of computer vision models may help improve their robustness, hence, in this paper we study the frequency sensitivity characteristics of deep neural networks using a principled approach.
We ﬁrst propose a general measure of a computer vision model’s spatial frequency sensitivity based on its input-Jacobian represented in the Fourier-basis.
When applied to deep neural networks, we ﬁnd that standard minibatch training consistently leads to increased sensitivity towards particular spatial frequencies independent of network architecture.
We further propose a family of spatial frequency regularizers based on our proposed measure to induce speciﬁc spatial frequency sensitivities in a model.
In various generalization tests involving shifts in the Fourier-statistics of the data, we ﬁnd that deep neural networks trained with our proposed regularizers obtain signiﬁcantly improved classiﬁcation accuracy while maintaining high accuracy on in-distribution test images.
I NTRODUCTION
While deep neural networks (DNN) achieve remarkable performance on many challenging image classiﬁcation tasks, they can suffer signiﬁcant drops in performance when evaluated on out-ofdistribution (o.o.d.) data.
Intriguingly, this lack of robustness has been partially attributed to the frequency characteristics of data shifts at test time in relation to the frequency sensitivity characteristics of the model (Yin et al., 2019; Jo & Bengio, 2017).
Distinct spatial frequencies in images contain features at different spatial scales; low spatial frequencies (LSF) carry global structure and shape information in a scene whereas high spatial frequencies (HSF) carry local information such as edges and borders of objects in a scene (Kauffmann et al., 2014); in fact, spatial frequencies are differentially processed in distinct channels of the visual cortex in the brain to learn features at different scales (Appendix B).
When information is destroyed or corrupted in frequency bands that a model relies on, performance suffers.
Hence, understanding the spatial frequency sensitivity of a DNN can help us characterise the features it relies on to make predictions.
DNNs have been demonstrated to be sensitive to Fourier-basis directions in the input (Tsuzuku & Sato, 2019; Yin et al., 2019) both empirically and using theoretical analysis of linear convolutional networks (Tsuzuku & Sato, 2019). In fact, the existence of so-called “universal adversarial perturbations” (Moosavi-Dezfooli et al., 2017), simple semantics-preserving distortions that can degrade models’ accuracy across inputs and architectures, is attributed to this structural sensitivity and the use of convolution operations. Yin et al.
(2019) also showed that many natural and digital image corruptions that degrade model performance may also be targeting this vulnerability.
The Fouriercharacteristics of adversarial examples are also known to closely match the Fourier-sensitivity of models (Yin et al., 2019). Hence, understanding and modifying Fourier-sensitivity can aid efforts to improve model robustness. While this Fourier-sensitivity has been studied empirically, the precise deﬁnition and measurement of a computer vision model’s spatial frequency sensitivity still lacks a rigorous approach across studies. In addition, there has been no principled method to speciﬁcally modify the spatial frequency sensitivity of a model.
Existing works have heuristically applied ﬁlters on convolution layer parameters (Wang et al., 2020; Saikia et al., 2021) and data augmentations (Yin et al., 2019) to modify a model’s frequency sensitivity.
In this work, we propose a novel and rigorous measure of a deep neural network’s spatial frequency sensitivity using the input-Jacobian represented in the Fourier-basis and show that deep neural networks demonstrate consistent spatial frequency sensitivities across samples, an observation that suggests DNNs are more likely to consistently use some frequencies more than others, and has implications for robustness.
In addition, using our proposed measure, which is differentiable with respect to model parameters, we propose a novel family of spatial frequency regularizers to directly induce speciﬁc frequency sensitivities in a model.
We hypothesize and show in empirical evaluations that spatial frequency regularization can modify the frequency sensitivity characteristics of computer vision models and can signiﬁcantly improve the generalization performance of models on o.o.d. datasets where the Fourier-statistics are unfavorably shifted.
In summary, the main contributions of this work are as follows:
We propose a novel and rigorous measure of a model’s spatial frequency sensitivity based on its input-Jacobian represented in the Fourier-basis 2.
We propose a novel family of spatial frequency regularizers to directly induce speciﬁc spatial frequency sensitivities 3.
We demonstrate that spatial frequency regularization can signiﬁcantly improve generalization performance on out-of-distribution data where Fourier-statistics are unfavorably shifted
R ELATED W ORK
Characterising frequency sensitivity: Yin et al.
; Tsuzuku & Sato (2019) characterised the
Fourier characteristics of trained CNNs using perturbation analysis of their test error under Fourierbasis noise.
They showed that a naturally trained model is most sensitive to all but low frequencies whereas models adversarially trained (Madry et al., 2018) models are sensitive to low-frequency noise.
They further showed that these Fourier characteristics relate to model robustness on corruptions and noise, with models biased towards low frequencies performing better under high frequency noise and vice versa.
Abello et al.
(2021) took a different approach by measuring the impact on accuracy of removing individual frequency components from the input using ﬁlters whereas OrtizJimenez et al.
(2020) computed the margin in input space along basis directions of the discrete cosine transform (DCT).
Wang et al.
(2020) made observations about the Fourier characteristics of CNNs in different training regimes including standard and adversarial training by evaluating accuracy on band-pass ﬁltered data.
In contrast to these disparate approaches, in this work, we propose a rigorous measure of a model’s spatial frequency sensitivity .
Regularizing frequency sensitivity: Yin et al.
(2019) proposed adversarial training (Madry et al., 2018) and gaussian noise augmentations as methods that induce a low-frequency sensitivity.
Wang et al.
(2020) proposed smoothing convolution ﬁlter parameters to induce a low-frequency sensitivity in models.
We note that, in general, techniques that apply ﬁlters on convolutional parameters to affect a model’s frequency sensitivity do not take into account complex operations such as nonlinearities, pooling and other transformations that often follow convolutional layers that can modify as well as undo the effects of such ﬁlters.
In addition, data augmentations do not provide precise control over the Fourier-sensitivity of a model.
In this work, we propose a family of spatial frequency regularizers that can precisely modify the overall spatial frequency sensitivity of any differentiable model.
Jacobian regularization: Methods that regularize the Jacobian of a model’s output-logits or loss with respect to its input can broadly be divided into two types; one regularizes the norm of the input-Jacobian and the other regularizes its direction or directional derivatives.
Drucker & Le Cun (1991) proposed a method that penalized the norm of the input-Jacobian to improve generalization, more recently this has been explored to improve robustness to adversarial perturbations (Ross & Doshi-Velez, 2018; Jakubovitz & Giryes Hoffman et al.
Simard et al.
(1992) proposed ”Tangent Prop”, which minimized directional derivatives of classiﬁers in the direction of local inputtransformations (e.g. rotations, translations; called ”tangent vectors”) to reduce model sensitivity to such transformations.
Czarnecki et al.
(2017) proposed Sobolev training of neural networks to improve model distillation by matching the input-Jacobian of the original model.
Regularizing the direction of the input-Jacobian has also been used to improve adversarial robustness (Chan et al., 2020).
In the present work, we regularize Fourier-components in the input-Jacobian to modify the spatial frequency sensitivity of models.
As such, we are directly modifying the direction of the input-Jacobian instead of its norm.
M ETHODS
Preliminaries: We introduce all relevant deﬁnitions and notations before describing the proposed methods.
Consider an image classiﬁcation task with input images x , target labels y , and the standard cross-entropy loss function L CE .
Let f denote any differentiable model, F ( · ) the unitary 2D discrete Fourier transform (DFT), F − 1 ( · ) its inverse, and F − 1 ∗ ( · ) the adjoint of the inverse-Fourier transform, and let x f denote the Fourier space representation of the input, i.e. x f = F ( x ) .
We denote the input-Jacobian in the standard input basis as ∂ L CE ∂x , and ∂ L CE ∂x f as the input-Jacobian in the Fourier-basis.
Let N denote the height of the input images x (although not necessary, all images used in this work are square).
The zero-shifted 2D-DFT of the input-Jacobian is denoted F = F ( ∂ L CE ∂x ) .
Since the input-Jacobian typically has three color channels, they are averaged before computing the 2D-DFT.
Fourier coefﬁcients in F are complex numbers with real and imaginary components;
F ( u, v ) =
Real ( u, v ) + Imag ( u, v ) , where ( u, v ) are indices of coefﬁcients.
The power in a coefﬁcient is its squared amplitude, P ( u, v ) =
| F ( u, v ) | 2 =
Real ( u, v )
+ Imag ( u, v ) 2 and the matrix of powers is denoted P (power-matrix).
Each coefﬁcient has a radial distance r ( u, v ) from the centre of the matrix, r ( u, v ) =
d (( u, v ) , ( c u , c v )) , where ( c u , c v ) denotes the centroid of P and d ( · , · ) is Euclidean distance.
Distinct radial distances, rounded to the nearest integer, of coefﬁcients in the matrix are the set of integers { 1 , . .
. , N/ √ 2 } and correspond to low to high spatial frequency bands, the highest spatial frequency being limited by the Nyquist frequency.
We denote P Total as the total power in P , excluding the zero-frequency coefﬁcient, P Total = (cid:80) r
( u,v ) >
=1 P ( u, v ) .
Similarly, we deﬁne ˜
P Total as the total power in P excluding the zero-frequency coefﬁcient as well as coefﬁcients with radial distance r ( u, v ) > N/ 2 , i.e. outside the largest circle inscribed in the power-matrix P ; ˜ P Total = (cid:80) 1 < = r ( u,v ) <
= N/
P ( u, v ) (please see Figure 5 in Appendix A. for an illustration).
We denote P k as the total power at radial distance k normalized by P Total ,
P k = 1 P Total (cid:80) r ( u,v )=
k
P ( u, v ) and ˜ P k as total power at radial distance k normalized by ˜ P Total instead, ˜ P k = 1 ˜ P Total (cid:80) r ( u,v )=
k
P ( u, v ) .
3. S PATIAL F REQUENCY S ENSITIVITY OF A MODEL
In this section, we deﬁne the proposed spatial frequency sensitivity (SFS) of any differentiable model using its input-Jacobian represented in the Fourier-basis.
As the input-Jacobian provides the direction of highest input sensitivity, we show below that its Fourier transform, which is simply a change of basis, provides the model’s sensitivities to spatial frequency components in the input.
The input-Jacobian in the Fourier-basis, ∂ L CE ∂x f , comprises the sensitivities of the model with respect to individual frequencies in the input and we obtain this by simply computing the Fourier transform of the input-Jacobian ∂ L CE ∂x .
In order to justify this, consider the computation graph where the input x is mapped to a scalar loss via a model f and loss function L CE (Figure 2).
We introduce an implicit operation (shown in red) that maps the Fourier space representation of the input, x f , to the standard input x through the inverse Fourier-transform; x f F − 1 −−−→ x , so in order to compute the input-Jacobian in the Fourier-basis, ∂ L CE ∂x f , we must differentiate through this implicit operation in the forward graph.
Since the inverse-Fourier transform is a unitary operator, its adjoint is also its inverse, i.e. F − 1 ∗ =
( F − 1 ) − 1 = F .
Hence, following the chain rule for complex operators, ∂ L CE ∂x f is simply the Fourier transform of the input-Jacobian ∂ L CE ∂x .
[Figure] [Figure]
[Equation]
Hence, even though we do not explicitly use the Fourier representation of the input, this shows that the Fourier transform of the input-Jacobian provides us the sensitivity of the model with respect to spatial frequencies of the input.
In fact, similar results hold for any unitary operation on the inputJacobian, such as the discrete cosine transform (DCT).
Interested readers can replace the DFT ( F ) with other such transforms if they would like to understand the sensitivity of a model with respect to other components in the input.
We now deﬁne the spatial frequency sensitivity f SFS ( x, y ) with respect to an individual input ( x, y ) as,
[Equation]
where P k is the proportion of total power in Fourier coefﬁcients at radial distance k in the power matrix P of the Fourier-transformed input-Jacobian.
The overall spatial frequency sensitivity of a model is deﬁned as the expectation of f SFS ( x, y ) over the data distribution, i.e. f SFS ( · ; θ )
=
E ( x,y ) ∼ p
[ f SFS ( x, y )]
(please see Figure 1 for an illustration and Algorithm 1 in Appendix A.1).
We note that although we use the Jacobian of the loss function to estimate the SFS of a model, this formulation is valid for other output functions and tasks as well.
An alternative approach that does not require image labels is to compute the Jacobian of the model’s output logits or softmax probabilities.
In addition, while the spatial frequency sensitivity assumes the input is spatial 2D data (e.g. images), we can extend this approach to any n-dimensional data by using the n-dimensional
Fourier-transform as well as to other computer vision tasks.
3. S PATIAL F REQUENCY R EGULARIZATION
In this section, we propose a novel family of spatial frequency regularizers that can modify the spatial frequency sensitivity , as deﬁned in Section 3.1, of a model directly.
As all computations involved in the SFS of a model (Algorithm 1 in Appendix A.1) are differentiable with respect to its parameters, we compute a loss function, L ( x, y )
=
L CE ( x, y ) + λ SFS L SFS ( x, y ) , where L SFS is the proposed loss that can be used to induce a speciﬁc SFS and λ SFS is a hyperparameter.
As L SFS is deﬁned on the SFS , which is a function of the input-Jacobian, optimizing it requires an additional backpropagation step to compute its derivatives with respect to model parameters, similar to other Jacobian-regularization methods.
We now deﬁne L SFS for three different SFS regularizers; SFS ∈ { LSF, MSF, ASF } .
LSF regularization trains a model to be insensitive to medium and high spatial frequencies and MSF regularization trains a model to be insensitive to low and high spatial frequencies.
We achieve this by penalizing the proportion of total power, P k , in frequency bands.
ASF regularization trains a model to be equally sensitive to all spatial frequency (ASF) bands.
The motivation behind ASF regularization model is to encourage a model to be sensitive to multiple frequency bands instead of being concentrated in a single frequency range.
Hence, the ASF-regularizer loss is the negative entropy of the distribution of power over spatial frequency bands.
[Equation]
The deﬁnitions of frequency ranges are based on equally dividing the largest circle inscribed in the power-matrix P into equal parts (Figure 5 in Appendix A.1).
For ASF-regularization, very high frequency bands ( r ( u, v ) >
N/ 2 are excluded.
Interested readers may modify these loss functions as they see ﬁt for their particular application.
E XPERIMENTS
In this section we explore the SFS of models as well as explore the effects of spatial frequency regularization by evaluating on datasets where the Fourier-statistics of predictive features are naturally or artiﬁcially altered (Sections 4.2, 4.4).
We also make some observations about the differences between spatial frequency regularization and training on Fourier-ﬁltered training data (Section D.3).
We benchmark against methods that have been proposed to modify the frequency sensitivity of models such as adversarial training and Gaussian Noise augmentation (Yin et al., 2019) to induce low-frequency sensitivity.
AugMix (Hendrycks et al., 2020) is also benchmarked as it was shown to achieve state-of-the-art performance on common corruptions.
We also trained models on band-pass ﬁltered datasets to demonstrate that SFS regularization is not equivalent to training on Fourier-ﬁltered data (discussion relegated to Appendix D.3). Experimental Setup:
We ﬁrst describe the experimental settings for all experiments that follow.
We trained the ResNet50 architecture, unless stated otherwise. For CIFAR10 and CIFAR (Krizhevsky & Hinton, 2009), we trained all models for 150 epochs using stochastic gradient descent (SGD) with momentum (0.9), an initial learning rate of 0.1 decayed by a factor of 10 every 50 epochs, weight decay parameter equal to 5e-4 and batch size equal to 128. For SVHN (Netzer et al., 2011), we trained models for 40 epochs using Nesterov momentum with an initial learning rate of 0.01 and momentum parameter 0.9.
The training batch size was 128, the L2 regularization parameter was 0.0005 and we decayed the learning rate at epochs 15 and 30 by dividing by 10.
Standard data augmentations random-crop, random-horizontal-ﬂip, random-rotation, and color-jitter were used during training.
We trained spatial frequency regularized models using Algorithm 2 2 in Appendix A.1, and λ SFS = 1 for all models.
We observed a simple trade-off between this hyperparameter and clean accuracy (higher values decreased clean accuracy more).
Values less than 1 did not always achieve the desired SFS in the evaluated datasets.
For adversarial training (AT), we used PGD (cid:96) 2 attacks ( (cid:15) = 1 , attack-steps = 7 attack-lr = 1/7 ).
To train models with Gaussian noise data augmentation, we added i.i.d.
Gaussian noise N (0 , σ 2 ) to each pixel in all the images in the training set ( σ = 0 . 1 ).
We used the robustness (Engstrom et al., 2019) library to train models as well as retrieve pre-trained ImageNet models of various architectures.
We trained AugMix models using the original code shared by the authors ( https://github.com/google-research/augmix ).
Please see Section D.2 in Appendix D for details about band-pass ﬁlters. [Figure]
4. CNN S DISPLAY CONSISTENT SPATIAL FREQUENCY SENSITIVITY ACROSS SAMPLES
We now visualize the spatial frequency sensitivity (SFS) , as deﬁned in Section 3.1, of models trained on CIFAR10 and ImageNet (Figure 3).
We note that training biases models to be sensitive to some frequencies more than others and this bias is consistent across samples (the error bars in Figurerepresent deviation across samples).
This consistency also suggests that, once trained, CNNs are more likely to use some frequencies more than others and may explain their tendency to suffer performance drops when evaluated on data with different Fourier-statistics.
CIFAR100 models have very similar SFS to those of CIFAR10 (plots relegated to Appendix C.2) and are also most sensitive to mid-to-high spatial frequencies.
In contrast, standard training on SVHN leads to a low-frequency sensitivity, which suggests a strong dataset dependence of spatial frequency sensitivity.
Please refer to Appendix C.3 for SFS plots of models trained on SVHN.
ImageNet models are sensitive to a wide range of the frequency spectrum with peak sensitivity to mid-range frequencies.
We also observed consistency in the spatial frequency sensitivity of various convolutional architectures trained on ImageNet (Appendix C.1).
Adversarially trained models (Madry et al., 2018) are most sensitive to low spatial frequencies across datasets and architectures which suggests they rely on coarse global features, in agreement with observations made in prior work.
Training on Stylized-ImageNet, proposed by Geirhos et al.
(2019) to train shape-biased in models, induces a lower peak-spatial-frequency-sensitivity and as well as lower sensitivity to high frequencies compared to training on ImageNet (Appendix C.2), which reﬂects the increased shapebias of such models.
Similarly, a model trained on low-pass Fourier-ﬁltered training data correspondingly has high sensitivity to low-frequencies in comparison to the baseline model (Appendix
C.2).
Gaussian noise augmentation N (0 , σ 2 ; σ = 0 . 1) , decreases the high-frequency sensitivity of the model (Figure 3) compared to baseline.
The SFS of the AugMix (Hendrycks et al., 2020) trained model was not signiﬁcantly different from baseline, which suggests it ﬁnds more robust features in similar frequency bands using data augmentations.
Interestingly, we also note that models trained on corrupted versions of the CIFAR10 training set (e.g. blurred or JPEG-compressed) can display different spatial frequency sensitivities to a model trained on clean CIFAR10 images (please see Appendix C.4 for plots).
For example, models trained on images with severe noise corruptions (gaussian, shot and speckle) display increased sensitivity to lower spatial frequencies.
Models trained on highly gaussian-blurred, glass-blurred, JPEGcompressed and pixelated images also display higher sensitivity to lower spatial frequencies.
These changes reﬂect the shift in the Fourier-statistics of predictive features in corrupted images.
4. P ERFORMANCE UNDER F OURIER FILTERING
Jo & Bengio (2017) showed that DNNs have a tendency to rely on superﬁcial Fourier-statistics of their training data.
In the vein of generalization evaluations they performed, we generate Fourierﬁltered CIFAR10/CIFAR100 validation images using radial masking in frequency space (please see
Appendix D.1 for examples).
A mask radius r determines Fourier components that are preserved with larger radii preserving more components.
We use ( c u , c v ) to denote the centre of the mask and [Figure] Euclidean distance
[Equation]
The mask is applied on the zero-shifted output of the Fourier transform of each image, denoted X , followed by the inverse transform, i.e. X filtered =
F − 1 ( F ( X ) (cid:12) M r ) , where (cid:12) is the elementwise product.
Fourier-ﬁltering is performed on each color channel independently.
A standard CNN suffers up to a 75% drop in accuracy on low-pass ﬁltered CIFAR10 data as it relies on high frequency information that is no longer present (Table 1).
On the other hand, the LSF- REGULARIZED model performs well on both clean images as well severely low-pass ﬁltered images.
This shows that regularized CNNs are able to exploit very low frequency features ( r ≤ 5 ) not easily recognizable even to humans, which reﬂects the limited spatial-frequency-sensitivity range of the human eye.
[Table]
The MSF- REGULARIZED model performs better for mild Fourier-ﬁltering ( r = 11 and r = 7 ) but suffers a big drop in accuracy ( 71 . 5% to 46 .
2% in CIFAR10 and 46 . 4% to 18 . 6% in CIFAR100) for strong low-pass ﬁltering ( r = 5 ).
We also observed that the ASF- REGULARIZED model is also signiﬁcantly more robust than the baseline model although not as robust as the LSF- REGULARIZED model to Fourier-ﬁltering.
This shows that the ASF- REGULARIZED model can use coarse lowspatial-frequency features in addition to medium and high spatial frequencies.
The low-frequency sensitivity of adversarially trained models makes it more robust than the baseline model but less so than the LSF- REGULARIZED model [Table] while the Gaussian noise augmented model suffers large drops in accuracy.
AugMix tends to be robust to mild Fourier-ﬁltering but suffers large drops in accuracy under more severe ﬁltering ( r = 5 ) due to its reliance on mid-to-high-frequency information, as suggested by its SFS (Figure 3), in comparison to the LSF- REGULARIZED model. 4. P ERFORMANCE UNDER F OURIER - NOISE CORRUPTIONS
Recently, data-agnostic corruptions along Fourier-basis directions have been identiﬁed as a threat to model robustness and security (Yin et al., 2019; Tsuzuku & Sato, 2019).
A Fourier-noise corruption is additive noise containing a single Fourier-mode (frequency). These corruptions are semanticspreserving and signiﬁcantly affect model performance while being recognizable to humans, please refer to Appendix E for examples in each dataset. We generated corruptions for each of 1, Fourier-modes in SVHN/CIFAR10/CIFAR100 and report the overall mean accuracy across all corruptions in Table 2.
We note that the baseline model suffers signiﬁcant drops in accuracy across all datasets. The LSF-regularized model achieves the highest mean accuracy in SVHN while Gaussian augmentation also improves over the baseline. AugMix (Hendrycks et al., 2020), a dataaugmentation method proposed for robustness, performs worse than baseline. In CIFAR10 and
CIFAR100, the MSF-regularized model achieves the highest mean accuracies.
Detailed heat maps containing error rates for each Fourier-mode can be found in Appendix E.
[Table]
P ERFORMANCE UNDER CORRUPTIONS IN
(H
ENDRYCKS &
D
We evaluated the robustness of spatial frequency regularized models to corruptions (please see Appendix F for examples) in CIFAR10-C (Hendrycks & Dietterich, 2019).
In addition, we applied these corruptions to SVHN to create SVHN-C to evaluate the performance of methods across datasets.
Overall, SFS regularization provided consistent improvements across both SVHN-C (Table 3) and CIFAR10-C (Table 4) due to its general and principled approach. We observed that AugMix’s (Hendrycks et al., 2020) high performance on CIFAR10-C does not translate to high performance on SVHN-C. Similarly, Gaussian augmentation’s high performance in SVHN-C did not translate to CIFAR10-C. Speciﬁcally, on SVHN-C, LSF-regularization, low-pass ﬁltering and Gaussian augmentation performed very well under noise corruptions whereas AugMix did not improve over the baseline.
In CIFAR10-C, we observed that the LSF- REGULARIZED model was signiﬁcantly more robust than the baseline model to all blurring corruptions as well as pixelate, elastic, snow and frost corruptions.
The baseline model suffered up to 45% drop in accuracy on these corruptions, whereas the LSF- REGULARIZED model suffered at most 5% drop in accuracy.
This is likely due to the robustness of low-spatial-frequency features to corruptions such as blurring.
We also observed that the MSF- REGULARIZED and ASF- REGULARIZED models were more robust to noise corruptions as well as JPEG compression (Table 4), which suggests multiple spatial frequency bands maybe needed to do well under such corruptions. We observed similar results in CIFAR100-C (Appendix F.2).
Overall, SFS regularization can provide consistent improvements in robustness to these corruptions across datasets due to its general and principled approach.
On the other hand, data augmentation based methods invariably perform better on some datasets than others depending on the statistics and features of each dataset. [Figure]
[Table]
[Figure]
[Table]
C by applying corruptions in CIFAR10-C to SVHN samples.
[Table]
4. E VALUATION OF GLOBAL FEATURE LEARNING
Here we wish to evaluate the extent to which models use global features by measuring their performance on patch-shufﬂed images, which have previously been used by Mummadi et al.
Zhang & Zhu (2019); Wang et al.
Patch-shufﬂing involves splitting an image into k × k squares and randomly swapping the positions of these squares.
This is intended to destroy global features and retain local features with larger values of k retaining less global structure in the image (please see Appendix G for examples).
As such, models that rely more on global rather than local structure suffer more from patch-shufﬂing.
Hence, in this benchmark, lower accuracy suggests increased reliance on and use of global structure.
We observed that spatial frequency regularized models, adversarially trained models suffered larger drops in accuracy compared to the standard trained, AugMix and Gaussian augmentation models, which suggests their reliance on global structure in images contributes to their robustness (Table 5)
[Table]
C ONCLUSION
We proposed a novel and rigorous measure of the spatial frequency sensitivity (SFS) of any differentiable computer model using its input-Jacobian represented in the Fourier-basis.
We showed that models display consistent SFS across samples as well as across architectures for a given task and when there is a shift in the Fourier-statistics of features relative to a model’s frequency sensitivity, performance can suffer.
We also proposed a novel method of spatial frequency regularization that can be used to improve robustness to many test-time corruptions.
Extending our method to other computer vision tasks (e.g. detection) as well as domains ( n -dimensional) are interesting directions for future work.
A PPENDICES A S PATIAL F REQUENCY S ENSITIVITY AND R EGULARIZER A. P SEUDO - CODES
Input: Labeled samples L = { ( x i , y i ) } ni =1 ; a model f with trained parameters θ
Output: Estimated Spatial Frequency Sensitivity of model, f SFS ( · ; θ ) for i = 1 , . .
. , n do compute loss L CE ( f ( x i ) , y i ) //
forward pass backpropagate L CE to obtain ∂ L CE ∂x
i // input-jacobian ∂ L CE ∂x fi = F ( ∂ L CE ∂x i ) //
2D DFT of input-jacobian (averaged across color channels)
f SFS ( x i , y i ) =
[ P k ; for k=1 to N/ √ 2 ] // P k is proportion of total power (excluding DC) in coefficients at radial distance k end f SFS
[Equation]
estimated SFS of model
Input: Labeled samples L = { ( x i , y i ) } ni =1 ; a model f with parameters θ ; SFS ∈ { LSF, MSF, ASF } ; hyperparameter λ SFS
Output: Regularized model with updated parameters θ for i = 1 , . .
. , n do compute loss L CE ( f ( x i ) , y i ) //
forward pass
L ( x i , y i ) =
L CE ( x i , y i ) + λ SFS L SFS
( x i , y i ) //
backpropagate L to update θ end r=N/ r=N/ √
N
N r=N/ r=N/ (cu,cv)
P(u,v)
N
[Figure]
B S PATIAL FREQUENCY CHANNELS IN THE BRAIN
To further motivate the spatial frequency perspective of visual representation learning, we brieﬂy describe some relevant ﬁndings from neuroscience and vision research.
While the brain does not strictly perform a Fourier analysis of visual scenes, there has been mounting evidence over decades for multiple spatial frequency channels in the early human visual system that are physiologically independent and selectively responsive to distinct spatial frequency bands (Campbell & Robson, 1964; 1968).
It has been posited that the use of different spatial frequency channels are determined by the demands of a given visual task through an attention mechanism (Schyns & Oliva, 1999;
Rotshtein et al., 2010; Julesz & Papathomas, 1984).
Spatial frequency channels enable us to attend to different spatial scales in a scene at a ﬁxed viewing distance, similar to the focus lens in a camera (Figure 6).
On the other hand, once trained, a CNN’s spatial frequency sensitivity (Section 3.1) is inﬂexible and does not vary much across samples (both in and out of its training data distribution).
EFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEF
EFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFonononononononononononononoEFEFEFEFEFEFEFEFEFEFonononononononononononononoEFEFEFEFEFEFEFEFEFEFonononononononononononononoEFEFEFEFEFEFEFEFEFEFonononononononononononononoEFEFEFEFEFEFEFEFEFEFonononononononononononononoEFEFEFEFEFEFEFEFEFEFEFEFEFEonononononEFEFEFEFEFEFEFEFEEFEFEFEFEFEFEFEFEonononononEFEFEFEFEFEFEFEFEEFEFEFEFEFEFEFEFEonononononEFEFEFEFEFEFEFEFEEFEFEFEFEFEFEFEFEonononononEFEFEFEFEFEFEFEFEEFEFEFEFEFEFEFEFEonononononEFEFEFEFEFEFEFEFEEFEFEFEFEFEFEFEFEonononononEFEFEFEFEFEFEFEFEEFEFEFEFEFEFEFEFEonononononEFEFEFEFEFEFEFEFEEFEFEFEFEFEFEFEFEonononononEFEFEFEFEFEFEFEFEEFEFEFEFEFEFEFEFEonononononEFEFEFEFEFEFEFEFEEFEFEFEFEFEFEFEFEonononononEFEFEFEFEFEFEFEFEEFEFEFEFEFEFEFEFEonononononEFEFEFEFEFEFEFEFEEFEFEFEFEFEFEFEFEonononononEFEFEFEFEFEFEFEFEEFEFEFEFEFEFEFEFEonononononEFEFEFEFEFEFEFEFEEFEFEFEFEFEFEFEFEonononononEFEFEFEFEFEFEFEFEEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEF
EFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEFEF
C S UPPLEMENTARY P LOTS C. S PATIAL FREQUENCY S ENSITIVITIES OF POPULAR I MAGENET MODELS
[Figure]
C. S PATIAL FREQUENCY S ENSITIVITIES OF MODELS TRAINED ON CIFAR10 AND CIFAR
[Figure]
[Figure]
FREQUENCY
[Figure]
ENSITIVITIES
[Figure]
[Figure]
F OURIER F [Figure] r
[Figure]
unﬁ lt e r e d
D. B AND - PASS FILTERING
[Figure]
unﬁ lt e r e d
D. S PATIAL FREQUENCY REGULARIZATION IS NOT EQUIVALENT TO TRAINING ON F OURIER - FILTERED DATA
We note that spatial frequency regularization is not equivalent to standard training on corresponding
Fourier-ﬁltered data.
In natural images, the amount of energy in frequency bands falls off rapidly as frequency increases (Hyv¨arinen et al., 2009), hence, medium and high-pass ﬁltered natural images typically appear completely empty to the human eye without additional contrast maximisation. Even still, such images are not easily recognizable to the human eye due to its and reliance on low frequency information to recognize objects optimally (Appendix D.2). We observed that standard training of CNNs on medium or high-pass Fourier-ﬁltered data did not achieve high clean accuracies (Tables 4, 6). A model trained on medium-pass Fourier-ﬁltered CIFAR10 achieved a clean accuracy of only ∼ 33% on clean test samples whereas the MSF- REGULARIZED model’s clean accuracy is ∼ 87%, and a model trained on high-pass ﬁltered CIFAR10 training samples achieved only ∼ 15% accuracy on clean test samples.
Due to the energy statistics over frequency bands in natural images, training on Fourier-ﬁltered data is not successful for all but the lowest frequency bands, where most of their energy resides.
In addition, we note that regularizers such as ASF-regularization cannot be recreated by Fourier-ﬁltering the training data due to the tendency of standard training to be sensitive to certain frequencies.
E B LACK BOX F OURIER - NOISE CORRUPTIONS E. CIFAR10 F OURIER - NOISE CORRUPTIONS
[Figure] E. SVHN F OURIER - NOISE CORRUPTIONS [Figure]
F C ORRUPTIONS IN (H ENDRYCKS & D IETTERICH , 2019)
F.
E
[Figure]
F. CIFAR100-C BENCHMARK
[Figure] [Table]
G P ATCH - SHUFFLING IMAGES
[Figure]
[Equation]
[Figure]
r i g i n
[Figure]
l