R EGULARIZED LINEAR CONVOLUTIONAL NETWORKS INHERIT FREQUENCY SENSITIVITY FROM IMAGE STATISTICS
Anonymous authors
Paper under double-blind review
A BSTRACT
It is widely acknowledged that trained convolutional neural networks (CNNs) have different levels of sensitivity to signals of different frequency.
In particular, a number of empirical studies have documented CNNs sensitivity to low-frequency signals.
In this work we show with theory and experiments that this observed sensitivity is a consequence of the frequency distribution of natural images, which is known to have most of its power concentrated in low-to-mid frequencies.
Our theoretical analysis relies on representations of the layers of a CNN in frequency space, an idea that has previously been used to accelerate computations and study implicit bias of network training algorithms, but to the best of our knowledge has not been applied in the domain of model robustness.
I NTRODUCTION
Since their rise to prominence in the early 1990s, convolutional neural networks (CNNs) have formed the backbone of image and video recognition, object detection, and speech to text systems (Lecun et al., 1998).
The success of CNNs has largely been attributed to their "hard priors" of spatial translation invariance and local receptive ﬁelds (Goodfellow et al., 2016, §9.3).
On the other hand, more recent research has revealed a number of less desirable and potentially data-dependent biases of
CNNs, such as a tendency to make predictions on the basis of texture features (Geirhos et al., 2019).
Moreover, it has been repeatedly observed that CNNs are sensitive to perturbations in targeted ranges of the Fourier frequency spectrum (Guo et al., 2019; Sharma et al., 2019) and further investigation has shown that these frequency ranges are dependent on training data (Yin et al., 2019;
Bernhard et al., 2021;
Abello et al., 2021; Maiya et al., 2022).
In this work, we provide a mathematical explanation for these frequency space phenomena, showing with theory and experiments that neural network training causes CNNs to be most sensitive to frequencies that are prevalent in the training data distribution.
Our theoretical results rely on representing an idealized CNN in frequency space, a strategy we borrow from (Gunasekar et al., 2018).
This representation is built on the classical convolution theorem,
[Equation]
where ˆ x and ˆ w denote the Fourier transform of x and w respectively, and ∗ denotes a convolution.
Equation 1.1 demonstrates that a Fourier transform converts convolutions into products.
As such, in a “cartoon” representation of a CNN in frequency space, the convolution layers become coordinate-wise multiplications (a more precise description is presented in section 3).
This suggests that in the presence of some form of weight decay, the weights ˆ w for high-power frequencies in the training data distribution will grow during training, while weights corresponding to low-power frequencies in the training data will be suppressed.
The resulting uneven magnitude of the weights ˆ w across frequencies can thus account for the observed uneven perturbation-sensitivity of CNNs in frequency space.
We formalize this argument for linear CNNs (without biases) in sections 3 and 4.
One interesting feature of the framework set up in section 4 is that the discrete Fourier transform (DFT) representation of a linear CNN is precisely a feedforward network with block diagonal weight matrices, where each block corresponds to a frequency index.
We show in theorem 4.9 that a learning objective for such a network of depth L with an (cid:96) 2 -norm penalty on weights is equivalent to an objective for the associated linear model with an (cid:96) p penalty on the singular values of each of its blocks, i.e. each frequency index — this result is new for CNNs with multiple channels and outputs.
In particular, the latter penalty is highly sparsity-encouraging, suggesting as depth increases these linearly-activated CNNs have an even stronger incentive to prioritize frequencies present in the training data.
It has long been known that the frequency content of natural images is concentrated in low-to-mid frequencies, in the sense that the power in Fourier frequency f is well-described by 1 / | f | α for a coefﬁcient α ≈ 1 (Lee et al., 2001).
Hence, when specialized to training data distributions of natural images, our results explain ﬁndings that CNNs are more susceptible to low frequency perturbations in practice (Guo et al., 2019; Sharma et al., 2019).
We use our theoretical results to derive speciﬁc predictions: CNN frequency sensitivity aligns with the frequency content of training data, and deeper models, as well as models trained with substantial weight decay, exhibit frequency sensitivity more closely reﬂecting the statistics of the underlying images.
We conﬁrm these predictions for nonlinear CNNs trained on the CIFAR10 and ImageNette datasets.
Figure 1 shows our experimental results for a variety of CNN models trained on CIFAR as well as a variant of CIFAR10 preprocessed with high pass ﬁltering (more experimental details will be provided in section 5).
To the best of our knowledge, ours is the ﬁrst work to connect the following research threads (see section 2 for further discussion):
• equivalences between linear neural networks and sparse linear models, • classical data-dependent “shrinkage” properties of sparse linear models, • statistical properties of natural images, and • sensitivity of CNNs to perturbations in certain frequency ranges.
R ELATED WORK
The following a brief synopsis of work most closely related to this paper; a more through survey can be found in appendix A.
Perturbations in frequency components : (Guo et al., 2019) found that adversarial perturbations constrained to low frequency Fourier components allowed for greater query efﬁciency and higher transferability between different neural networks, and (Sharma et al., 2019) demonstrated that constraining to high or or midrange frequencies did not produce similar effects.
(Jo & Bengio, 2017), (Yin et al., 2019), (Bernhard et al., 2021), (Abello et al., 2021) and (Maiya et al., 2022) all found in one way or another that model frequency sensitivity depends on the underlying training data.
Our work began as an attempt to explain this phenomenon mathematically.
Implicit bias and representation cost of CNNs : Our analysis of (linear) convolutional networks leverages prior work on implicit bias and representational cost of CNNs, especially (Gunasekar et al., 2018).
There it was found that for a linear CNN trained on a binary linear classiﬁcation task with exponential loss, with linear effective predictor β , the Fourier transformed predictor ˆ β converges (in direction) to a ﬁrst-order stationary point of
[Equation]
Our general setup in section 3 closely follows these authors’, and our theorem 4.9 partially conﬁrms a suspicion of (Gunasekar et al., 2018, §6) that “with multiple outputs, as more layers are added, even fully connected networks exhibit a shrinking sparsity penalty on the singular values of the effective linear matrix predictor ...”
While the above result describes a form of implicit regularization imposed by gradient descent, we instead consider explicit regularization imposed by auxiliary (cid:96) 2 norm penalties in objective functions, and prove equivalences of minimization problems.
In this sense our analysis is technically more closely related to that of (Dai et al., 2021), which considers parametrized families of functions f ( x, w ) and deﬁnes the representation cost of a function g ( x ) appearing in the parametric family as
[Equation]
[Figure]
While this approach lacks the intimate connection with the gradient descent algorithms used to train modern neural networks, it comes with some beneﬁts: for example, results regarding representation cost are agnostic to the choice of primary loss function (e.g. squared error, cross entropy, even contrastive losses).
Data-dependent bias : Perhaps the work most closely related to ours is that of (Hacohen & Weinshall, 2022) on principal component bias , where it is shown that rates of convergence in deep (and wide) linear networks are governed by the spectrum of the input data covariance matrix.
T HE DISCRETE F OURIER TRANSFORM OF A CNN
In this section we ﬁx the notation and structures we will be working with.
We deﬁne a class of idealized, linear convolutional networks and derive a useful representation of these networks in the frequency space via the discrete Fourier transform.
Consider a linear, feedforward 2D-CNN f ( x ) of the form
[Equation]
(3.1) where w l ∗ x denotes the convolution operation between tensors
w l ∈ R C
l × H × W × C l − 1 and x ∈
R C l − 1 × H × W , deﬁned by
[Equation]
and w L,T x denotes a contraction (a.k.a.
Einstein summation) of the tensor w L ∈ R K × H × W × C L −with the tensor x ∈ R C L − 1 × H × W over the last 3 indices (the ( − ) T denotes a transpose operation described momentarily).
Explicitly,
[Equation]
Thus, the model eq.
(3.1) has weights
w l ∈ R C
l × H × W × C l − 1 for l = 1 , . . .
, L − 1 and w L ∈ R K × H × W × C L − 1 .
Remarks 3.4 .
For tensors with at least 3 indices (such as x and the weights w l above) we will always use the transpose notation − T to denote reversing the second and third tensor indices, which will always be the 2D spatial indices.
For matrices and vectors it will be used according to standard practice.
In eq.
(3.3) the transpose ensures that the indices in Einstein sums move from “inside to out” as is standard practice.
Equivalently, w L,T x can be described as a usual matrix product ˜ w L vec( x ) where vec( x ) is the vectorization (ﬂattening) of x and ˜ w L is obtained by ﬂattening the last 3 tensor indices of w L (compatibly with those of x as dictated by eq.
Hence it represents a typical “ﬂatten and then apply a linear layer” architecture component.
Our reason for adopting the tensor contraction perspective is that it is more amenable to the Fourier analysis described below.
Note that in this network the number of channels is allowed to vary but the heights and widths remain ﬁxed, and that we use full H × W convolutions throughout as opposed to the local (e.g. 3 × 3 ) convolutions often occurring in practice.
Given an array x ∈ R C l × H × W , we may consider its discrete Fourier transform (DFT) ˆ x , whose entries are computed as
[Equation]
Similarly, for an array w ∈ R C l × H × W × C l − 1 the DFT ˆ w is deﬁned to be
[Equation]
In what follows the DFT will always be taken with respect to the two spatial dimensions and no others.
The mapping x
(cid:55)→ ˆ x deﬁnes an orthogonal linear transformation of R C l × H × W .
In addition, it satisﬁes the following two properties:
Lemma 3.7 (Cf.
(Kiani et al., 2022, Lem.
C.2)) .
[Equation] [Equation]
Explicitly, the products on the right hand sides of eqs.
(3.8) and (3.9) are deﬁned as
[Equation]
Remark 3.11 .
In the terminology of CNNs, this says that the DFT converts full H × W convolutions to spatially pointwise matrix multiplications, and preserves dot products.
Our ﬁrst lemma is a mild generalization of (Gunasekar et al., 2018, Lem. 3); we defer all proofs to appendix D.
Lemma 3.12.
The CNN f ( x ) is functionally equivalent to the network ˆ f
(ˆ x ) deﬁned as
[Equation]
where the ﬁrst map ˆ − denotes the DFT x (cid:55)→ ˆ x .
R EGULARIZED CNN OPTIMIZATION PROBLEMS IN FREQUENCY SPACE
Consider a learning problem described as follows: given a dataset D
= { ( x n , y n ) ∈ R C × H × W × R K
|
i
= 1 , . .
. , N } , and a loss function L : ( R K )
N ×
( R K ) N
→ R , we seek weights w solving the (cid:96) 2 -regularized minimization problem
[Equation]
or
[Equation]
lemma 3.
[Equation]
This setup allows a wide variety of loss functions.
In the experiments of section 5 and appendix B we consider two important special case: ﬁrst, supervised-style empirical risk minimization with respect to a sample-wise loss function (cid:96) : R K × R K
→ R (in our experiments, cross entropy), where
[Equation]
Second, contrastive losses such as the alignment and uniformity objective of (Wang & Isola, 2020), which encourages the features f ( x n ) to be closely aligned with their corresponding “labels” y n , and the set of features { f ( x n ) }
Nn =1 to be uniformly distributed on the unit sphere S K − 1 ⊂ R K .
According to lemma 3.12,
[Equation]
In the case where the numbers of channels C, C 1 , . . .
, C L − 1 are all 1 and the number of classes
K = 1 , networks of this form were studied in (Tibshirani, 2021) where they were termed “simply connected.”
In the case where the the number of classes K =
but the numbers of channels C l may be larger, such networks were studied in (Gunasekar et al., 2018).
Of course, eq. (4.3) is just an over-parametrized linear function.
We can describe it more succinctly by introducing a new tensor ˆ v ∈ R K × H × W × C such that ˆ f
(ˆ x )
=
ˆ v T ˆ x .
With a little manipulation of eq. (4.3), we can obtain a formula for ˆ v in terms of the ˆ w l for l = 1 , . . .
, L .
Lemma 4.4.
[Equation]
The following theorem shows that the regularization term of eq.
(4.1), which penalizes the (cid:96) 2 -norms of the factors ˆ w l , is equivalent to a penalty using more sparsity-encouraging norms of ˆ v .
To state it we need a deﬁnition.
Deﬁnition 4.6 (Schatten p -norms) .
Let A =
( a ij ) ∈ M ( n × n, C ) be a square matrix and let
A =
UDV T be a singular value decomposition of A , where U, V ∈ U ( n ) and D = diag( λ i ) is a non-negative diagonal matrix with diagonal entries
λ 1 , . .
λ n ≥ 0 .
For any p > 0 the Schatten p -norm of A is
[Equation]
Remark 4.8 .
In the case p = 2 , the 2 -norm of deﬁnition 4.6 agrees with the usual Euclidean 2 -norm ( (cid:80) i,j | a ij | 2 ) 12 , since left (resp.
right) multiplication by a unitary matrix U T (resp.
V ) preserves the
Euclidean 2 -norms of the columns (resp.
rows) of A .
Theorem 4.9.
The optimization problem eq.
(4.1) is equivalent to an optimization problem for ˆ v of the form
[Equation]
where 1 ˆ v ij denotes the K × C matrix obtained by ﬁxing the spatial indices of ˆ v
= (ˆ v cijd ) , and the min is taken over the space of tensors ˆ v such that each matrix ˆ v ij has rank at most min { C, C 1 , . . .
, C L − 1 , K } .
The essential ingredient of our proof is a generalized non-commutative Hölder inequality.
Lemma 4.11.
If B ∈ M ( m × n, C ) is a matrix with complex entries, A 1 , . .
A k is a composable sequence of complex matrices such that A L · · · A 1 = B and (cid:80) i 1 p
i = 1 r where p 1 , . .
. , p L , r > 0 are positive real numbers,
[Equation]
Such inequalities are not new: in the case r = 1 , lemma 4.11 follows from (Dixmier, 1953, Thm.
6), and in the case L = 2 it is an exercise in (Bhatia, 1996).
However, we suspect (and our proof of theorem 4.9 suggests) that lemma 4.11 underpins many existing results on implicit bias and representation costs of (linear) neural networks, such as those of (Gunasekar et al., 2018;
Yun et al., 2021; Dai et al., 2021; Lawrence et al., 2022; Jagadeesan et al., 2022).
In the case where the numbers of channels C, C 1 , . . .
, C L − 1 are all 1 and and the number of outputs
K = 1 , and where the loss (cid:96) is squared error, the problem eq.
(4.10) reduces to
[Equation]
The sum in the regularization term of eq.
(4.13) is | ˆ v | pp where p = 2 L — in particular when L = 1 , eq.
(4.13) is a ridge regression problem (Hastie et al., 2001; Tibshirani & Wasserman) and when
L = 2 (the one hidden layer case) eq.
(4.13) is a LASSO problem.
We can analyze these two tractable cases to obtain qualitative predictions which will be tested empirically in section 5.
Since the qualitative predictions from both cases are similar, we devote the following section to ridge, and defer LASSO to appendix C.
4. L = 1 : RIDGE REGRESSION
In this case, eq.
(4.13) is the usual ridge regression objective; the closed-form solution is
[Equation]
where ˆ X is a N × C × H × W batch tensor with “rows” the ˆ x n and the entries of Y are the y n (see e.g. (Hastie et al., 2001)).
When λ
=
this reduces to the usual (unpenalized) least squares solution ˆ v LS :
=
( ˆ X T ˆ X ) − 1 ˆ X T Y , and substituting ˆ X T Y =
ˆ X T
ˆ X ˆ v LS in eq.
[Equation] [Equation]
14) we obtain
[Equation]
where strictly speaking ˆ X T
ˆ X is a tensor product of ˆ X with itself in which we contract over the batch index of length N , hence it is of shape W × H ×
C × C
× H × W .
The frequency properties of images enter into the structure of the symmetric tensor 1 N ˆ X T ˆ X , which (if the dataset D is centered, i.e. preprocessed by subtracting the mean 1 N X T 1 N ) serves as a generalized covariance matrix for the frequency space representation of D .
To ease notation, let Σ
= 1 N ˆ X T ˆ X , and suppose that
[Equation]
In other words, proper covariances between distinct frequency components are negligible and we retain only the variances , i.e. the diagonal entries of the covariance matrix.
In ﬁg. 4 we demonstrate that this assumption is not unrealistic in the case where X is a dataset of natural images.
With the assumption of eq.
(4.16), eq.
(4.15) reduces to
[Equation]
Equation (4.17) is an instance of the classic fact that ridge regression shrinks coefﬁcients more in directions of low input variance.
In words, τ cij is the variance of training images in Fourier component cij , and eq.
(4.17) says | ˆ v cij | shrinks more when the variance τ cij is low; in the limiting case τ cij → 0 the coefﬁcient ˆ v cij → 0 as well.
Returning to the subject of frequency sensitivity, observe that ˆ v cij is the directional derivative of f with respect to the cij -th Fourier component.
Proposition 4.18 (Data-dependent frequency sensitivity, L = 1 ) .
With the notations and assumptions introduced above, the magnitude of the directional derivative of f with respect to the cij
-th
Fourier component scales with λ and τ according to
[Equation]
Empirically it has been found that for natural distributions of images the variances τ cij follow a power law of the form τ cij ≈
[Equation]
(Lee et al., 2001; Baradad et al., 2021).
2 Under this model, eq. (4.17) becomes
[Equation]
that is, sensitivity is monotonically decreasing with respect to both frequency magnitude and the regularization coefﬁcient λ .
This is consistent with ﬁndings that CNNs trained on natural images are vulnerable to low frequency perturbations (Guo et al., 2019; Sharma et al., 2019).
E XPERIMENTS
When L > 2 (that is, when there are more than 1 convolutional layers), the p “norm” of eq.
(4.10) is non-convex (hence the quotes), and in the limit as L → ∞ the norms appearing in eq.
(4.10) converge to the Schatten 0-“norm,” which is simply the number of non-0 singular values of a matrix.
Moreover, we see that the regularization coefﬁcient of eq.
(4.10) is effectively multiplied by L .
Even in the case where K = 1 so that ˆ v is a vector, it is known that solving eq.
(4.10) for L > 1 is
NP-hard (Chen et al., 2017), so we have no hope of ﬁnding closed form solutions as in section 4.1.
However, we can use the analysis in section 4 to derive three testable hypotheses:
I. CNN frequency sensitivity depends on the frequency content of training data (proposition 4.18).
II.
The fact that the regularization term of eq.
(4.10) becomes more
sparsity-encouraging as
L increases suggests that the data-dependent frequency sensitivity observed in section 4. and appendix C becomes even more pronounced as the number of convolutional layers increases.
[Figure]
III.
Moreover, the functional forms of eqs.
(4.17) and (C.7) suggest that the data-dependent frequency sensitivity will increase monotonically with the weight decay parameter λ .
We empirically validate these hypotheses with experiments using CNNs trained on multiple datasets.
The datasets used in these experiments are CIFAR10 (Krizhevsky, 2009), ImageNette (the 10 easiest classes from ImageNet (Deng et al., 2009))
(FastAI, 2019), and synthetic images generated using the wavelet marginal model (WMM) of (Baradad et al., 2021).
The later dataset is of interest since the generative model is explicitly designed to capture the frequency statistics of natural images, and allows for varying the exponents α and β in the power law τ cij ≈
[Figure]
described above.
Figure 2a display the variances CIFAR10, ImageNette and their high pass ﬁltered variants in frequency space; for those of the WMM datasets see ﬁg. 3.
In addition, we experiment with high pass ﬁltered versions of the CIFAR10 and ImageNette datasets, which we refer to as hpf-CIFAR10 and hpf-ImageNette respectively; the frequency space statistics of these are shown in ﬁg. 2b.
4 For implementation details we refer to appendix B. We can further summarize the data displayed in ﬁgs.
2a and 2b by averaging over circles with varying radii r , i.e. computing expectations E
[ (cid:112) Σ jiccij | | ( i, j ) | = r ] , to obtain the frequency magnitude statistics curves shown in ﬁg. 1 (see appendix B.2 for implementation details).
From this point forward we focus on such curves with respect to frequency magnitude.
The CNNs used in these experiments are ConvActually (a CNN that closely approximates eq.
(3.1), the differences being the addition of biases and ReLU non-linearities), Myrtle CNN (a small feedforward CNN known to achieve high performance on CIFAR10 obtained from (Page)), VVG (a family of CIFAR10 VGGs (Simonyan & Zisserman, 2015) obtained from (Fu, 2017) and ImageNet
VGGs from (Marcel & Rodriguez, 2010)), ResNet (a family of CIFAR10 ResNets
(He et al., 2016) obtained from (Team, 2021)), and AlexNet (a small AlexNet (Krizhevsky et al., 2012) adapted to contrastive training obtained from (Baradad et al., 2021))
For more detailed descriptions of datasets and model architectures we refer to appendices B.1 and B.5.
We measure frequency sensitivity of a CNN f in terms of the magnitudes of the directional derivatives ∇ x f ( x ) T ˆ e cij , where ˆ e cij is the cij
-th
Fourier basis vector.
These magnitudes are averaged over all the input images x in the relevant validation set, and as in the case of image statistics we can average them over circles of varying radii,
i.e. compute expectations E [ |∇ x f ( x ) T
ˆ e cij | | | ( i, j ) | = r ] (see appendix B.3 for implementation details).
By “data-dependent frequency sensitivity” we mean the extent to which the frequency sensitivity of a CNN f reﬂects the statistics of the images it was trained on.
Figures 1, 2a, 3 and 5 show that the variance of DFTed CIFAR10, ImageNette and
WMM images is heavily concentrated at low frequencies, in agreement with the power law form τ cij ≈ γ | i | α
+ | j | β described in section 4.
Hence in the absence of any modiﬁcations to the underlying images, we expect that training on this data will emphasize sensitivity of f to perturbations along the lowest frequency Fourier basis vectors, with the effect increasing along with model depth and the weight decay parameter λ .
On the other hand, the variance of hpf-CIFAR10 and hpf-ImageNette is concentrated in mid-range frequencies, and so here we expect training will emphasize sensitivity of f to perturbations along mid range Fourier basis vectors (again with more pronounced effect as depth/ λ increase).
5. F REQUENCY SENSITIVITY AND DEPTH
Figure 1a shows sensitivity of ConvActually models of varying depth to perturbations of varying
DFT frequency magnitudes.
These curves illustrate that as depth increases, frequency sensitivity
E [ |∇ x f ( x ) T ˆ e cij | | | ( i, j ) | = r ] more and more closely matches the frequency magnitude statistics
E [ (cid:112) Σ jiccij | | ( i, j ) | = r ] of the training data set, both in the case of natural and high pass ﬁltered images — hence, these empirical results corroborate hypotheses I. and II..
Figure 1c shows results of a similar experiment with VGG models of varying depth; models trained on natural images have sensitivity generally decreasing with frequency magnitude, whereas those trained on high pass ﬁltered data have “U” shaped sensitivity curves with minima near the ﬁlter cutoff, corroborating item I..
Here it is not clear whether models trained on natural images follow that pattern predicted by II.
(the deepest models are less sensitive to both low and high frequencies), however when trained on high pass ﬁltered images deeper models do seem to most closely follow the frequency statistics of the dataset.
Figure 5 includes a similar experiment with VGG models of varying depth trained on (hpf-)ImageNette.
5. F REQUENCY SENSITIVITY AND WEIGHT DECAY
Figure 1b shows radial frequency sensitivity curves for ConvActually models of depth 4 trained on (hpf-)CIFAR10 with varying weight decay coefﬁcient λ .
We see that as λ increases, model frequency sensitivity more and more closely reﬂects the statistics of the training data images, corroborating hypotheses I. and III..
Figure 1b shows results for a similar experiment with Myrtle CNNs, with a similar conclusion.
Figure 5 shows results for VGG models trained with varying weight decay on (hpf-)ImageNette.
5. I MPACT OF THE LEARNING OBJECTIVE
So far, our analysis and experiments have only shown that the regularization term in eq.
(4.1) encourages CNN gradients with respect to spatial Fourier basis vectors to reﬂect the frequency statistics of the training data.
It is of course possible that the ﬁrst term of eq.
(4.1) deﬁning the learning objective overwhelms the regularization term resulting in different model frequency sensitivity.
In ﬁg. 7 we show this occurs in CNNs trained on WMM synthetic data with an alignment and uniformity contrastive loss; see appendix B.4 for a possible explanation in the framework of sections 3 and 4.
L IMITATIONS AND OPEN QUESTIONS
In order to obtain an optimization problem with some level of analytical tractibility, we made many simplifying assumptions in sections 3 and 4, most notably omitting nonlinearities from our idealized
CNNs.
While the experimental results of section 5 illustrate that multiple predictions derived from sections 3 and 4 hold true for CNNs more closely resembling those used in practice trained with supervised learning, ﬁg. 6 shows that hypotheses I-III can fail in the presence of residual connections — see appendix B.4 for further discussion.
As previously mentioned, ﬁg. 7 shows that a contrastive alignment and uniformity learning objective results in far different CNN representations.
Perhaps more signiﬁcantly, it must be emphasized that model sensitivity as measured by gradients represents a very small corner of a broader picture of model robustness (or lack therof).
For example, it does not encompass model behaviour on corruptions (see e.g. (Hendrycks & Dietterich, 2019)) or shifted distributions (see e.g. (Recht et al., 2019)).
A R ELATED WORK IN GREATER DETAIL
CNN sensitivity to
Fourier frequency
components : (Jo & Bengio, 2017) computed transfer accuracy of image classiﬁers trained on data preprocessed with various Fourier ﬁltering schemes (e.g. train on low pass ﬁltered images, test on unﬁltered images or vice versa).
They found signiﬁcant generalization gaps, suggesting that models trained on images with different frequency content learned different patterns.
(Guo et al., 2019) proposed algorithms for generating adversarial perturbations constrained to low frequency Fourier components, ﬁnding that they allowed for greater query efﬁciency and higher transferability between different neural networks.
(Sharma et al., 2019) demonstrated empirically that constraining to high or or midrange frequencies did not produce similar effects, suggesting convolutional networks trained on natural images exhibit a particular sensitivity to low frequency perturbations.
(Yin et al., 2019) showed different types of corruptions of natural images (e.g. blur, noise, fog) have different effects when viewed in frequency space, and models trained with different augmentation strategies (e.g. adversarial training, gaussian noise augmentation) exhibit different sensitivities to perturbations along Fourier frequency components.
(Bernhard et al., 2021) investigated the extent to which constraining models to use only the lowest (or highest)
Fourier frequency components of input data provided perturbation robustness, also ﬁnding signiﬁcant variability across datasets.
(Abello et al., 2021) tested the extent to which CNNs relied on various frequency bands by measuring model error on inputs where certain frequencies were removed, again ﬁnding a striking amount of variability across datasets.
(Maiya et al., 2022) analyzed the sensitivity of networks to perturbations in various frequencies, ﬁnding signiﬁcant variation across a variety of datasets and model architectures.
All of these works suggest that model frequency sensitivity depends heavily on the underlying training data.
— our work began as an attempt to explain this phenomenon mathematically.
Implicit bias and representation cost of CNNs : Our analysis of (linear) convolutional networks leverages prior work on implicit bias and representational cost of CNNs, especially (Gunasekar et al., 2018).
There it was found that for a linear one-dimensional convolutional network where inputs and all hidden layers have one channel (in the notation of section 3, C = C 1 = · · · = C L − 1 = K = 1 ) trained on a binary linear classiﬁcation task with exponential loss, with linear effective predictor β , the Fourier transformed predictor ˆ β converges in direction to a ﬁrst-order stationary point of an optimization problem of the form
[Equation]
A generalization to arbitrary group-equivariant CNNs (of which the usual CNNs are a special case) appears in (Lawrence et al., 2022, Thm. 1) — while we suspect that some of our results generalize to more general equivariant networks we leave that to future work.
For generalizations in different directions see (Lyu & Li, 2020; Yun et al., 2021), and for additional follow up work see (Jagadeesan et al., 2022).
Our general setup in section 3 closely follows these authors’, and our theorem 4.
partially conﬁrms a suspicion of (Gunasekar et al., 2018, §6) that “with multiple outputs, as more layers are added, even fully connected networks exhibit a shrinking sparsity penalty on the singular values of the effective linear matrix predictor ...”
While the aforementioned works study the implicit regularization imposed by gradient descent, we instead consider explicit regularization imposed by auxiliary (cid:96) 2 norm penalties in objective functions, and prove equivalences of minimization problems.
In this sense our analysis is perhaps more closely related to that of (Dai et al., 2021), which considers parametrized families of functions f ( x, w ) and deﬁnes the representation cost of a function g ( x ) appearing in the parametric family as
[Equation]
While this approach lacks the intimate connection with the gradient descent algorithms used to train modern neural networks, it comes with some beneﬁts: for example, results regarding representation cost are agnostic to the choice of per-sample loss function (in particular they apply to both squared error and cross entropy loss).
In the case where the number of channels C = C 1 = · · · = C L − 1 =(but the number of outputs may be > 1 ), theorem 4.9 can be deduced from (Dai et al., 2021, Thm. 3).
Lastly, while in this paper we focus on spatial frequency properties of image data, there is a large and growing body of work on the impact of frequency properties of training data more broadly interpreted.
(Rahaman et al., 2019) gave a formula for the continuous Fourier transform of a ReLU network f : R n → R , and showed in a range of experiments that ReLU networks learn low frequency modes of input data ﬁrst.
(Xiao, 2022) proves theoretical results on low frequency components being learned ﬁrst for networks f : (cid:81) i S n i → R on products of spheres, where the role of frequency is played by spherical harmonic indices (see also (Xiao & Pennington, 2022) for some related results).
Perhaps the work most closely related to ours is that of (Hacohen & Weinshall, 2022) on principal component bias , where it is shown that rates of convergence in deep (and wide) linear networks are governed by the spectrum of the input data covariance matrix.
6 (Hacohen & Weinshall, 2022) also includes experiments connecting PC bias with spectral bias (learning low frequency modes ﬁrst, as in the preceding paragraph) and a phenomenon known as learning order consistency .
However, it is worth noting that in their work there is no explicit theoretical analysis of CNNs and no consideration of the statistics of natural images in Fourier frequency space.
Other applications of Fourier transformed CNNs : (Zhu et al., 2021;
Pratt et al., 2017; Mathieu et al., 2014; Vasilache et al., 2015) all, in one way or another, leverage frequency space representations of convolutions to accellerate computations, e.g. neural network training.
Since this is not our main focus, we omit a more detailed synopsis.
B E XPERIMENTAL DETAILS B. D ATASETS
Figure 3 illustrates the frequency content of the image datasets used in this paper.
For CIFAR10 we use canonical train/test splits (imported using (Marcel & Rodriguez, 2010)).
As described at (FastAI, 2019),
Imagenette is a subset of 10 easily classiﬁed classes from Imagenet (tench, English springer, cassette player, chain saw, church, French horn, garbage truck, gas pump, golf ball, parachute).
We use the “full size” version of the dataset with standard (224-by-224)
ImageNet preprocessing.
We implement high pass ﬁltering by passing each image x through the following preprocessing steps (before any other preprocessing other than loading a JPEG image ﬁle as a tensor): (i) take the DFT ˆ x , (ii) multiply with a mask m where m cij =
if | i | , | j | ≤ some ﬁxed threshold, andotherwise, 7 (iii) applying the inverse DFT.
For both CIFAR10 and ImageNette, our threshold ispixels.
This means that while 50% of frequency indices are ﬁltered out for CIFAR10, only about
[Figure]
0.13% are ﬁltered for ImageNette.
The motivation for this approach was to remove a similar amount of absolute frequency content in both cases; we also experimented with a threshold of 112 in the case of ImageNette, resulting in removal of 50% of frequency indices in both cases
We use the wavelet marginal model dataset generated using the implementation in (Baradad et al., 2021) — details of this model are described in (Baradad et al., 2021, §3.3).
The generated images are of resolution 128-by-128; they are preprocessed with downsampling to 96-by-96 and cropping to 64-by-64.
For further preprocessing details we refer to (Baradad et al., 2021, §4).
All pipelines described in this paper implement the standard preprocessing step of subtracting the mean RGB value of the training dataset, and dividing by the standard deviation of the training set
RGB values.
8 Note that this does not ﬂatten the variances of the image distributions in frequency space displayed in ﬁgs.
1 and 5 — in fact, it (provably) only impacts the variance of the 0th Fourier component, corresponding to constant images.
Ensuring that the variance in each Fourier component is (approximately) 1 would require the far less standard preprocessing step to each image x consisting of (i) apply the DFT to get ˆ x , (ii) subtract ˆ µ , where ˆ µ is the mean of the DFTed images in the training dataset ( not just the mean of their RGB values), 9 (iii) divide by the standard deviation ˆ σ of the DFTed images in the training dataset, and ﬁnally (iv) apply the inverse DFT.
It should be emphasized that ˆ µ and ˆ σ have the same shape as x , e.g. (3 , 32 , 32) for the CIFAR10 dataset.
The only work we are aware of that implements such preprocessing on images is (Hacohen & Weinshall, 2022, see §4), though of course there may be others.
B. C OVARIANCE MATRICES OF DFT ED IMAGE DATASETS
In this section we provide further details on our computations of (co)variances and standard deviations of DFTed image datasets.
To compute the covariances Σ jicc (cid:48) i (cid:48) j (cid:48) for ﬁg. 4, we begin with a dataset of natural images, say X .
We subtract its mean RGB pixel value (a vector in R 3 ) and divide by the standard deviation of RGB pixel values (also a vector in R 3 ) as is standard.
We next apply the DFT to every image in X , to obtain a DFTed dataset ˆ X .
We then compute the mean ˆ µ of the images in ˆ X , not the RGB pixel values — this is a 3 × H × W tensor, where H, W are the heights/widths of the images in X (e.g., 32 for CIFAR10).
This mean is then subtracted from ˆ X to obtain a centered dataset.
Next, we sample batches of size B , say ˆ x 1 , . .
, ˆ x B , from ˆ X (these are tensors of shape B × C × H × W ).
For each batch, we subtract the mean, contract over the batch index and divide by B to obtain an estimate for Σ , say ˜Σ like so: [Equation] [Equation]
); explicitly
[Equation]
Finally, we average over an entire dataset’s worth of batches to get our ﬁnal estimate of Σ jicc (cid:48) i (cid:48)
j (cid:48) .
To estimate the expectations E [ | Σ jicc (cid:48) i (cid:48)
j (cid:48) | | | ( c, i, j ) − ( c, i (cid:48) , j (cid:48) ) | = r ] , we average absolute values | Σ jicc (cid:48) i (cid:48)
j (cid:48) | over all indices jicc (cid:48) i (cid:48)
j (cid:48) satisfying | ( c, i, j ) − ( c (cid:48) , i (cid:48) , j (cid:48) )
|
= r .
The number of such entries varies signiﬁcantly with r , which is why we have not explicitly written down the average.
We compute “modular” distances, i.e. distances on the discrete torus, using the formulae d ( i, j (cid:48) ) =
min {| ( h − h (cid:48) ) mod H | , | H − (( h − h (cid:48) ) mod H ) |} d ( w, w (cid:48) )
= min {| ( w − w (cid:48) ) mod W | , | W − (( w − w (cid:48) ) mod W ) |} (B.2) and ﬁnally d (( c, h, w ) , ( c (cid:48) , h (cid:48) , w (cid:48) ))
= (cid:112) ( c − c (cid:48) )
+ d ( h, h (cid:48) )
+ d ( w, w (cid:48) ) 2 .
In ﬁg. 2a, we begin with a dataset of natural images, say X .
We subtract its mean RGB pixel value (a vector in R 3 ) and divide by the standard deviation of RGB pixel values (also a vector in R 3 ) as is standard.
We next apply the DFT to every image in X , to obtain a DFTed dataset ˆ X .
We then compute the standard deviation of the images in ˆ X , not the RGB pixel values — this is a 3 × H × W tensor, where H, W are the heights/widths of the images in X (e.g., 32 for CIFAR10).
B. G RADIENT SENSITIVITY IMAGES
To compute the sensitvity curves in ﬁgs. 1 and 5 to 7, we subsample 5,000 images from the underlying validation dataset in the case of CIFAR10 and WMM use the entire validation dataset in the case of ImageNette.
For each such image x , we compute the DFT ˆ x , and then backpropagate gradients through the composition ˆ x → ˆˆ x = x → f ( x ) .
The result is a C × H × W × K Jacobian matrix expressing the derivative of f with respect to Fourier basis vectors.
We take (cid:96) 2 norms over the class index (corresponding to K ) to obtain a C × H × W gradient norm image, with c, h, w component |∇ x f ( x ) T ˆ e cij | .
Finally, we average these gradient norms over the (subsampled in the case of
CIFAR10 and WMM) dataset.
Next, we average radially much as we did in ﬁg. 4.
Given an expected gradient norm image
E [ |∇ x f ( x ) T ˆ e cij | ] obtained as above, we further average over all indices cij such that (cid:112)
i
+ j 2 = r .
Again, the number of such indices is highly variable.
To provide error estimates, for each set of trained model weights and each image dataset, we compute radial frequency sensitivity curves as in the paragraph above, apply post-processing consisting of:
Dividing the curve by its integral (to obtain a probability distribution)•
Taking the logarithm of the resulting probability distribution.
This step is motivated by the empirical observation that the variance of Fourier transformed images follows a power law with respect to frequency magnitude.
• Smoothing by averaging with 3 nearest neighbors on each side.
This step is motivated by the aforementioned highly variable number of DFT frequency indices corresponding to a given radius, which means that some radius values correspond to an average over far fewer samples which as a result has high variance.
A possible alternative would have been to bin radii, thus averaging over DFT components with frequency magnitudes in a small interval.
However, the automatic binning strategies we tried yielded bins that were too large, giving an undesirably low resolution view of frequency sensitivity curves.
We then repeat the entire pipeline above for 5 sets of model weights trained from independent random initializations, to obtain 5 curves, and display the standard deviation of their y -values.
For more details on our training procedures, see appendix B.5.
B. M ORE EXPERIMENTAL RESULTS
Figure 5 shows radial frequency sensitivity curves from experiments training VGGs with variable depth and weight decay on ImageNette, with and without high pass ﬁltering.
Here we do see signiﬁcant differences between models trained on natural vs. high pass ﬁltered images, including in the later case (small) local peaks in near the ﬁlter cutoff, however the effect of ﬁltering is not nearly as noticeable as in the CIFAR10 experiments of ﬁg. 1.
Of course, we only tested one ﬁlter cutoff, chosen somewhat arbitrarily.
Moreover, effects of depth and decay are not as dramatic in these experiments, although we do see the curves corresponding to high depth/decay for VGGs trained on natural images dropping off most severely at high frequencies, and in the case of high pass ﬁltered images deep VGGs are the least sensitive in the low frequency range (where the training images have no variance).
Interestingly, we see a range of frequency radii, roughly r ∈ [10 , 100] , where the ImageNette frequency statistics exhibit signiﬁcant noise around the overall power law pattern, and it does appear that all VGG models concentrate frequency sensitivity in this range, and more so with greater depth/decay.
It would be interesting to understand what sorts of natural image features contribute to the observed noise in the r ∈ [10 , 100] range, and if they are for some reason useful to the ImageNette classiﬁcation task.
Figure 6 shows radial frequency sensitivity curves from experiments training ResNets with variable depth and weight decay on CIFAR10, with and without high pass ﬁltering.
The frequency sensitivity curves are clearly different when trained on natural images (where they drop off at the highest frequencies) versus high pass ﬁltered images (where they have a “U”-shape similar to those of the
VGG models trained on high pass ﬁltered CIFAR10 in ﬁg. 1).
These observations seems somewhat consistent with item I..
However, the trends with depth and decay do not conform with items II. and III.
: it is difﬁcult to see trends in the depth experiment, and in the decay experiment the frequency sensitivity curves seem to become more increasing as decay increases, rather than adhering to the frequency content of the dataset.
It is not immediately clear to the authors what causes this behavior — further investigation would be an interesting direction for future work.
Lastly, ﬁg. 7 shows frequency sensitivity curves for AlexNets trained with alignment and uniformity loss (Wang & Isola, 2020) on synthetic data generated by the wavelet marginal model (WMM) of (Baradad et al., 2021) with varying α parameter; to be speciﬁc, for simplicity we set α = β in the power law τ cij ≈ γ | i | α + | j | β .
These results initially surprised us: for all α the variance of the synthetic data is concentrated in low frequencies (ﬁg. 3), with the level of concentration decreasing as α increases (i.e. as α increases, the frequency curve of the synthetic data spreads out).
However , for all AlexNets trained on WMM data the frequency sensitivity curves increase with frequency magnitude, with slope roughly increasing with respect to α !
This result, which shows that CNNs trained with contrastive learning objectives can respond quite differently to the statistics of their training data, admits a simple explanation in terms of sectionsand 4.
Indeed, the uniformity part of the objective encourages the set of vectors
[Equation]
to be uniformly distributed on the unit sphere S K − 1 ⊆ R K .
Suppose now for the sake of simplicity that that the vectors ˆ x n = (ˆ x ncij ) are normally distributed and their covariance matrix Σ is diagonal, with diagonal entries of the form
[Equation]
Then the feature vectors ˆ f
( ˆ x n )
=
ˆ v T ˆ x n are also normally distributed, with covariance matrix
[Equation]
Here as above ˆ v ij is a K × C matrix, so each ˆ v ij ˆ v Tij is indeed a K × K positive semi-deﬁnite matrix, and in cases of interest where C (cid:28) K it will have very low rank.
Considering eq.
(B.5), we see for example that if C (cid:28) K and the ˆ v ij are of roughly constant magnitude, then the covariance of the features
ˆ f
( ˆ x n ) will be dominated by the terms
[Equation]
making it impossible for the ˆ f ( ˆ x n ) to be uniformly distributed on the unit sphere.
It seems that perhaps the only way for such uniform distribution to occur is for the magnitudes of the ˆ v ij to increase [Figure] with the frequency magnitude | ( i, j ) | , in such a way as to offset the denominator | i | α + | j | α .
We do not have a proof of this fact (it seems such a proof would have to involve analysis of the functional form of the contrastive loss in (Wang & Isola, 2020), which we have not carried out), but it does offer one potential explanation of ﬁg. 7.
At the risk of being overzealous, the above discussion would suggest that
[Equation]
i.e. the norm of the gradient of f with respect to the ij -th DFT basis vector, follows a power law in α , and indeed in ﬁg. 7, where these gradient norms are plotted with respect to frequency magnitude on a log-log scale, we see slope roughly increasing with α .
B. M ODEL ARCHITECTURES AND TRAINING PARAMETERS
Recall that our CNNs are
ConvActually A CNN that closely approximates eq.
(3.1), the differences being the addition of biases and ReLU non-linearities.
This is accomplished by applying convolutions with 32 × 32 kernels (i.e., kernels of the same size as the input images) with circular padding.
Myrtle CNN
A small feed-forward CNN known to achieve high performance on CIFAR10 obtained from (Page).
This CNN has small kernels, ReLU non-linearities and max -pooling, as well as exponentially varying channel dimension.
VVG A family of CIFAR10 VGGs obtained from (Fu, 2017) and ImageNet VGGs from (Marcel &
Rodriguez,
ResNet A family of CIFAR10 ResNets (He et al., 2016) obtained from (Team, 2021).
AlexNet A small AlexNet adapted to contrastive training obtained from (Baradad et al., 2021)
Tables 1 and 2 describe training hyperparameters.
In all experiments we optimize using stochastic gradient descent (SGD) with momentum 0.9, using a “reduce on plateau” learning rate schedule where the learning rate is multiplied by 0 .
1 after “patience” epochs without a 1% improvement in validation accuracy, where “patience” is some ﬁxed integer (i.e., a hyperparameter); we use patience = [Figure] 20 throughout.
This schedule proceeds until either a minimum learning rate (in all of our experiments, 10 − 6 ) or a maximum number of epochs is hit, at which point training stops.
We use the PyTorch library (Paszke et al., 2019) on a cluster environment with Nvidia GPUs.
All models are trained on CIFAR10 except the ImageNette VGGs and AlexNets.
The ImageNette
VGGs are optimized as above, with the exception that we use distributed training on 8 GPUs (the batch size of 256 corresponds to a batch of size 32 on each GPU).
The AlexNets are trained using an unsupervised alignment and uniformity objective as described in (Baradad et al., 2021, §4, §A); we use the ofﬁcial implementation of (Baradad et al., 2021).
For CIFAR10 and ImageNette, we use the same hyperparameters when training on natural and high pass ﬁltered images.
For each model architecture and choice of hyperparameters, 11 i.e. instance appearing in tablesand 2, and each training dataset, we train 5 models from different random weight initializations.
Efﬁciently training this many CNNs was facilitated by the excellent FFCV library (Leclerc et al., 2022). [Table] [Table]
10 ** linspace(-5, -1, 4)
[Table]
[Table]
10**linspace(-5, -2,10) [Table] [Table] 10**linspace(-5, -2,10)
[Table]
[Table] [Table] 10**linspace(-5, -2,3) [Table]
[Table]
10**linspace(-5, -2,10)
[Table]
C L = 2 : LASSO PARAMETER SHRINKING AND SELECTION
When C l = 1 for all l and L = 1 , we may simplify eqs.
(4.10) and (4.13) to min ˆ v
[Equation]
(C.1) where ˆ X and Y are as in section 4.1.
The optimality criterion for eq.
(4.10) becomes (see e.g.
(Tibshirani & Wasserman))
[Equation]
(C.2) where ∇| ˆ v | 1 is the sub-gradient of the (cid:96) 1 -norm: ( ∇| ˆ v | 1 ) i (cid:26)
= sign(ˆ v i ) if ˆ v i (cid:54) = ∈ [ − 1 , 1] if ˆ v i = 0 . (C.3)
When λ =
this again reduces to the unpenalized least squares solution ˆ v LS :
= ( ˆ X T ˆ X ) − 1 ˆ X T Y , and substituting this in eq.
(C.2) we obtain
[Equation]
If we again make the assumption that Σ = 1 N X T X is “diagonal” as in eq.
(4.16), eq.
(C.2) simpliﬁes to
[Equation]
where strictly speaking d | ˆ v cij | d ˆ v cij is a subgradient as in eq.
(C.3).
From this we conclude
[Equation]
(the second case is equivalent to: if the LASSO solution ˆ v cij = 0 , it must be that the least squares solution satisﬁes | ˆ v LS ,cij | ≤ λτ cij ).
In the case where sign(ˆ v cij )
=
sign(ˆ v LS ,cij ) , we obtain a particularly nice conclusion:
[Equation]
Proposition C.8 (Data-dependent frequency sensitivity, L = 2 ) .
With the notations and assumptions introduced above, the magnitude of the directional derivative of f with respect to the cij -th Fourier component is linear in λτ cij with slope − 1 .
If we again plug in the empirically determined power law observed in natural imagery, τ [Equation] ≈ γ | i | α + | [Equation]
| β , eq.
(C.7) becomes
[Equation]
Here, the sensitivity decreases monotonically with respect to both frequency magnitude and the regularization coefﬁcient λ .
Note that compared to eq.
(4.19) from section 4, eq.
(C.9) implies a greater shrinking effect when | ˆ v cij | < 1 , and less shrinking when | ˆ v cij | > 1 .
12 We conjecture that shrinkage due to | ˆ v cij | < 1 is the dominant effect, due to the initial distribution of weights w in modern neural networks, which are often sampled from uniform or normal distributions with variance (cid:28) 1 (e.g., according to He initialization; He et al., 2016).
D P ROOFS D. P ROOF OF LEMMAS 3.7, 3.12 AND 4.
Proof of lemma 3.7.
For a direct proof of the convolution part, see (Kiani et al., 2022, Lem. C.2).
We reduce to the “single channel” cases of these formulae ( C l = C l − 1 = 1 ), which we take to be well known.
For the purposes of legibility, in this proof we denote the DFT (eqs.
(3.5) and (3.6)) by
F .
For the convolution part we must apply the DFT to eq.
Let w c,...,d denote the single channel tensor obtained by ﬁxing the input/output channels to indices c, d and similarly let x d,... be the single channel tensor ﬁxing the input channel to index d .
Then by interchanging the order of sums ( w ∗ x ) cij = (cid:88)
[Equation]
In other words, ( w ∗ x ) c,...
= (cid:80) d w c,...
,d ∗ x d,...
Now linearity of the DFT gives
[Equation]
where on the right hand side we’ve applied the standard single channel convolution theorem.
But this means
[Equation]
as desired.
Our proof of the Parseval identity is similar: we start with eq.
(3.3), that is, ( w T x ) k
= [Equation] l,m,n w kmnl x lmn .
With notation as above, we can write
[Equation]
where each term on the right hand side is an ordinary inner product of single channel signals.
Taking the Parseval identity for these as well known, we get
[Equation]
as desired.
Proof of lemmas 3.12 and 4.4.
We proceed by induction on L .
In the base case where L = 0 , lemma 3.12 is equivalent to Parseval’s theorem eq.
So, suppose L > 0 .
We may decompose f as a convolution followed by a linear CNN of the form eq.
(3.1), say g , with one fewer layers.
Explicitly, g has weights w 2 , . . .
, w L and
[Equation]
By inductive hypothesis, we may assume that
[Equation]
Applying the convolution theorem eq.
(3.8) to obtain (cid:92)
[Equation]
completes the proof.
Proof of lemma 4.4.
For each weight ˆ w l let ˆ w lij denote the C l × C l − 1 matrix obtained by ﬁxing the spatial indices of ˆ w l to ij .
Unpacking deﬁnitions,
[Equation]
ˆ
[Equation]
ˆ
[Equation]
and so
[Equation]
We can recognize the inside sum as performing a matrix product of ˆ w Lij with ˆ w L − 1 ij · ˆ w L − 1 ij · · · ˆ w 1 ij · ˆ x ij .
Since matrix multiplication is associative, we can just as well multiply the ˆ w l ij ﬁrst and then act on the vector ˆ x ij .
Thus (cid:88) i
[Equation]
Now we can recognize the right hand side as ( ˆ w L · ˆ w L − 1 · · · ˆ w 1 ) T ˆ x , as claimed.
D. P ROOFS OF THEOREM 4.9 AND LEMMA 4.
Recall we aim to prove: the optimization problem
[Equation]
is equivalent to the following optimization problem for the product ˆ v ...ij = (cid:81)
Ll
[Equation]
ˆ w L − l,...ij :
[Equation]
where the minima runs over the space of tensors ˆ v such that each matrix ˆ v ij has rank at most min { C, C 1 , . .
, C L − 1 , K } .
We proceed by a series of reductions; as a ﬁrst step we observe
[Equation]
and hence theorem 4.9 will follow if we can prove
[Equation]
where the min on the right hand side runs over all ˆ w such that ˆ w L · ˆ w L − 1 · ˆ w L − 1 · · · ˆ w
= ˆ v .
We can make life slightly simpler by noticing eq.
(D.14) decomposes over the i, j index, and will follow from
[Equation]
22 for [Equation] which we will show in lemma D.18 — note that at this point we are arriving at a statement about norms of matrix products and can dispense with the baggage of i, j indices and ˆ s. To formally state that lemma, we introduce a convenient deﬁnition.
Deﬁnition D.16.
A sequence of matrices A 1 ∈ M ( m 1 × n 1 , C ) , . . .
, A L ∈ M ( m L × n L , C ) is composable if and only if
[Equation]
In other words, A 1 , . . .
, A L is composable if and only if the product A L · · · A 1 makes sense.
Lemma D.18.
If B ∈ M ( m × n, C ) is a matrix with complex entries and L ∈ N is a non-negative integer, then
[Equation]
where both minima are taken over all composable sequences of complex matrices A 1 , . .
A L such that A L · · · A 1 = B .
We ﬁrst deal with the elementary aspects of lemma D.18: it will sufﬁce to show that whenever
A L · · · [Equation] B ,
[Equation]
l | 22 and that
[Equation]
[Equation]
for some composable sequence A 1 , . . .
, A L such that A L · · · A 1 = B .
As noted in (Gunasekar et al., 2018)
the second inequality of eq.
(D.20) is simply the arithmetic-geometric mean inequality applied to | A 1 | 22 , . . .
|
A L | 22 ∈ R ≥ 0 .
Furthermore, we can obtain eq.
(D.21) using the singular value decomposition of B : let B =
USV ∗ where, letting r :=
min { m, n } , U ∈ U ( r, m ) and V ∈ U ( n, r ) are unitary and
S = diag
( λ 1 , . . .
, λ r ) ,
where λ
i ≥ 0 for all i .
We may decompose B into L factors like
[Equation]
and by unitary invariance of the C 2 (a.k.a.
Frobenius) norm,
[Equation]
We now note that by deﬁnition 4.6 ,
[Equation]
and on the other hand
[Equation]
so that combining eqs.
(D.23) and (D.24) gives eq.
(D.21) for the composable sequence of eq.
(D.22).
It remains to prove the ﬁrst inequality of eq.
(D.20) — this is a special case of the non-commutative generalized Hölder inequality lemma 4.11.
In fact we will prove a slightly more general statement — to state it we need a couple more deﬁnitions:
Deﬁnition D.25.
For a (not necessarily square) complex matrix A
[Equation]
where A ∗ is the conjugate transpose of A .
The matrix | A | is Hermitian and positive semi-deﬁnite, and often referred to as the polar part of A .
Given any Hermitian matrix H and complex number z , we may form the matrix H z ; explicitly it can be deﬁned as U diag( λ zi ) U ∗ where
H =
U diag( λ i )
U ∗ is a diagonalization of
H .
We next deﬁne unitary invariant norms on spaces of matrices.
For technical reasons to be encountered shortly, we actually introduce families of norms compatible with the natural inclusions
M ( m × n, C ) ⊆
M ( m (cid:48) × n (cid:48) , C ) for m (cid:48) ≥ m, n (cid:48) ≥ n .
Deﬁnition D.27 (cf.
(Bhatia, 1996, §IV)) .
A compatible family of matrix norms is a function
[Equation]
[Equation] such that
(i) the restriction of (cid:107)−(cid:107) to M ( m × n, C ) is a norm (in the sense of functional analysis) for all m, n ∈ N , and (ii) whenever m (cid:48) ≥ m, n (cid:48) ≥ n and ι : M ( m × n, C ) ⊆
M ( m (cid:48) × n (cid:48) , C ) is the “upper left block inclusion” sending
[Equation]
we have (cid:107) ι ( A ) (cid:107) =
(cid:107) A (cid:107) .
A compatible family of norms is unitary invariant if and only if for any A ∈ M ( m × n, C ) and any unitary matrices
U ∈
U ( m ) and V ∈ U ( n )
[Equation]
We will show below that for any p > 0 the Schatten p -norms form a unitary invariant compatible family.
Lemma D.31.
If B ∈ M ( m × n, C ) is a matrix with complex entries, A 1 , . .
A L is a composable sequence of complex matrices such that A L · · · A 1 = B and (cid:80) i 1 p 1 = 1 r where p 1 , . . .
, p L > 0 are positive real numbers, then for every unitary invariant compatible family of norms (cid:107)−(cid:107) ,
[Equation]
We note that this non-commutative generalized Hölder inequality is “non-commutative” since we work with products of matrices as opposed to inner products of vectors, and “generalized” since we consider (cid:96) p exponents p i where (cid:80) 1 p i > 1 .
To be speciﬁc, lemma 4.11 is the case of lemma D. where the unitary invariant compatible family of norms (cid:107)−(cid:107) consists of the C 1 norms 14 and p i
=for all i .
Here we are implicitly using the relationship between Schatten norms
[Equation]
As mentioned in section 4, when L = 2 lemma D.31 is (Bhatia, 1996, Exercise IV.2.7).
In the case of Schatten norms, it is essentially derived in the course of the proof of (Dai et al., 2021, Thm. 1).
Below, we solve (Bhatia, 1996, Exercise IV.2.7) and show that the case L > 2 follows by induction.
We ﬁrst address a subtle difference between lemma D.31 and the setup of (Bhatia, 1996) (and indeed most work on matrix analysis).
(Bhatia, 1996) considers square matrices throughout, whereas in lemma D.31 we allow all matrices to be rectangular — this is essential in our applications, since neural networks have variable width (even if we made our CNNs of section 3 “constant width” by requiring C
= C 2 = · · · = C L − 1 , the ﬁrst and last layers would still change width in general).
Thankfully, there is a simple trick that allows us to reduce to the case of square matrices.
Say
A l ∈ M ( m l × n l ) as in deﬁnition D.16, and let [Equation]
[Equation]
Since N ≥ m l , n l by deﬁnition, for each matrix A l we may deﬁne a new block diagonal matrix
[Equation]
35) (that is, we push A l to the top left corner).
Note that by deﬁnition D.16 and the hypotheses of lemma D.31 m = m L , n = n 1 and a straightforward inspection of the mechanics of block diagonal matrix multiplication shows
[Equation]
Now ˜
A 1 , . .
˜
A L and ˜ B are all square, and to reduce to the square case it will sufﬁce to argue that
[Equation]
i for all i and (cid:107)| ˜ B | r (cid:107) =
[Equation]
|
[Equation]
|
[Equation]
Since by hypothesis (cid:107)−(cid:107) is a compatible family of unitary invariant norms , it will sufﬁce to show that
[Equation]
) and this is follows from the identities
[Equation]
(cid:21)
[Equation]
(cid:21) =
(cid:20) A
[Equation]
A  (cid:21) and (cid:20) H   (cid:21) z = (cid:20) H z   (cid:21) (D.39) valid for any complex matrix A , Hermitian matrix H and complex number z , the proofs of which we omit.
Before continuing with the proof of lemma D.31, we pause to verify that Schatten norms are indeed a unitary invariant compatible family.
In doing so we prove a lemma that will be of further use in the sequel.
Deﬁnition D.40.
A compatible family of guage functions is a function Φ : (cid:96) n R n → R ≥ 0 such that
(i) for each n ∈ N the restriction of Φ to R n is a norm (in the sense of functional analysis) and (ii) whenever n (cid:48) ≥ n and ι : R n → R n (cid:48) is the inclusion mapping ( x 1 , . . .
, x n ) (cid:55)→ ( x 1 , . . .
x n , 0 , . . . , 0) , we have Φ( ι ( x ))
=
Φ( x ) .
A compatible family of guage functions is symmetric if and only if for each n ∈ N the restriction of Φ to R n is invariant under the action of matrices of the form PD where P is a permutation and D is diagonal, with diagonal entries in {± 1 } .
Lemma D.41 (cf.
(Bhatia, 1996, Thm.
IV.2.1)) .
There is a natural one-to-one correspondence between unitary invariant compatible families of matrix norms and symmetric compatible families of guage functions.
Proof.
Given a unitary invariant compatible family of matrix norms (cid:107)−(cid:107) , a symmetric compatible family of guage functions Φ can be deﬁned using the maps
[Equation]
compatibility of Φ comes from compatibility of (cid:107)−(cid:107) and the identity “ diag ◦ ι = ι ◦ diag ” (suitably interpreted), and symmetry of Φ follows from the identity diag( PDx ) =
( PD ) diag( x )( PD ) ∗ for matrices PD as above, the fact such matrices PD are unitary and unitary invariance of (cid:107)−(cid:107) .
Conversely, given a symmetric compatible family of guage functions Φ one may deﬁne a unitary invariant compatible family of matrix norms (cid:107)−(cid:107) as (cid:107) A (cid:107) :
= Φ( s ( A )) where s ( A ) denotes the singular values of A .
Compatibility comes from the fact that the singular values of a block diagonal matrix
[Equation]
are the concatenation of s ( A 1 ) and s ( A 2 ) , and unitary invariance follows from the fact that singular values unitary invariant up to permutations (and Φ is symmetric).
The proof that the maps (cid:107)−(cid:107) :
M ( m × n, C )
→ R ≥ 0 are indeed norms is as in (Bhatia, 1996, Thm.
IV.2.1).
It can be veriﬁed that these maps are mutual inverses — we omit this ﬁnal step.
Corollary D.43.
For any p > 0 the Schatten p -norms form a unitary invariant compatible family of matrix norms.
Proof.
By deﬁnition , (cid:107) A (cid:107)
Sp =
( (cid:80) i s ( A ) pi ) 1 p , i.e. the (cid:96) p norm of the singular values of A .
By lemma D.41 it sufﬁces to show that the (cid:96) p -norms form a symmetric compatible family — this is straightforward and omitted.
We now resume proving lemma D.31, ﬁrst in the case L = 2 (later we will prove the general case by induction).
Recall that at this point we have reduced to the case where all matrices in sight are n × n for some ﬁxed n ∈ N , so in particular we are dealing with a ﬁxed unitary invariant norm (no further need for compatible families).
By the above lemma, (cid:107) A (cid:107) =
Φ( s ( A )) for some symmetric guage function Φ , for all A ∈ M ( n × n, C ) .
By (Bhatia, 1996, Thm. IV.2.5),
[Equation]
<
[Equation]
s [Equation] r (D.44) where < w denotes weak submajorization.
Now the “strongly isotone” property of the symmetric guage function Φ implies
[Equation]
We need a generalized Hölder inequality for symmetric guage functions.
Lemma D.46 ((Bhatia, 1996, Ex. IV.1.7)) .
If p 1 , p 2 > 0 and 1 p 1 + 1 p 2 = 1 r then for every symmetric guage function Φ : R n → R ≥ 0 and every x, y ∈ R n we have
[Equation]
where abs denotes the coordinatewise absolute value.
Proof.
Apply the regular r = 1 Hölder inequality (Bhatia, 1996, Thm. IV.1.6) to the vectors ˜ x = abs( x ) r , ˜ y =
abs( y ) r
(coordinatewise r -th powers) with the exponents ˜ p 1 = p 1 /r, ˜ p 2 =
p 2 /r (note that 1˜ p
+ 1˜ p 2 = rp
+ rp 2 = 1 ) to obtain
[Equation]
where in the last equality we have just used the deﬁnitions of ˜ x, ˜ y, ˜ p 1 and ˜ p 2 .
Taking r -th roots completes the proof.
Now applying lemma D.46 to eq.
(D.45) gives
[Equation]
(the ﬁrst inequality is just taking r -th roots of eq.
(D.45), the second is applying lemma D.46).
Using the identity s ( | A | r )
=
s ( A ) r , we ﬁnally obtain
[Equation]
which is lemma D.31 when L = 2 .
Now suppose L > 2 and assume by inductive hypothesis that lemma D.31 holds for all smaller values of L .
Deﬁne
[Equation]
.51) (note that 1 p L + 1 q = 1 r ).
By the L = 2 case of lemma D.
[Equation]
and by inductive hypothesis
[Equation]