P HASE C OLLAPSE IN N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Deep convolutional classiﬁers separate linearly image classes, with an improving accuracy as depth increases. The spatial dimension is progressively reduced whereas the number of channels grows with depth. Spatial variability is therefore transformed into variability along channels.
A fundamental challenge is to understand the role of non-linearities together with convolutional ﬁlters in this transformation.
ReLUs with biases are often interpreted as thresholding operators that improve discrimination through sparsity.
This paper demonstrates that it is a different mechanism called phase collapse which eliminates spatial variability while linearly separating classes.
This is explained and numerically demonstrated by deﬁning a simpliﬁed complex-valued convolutional network architecture.
It implements spatial convolutions with wavelet ﬁlters and uses a complex modulus to collapse phase variables.
This phase collapse network reaches the classiﬁcation accuracy of ResNets of similar depths. Its performance is considerably degraded when replacing the phase collapses with thresholding operators that enforce sparsity.
This is justiﬁed by explaining how the iteration of phase collapses progressively improve separation of class means, as opposed to thresholding non-linearities.
I NTRODUCTION
CNN image classiﬁers progressively eliminate spatial variables through iterated ﬁlterings and subsamplings, while linear classiﬁcation accuracy improves as depth increases (Oyallon, 2017).
It has also been observed numerically that CNNs concentrate training samples of each class in small separated regions of a progressively lower-dimensional space.
It can ultimately produce a neural collapse (Papyan et al., 2020), where all training samples of each class are mapped to a single point.
In this case, the elimination of spatial variables comes with a collapse of within-class variability and perfect linear separability.
This increase in linear classiﬁcation accuracy is obtained in standard CNN architectures like ResNets from the iteration of linear convolutional operators and ReLUs with biases.
A difﬁculty in understanding the underlying mathematics comes from the ﬂexibility of ReLUs.
Any non-linearity can indeed be approximated by a linear combination of biased ReLUs.
Many papers interpret iterations on ReLUs and linear operators as sparse code computations (Sun et al., 2018; Sulam et al., 2018; 2019; Mahdizadehaghdam et al., 2019; Zarka et al., 2020; 2021).
We show that it is a different mechanism, called phase collapse , which underlies the increase in classiﬁcation accuracy of these architectures. A phase collapse is the elimination of phases of complex-valued network coefﬁcients with a modulus, which we show to concentrate spatial variability.
This is demonstrated by introducing a structured convolutional neural network, with wavelet ﬁlters and no biases.
Section 2 introduces and explains phase collapses, and contrasts them with thresholding operators.
Complex-valued representations are used because they reveal the mathematics of spatial variability. Indeed, translations are diagonalized in the Fourier basis, where they become a complex phase shift.
Invariants to translations are computed with a modulus which collapses the phases of this complex representation.
Section 2 explains how this can improve linear classiﬁcation. Phase collapses can also be calculated with ReLUs and real ﬁlters.
A CNN with complex-valued ﬁlters is indeed just a particular instance of a real-valued CNN, whose channels are paired together to deﬁne complex numbers.
Section 3 demonstrates the role of phase collapse in deep classiﬁcation architectures, as opposed to thresholdings or more general amplitude reduction operators.
It introduces a Learned Scattering network with phase collapses. This network applies a learned 1 × 1 convolutional complex operator P j on each layer x j , followed by a complex wavelet ﬁltering operator W and a phase collapse:
[Equation]
It does not use biases, which are also not needed in deep networks for image denoising (Mohan et al., 2019). This network architecture is illustrated in Figure 1.
With the addition of skip-connections, we show that this phase collapse network reaches ResNet accuracy on ImageNet and CIFAR-10.
On the contrary, the accuracy is considerably reduced when replacing the phase collapse modulus by soft-thresholdings with learned biases, which can enforce sparsity but do not modify the phase. This is also true of more general phase-preserving non-linearities.
Section 4 explains the performance of iterated phase collapses by showing that each phase collapse progressively improves linear discriminability.
On the opposite, the improvements in classiﬁcation accuracy of successive sparse code computations are shown to quickly saturate.
The main contribution of this paper is a demonstration that the classiﬁcation accuracy of deep neural networks mostly relies on phase collapses, which are sufﬁcient to linearly separate the different classes on complex image databases.
This is captured by the Learned Scattering architecture which reaches
ResNet-18 accuracy on ImageNet and CIFAR-10.
We also show that phase collapses are necessary to reach this accuracy, by showing numerically and theoretically that iterating phase-preserving non-linearities leads to a signiﬁcantly worse performance. × 1 P J reduces the dimension before a linear classiﬁer.
P HASE C OLLAPSE V ERSUS T HRESHOLDINGS AND R E LU S
Deep convolutional classiﬁers achieve linear separation of image classes. We show that linear classiﬁcation on raw images has a poor accuracy because image classes are invariant to local translations. This geometric within-class variability takes the form of random phase ﬂuctuations, which induce zero class means.
To improve classiﬁcation accuracy, non-linear operators must therefore eliminate these ﬂuctuations with phase collapses, so as to separate class means.
We also analyze the effect of other non-linearities such as soft-thresholdings and ReLUs.
Translations and phase shifts Translations capture the spatial topology of the grid on which the image is deﬁned.
They are transformed into phase shifts by a Fourier transform. We prove that this remains approximately valid for images convolved with appropriate complex ﬁlters.
Let x be an image indexed by u ∈ Z 2 .
We write x τ ( u ) =
x ( u − τ ) the translation of x by τ .
It is diagonalized by the Fourier transform (cid:98) x ( ω ) =
(cid:80) u
x ( u ) e − iω · u , which creates a phase shift:
[Equation]
This diagonalization explains the need to introduce complex numbers to analyze the mathematical properties of geometric within-class variabilities. Computations can however be carried with real numbers, as we will show.
A Fourier transform is computed by ﬁltering x with complex exponentials e iω · u .
One may replace these by complex ﬁlters ψ that are localized in space and in the Fourier domain.
The following theorem proves that small translations can still be approximated by a phase shift in this case.
A convolution is written ∗ .
Theorem 1.
Let ψ : Z 2 → C be a ﬁlter with (cid:107) ψ (cid:107) 2 = 1 , whose center frequency ξ and bandwidth σ are deﬁned by:
[Equation]
Then, for any τ ∈ Z 2 ,
[Equation]
The proof is in Appendix B.
This theorem proves that if | τ | (cid:28) 1 /σ then x τ ∗ ψ ≈ e − iξ · τ x ∗ ψ .
In this case, a translation by τ produces a phase shift by ξ · τ .
Phase collapse and stationarity We deﬁne a phase collapse as the elimination of the phase of a complex number with a modulus.
We now show that phase collapses improve linear classiﬁcation of classes that are invariant to translations or to local translations.
Each image class indexed by y may be represented by a random vector X y , whose realizations are the class samples.
To achieve linear separation, it is sufﬁcient that class means E (cid:2)
X y (cid:3) are sufﬁciently separated, and that each class mean can be estimated from a single realization, with a sufﬁciently small variance E [ (cid:13) (cid:13) X y − E (cid:2)
X y (cid:3)(cid:13)(cid:13) 2 ] (Hastie et al., 2009).
The goal of classiﬁcation is to ﬁnd a representation of the input images in which these properties hold.
To simplify the analysis, we consider the particular case where each class y is invariant to translations. In this case, each random vector X y is stationary, which means that its probability distribution is invariant to translations.
Equation (2) then implies that the phases of Fourier coefﬁcients of X y are uniformly distributed in [0 , 2 π ] , leading to E [ (cid:98) X y ( ω )] = 0 for ω (cid:54) = 0 .
If class-average pixel values of images are the same, then E [ (cid:98) X y (0)] does not depend on y either. A linear classiﬁer thus cannot discriminate such translation-invariant classes, as class means are all equal.
A phase collapse is then necessary to create non-zero means from these Fourier coefﬁcients.
It deﬁnes translation-invariant coefﬁcients: | (cid:98) x τ | = | (cid:98) x | . It improves linear discriminability of stationary classes, because E [ | (cid:98) X y ( ω ) | ] may be different for different y .
However, one can not estimate accurately the value of E [ | (cid:98) X y ( ω ) | ] at all frequencies ω from a single image. Indeed, there are as many frequencies as the number of image pixels.
Moreover, the Fourier transform is unstable to small deformations (Bruna and Mallat, 2013) at high frequencies.
This explains the limited performances of a Fourier transform modulus for linear classiﬁcation.
Fourier modulus descriptors can be improved by using ﬁlters ψ that have a localized support in space.
If (cid:80) u ψ ( u ) = 0 , then E [ X y ∗ ψ ] =
if X y is stationary.
Theorem 1 shows that the phase of X y ∗ ψ is again uniformly distributed in [0 , 2 π ] .
A phase collapse similarly computes approximate invariants to small translations: | x τ ∗ ψ | ≈ | x ∗ ψ | with an error bounded by σ | τ | (cid:107) x (cid:107) 2 .
This representation improves linear classiﬁcation by creating different non-zero class means E [ | X y ∗ ψ | ] . It deﬁnes better descriptors than Fourier coefﬁcients because E [ | X y ∗ ψ | ] can be accurately estimated from a single image with a spatial average of | X y ∗ ψ ( u ) | , if the support of ψ is not too large. Moreover,these expected values are stable to deformations when the support of ψ is small (Bruna and Mallat, 2013). Image classes are usually not invariant to global translations.
For instance, subjects are generally centered or the sky is usually at the top. However, the classes are often invariant to local translations, up to an unknown maximum scale. This is captured by the notion of local stationarity, which means that the probability distribution of X y is nearly invariant to translations smaller than some maximum scale (Priestley, 1965).
The above discussion remains valid if X y is only locally stationary over a domain larger than the support of ψ . The use of so-called “windowed absolute spectra” E (cid:2)(cid:12)(cid:12) X y ∗ ψ (cid:12)(cid:12)(cid:3) for locally stationary processes has previously been studied in Tygert et al.
Complex versus real in CNNs: phases and ReLUs
The use of complex numbers is a mathematical abstraction to diagonalize translations, which are captured by complex phases.
It provides a mathematical interpretation of ﬁltering operations performed on real numbers.
We show that a real network can still implement complex phase collapses.
In the ﬁrst layer of a CNN, one can observe that ﬁlters are usually oscillatory patterns with small supports, where some ﬁlters have nearly the same orientations and frequency but have a phase shifted by some α (Krizhevsky et al., 2012). We reproduce in Appendix F a ﬁgure from Shang et al. (2016) which evidences this phenomenon.
This means that such ﬁlters can be written ψ α
= Re( e − iα ψ ) for a single complex ﬁlter ψ and different α .
Grouping such channels speciﬁes the complex ﬁlter ψ .
A CNN with complex ﬁlters is thus a structured real-valued CNN, whose channels correspond to phase shifts of these complex ﬁlters.
This structure simpliﬁes the mathematical interpretation of non-linearities by explicitly deﬁning the phase, which is otherwise a hidden variable relating multiple ﬁlter outputs within each layer.
A phase collapse is computed with a modulus.
It can be calculated by real-valued CNNs by applying
ReLUs without biases over ﬁlters having different phases: ψ α = Re( e − iα ψ ) .
Indeed, for any real-valued signal x , we have:
[Equation]
Furthermore, this integral is well approximated by a sum over 4 phases, allowing to compute complex moduli with ReLUs without biases over real-valued CNN ﬁlters. See Appendix A for a proof of eq.
(4) and its approximation. Thresholding and sparsity versus phase collapse The action of a ReLU can be linearly decomposed into an elimination of sign and a soft-thresholding, which attenuates the amplitude.
As both can improve linear classiﬁcation in different cases, an important issue is to understand which property is important for classiﬁcation.
If z ∈ R and b ≥ 0 , then the even part of ReLU( z − b ) is ReLU( z − b ) + ReLU( − z − b ) , which is an absolute value with a dead-zone [ − b, b ] .
When b = 0 , it becomes a phase collapse | z | .
The odd part is a soft-thresholding ρ b ( z ) =
ReLU( z − b ) − ReLU( − z − b ) .
A soft-thresholding operator over a complex z = | z | e iϕ reduces the amplitude by b while preserving the phase: [Equation]
Similarly to its real counterpart, this amounts to deﬁning the complex soft-thresholding as the proximal operator of the complex modulus (Yang et al., 2012):
[Equation]
Soft-thresholdings and phase collapses have opposite properties, since soft-thresholdings preserve the phase while attenuating the amplitude, whereas phase collapses preserve the amplitude while eliminating the phase.
We have explained how phase collapses can improve the classiﬁcation accuracy of locally stationary processes by separating class means E (cid:2)(cid:12)(cid:12)
X y ∗ ψ (cid:12)(cid:12)(cid:3) . In contrast, if X y ∗ ψ has a distribution which is symmetric around 0 then E (cid:2) ρ b ( X y ∗ ψ ) (cid:3) = 0 for all b because the thresholding preserves the phase and hence this symmetry around 0 .
This symmetry is often observed when (cid:80) u ψ ( u ) = 0 , in which case class means are not separated by a thresholding.
If E (cid:2) X y ∗ ψ (cid:3) (cid:54) = [-
-]0 and X y ∗ ψ is sparse, then a soft-thresholding of X y ∗ ψ may reduce the variance of class mean estimators (Donoho and Johnstone, 1994).
In this case, it can improve linear classiﬁcation.
Coefﬁcients below the threshold may be assimilated to unnecessary “clutter” which is set to 0 .
To improve classiﬁcation, convolutional ﬁlters must produce high-amplitude coefﬁcients corresponding to discriminative “features”, whereas small-amplitude clutter coefﬁcients are eliminated (Zarka et al., 2021).
The dimensionality of sparse representations can also be reduced with random ﬁlters which implement a form of compressed sensing (Donoho, 2006; Candes et al., 2006).
The interpretation of
CNNs as compressed sensing machines with random ﬁlters has been studied (Giryes et al., 2015), but it never led to classiﬁcation results close to e.g. ResNet accuracy.
On the contrary, we shall see that the use of phase collapse alone reaches this performance.
L EARNED S CATTERING N ETWORK WITH P HASE C OLLAPSE
This section introduces a learned scattering transform, which reaches ResNet accuracy on the
ImageNet (Russakovsky et al., 2015) and CIFAR-10 (Krizhevsky, 2009) datasets.
It is a highly structured CNN architecture computed with wavelet ﬁlters and phase collapses.
We describe this architecture and then compare the performance of phase collapses and thresholding non-linearities.
Scattering transform Theorem 1 proves that a phase collapse applied to the output of a complex ﬁlter produces a locally invariant descriptor that can then be subsampled, depending upon the ﬁlter’s bandwidth.
We brieﬂy review the scattering transform (Mallat, 2012; Bruna and Mallat, 2013), which iterates such convolutions with wavelet ﬁlters and moduli.
A scattering transform over J scales is implemented with a network of depth J , whose ﬁlters are speciﬁed by the choice of wavelet.
Let x 0 = x .
For 0 ≤ j < J , the j + 1 -th layer x j +1 is computed by applying a wavelet ﬁltering operator W and a modulus on the j -th layer x j :
[Equation]
The operator W is deﬁned with Morlet ﬁlters (Bruna and Mallat, 2013).
It has one low-pass ﬁlter g 0 , and L zero-mean complex band-pass ﬁlters g (cid:96) , having an angular direction (cid:96)π/L for 0 < (cid:96) ≤ L .
It thus transforms an input image x ( u ) into L + 1 sub-band images which are subsampled by 2 : Wx ( u, (cid:96) )
=
x ∗ g (cid:96) (2 u ) . The cascade of j low-pass ﬁlters g 0 with a ﬁnal band-pass ﬁlter g (cid:96) , each followed by a subsampling, computes wavelet coefﬁcients at each scale 2 j .
One can also modify the wavelet ﬁltering W to compute intermediate scales 2 j/ 2 , as explained in Appendix E.
The spatial subsampling is then only implemented every other layer, and the depth of the network becomes twice larger.
One can restrict the channels of x J to the ones obtained from less than 2 moduli (Bruna and Mallat, 2013).
Applying a linear classiﬁer on such a scattering transform classiﬁer gives good results on simple classiﬁcation problems such as MNIST (LeCun et al., 2010).
However, results are well below ResNet performances on CIFAR-10 and ImageNet, as shown in Table 1.
Learned Scattering Zarka et al.
(2021) showed that a scattering transform can reach ResNet accuracy by alternating wavelet transform ﬁlters with two learned 1 × 1 convolutional operators.
The ﬁrst is preceded by a ReLU with no bias, and the second is followed by a soft-thresholding non-linearity.
However, this learned scattering architecture is difﬁcult to interpret in the context of phase collapses versus thresholdings, because it mixes both.
In line with several papers on the use of sparsity for classiﬁcation (Sun et al., 2018;
Sulam et al., 2018; 2019; Mahdizadehaghdam et al., 2019; Zarka et al., 2020), Zarka et al.
(2021) made the hypothesis that sparsifying neural responses with thresholdings is a major mechanism for improving classiﬁcation accuracy. Our
Learned Scattering architecture with phase collapse modulus shows that it is not necessary to use any biases nor thresholdings.
We demonstrate numerically that high image classiﬁcation performance mostly results from iterated phase collapses which separate class means.
The Learned Scattering (LScat) network inserts in eq. (7) a learned complex 1 × 1 convolutional operator P j which reduces the channel dimensionality of each layer x j , before applying wavelet ﬁlters and a phase collapse:
[Equation]
Similar architectures which separate space-mixing and channel-mixing operators had previously been studied in the context of basis expansion (Qiu et al., 2018; Ulicny et al., 2019) or to ﬁlter scattering channels (Cotter and Kingsbury, 2019).
This separation is also a major feature of recent architectures such as Vision Transformers (Dosovitskiy et al., 2021) or MLP-Mixer (Tolstikhin et al., 2021).
Each P j computes discriminative channels whose spatial variability is eliminated by the phase collapse of complex wavelet coefﬁcients.
Their role is further discussed in Section 4.
Table 1 gives the accuracy of a linear classiﬁer applied to the last layer of this Learned Scattering.
It provides an important improvement over a scattering transform, but it does not yet reach the accuracy of ResNet-18.
The architecture uses a total number of layers J = 11 for ImageNet and J = 8 for CIFAR, by introducing intermediate scales 2 j/ 2 .
The number of channels of P j x j are the same as in a standard ResNet architecture (He et al., 2016) and remain no larger than 512 .
Details of this Learned Scattering architecture are provided in Appendix E.
Skip-connections across moduli The architecture of eq.
(9) imposes that all phases are collapsed at each layer, after computing a wavelet transform.
More ﬂexibility is provided by adding a skipconnection which concatenates WP j x j with its modulus:
[Equation]
[Table]
The skip-connection produces a cascade of convolutional ﬁlters W without non-linearities in-between.
The resulting convolutional operator WW · · · W is a “wavelet packet” transform which generalizes the wavelet transform (Coifman and Wickerhauser, 1992; Mallat, 2008).
Wavelet packets are obtained as the cascade of low-pass and band-pass ﬁlters ( g (cid:96) ) (cid:96) , each followed by a subsampling.
Besides wavelets, wavelet packets include ﬁlters having a larger spatial support than wavelets and a narrower Fourier bandwidth.
A wavelet packet transform is then similar to a local Fourier transform.
Applying a modulus on such wavelet packet coefﬁcients deﬁnes local spatial invariants over wider domains.
As discussed in Section 2, image classes are usually invariant to local rather than global translations.
Section 2 explains that a phase collapse improves discriminability for image classes that are locally translation-invariant over the ﬁlter’s support.
Indeed, phases of wavelet coefﬁcients are then uniformly distributed over [0 , 2 π ] , yielding zero-mean coefﬁcients for all classes.
At a scale where there is no local translation-invariance, these phases are no longer uniformly distributed, and they encode information about the spatial localization of features.
Introducing a skip-connection provides the ﬂexibility to choose whether to eliminate phases at different scales or to propagate them up to the last layer. Indeed, the next 1 × 1 operator P j +1 linearly combines (cid:12)(cid:12) WP j x j (cid:12)(cid:12) and
WP j x j and may learn to use only one of these.
This adds some localization information, which appears to be important.
Table 1 shows that the skip-connection improves classiﬁcation accuracy.
Applying a linear classiﬁer on this Learned Scattering reaches ResNet-18 accuracy on CIFAR-10 and ImageNet.
It demonstrates that collapsing appropriate phases is sufﬁcient to obtain 89% Top-5 accuracy on large-scale classiﬁcation problems.
Learning is reduced to 1 × 1 ﬁlters ( P j ) j across channels.
Phase collapse versus amplitude reduction A Learned Scattering with phase collapse preserves the amplitudes of wavelet coefﬁcients and eliminates their phases.
Section 2 explains that it improves separation of class means.
On the opposite, one may use a non-linearity which preserves the phases of wavelet coefﬁcients but attenuates their amplitudes.
Among such non-linearities is the softthresholding, which can improve classiﬁcation by reducing the variance of class mean estimators, within a sparse representation.
These sparse representations are usually considered important for image classiﬁcation (Sun et al., 2018; Sulam et al., 2018; 2019; Mahdizadehaghdam et al., 2019).
The following results show that it is not the case.
Several amplitude reduction non-linearities are used, including soft-thresholdings, sigmoids, and an amplitude collapse.
We show that it considerably degrades the accuracy compared to a phase collapse, to the point that the classiﬁcation error becomes larger than with a scattering without learning.
A Learned Amplitude Reduction Scattering applies a non-linearity ρ ( z ) on each wavelet coefﬁcient z = | z | e iϕ and preserves the phase: ρ ( z ) =
e iϕ ρ ( | z | ) .
Without skip-connections, each layer x j +1 is computed from x j by:
[Equation]
and with skip-connections:
[Equation]
A soft-thresholding is deﬁned by ρ ( | z | ) =
ReLU( | z | − b ) for some threshold b .
For an amplitude sigmoid we have ρ ( | z | ) =
(1 + e − a | z |− b ) − 1 , and an amplitude collapse is ρ ( | z | )
= 1 .
The softthresholding and sigmoid parameters a and b are learned for each layer and each channel.
[Table] The classiﬁcation performance of a Learned Amplitude Reduction Scattering is evaluated on CIFAR10, by applying a linear classiﬁer on the last layer.
Classiﬁcation results are given in Table 2 for different amplitude reductions, with or without skip-connections.
Learned Amplitude Reduction Scatterings yield much larger errors than a Learned Scattering with phase collapses. They are even above a scattering transform which also uses phase collapses but without learned 1 × 1 convolutional projections P j .
It demonstrates that high accuracies result from phase collapses without biases, as opposed to amplitude reduction operators including thresholdings, which learn bias parameters.
ReLUs with biases Most CNNs, including ResNets, use ReLUs with biases.
As explained in
Section 2, a ReLU with bias simultaneously affects the sign and the amplitude of a real coefﬁcient.
When decomposing it into its even and odd part, we separate the effects on sign and on amplitude.
Over complex numbers, it amounts to transforming the phase or the amplitude.
These numerical experiments show that accuracy improvements result from phase collapses rather than amplitude reductions, which is a surprising new result.
Several CNN architectures have demonstrated a good classiﬁcation accuracy with iterated thresholding algorithms, which increase sparsity.
However, all these architecture also modiﬁed the sign of coefﬁcients by computing non-negative sparse codes (Sun et al., 2018; Sulam et al., 2018; Mahdizadehaghdam et al., 2019) or with additional ReLU or modulus layers (Zarka et al., 2020; 2021).
It seems that it is the sign or phase collapse of these non-linearities which is responsible for good classiﬁcation accuracies, as opposed to the calculation of sparse codes through iterated amplitude reductions.
I TERATING P HASE C OLLAPSES AND A MPLITUDE R EDUCTIONS
We now provide a theoretical justiﬁcation to the above numerical results in simpliﬁed mathematical frameworks.
This section studies the behavior of phase collapses and amplitude reductions when they are iterated over several layers.
It shows that phase collapses beneﬁt from iterations over multiple layers, whereas there is no signiﬁcant gain in performance when iterating amplitude reductions.
4. I TERATED P HASE C OLLAPSES
We explain the importance of phase collapses to improve linear classiﬁcation, as well as the role of iterated phase collapses with multiple ﬁlters at each layer, together with learned projectors P j .
For this purpose, we consider the classiﬁcation of ergodic stationary processes X y ∈ R d corresponding to different image textures indexed by y .
Given a realization x of X y , and because of ergodicity, the optimal linear classiﬁer is calculated from the empirical mean 1 /d (cid:80) u x ( u ) .
It computes an optimal linear estimation of E (cid:2)
X y ( u ) (cid:3) = µ y .
If all textures have the same mean µ y = µ , then all linear estimators fail.
More precisely, the textures can be linearly separated only if the distance between different means (cid:107) µ y − µ y (cid:48) (cid:107) is greater than the average estimation error of the empirical means.
As explained in Section 2, linear classiﬁcation can be improved by computing ( | x ∗ ψ k | ) k for some ﬁlters ( ψ k ) k
such that (cid:80) u ψ k ( u ) = 0 , because phase collapses create non-zero means.
If X y [-∗ ψ k (cid:3) = 
 but E (cid:2)-]is stationary, then | X y ∗ ψ k | remains stationary for any ψ k .
An optimal linear classiﬁer applied to ( | x ∗ ψ k ( u ) | ) k is thus obtained by a linear combination of all empirical means (1 /d (cid:80) u | x ∗ ψ k ( u ) | ) k .
They are proportional to the (cid:96) 1 norm (cid:107) x ∗ ψ (cid:107) 1 , which is a measure of sparsity of x ∗ ψ k .
If linear classiﬁcation on ( | x ∗ ψ k ( u ) | ) k fails, it reveals that the means E (cid:2) | X y ∗ ψ k ( u ) | (cid:3) =
µ y,k are not sufﬁciently different.
Separation can be improved by considering the spatial variations of | X y ∗ ψ k ( u ) | for different y .
These variations can be revealed by a new set of ﬁlters ψ k (cid:48) having a zero mean, such as a new set of wavelet ﬁlters.
Since E (cid:2) | X y ∗ ψ k | ∗ ψ k (cid:48) (cid:3) = 0 , we must again cancel the phase and compute E (cid:2)
||
X y ∗ ψ k | ∗ ψ k (cid:48) | (cid:3) .
This phase collapse iteration is the principle used by scattering transforms to discriminate textures (Bruna and Mallat, 2013; Sifre and Mallat, 2013).
However, this may still not be sufﬁciently accurate.
More discriminant statistical properties may be obtained by linearly combining | X y ∗ ψ k | across k before applying a new ﬁlter ψ k (cid:48) .
In a Learned Scattering with phase collapse, the x ∗ ψ k are combined with a linear projector Pacross the channel indices k , before computing a convolution with the next ﬁlter ψ k (cid:48) .
The 1 × 1 ﬁlter
P 1 is optimized to improve the linear classiﬁcation accuracy.
It amounts to computing weights w k such that E [ (cid:12)(cid:12)(cid:80) k w k (cid:12)(cid:12)
X y ∗ ψ k (cid:12)(cid:12) ∗ ψ k (cid:48)
(cid:12)(cid:12) ] is as different as possible for different y .
Because these are proportional to the (cid:96) 1 norms (cid:13)(cid:13)(cid:80) k w k | x ∗ ψ k | ∗ ψ k (cid:48)
(cid:13)(cid:13) 1 , it means that the images (cid:80) k w k | x ∗ ψ k |∗ ψ k (cid:48) have different sparsity levels depending upon the class y of x .
The ﬁlter weights of P 1 can thus be interpreted as features along channels providing different sparsiﬁcations for different classes.
A Learned Scattering computes such P j at each scale j .
4. I TERATED A MPLITUDE R EDUCTIONS
Sparse representations and amplitude reduction algorithms may improve linear classiﬁcation by reducing the variance of class mean estimations, which can be interpreted as clutter removal.
Such approaches are studied in Zarka et al.
by modeling the clutter as an additive white noise.
Although a single thresholding step may improve linear classiﬁcation, we show that iterating more than one thresholding does not improve the classiﬁcation accuracy, if no phase collapses are inserted.
To understand these properties, we consider the discrimination of classes represented by non-stationary random processes X y ( u ) .
We suppose that class means are linearly separable and hence that all E ( X y ) =
µ y are different for different y .
If there exists y (cid:48) such that (cid:107) µ y − µ y (cid:48) (cid:107) is small, then the class y can still be discriminated, if we can estimate E ( X y ) sufﬁciently accurately from a single realization x of X y .
This is a mean estimation problem.
Suppose that X y = µ y + N (0 , σ 2 ) is contaminated with Gaussian white noise, where the noise models some clutter.
Suppose also that there exists a linear orthogonal operator D such that Dµ y is sparse for every y , and hence has its energy concentrated in few non-zero coefﬁcients.
Such a D may be computed by minimizing the expected (cid:96) 1 norm (cid:80) y E (cid:2) (cid:107) DX y (cid:107) 1 (cid:3) .
The estimation of µ y can be improved with a soft-thresholding estimator (Donoho and Johnstone, 1994), which sets to zero all coefﬁcients below a threshold b proportional to σ .
It amounts to computing ρ b ( Dx ) , with ρ b ( x ) =
ReLU( x − b ) − ReLU( − x − b ) .
However, we explain why this approach cannot be further iterated without inserting phase collapses.
The reason is that sparsity and entropy of phases are linked, in the sense that a sparse representation concentrates its entropy in the phases of the coefﬁcients, rather than their amplitude.
If the entropy is concentrated in the phases, then the process cannot be further sparsiﬁed.
The entropy of the phases must then be reduced before the next sparsiﬁcation.
This entails that a model of intra-class variability relying on sparsity and amplitude reduction cannot be the sole mechanism behind the performance of deep networks.
Iterating sparsity may however be useful if it is alternated with another non-linearity which partly or fully collapses phases.
As mentioned in Section 3, this is the case for previous work which used iterated sparsiﬁcation operators (Sun et al., 2018; Sulam et al., 2018; Mahdizadehaghdam et al., 2019).
Indeed, these networks compute non-negative sparse codes where sparsity is enforced with a ReLU, which acts both on phases and amplitudes.
Our results shows that the beneﬁt of iterating non-negative sparse coding comes from the sign collapse due to the non-negativity constraint.
We now qualitatively demonstrate the two directions of this link with two theorems.
We ﬁrst show that ﬁnding the sparsest representation of a random process (i.e., minimizing its (cid:96) 1 norm) is the same as maximizing a lower bound on the entropy of its phases.
Theorem 2.
Let X denote a random vector in C d with a probability density p .
Let H ( X ) be the entropy of X with respect to the Lebesgue measure:
[Equation]
If D ∈ U ( d ) is a unitary operator then:
[Equation]
where ϕ ( DX ) ∈ [0 , 2 π ] d (resp.
| DX | ∈ R d + ) is the random process of the entry-wise phases (resp. moduli) of DX .
The proof is in Appendix C.
This theorem gives a lower-bound of the conditional entropy of the phases of DX with a decreasing function of the expected (cid:96) 1 norm of DX .
Minimizing over D this expected (cid:96) 1 norm amounts to maximizing the lower bound on H (cid:16) ϕ ( DX ) (cid:12)(cid:12)(cid:12)
| DX | (cid:17) .
An extreme situation arises when this entropy reaches its maximal value of d log(2 π ) .
In this case, the phase ϕ ( DX ) has a maximum-entropy distribution and is therefore uniformly distributed in [0 , 2 π ] d . Moreover, in this extreme case ϕ ( DX ) is independent from | DX | , since its conditional distribution does not depend on | DX | .
Such statistical properties have previously been observed on wavelet coefﬁcients of natural images (Rao et al., 2001), where the wavelet transform seems to be a nearly optimal unitary dictionary to enforce sparsity.
The second theorem considers the extreme case of a random process whose phases are conditionally independent and uniform. It proves show that such a process cannot be signiﬁcantly sparsiﬁed with a change of basis.
Theorem 3.
Let X be a random vector of C d such that ϕ ( X ) is uniformly distributed in [0 , 2 π ] d and independent from | X | .
Then there exists a constant C d > 0 which depends on the dimension d , such that for any D ∈ U ( d ) ,
[Equation]
DX (cid:107) 1 ] ≥ C d E [ (cid:107) X (cid:107) 1 ] . The proof is in Appendix D. This theorem shows that random processes with conditionally independent and uniform phases have an (cid:96) 1 norm which cannot be signiﬁcantly decreased by any unitary transformation.
Numerical evaluations suggest that the constant C d may be chosen to be √ π/ 2 ≈ 0 .
886 , independently of the dimension d .
This constant arises as the value of E [ | Z | ] when Z is a complex normal random variable with E [ | Z | 2 ] = 1 .
These two theorems explain qualitatively that linear classiﬁcation on ρ b ( Dx ) cannot be improved by another thresholding that would take advantage of another sparsiﬁcation operator.
Indeed, Theoremshows that if Dx is sparse, then its phases have random ﬂuctuations of high entropy.
Theoremindicates that such random phases prevent a further sparsiﬁcation of ρ b ( Dx ) with some linear operator D (cid:48) .
Applying a second thresholding ρ b (cid:48) ( D (cid:48) ρ b ( Dx )) thus cannot signiﬁcantly reduce the variance of class mean estimators.
C ONCLUSION
This paper studies the improvement of linear separability for image classiﬁcation in deep convolutional networks. We show that it mostly relies on a phase collapse phenomenon.
Eliminating the phase of wavelet coefﬁcients improves the separation of class means.
We introduced a Learned Scattering network with a phase collapse on the output of wavelet ﬁlters and learned 1 × 1 convolutional ﬁlters P j , which reaches ResNet accuracy.
The learned 1 × 1 operators P j enhance discriminability by computing channels that have different levels of sparsity for different classes.
When class means are different, thresholding non-linearities can improve classiﬁcation by reducing the variance of class mean estimators.
When used alone, the classiﬁcation performance is poor over complex datasets such as ImageNet or CIFAR-10, because class means are not sufﬁciently separated.
The iteration of thresholdings on sparsiﬁcation operators indeed requires intermediary phase collapses.
These results show that the elimination of spatial variability with phase collapses is both necessary and sufﬁcient to linearly separate classes on complex image datasets R EPRODUCIBILITY STATEMENT
The code to reproduce the experiments of the paper is available in the supplementary material.
All experimental details and hyperparameters are also provided in Appendix E.
A P ROOF OF E QUATION (4)
We have:
[Equation]
since x is real.
By writing: x ∗ ψ = | x ∗ ψ | e iϕ ( x ∗ ψ ) where ϕ ( x ∗ ψ ) is the phase of x ∗ ψ , this leads to:
[Equation]
since ReLU activation is positive-homogeneous of degree 1.
Thus:
[Equation]
with a change of variable ,
[Equation]
since cos is 2 π periodic and even, =
[Equation]
For z ∈ C
[Equation]
we have | z | = (cid:113) | Re( z ) | 2 + | Im( z ) | 2 ≈ | Re( z ) | + | Im( z ) | in the following sense:
[Equation]
We can write:
[Equation]
= ReLU(Re( z ))
+ ReLU( − Re( z )) , | Im( z ) | = ReLU(Im( z ))
+ ReLU( − Im( z )) .
and then, using Im( z ) =
Re( e iπ/ 2 z ) and e iπ = − 1 : | z | ≈ ReLU(Re( z )) + ReLU(Re( e − iπ z ))
+ ReLU(Re( e − iπ/ 2 z ))
+ ReLU(Re( e iπ/ 2 z )) .
Finally, |
[Equation]
which shows that the integral can be well approximated with a sum of 4 phases α of the complex ﬁlter ψ .
B P ROOF OF T HEOREM
We have:
[Equation]
by covariance of convolution,
[Equation]
by Young’s inequality, and then:
[Equation]
by Plancherel,
[Equation]
since ψ τ ( u ) = ψ ( u − τ ) ,
[Equation]
since x ∈ R (cid:55)→ e ix is 1-Lipschitz,
[Equation]
(cid:98) ψ ( ω ) | 2 d ω by Cauchy-Schwarz,
[Equation]
which leads to the desired result of eq.
[Equation]
C P ROOF OF T HEOREM
We ﬁrst use the chain rule for the entropy:
[Equation]
The ﬁrst term is rewritten with a change of variable:
[Equation]
k
[Equation]
E
[Equation]
DX ) k | [Equation] as D is unitary and hence | det( D ) | = 1 ,
[Equation]
by concavity,
[Equation]
by concavity.
The second term is bounded using the fact that the exponential distribution E ( λ ) is the maximumentropy distribution on R + with mean 1 λ :
[Equation]
by concavity.
Combining both inequalities and rearranging terms yields the stated bound:
[Equation]
D P ROOF OF T HEOREM
We begin with the following lemma:
Lemma 1.
Let ( θ 1 , . . .
, θ d ) be i.i.d.
uniform random variables in [0 , 2 π ] .
Then there exists a constant
C d > 0 such that for all ( ρ 1 , . .
. , ρ d ) ∈ R d , then:
[Equation]
This is proved by observing that the left-hand side is a norm on R d .
One can indeed verify that it is positive deﬁnite, homogeneous and satisﬁes the triangle inequality.
Since all norms on R d are equivalent, there exists a constant C d > 0 such that:
[Equation]
for all ( ρ 1 , . . .
, ρ d ) ∈ R d .
Going back to the proof of Theorem 3, we then have:
[Equation]
2 | X k |by the above lemma, ≥ C d
[Equation]
m =
[Equation]
{+k = (cid:12)(cid:12) D m,k (cid:12)(cid:12) 
+}
| X k | by concavity, because d (cid:88) k
= (cid:12)(cid:12) D m,k (cid:12)(cid:12)
= 1 , =
C d (cid:107) X (cid:107)because d (cid:88) m
= (cid:12)(cid:12) D m,k (cid:12)(cid:12) 2 = 1 .
Taking the expectation ﬁnishes the proof:
[Equation]
[Equation]
E E XPERIMENTAL D ETAILS
Channel operators In all experiments we set P
= Id , and factorize the classiﬁer with an additional complex 1 × 1 convolutional operator P J , which reduces the dimension before all channels and positions are linearly combined.
The architectures implemented are thus also written as (cid:81)
Jj =1 P j ρW , where ρ is the non-linearity.
Each operator ( P j ) 1 ≤ j ≤ J is preceded by a standardization.
It sets the complex mean µ = E [ z ] of every channel to zero, and the real variance σ 2 = E
[ | z | 2 ] of every channel to one.
This is similar to a complex 2D batch-normalization layer (Ioffe and Szegedy, 2015), but without learned afﬁne parameters.
Each operator ( P j ) 1 ≤ j ≤ J is additionally followed by a spatial divisive normalization (Rao et al., 2001), similarly to the local response normalization of Krizhevsky et al.
It sets the norm across channels of each spatial position to one.
The sizes of the ( P j ) j are speciﬁed in Table 3.
The total numbers of parameters for each architecture are speciﬁed in Table 4.
Learned Scattering with phase collapse have a large number of parameters compared to ResNet, despite the comparable width.
This is because the predeﬁned wavelet operator W expands the dimension by a factor of
L + 1 , which means that the input dimension of the learned ( P j ) j is higher than in ResNet.
The skip-connection further increases this input dimension by a factor of 2 .
Spatial ﬁlters We use elongated Morlet ﬁlters for the L complex band-pass ﬁlters ( g (cid:96) ) (cid:96) which are rotated versions of a mother wavelet g : g (cid:96) ( u ) =
g ( r − π(cid:96)/L u ) , with r θ the rotation by angle θ .
The mother wavelet g is deﬁned as:
[Equation]
[Table]
[Table]
Its parameters are its center frequency ξ =
((3 π/ 4) / 2 γ , 0) , its bandwidth σ = 1 .
25 × 2 − γ , and its slant s = 0 .
5 , where 2 γ designates the scale of the band-pass ﬁlter and is to be adjusted.
g is rotated along L = 8 angles for Imagenet and L = 4 angles for CIFAR: θ (cid:96) =
( π(cid:96)/L ) 1 ≤ (cid:96) ≤ L .
The g (cid:96) are then discretized for numerical computations, and K is adjusted so that they have a zero mean.
Finally, we use for the low frequency g
a Gaussian window:
[Equation]
The ﬁlters are implemented with the Kymatio package (Andreux et al., 2020).
Intermediate scales 2 j/ 2 are obtained by applying a subsampling by 2 after each block of 2 layers.
This introduces intermediate scales and generates a wavelet ﬁlterbank with 2 scales per octave: the ﬁlters are designed so that when j low-pass ﬁlters and one band-pass ﬁlter are cascaded, with a subsampling every 2 layers, the scale of the resulting wavelet is 2 j/ 2 .
Each block comprises in its ﬁrst layer a low-frequency ﬁlter g 10 with γ = − 1 / 2 and band-pass ﬁlters with γ = 0 .
In the second layer, we use the same low-frequency ﬁlter g 20 = g 10 with γ = − 1 / 2 .
The band-pass ﬁlters g 2 (cid:96) are obtained with parameters ξ (cid:48)
= ( π/ √ 2 , 0) , σ (cid:48)
= 1 .
25 (cid:112) 2 / 3 , and s (cid:48) =
√ 0 .
For CIFAR experiments, the J = 8 layers are grouped in 4 successive blocks of 2 layers.
For
ImageNet experiments, the ﬁrst layer consists of band-pass elongated Morlet ﬁlters g (cid:96) and a low-pass Gaussian window g 0 with γ = 0 , followed by a subsampling of 2 .
The 10 following layers are grouped in 5 blocks of 2 layers.
Optimization We use the optimizer SGD with an initial learning rate of 0 .
01 , a momentum of 0 .
9 , a weight decay of 0 . 0001 , and a batch size of 128 .
The classiﬁer is preceded by a 2D batchnormalization layer.
We use traditional data augmentation: horizontal ﬂips and random crops for
CIFAR, random resized crops of size 224 and horizontal ﬂips for ImageNet.
Classiﬁcation error on
ImageNet validation set is computed on a single center crop of size 224 .
On CIFAR, training lasts for 300 epochs and the learning rate is divided by 10 every 70 epochs.
On ImageNet, training lasts for 150 epochs and the learning rate is divided by 10 every 45 epochs.
The experiments in the paper required around 10k 32GB NVIDIA V100 GPU-hours.
F P AIRED A LEX N ET F ILTERS
Section 2 explains that real networks can still implement phase collapses.
This is done with several ﬁlters ψ α =
Re( e − iα ψ ) which correspond to several phases α of the same complex ﬁlter ψ .
Shang et al.
(2016) showed that the ﬁlters in e.g. the ﬁrst layer of AlexNet (Krizhevsky et al., 2012) can indeed be grouped in such a way.
For the sake of completeness, we reproduce in Figure 2 a ﬁgure from Shang et al.
This suggests that real-valued networks may indeed implement phase collapses using eq.