P HASE C OLLAPSE IN N EURAL N ETWORKS
A BSTRACT
Deep convolutional classiﬁers linearly separate image classes and improve accuracy as depth increases.
They progressively reduce the spatial dimension whereas the number of channels grows with depth.
Spatial variability is therefore transformed into variability along channels.
A fundamental challenge is to understand the role of non-linearities together with convolutional ﬁlters in this transformation.
ReLUs with biases are often interpreted as thresholding operators that improve discrimination through sparsity.
This paper demonstrates that it is a different mechanism called phase collapse which eliminates spatial variability while linearly separating classes.
We show that collapsing the phases of complex wavelet coefﬁcients is sufﬁcient to reach the classiﬁcation accuracy of ResNets of similar depths.
However, replacing the phase collapses with thresholding operators that enforce sparsity considerably degrades the performance.
We explain these numerical results by showing that the iteration of phase collapses progressively improves separation of classes, as opposed to thresholding non-linearities.
I NTRODUCTION
CNN image classiﬁers progressively eliminate spatial variables through iterated ﬁlterings and subsamplings, while linear classiﬁcation accuracy improves as depth increases (Oyallon, 2017).
It has also been numerically observed that CNNs concentrate training samples of each class in small separated regions of a progressively lower-dimensional space.
It can ultimately produce a neural collapse (Papyan et al., 2020), where all training samples of each class are mapped to a single point.
In this case, the elimination of spatial variables comes with a collapse of within-class variability and perfect linear separability.
This increase in linear classiﬁcation accuracy is obtained in standard CNN architectures like ResNets from the iteration of linear convolutional operators and ReLUs with biases.
A difﬁculty in understanding the underlying mathematics comes from the ﬂexibility of ReLUs.
Indeed, a linear combination of biased ReLUs can approximate any non-linearity.
Many papers interpret iterations on ReLUs and linear operators as sparse code computations (Sun et al., 2018; Sulam et al., 2018; 2019; Mahdizadehaghdam et al., 2019; Zarka et al., 2020; 2021).
We show that it is a different mechanism, called phase collapse , which underlies the increase in classiﬁcation accuracy of these architectures.
A phase collapse is the elimination of phases of complex-valued wavelet coefﬁcients with a modulus, which we show to concentrate spatial variability.
This is demonstrated by introducing a structured convolutional neural network with wavelet ﬁlters and no biases.
Section 2 introduces and explains phase collapses.
Complex-valued representations are used because they reveal the mathematics of spatial variability.
Indeed, translations are diagonalized in the Fourier basis, where they become a complex phase shift.
Invariants to translations are computed with a modulus, which collapses the phases of this complex representation.
Section 2 explains how this can improve linear classiﬁcation.
Phase collapses can also be calculated with ReLUs and real ﬁlters.
A
CNN with complex-valued ﬁlters is indeed just a particular instance of a real-valued CNN, whose channels are paired together to deﬁne complex numbers.
Section 3 demonstrates the role of phase collapse in deep classiﬁcation architectures.
It introduces a
Learned Scattering network with phase collapses.
This network applies a learned 1 × 1 convolutional complex operator P j on each layer x j , followed by a phase collapse, which is obtained with a complex wavelet ﬁltering operator W and a modulus:
[Equation]
It does not use any bias.
This network architecture is illustrated in Figure 1.
With the addition of skip-connections, we show that this phase collapse network reaches ResNet accuracy on ImageNet and CIFAR-10.
Section 4 compares phase collapses with other non-linearities such as thresholdings or more general amplitude reduction operators.
Such non-linearities can enforce sparsity but do not modify the phase.
We show that the accuracy of a Learned Scattering network is considerably reduced when the phase collapse modulus is replaced by soft-thresholdings with learned biases.
This is also true of more general phase-preserving non-linearities and architectures.
Section 5 explains the performance of iterated phase collapses by showing that each phase collapse progressively improves linear discriminability.
On the opposite, the improvements in classiﬁcation accuracy of successive sparse code computations are shown to quickly saturate.
The main contribution of this paper is a demonstration that the classiﬁcation accuracy of deep neural networks mostly relies on phase collapses, which are sufﬁcient to linearly separate the different classes on natural image databases.
This is captured by the Learned Scattering architecture which reaches
ResNet-18 accuracy on ImageNet and CIFAR-10.
We also show that phase collapses are necessary to reach this accuracy, by demonstrating numerically and theoretically that iterating phase-preserving non-linearities leads to a signiﬁcantly worse performance.
E LIMINATING S PATIAL V ARIABILITY WITH P HASE C OLLAPSES
Deep convolutional classiﬁers achieve linear separation of image classes.
We show that linear classiﬁcation on raw images has a poor accuracy because image classes are invariant to local translations.
This geometric within-class variability takes the form of random phase ﬂuctuations, and as a result all classes have a zero mean.
To improve classiﬁcation accuracy, non-linear operators must separate class means, which therefore requires to collapse these phase ﬂuctuations.
Translations and phase shifts Translations capture the spatial topology of the grid on which the image is deﬁned.
These translations are transformed into phase shifts by a Fourier transform.
We prove that this remains approximately valid for images convolved with appropriate complex ﬁlters.
Let x be an image indexed by u ∈ Z 2 .
We write x τ ( u ) =
x ( u − τ ) the translation of x by τ .
It is diagonalized by the Fourier transform (cid:98) x
[Equation]
ω
[Equation]
= (cid:80) u
x ( u
[Equation]
e − iω · u , which creates a phase shift:
[Equation]
This diagonalization explains the need to introduce complex numbers to analyze the mathematical properties of geometric within-class variabilities.
Computations can however be carried with real numbers, as we will show.
A Fourier transform is computed by ﬁltering x with complex exponentials e iω · u .
One may replace these by complex wavelet ﬁlters ψ that are localized in space and in the Fourier domain.
The following theorem proves that small translations can still be approximated by a phase shift in this case.
We denote by ∗ the convolution of images.
Theorem 1.
Let ψ : Z 2 → C be a ﬁlter with (cid:107) ψ (cid:107) 2 = 1 , whose center frequency ξ and bandwidth σ are deﬁned by:
[Equation]
Then, for any τ ∈ Z 2 ,
[Equation]
The proof is in Appendix C.
This theorem proves that if | τ | (cid:28) 1 /σ then x τ ∗ ψ ≈ e − iξ · τ x ∗ ψ .
In this case, a translation by τ produces a phase shift by ξ · τ .
Phase collapse and stationarity We deﬁne a phase collapse as the elimination of the phase created by a spatial ﬁltering with a complex wavelet ψ .
We now show that phase collapses improve linear classiﬁcation of classes that are invariant to global or local translations.
The training images corresponding to the class label y may be represented as the realizations of a random vector X y .
To achieve linear separation, it is sufﬁcient that class means E (cid:2)
X y (cid:3) are separated and within-class variances around these means are small enough (Hastie et al., 2009).
The goal of classiﬁcation is to ﬁnd a representation of the input images in which these properties hold.
To simplify the analysis, we consider the particular case where each class y is invariant to translations.
More precisely, each random vector X y is stationary, which means that its probability distribution is invariant to translations.
Equation (2) then implies that the phases of Fourier coefﬁcients of X y are uniformly distributed in [0 , 2 π ] , leading to E [ (cid:98)
X y ( ω )] = 0 for ω (cid:54) = 0 .
The class means E [ X y ] are thus constant images whose pixel values are all equal to E [ (cid:98) X y (0)] .
A linear classiﬁer can then only rely on the average colors of the classes, which are often equal in practice.
It thus cannot discriminate such translation-invariant classes.
Eliminating uniform phase ﬂuctuations of non-zero frequencies is thus necessary to create separated class means, which can be achieved with the modulus of the Fourier transform.
It is a translationinvariant representation: | (cid:98) x τ | = | (cid:98) x | .
This improves linear discriminability of stationary classes, because E [ | (cid:98) X y | ] may be different for different y .
However, | (cid:98) X y | has a high variance, because the
Fourier transform is unstable to small deformations (Bruna and Mallat, 2013).
Fourier modulus descriptors can be improved by using ﬁlters ψ that have a localized support in space.
Theorem 1 shows that the phase of X y ∗ ψ is also uniformly distributed in [0 , 2 π ] .
It results that E [ X y ∗ ψ ] = 0 , and x ∗ ψ still provides no information for linear classiﬁcation.
Applying a modulus similarly computes approximate invariants to small translations: | x τ ∗ ψ | ≈ | x ∗ ψ | , with an error bounded by σ | τ | (cid:107) x (cid:107) 2 .
More generally, these phase collapses compute approximate invariants to deformations which are well approximated by translations over the support of ψ .
This representation improves linear classiﬁcation by creating different non-zero class means E [ | X y ∗ ψ | ] while achieving a lower variance than Fourier coefﬁcients, as it is stable to deformations (Bruna and Mallat, 2013).
Image classes are usually not invariant to global translations, because of e.g. centered subjects or the sky located in the topmost part of the image.
However, classes are often invariant to local translations, up to an unknown maximum scale.
This is captured by the notion of local stationarity, which means that the probability distribution of X y is nearly invariant to translations smaller than some maximum scale (Priestley, 1965).
The above discussion remains valid if X y is only locally stationary over a domain larger than the support of ψ .
The use of so-called “windowed absolute spectra” E (cid:2)(cid:12)(cid:12) X y ∗ ψ (cid:12)(cid:12)(cid:3) for locally stationary processes has previously been studied in Tygert et al.
Real or complex networks The use of complex numbers is a mathematical abstraction which allows diagonalizing translations, which are then represented by complex phases.
It provides a mathematical interpretation of ﬁltering operations performed on real numbers.
We show that a real network can still implement complex phase collapses.
In the ﬁrst layer of a CNN, one can observe that ﬁlters are often oscillatory patterns with small supports, where some ﬁlters have nearly the same orientation and frequency but with a phase shifted by some α (Krizhevsky et al., 2012).
We reproduce in Appendix A a ﬁgure from Shang et al.
(2016) which evidences this phenomenon.
It shows that real ﬁlters may be arranged in groups ( ψ α ) α that can be written ψ α =
Re( e − iα ψ ) for a single complex ﬁlter ψ and several phases α .
A CNN with complex ﬁlters is thus a structured real-valued CNN, where several real ﬁlters ( ψ α ) α have been regrouped into a single complex ﬁlter ψ .
This structure simpliﬁes the mathematical interpretation of non-linearities by explicitly deﬁning the phase, which is otherwise a hidden variable relating multiple ﬁlter outputs within each layer.
A phase collapse is explicitly computed with a complex wavelet ﬁlter and a modulus.
It can also be implicitly calculated by real-valued CNNs.
Indeed, for any real-valued signal x , we have:
[Equation]
Furthermore, this integral is well approximated by a sum over 4 phases, allowing to compute complex moduli with real-valued ﬁlters and ReLUs without biases.
See Appendix D for a proof of eq.
(4) and its approximation.
L EARNED S CATTERING N ETWORK WITH P HASE C OLLAPSES
This section introduces a learned scattering transform, which is a highly structured CNN architecture relying on phase collapses and reaching ResNet accuracy on the ImageNet (Russakovsky et al., 2015) and CIFAR-10 (Krizhevsky, 2009) datasets.
Scattering transform Theorem 1 proves that a modulus applied to the output of a complex wavelet ﬁlter produces a locally invariant descriptor.
This descriptor can then be subsampled, depending upon the ﬁlter’s bandwidth.
We brieﬂy review the scattering transform (Mallat, 2012; Bruna and Mallat, 2013), which iterates phase collapses.
A scattering transform over J scales is implemented with a network of depth J , whose ﬁlters are speciﬁed by the choice of wavelet.
Let x 0 = x .
For 0 ≤ j < J , the ( j +1) -th layer x j +1 is computed by applying a phase collapse on the j -th layer x j .
It is implemented by a modulus which collapses the phases created by a wavelet ﬁltering operator W :
[Equation]
The operator W is deﬁned with Morlet ﬁlters (Bruna and Mallat, 2013).
It has one low-pass ﬁlter g 0 , and L zero-mean complex band-pass ﬁlters ( g (cid:96) ) (cid:96) , having an angular direction (cid:96)π/L for 0 < (cid:96) ≤ L .
It thus transforms an input image x ( u ) into L + 1 sub-band images which are subsampled by 2 :
[Equation]
The cascade of j low-pass ﬁlters g 0 with a ﬁnal band-pass ﬁlter g (cid:96) , each followed by a subsampling, computes wavelet coefﬁcients at a scale 2 j .
One can also modify the wavelet ﬁltering W to compute intermediate scales 2 j/ 2 , as explained in
Appendix G.
The spatial subsampling is then only computed every other layer, and the depth of the network becomes twice larger.
Applying a linear classiﬁer on such a scattering transform gives good results on simple classiﬁcation problems such as MNIST (LeCun et al., 2010).
However, results are well below ResNet accuracy on CIFAR-10 and ImageNet, as shown in Table 1.
Learned Scattering The prior work of Zarka et al.
(2021) showed that a scattering transform can reach ResNet accuracy by incorporating learned 1 × 1 convolutional operators and soft-thresholding non-linearities in-between wavelet ﬁlters.
In contrast, we introduce a Learned Scattering architecture whose sole non-linearity is a phase collapse.
It shows that neither biases nor thresholdings are necessary to reach a high accuracy in image classiﬁcation.
A similar result had previously been obtained on image denoising (Mohan et al., 2019).
The Learned Scattering (LScat) network inserts in eq.
(5) a learned complex 1 × 1 convolutional operator P j which reduces the channel dimensionality of each layer x j before each phase collapse:
[Equation]
Similar architectures which separate space-mixing and channel-mixing operators had previously been studied in the context of basis expansion (Qiu et al., 2018;
Ulicny et al., 2019) or to ﬁlter scattering
[Table]
channels (Cotter and Kingsbury, 2019).
This separation is also a major feature of recent architectures such as Vision Transformers (Dosovitskiy et al., 2021) or MLP-Mixer (Tolstikhin et al., 2021).
Each P j computes discriminative channels whose spatial variability is eliminated by the phase collapse operator.
Their role is further discussed in Section 5.
Table 1 gives the accuracy of a linear classiﬁer applied to the last layer of this Learned Scattering.
It provides an important improvement over a scattering transform, but it does not yet reach the accuracy of ResNet-18.
Including the linear classiﬁer, the architecture uses a total number of layers J + 1 = 12 for ImageNet and J + 1 = 9 for CIFAR, by introducing intermediate scales.
The number of channels of P j x j is the same as in a standard ResNet architecture (He et al., 2016) and remains no larger than 512 .
More details are provided in Appendix G.
Skip-connections across moduli Equation (7) imposes that all phases are collapsed at each layer, after computing a wavelet transform.
More ﬂexibility is provided by adding a skip-connection which concatenates WP j x j with its modulus:
[Equation]
The skip-connection produces a cascade of convolutional ﬁlters W without non-linearities in-between.
The resulting convolutional operator WW · · · W is a “wavelet packet” transform which generalizes the wavelet transform (Coifman and Wickerhauser, 1992).
Wavelet packets are obtained as the cascade of low-pass and band-pass ﬁlters ( g (cid:96) ) (cid:96) , each followed by a subsampling.
Besides wavelets, wavelet packets include ﬁlters having a larger spatial support and a narrower Fourier bandwidth.
A wavelet packet transform is then similar to a local Fourier transform.
Applying a modulus on such wavelet packet coefﬁcients deﬁnes local spatial invariants over larger domains.
As discussed in Section 2, image classes are usually invariant to local rather than global translations.
Section 2 explains that a phase collapse improves discriminability for image classes that are locally translation-invariant over the ﬁlter’s support.
Indeed, phases of wavelet coefﬁcients are then uniformly distributed over [0 , 2 π ] , yielding zero-mean coefﬁcients for all classes.
At scales where there is no local translation-invariance, these phases are no longer uniformly distributed, and they encode information about the spatial localization of features.
Introducing a skip-connection provides the ﬂexibility to choose whether to eliminate phases at different scales or to propagate them up to the last layer.
Indeed, the next 1 × 1 operator P j +1 linearly combines (cid:12)(cid:12)
WP j x j (cid:12)(cid:12) and
WP j x j and may learn to use only one of these.
This adds some localization information, which appears to be important.
Table 1 shows that the skip-connection indeed improves classiﬁcation accuracy.
A linear classiﬁer on this Learned Scattering reaches ResNet-18 accuracy on CIFAR-10 and ImageNet.
It demonstrates that collapsing appropriate phases is sufﬁcient to obtain a high accuracy on large-scale classiﬁcation problems.
Learning is reduced to 1 × 1 convolutions ( P j ) j across channels.
P HASE C OLLAPSES V ERSUS A MPLITUDE R EDUCTIONS
We now compare phase collapses with amplitude reductions, which are non-linearities which preserve the phase and act on the amplitude.
We show that the accuracy of a Learned Scattering network is considerably reduced when the phase collapse modulus is replaced by soft-thresholdings with learned biases.
This result remains true for other amplitude reductions and architectures.
Thresholding and sparsity A complex soft-thresholding reduces the amplitude of its input z
= | z | e iϕ by b while preserving the phase:
ρ b ( z ) = ReLU( | z | − b ) e iϕ .
Similarly to its real counterpart, it is obtained as the proximal operator of the complex modulus (Yang et al., 2012):
[Equation]
Soft-thresholdings and moduli have opposite properties, since soft-thresholdings preserve the phase while attenuating the amplitude, whereas moduli preserve the amplitude while eliminating the phase.
In contrast, ReLUs with biases are more general non-linearities which can act both on phase and amplitude.
This is best illustrated over R where the phase is replaced by the sign, through the even-odd decomposition.
If z ∈ R and b ≥ 0 , then the even part of ReLU( z − b ) is ReLU( | z | − b ) , which is an absolute value with a dead-zone [ − b, b ] .
When b = 0 , it becomes an absolute value | z | .
The odd part is a soft-thresholding ρ b ( z ) = sign( z ) ReLU( | z | − b ) .
Over C , a similar result can be obtained through the decomposition into phase harmonics (Mallat et al., 2019).
We have explained how phase collapses can improve the classiﬁcation accuracy of locally stationary processes by separating class means E (cid:2)(cid:12)(cid:12)
X y ∗ ψ (cid:12)(cid:12)(cid:3) .
In contrast, since the phase of X y ∗ ψ is uniformly distributed for such processes, then it is also true of ρ b ( X y ∗ ψ ) .
This implies that E (cid:2) ρ b ( X y ∗ ψ ) (cid:3) =for all b .
Class means of locally stationary processes are thus not separated by a thresholding.
When class means E [ X y ∗ ψ ] are separated, a soft-thresholding of X y ∗ ψ may however improve classiﬁcation accuracy.
If X y ∗ ψ is sparse, then a soft-thresholding ρ b ( X y ∗ ψ ) reduces the withinclass variance (Donoho and Johnstone, 1994; Zarka et al., 2021).
Coefﬁcients below the threshold may be assimilated to unnecessary “clutter” which is set to 0 .
To improve classiﬁcation, convolutional ﬁlters must then produce high-amplitude coefﬁcients corresponding to discriminative “features”.
Phase collapses versus amplitude reductions A Learned Scattering with phase collapses preserves the amplitudes of wavelet coefﬁcients and eliminates their phases.
On the opposite, one may use a non-linearity which preserves the phases of wavelet coefﬁcients but attenuates their amplitudes, such as a soft-thresholding.
We show that such non-linearities considerably degrade the classiﬁcation accuracy compared to phase collapses.
Several previous works made the hypothesis that sparsifying neural responses with thresholdings is a major mechanism for improving classiﬁcation accuracy (Sun et al., 2018; Sulam et al., 2018; 2019; Mahdizadehaghdam et al., 2019; Zarka et al., 2020; 2021).
The dimensionality of sparse representations can then be reduced with random ﬁlters which implement a form of compressed sensing (Donoho, 2006; Candes et al., 2006).
The interpretation of CNNs as compressed sensing machines with random ﬁlters has been studied (Giryes et al., 2015), but it never led to classiﬁcation results close to e.g. ResNet accuracy.
To test this hypothesis, we replace the modulus non-linearity in the Learned Scattering architecture with thresholdings, or more general phase-preserving non-linearities.
A Learned Amplitude Reduction
Scattering applies a non-linearity ρ ( z ) which preserves the phases of wavelet coefﬁcients z = | z | e iϕ : ρ ( z )
=
e iϕ ρ ( | z | ) .
Without skip-connections, each layer x j +1 is computed from x j by:
[Equation]
and with skip-connections:
[Equation]
[Equation]
A soft-thresholding is deﬁned by ρ ( | z | ) =
ReLU( | z | − b ) for some threshold b .
We also deﬁne an amplitude hyperbolic tangent ρ ( | z | ) =
( e | z | − e −| z | ) /
( e | z | + e −| z | ) , an amplitude sigmoid as ρ ( | z | ) =
(1 + e − a log | z |− b ) − 1 and an amplitude soft-sign as ρ ( | z | ) = | z | / (1 + | z | ) .
The softthresholding and sigmoid parameters a and b are learned for each layer and each channel.
We evaluate the classiﬁcation performance of a Learned Amplitude Reduction Scattering on CIFAR10, by applying a linear classiﬁer on the last layer.
Classiﬁcation results are given in Table 2 for different amplitude reductions, with or without skip-connections.
Learned Amplitude Reduction
Scatterings yield much larger errors than a Learned Scattering with phase collapses.
Without skipconnections, they are even above a scattering transform, which also uses phase collapses but does not [Table] have learned 1 × 1 convolutional projections ( P j ) j .
It demonstrates that high accuracies result from phase collapses without biases, as opposed to amplitude reduction operators including thresholdings, which learn bias parameters.
Similar experiments in the real domain with a standard ResNetarchitecture on the ImageNet dataset can be found in Appendix B.
ReLUs with biases Most CNNs, including ResNets, use ReLUs with biases.
A ReLU with bias simultaneously affects the sign and the amplitude of its real input.
Over complex numbers, it amounts to transforming the phase and the amplitude.
These numerical experiments show that accuracy improvements result from acting on the sign or phase rather than the amplitude.
Furthermore, this can be constrained to collapsing the phase of wavelet coefﬁcients while preserving their amplitude.
Several CNN architectures have demonstrated a good classiﬁcation accuracy with iterated thresholding algorithms, which increase sparsity.
However, all these architecture also modiﬁed the sign of coefﬁcients by computing non-negative sparse codes (Sun et al., 2018; Sulam et al., 2018; Mahdizadehaghdam et al., 2019) or with additional ReLU or modulus layers (Zarka et al., 2020; 2021).
It seems that it is the sign or phase collapse of these non-linearities which is responsible for good classiﬁcation accuracies, as opposed to the calculation of sparse codes through iterated amplitude reductions.
I TERATING P HASE C OLLAPSES AND A MPLITUDE R EDUCTIONS
We now provide a theoretical justiﬁcation to the above numerical results in simpliﬁed mathematical frameworks.
This section studies the behavior of phase collapses and amplitude reductions when they are iterated over several layers.
It shows that phase collapses beneﬁt from iterations over multiple layers, whereas there is no signiﬁcant gain in performance when iterating amplitude reductions.
5. I TERATED P HASE C OLLAPSES
We explain the role of iterated phase collapses with multiple ﬁlters at each layer.
Classiﬁcation accuracy is improved through the creation of additional dimensions to separate class means.
The learned projectors ( P j ) j are optimized for this separation.
We consider the classiﬁcation of stationary processes X y ∈ R d , corresponding to different image classes indexed by y .
Given a realization x of X y , and because of stationarity, the optimal linear classiﬁer is calculated from the empirical mean 1 /d (cid:80) u x ( u ) .
It computes an optimal linear estimation of E (cid:2)
X y ( u ) (cid:3) = µ y .
If all classes have the same mean µ y = µ , then all linear classiﬁers fail.
As explained in Section 2, linear classiﬁcation can be improved by computing ( | x ∗ ψ k | )
k for some wavelet ﬁlters ( ψ k ) k .
These phase collapses create additional directions with non-zero means which may separate the classes.
If X y is stationary, then | X y ∗ ψ k | remains stationary for any ψ k .
An optimal linear classiﬁer applied to ( | x ∗ ψ k ( u ) | ) k is thus obtained by a linear combination of all empirical means (1 /d (cid:80) u | x ∗ ψ k ( u ) | ) k .
They are proportional to the (cid:96) 1 norm (cid:107) x ∗ ψ k (cid:107) 1 , which is a measure of sparsity of x ∗ ψ k .
If linear classiﬁcation on ( | x ∗ ψ k ( u ) | ) k fails, it reveals that the means E (cid:2) | X y ∗ ψ k ( u ) | (cid:3) =
µ y,k are not sufﬁciently different.
Separation can be improved by considering the spatial variations of | X y ∗ ψ k ( u ) | for different y .
These variations can be revealed by a phase collapse on a new set of wavelet ﬁlters ψ k (cid:48) , which computes ( || x ∗ ψ k |∗ ψ k (cid:48) | ) k,k (cid:48) .
This phase collapse iteration is the principle used by scattering transforms to discriminate textures (Bruna and Mallat, 2013; Sifre and Mallat, 2013): each successive phase collapse creates additional directions to separate class means.
However, this may still not be sufﬁcient to separate class means.
More discriminant statistical properties may be obtained by linearly combining ( | x ∗ ψ k | ) k across k before applying a new ﬁlter ψ k (cid:48) .
In a Learned Scattering with phase collapse, this is done with a linear projector P 1 across the channel indices k , before computing a convolution with the next ﬁlter ψ k (cid:48) .
The 1 × 1 operator
P 1 is optimized to improve the linear classiﬁcation accuracy.
It amounts to learning weights w k such that E [ (cid:12)(cid:12)(cid:80) k w k (cid:12)(cid:12)
X y ∗ ψ k (cid:12)(cid:12) ∗ ψ k (cid:48)
(cid:12)(cid:12) ] is as different as possible for different y .
Because these are proportional to the (cid:96) 1 norms (cid:13)(cid:13)(cid:80) k w k | x ∗ ψ k | ∗ ψ k (cid:48)
(cid:13)(cid:13) 1 , it means that the images (cid:80) k
w k | x ∗ ψ k |∗ ψ k (cid:48) have different sparsity levels depending upon the class y of x .
The weights ( w k ) k of P 1 can thus be interpreted as features along channels providing different sparsiﬁcations for different classes.
A
Learned Scattering network learns such P j at each scale j .
5. I TERATED A MPLITUDE R EDUCTIONS
Sparse representations and amplitude reduction algorithms may improve linear classiﬁcation by reducing the variance of class mean estimations, which can be interpreted as clutter removal.
Such approaches are studied in Zarka et al.
by modeling the clutter as an additive white noise.
Although a single thresholding step may improve linear classiﬁcation, we show that iterating more than one thresholding does not improve the classiﬁcation accuracy, if no phase collapses are inserted.
To understand these properties, we consider the discrimination of classes X y for which class means
E ( X y ) =
µ y are all different.
If there exists y (cid:48) such that (cid:107) µ y − µ y (cid:48) (cid:107) is small, then the class y can still be discriminated from y (cid:48) if we can estimate E ( X y ) sufﬁciently accurately from a single realization x of X y .
This is a mean estimation problem.
Suppose that X y = µ y + N (0 , σ 2 ) is contaminated with
Gaussian white noise, where the noise models some clutter.
Suppose also that there exists a linear orthogonal operator D such that Dµ y is sparse for every y , and hence has its energy concentrated in few non-zero coefﬁcients.
Such a D may be computed by minimizing the expected (cid:96) 1 norm (cid:80) y E (cid:2) (cid:107) DX y (cid:107) 1 (cid:3) .
The estimation of µ y can be improved with a soft-thresholding estimator (Donoho and Johnstone, 1994), which sets to zero all coefﬁcients below a threshold b proportional to σ .
It amounts to computing ρ b ( Dx ) , where ρ b is a soft-thresholding.
However, we explain below why this approach cannot be further iterated without inserting phase collapses.
The reason is that a sparse representation ρ b ( Dx ) concentrates its entropy in the phases of the coefﬁcients, rather than their amplitude.
We then show that such processes cannot be further sparsiﬁed, which means that a second thresholding ρ b (cid:48)
( D (cid:48) ρ b ( Dx )) will not reduce further the variance of class mean estimators.
This entails that a model of within-class variability relying on amplitude reductions cannot be the sole mechanism behind the performance of deep networks.
Iterating amplitude reductions may however be useful if it is alternated with another non-linearity which partly or fully collapses phases.
Reducing the entropy of the phases of ρ b ( Dx ) allows ρ b (cid:48) D (cid:48) to further sparsify the process and hence further reduce the within-class variability.
As mentioned in Section 4, this is the case for previous work which used iterated sparsiﬁcation operators (Sun et al., 2018; Sulam et al., 2018; Mahdizadehaghdam et al., 2019).
Indeed, these networks compute non-negative sparse codes where sparsity is enforced with a ReLU, which acts both on phases and amplitudes.
Our results shows that the beneﬁt of iterating non-negative sparse coding comes from the sign collapse due to the non-negativity constraint.
We now qualitatively demonstrate these claims with two theorems.
We ﬁrst show that ﬁnding the sparsest representation of a random process (i.e., minimizing its (cid:96) 1 norm) is the same as maximizing a lower bound on the entropy of its phases.
Theorem 2.
Let X denote a random vector in C d with a probability density p .
Let H ( X ) be the entropy of X with respect to the Lebesgue measure:
[Equation]
If D ∈ U ( d ) is a unitary operator then:
[Equation]
where ϕ ( DX ) ∈ [0 , 2 π ] d (resp.
| DX | ∈ R d + ) is the random process of the entry-wise phases (resp. moduli) of DX .
The proof is in Appendix E.
This theorem gives a lower-bound on the conditional entropy of the phases of DX with a decreasing function of the expected (cid:96) 1 norm of DX .
Minimizing over D this expected (cid:96) 1 norm amounts to maximizing the lower bound on H (cid:16) ϕ ( DX ) (cid:12)(cid:12)(cid:12)
| DX | (cid:17) .
An extreme situation arises when this entropy reaches its maximal value of d log(2 π ) .
In this case, the phase ϕ ( DX ) has a maximum-entropy distribution and is therefore uniformly distributed in [0 , 2 π ] d .
Moreover, in this extreme case ϕ ( DX ) is independent from | DX | , since its conditional distribution does not depend on | DX | .
Such statistical properties have previously been observed on wavelet coefﬁcients of natural images (Rao et al., 2001), where the wavelet transform seems to be a nearly optimal sparsifying unitary dictionary.
The second theorem considers the extreme case of a random process whose phases are conditionally independent and uniform.
It proves that such a process cannot be signiﬁcantly sparsiﬁed with a change of basis.
Theorem 3.
Assume that ϕ ( ρ b ( DX )) is uniformly distributed in [0 , 2 π ] d and independent from | ρ b ( DX ) | .
Then there exists a constant C d > 0 which depends on the dimension d , such that for any
D (cid:48)
∈
U ( d ) ,
E (cid:2)
[Table]
The proof is in Appendix F.
This theorem shows that random processes with conditionally independent and uniform phases have an (cid:96) 1 norm which cannot be signiﬁcantly decreased by any unitary transformation.
Numerical evaluations suggest that the constant C d may be chosen to be √ π/ 2 ≈ 0 .
886 , independently of the dimension d .
This constant arises as the value of E [ | Z | ] when
Z is a complex normal random variable with E [ | Z | 2 ] = 1 .
These two theorems explain qualitatively that linear classiﬁcation on ρ b ( Dx ) cannot be improved by another thresholding that would take advantage of another sparsiﬁcation operator.
Indeed, Theoremshows that if ρ b ( Dx ) is sparse, then its phases have random ﬂuctuations of high entropy.
Theoremindicates that such random phases prevent a further sparsiﬁcation of ρ b ( Dx ) with some linear operator
D (cid:48) .
Applying a second thresholding ρ b (cid:48)
( D (cid:48) ρ b ( Dx )) thus cannot signiﬁcantly reduce the variance of class mean estimators.
C ONCLUSION
This paper studies the improvement of linear separability for image classiﬁcation in deep convolutional networks.
We show that it mostly relies on a phase collapse phenomenon.
Eliminating the phase of wavelet coefﬁcients improves the separation of class means.
We introduced a Learned Scattering network with wavelet phase collapses and learned 1 × 1 convolutional ﬁlters ( P j ) j , which reaches
ResNet accuracy.
The learned 1 × 1 operators ( P j ) enhance discriminability by computing channels that have different levels of sparsity for different classes.
When class means are separated, thresholding non-linearities can improve classiﬁcation by reducing the variance of class mean estimators.
When used alone, the classiﬁcation performance is poor over complex datasets such as ImageNet or CIFAR-10, because class means are not sufﬁciently separated.
Furthermore, the iteration of thresholdings on sparsiﬁcation operators requires intermediary phase collapses.
These results show that linear separation of classes result from acting on the sign or phase of network coefﬁcients rather than their amplitude.
Furthermore, this can be constrained to collapsing the phase of wavelet coefﬁcients while preserving their amplitude.
The elimination of spatial variability with phase collapses is thus both necessary and sufﬁcient to linearly separate classes on complex image datasets.
R EPRODUCIBILITY STATEMENT
The code to reproduce the experiments of the paper is available at https://github.com/
FlorentinGuth/PhaseCollapse .
All experimental details and hyperparameters are also provided in Appendix G.
A CKNOWLEDGMENTS
This work was supported by a grant from the PRAIRIE 3IA Institute of the French ANR-19-P3IA0001 program.
We would like to thank the Scientiﬁc Computing Core at the Flatiron Institute for the use of their computing resources.
We also thank Antoine Brochard, Brice Ménard and Rudy Morel for heplful comments.
A P AIRED A LEX N ET F ILTERS
Section 2 explains that real networks can still implement phase collapses.
This is done with several real ﬁlters ψ α =
Re( e − iα ψ ) which correspond to several phases α of the same complex ﬁlter ψ .
Shang et al.
(2016) showed that the ﬁlters in e.g. the ﬁrst layer of AlexNet (Krizhevsky et al., 2012) can indeed be grouped in such a way.
For the sake of completeness, we reproduce in Figure 2 a ﬁgure from Shang et al.
This suggests that real-valued networks may indeed implement phase collapses using eq.
B P HASE C OLLAPSE V ERSUS A MPLITUDE R EDUCTION W ITH R ES N ET
We now evaluate the classiﬁcation error of phase collapses and amplitude reduction non-linearities in the real domain.
We use a standard ResNet-18 architecture without biases.
We replace the ReLU non-linearity by an absolute value or sign collapse | x | and several sign-preserving (i.e., odd) nonlinearities.
They include a soft-thresholding ρ b ( x ) = sign( x ) ReLU( | x | − b ) , an hyperbolic tangent ρ ( x ) =
( e x − e − x ) / ( e x + e − x ) , and a soft-sign ρ ( x ) = x/ (1 + | x | ) . We do not report results for an amplitude sigmoid ρ ( x ) = sign( x )( 1 + e − a log | x |− b ) − 1 because of optimization instabilities when learning the parameters a and b .
Classiﬁcation results on the ImageNet dataset are given in Table 3.
The error of bias-free ReLUs and sign collapses are comparable to a standard ResNet-18, and conﬁrm that sign collapses are sufﬁcient to reach such accuracies.
In contrast, the performance of amplitude reduction non-linearities, which preserve the sign of network coefﬁcients, is signiﬁcantly worse.
The conclusions of Section 4 thus still hold in the real domain and when the spatial ﬁlters are not constrained to be wavelets
[Table]
[Table]
C P ROOF OF T HEOREM
We have:
[Equation]
by covariance of convolution,
[Equation]
by Young’s inequality, and then:
[Equation]
by Plancherel,
[Equation]
since ψ τ ( u ) = ψ ( u − τ ) ,
[Equation]
ψ
[Equation]
ω )
[Equation]
since x ∈ R (cid:55)→ e ix is 1-Lipschitz,
[Equation]
by Cauchy-Schwarz,
[Equation]
which leads to the desired result of eq.
[Equation]
− iξ · τ
[Equation]
ψ
[Equation]
|
[Equation]
|
[Equation]
D P ROOF OF E QUATION (4)
We have:
[Equation]
since x is real.
By writing: x ∗ ψ = | x ∗ ψ | e iϕ ( x ∗ ψ ) where ϕ ( x ∗ ψ ) is the phase of x ∗ ψ , this leads to:
[Equation]
since ReLU activation is positive-homogeneous of degree 1.
Thus:
[Equation]
a change of variable
[Equation]
since cos is 2 π periodic and even, =
[Equation]
= | x
[Equation]
| .
For z ∈ C
[Equation]
we have | z | = (cid:113) | Re( z ) | 2 + | Im( z ) | 2 ≈ | Re( z ) | + | Im( z ) | in the following sense:
[Equation]
z )
Im( z )
We can write:
[Equation]
Re
[Equation] [Equation]
and then, using Im( z ) =
Re( e iπ/ 2 z ) and e iπ = − 1 :
[Equation]
Finally,
[Equation]
which shows that the integral can be well approximated with a sum of 4 phases α of the complex ﬁlter ψ .
E P ROOF OF T HEOREM
We ﬁrst use the chain rule for the entropy:
[Equation]
The ﬁrst term is rewritten with a change of variable:
H ( |
[Equation]
|
[Equation]
DX
[Equation]
− d (cid:88) k
[Equation]
E [log | ( DX ) k | ]
= H ( X ) − d (cid:88) k
[Equation]
E [log | ( DX ) k | ] as D is unitary and hence | det( D ) | = 1 ,
[Equation]
by concavity,
[Equation]
by concavity.
The second term is bounded using the fact that the exponential distribution E ( λ ) is the maximumentropy distribution on R + with mean 1 λ :
[Equation]
by concavity
[Equation]
Combining both inequalities and rearranging terms yields the stated bound:
[Equation]
F P ROOF OF T HEOREM
We begin with the following lemma:
Lemma 1.
Let ( θ 1 , . . .
, θ d ) be i.i.d.
uniform random variables in [0 , 2 π ] .
Then there exists a constant
C d > 0 such that for all ( ρ 1 , . .
. , ρ d ) ∈ R d , then:
[Equation]
This is proved by observing that the left-hand side is a norm on R d .
One can indeed verify that it is positive deﬁnite, homogeneous and satisﬁes the triangle inequality.
Since all norms on R d are equivalent, there exists a constant C d > 0 such that:
[Equation]
for all ( ρ 1 , . . .
, ρ d ) ∈ R d .
Going back to the proof of Theorem 3, and letting X (cid:48)
=
ρ b ( DX ) , we then have:
[Equation]
by the above lemma,
[Equation]
by concavity, because
[Equation]
because
[Equation]
Taking the expectation ﬁnishes the proof:
[Equation]
G E XPERIMENTAL D ETAILS
Channel operators In all experiments we set P
= Id , and factorize the classiﬁer with an additional complex 1 × 1 convolutional operator P J , which reduces the dimension before all channels and positions are linearly combined.
The architectures implemented are thus also written as (cid:81)
Jj =1 P j ρW , where ρ is the non-linearity.
Each operator ( P j ) 1 ≤ j ≤ J is preceded by a standardization.
It sets the complex mean µ = E [ z ] of every channel to zero, and the real variance σ 2 = E
[ | z | 2 ] of every channel to one.
This is similar to a complex 2D batch-normalization layer (Ioffe and Szegedy, 2015), but without learned afﬁne parameters.
Each operator ( P j ) 1 ≤ j ≤ J is additionally followed by a spatial divisive normalization (Rao et al., 2001), similarly to the local response normalization of Krizhevsky et al.
It sets the norm across channels of each spatial position to one.
The sizes of the ( P j ) j are speciﬁed in Table 4.
The total numbers of parameters for each architecture are speciﬁed in Table 5.
Learned Scattering with phase collapse have a large number of parameters compared to ResNet, despite the comparable width.
This is because the predeﬁned wavelet operator W expands the dimension by a factor of
L + 1 , which means that the input dimension of the learned ( P j ) j is higher than in ResNet.
The skip-connection further increases this input dimension by a factor of 2 .
j
CIFARc j        - - -
ImageNet c j
[Table]
Spatial ﬁlters We use elongated Morlet ﬁlters for the L complex band-pass ﬁlters ( g (cid:96) ) (cid:96) which are rotated versions of a mother wavelet g : g (cid:96) ( u ) =
g ( r − π(cid:96)/L u ) , with r θ the rotation by angle θ .
The mother wavelet g is deﬁned as:
[Equation]
Its parameters are its center frequency ξ =
((3 π/ 4) / 2 γ , 0) , its bandwidth σ = 1 .
25 × 2 − γ , and its slant s = 0 .
5 , where 2 γ designates the scale of the band-pass ﬁlter and is to be adjusted.
g is rotated along L = 8 angles for Imagenet and L = 4 angles for CIFAR: θ (cid:96) =
( π(cid:96)/L ) 1 ≤ (cid:96) ≤ L .
The ( g (cid:96) ) (cid:96) are then discretized for numerical computations, and K is adjusted so that they have a zero mean.
Finally, we use for the low frequency g
a Gaussian window:
[Equation]
The ﬁlters are implemented with the Kymatio package (Andreux et al., 2020).
Intermediate scales 2 j/ 2 are obtained by applying a subsampling by 2 after each block of 2 layers.
This introduces intermediate scales and generates a wavelet ﬁlterbank with 2 scales per octave: the ﬁlters are designed so that when j low-pass ﬁlters and one band-pass ﬁlter are cascaded, with a subsampling every 2 layers, the scale of the resulting wavelet is 2 j/ 2 .
Each block comprises in its ﬁrst layer a low-frequency ﬁlter g 10 with γ = − 1 / 2 and band-pass ﬁlters with γ = 0 .
In the second layer, we use the same low-frequency ﬁlter g 20 = g 10 with γ = − 1 / 2 .
The band-pass ﬁlters g 2 (cid:96) are obtained with parameters ξ (cid:48)
= ( π/ √ 2 , 0) , σ (cid:48)
= 1 .
25 (cid:112) 2 / 3 , and s (cid:48) =
√ 0 .
For CIFAR experiments, the J = 8 layers are grouped in 4 successive blocks of 2 layers.
For
ImageNet experiments, the ﬁrst layer consists of band-pass elongated Morlet ﬁlters g (cid:96) and a low-pass
Gaussian window g 0 with γ = 0 , followed by a subsampling of 2 .
The 10 following layers are grouped in 5 blocks of 2 layers.
Optimization We use the optimizer SGD with an initial learning rate of 0 .
01 , a momentum of 0 .
9 , a weight decay of 0 . 0001 , and a batch size of 128 .
The classiﬁer is preceded by a 2D batchnormalization layer.
We use traditional data augmentation: horizontal ﬂips and random crops for
CIFAR, random resized crops of size 224 and horizontal ﬂips for ImageNet.
Classiﬁcation error on
ImageNet validation set is computed on a single center crop of size 224 .
On CIFAR, training lasts for 300 epochs and the learning rate is divided by 10 every 70 epochs.
On ImageNet, training lasts for 150 epochs and the learning rate is divided by 10 every 45 epochs.
All experiments ran during the preparation of this paper, including preliminary ones, required around 10k 32GB NVIDIA V
GPU-hours.