R INGING R E LU S : H ARMONIC D ISTORTION A NALYSIS OF N ONLINEAR F EEDFORWARD N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper, we apply harmonic distortion analysis to understand the effect of nonlinearities in the spectral domain.
Each nonlinear layer creates higherfrequency harmonics, which we call "blueshift", whose magnitude increases with network depth, thereby increasing the “roughness” of the output landscape.
Unlike differential models (such as vanishing gradients, sharpness), this provides a more global view of how network architectures behave across larger areas of their parameter domain.
For example, the model predicts that residual connections are able to counter the effect by dampening corresponding higher frequency modes.
We empirically verify the connection between blueshift and architectural choices, and provide evidence for a connection with trainability.
I NTRODUCTION
In the past decade, the emergence of practical deep neural networks arguably has had disruptive impact on applications of machine learning.
Depth as such appears to be key to expressive models (Raghu et al., 2017).
However, depth also comes with challenges concerning training stability.
Theoretical problems include vanishing and exploding gradients (Hochreiter, 1991), chaotic feedforward dynamics (Poole et al., 2016), or decorrelation of gradients (Balduzzi et al., 2017).
In practice, a number of “recipes” are widely used, such as speciﬁc nonlinearities (Glorot et al., 2011;
He et al., 2015), normalization methods such as batch normalization (Ioffe & Szegedy, 2015), shortcut architectures (Srivastava et al., 2015;
He et al., 2016a;b), or multi-path architecture with (Huang et al., 2017) and without shortcuts (Szegedy et al., 2016).
Broadly speaking, a key research question is to understand how the shape of the network function, i.e., the map from inputs and parameters to outputs, is affected by architectural choices.
Our paper considers speciﬁcally the roughness of the weights-to-outputs function (“ w-o function ”) of non-linear feed-forward networks.
Motivated by the recent visualizations of (Li et al., 2018), which show how depth increases roughness and residual connections smoothen the output again, our goal is to provide an analytical explanation of this effect, and study its implications on network design and trainability.
To this end, we ﬁrst formalize “roughness” as the decay-rate of the expected power spectrum of a function class.
Our main contribution is to then apply harmonic distortion analysis to non-linear feedforward networks, which predicts the creation of high-frequency “harmonics” (thereby “blueshifting” the power spectrum) by polynomial non-linearities with large higher-order coefﬁcients.
Based on this model, we discuss how network depth increases blueshift and thus roughness, while shortcut connections, low-degree nonlinearities and parallel computation paths dampen it.
In relation to trainability, we show an analytic link between blueshift and exploding gradients.
Unlike the former model, the spectral view describes a more global behavior of the w-o function over regions in the parameter domain.
Experiments conﬁrm the theoretical predictions: We observe the predicted effects of depth, shortcuts and parallel computation on blueshift, and are able to differentiate different types of non-linearities by the decay rate of coefﬁcients of a polynomial approximation.
The ﬁndings are in-line with known advantages in trainability of the different architectures.
We further strengthen the evidence by training a large set of networks with a different amount of non-linearity and depth, which shows a clear correlation between blueshift and training-problems, as well as a trade-off with expressivity.
In summary, our paper explains how network architecture affects roughness, shows a connection to trainability, and thereby provides a new tool for analyzing the design of deep networks.
R ELATED W ORK
Vanishing or exploding gradients are a central numerical problem (Hochreiter, 1991;
Pascanu et al., 2013; Yang et al., 2019):
If the the magnitudes of the singular values of layer Jacobians deviates from one, subspaces are attenuated ( | σ | < 1 ) or ampliﬁed ( | σ | > 1 ), potentially cascading exponentially over multiple layers (Pennington et al., 2017a).
Formally, the behavior of stacks of matrices and nonlinear functions can be modeled by random matrix theory or Gaussian mean-ﬁeld approximations (Poole et al., 2016; Pennington et al., 2017b; 2018).
The gist is that at initialization, orthogonal weight matrices are needed, which is challenging for convolutional architectures.
A solution for tanh -networks is given by Xiao et al.
(2018); for ReLU , there is a negative result (Pennington et al., 2017b).
Using mean-ﬁeld theory, it can be shown that batch normalization (Ioffe & Szegedy, 2015) leads to exploding gradients at initialization (Yang et al., 2019)
(which equalize after a few steps, but that might be too late (Frankle et al., 2020)).
A different route is taken by Balduzzi et al.
(2017), who observe an increasing decorrelation of gradients in the input space.
Similar to our paper, they show that deeper networks lead to spectral whitening (starting from brown noise); however, the analysis is performed with respect to the inputs x , not weights W .
The scale-space structure shown by our model might give further hints on the mechanisms behind training difﬁculties.
By visualizing random slices of the loss surface, Li et al.
(2018) observe that the loss surface of deep feedforward networks transitions between nearly convex to chaotic with increasing depth; our work explains these observations by spectral analysis. Duvenaud et al. (2014) visualize pathologies on the landscape of deep gaussian processes that model wide-limit nonlinear networks.
Fourier analysis of network functions (Candès, 1999) wrt.
input (Rahaman et al., 2019; Xu et al., 2019; Xu, 2018;
Basri et al., 2019; Yang & Salman, 2019) has been used to show an inductive bias towards low-frequency functions (wrt.
input x ), as well as a strong anisotropy of this spectrum.
Wang et al.
(2020) prove under some assumptions that all "bad" local minima of a deep residual network are very shallow.
H ARMONIC D ISTORTION
We now analyze the effect of a nonlinearity by relating the Fourier spectrum of a preactivation with that of its postactivation.
Let f denote the preactivation of a single neuron of a neural network consisting of L -layers.
We use x to denote the input to the whole network and thus to f , W to denote the weights, and φ to denote the employed nonlinearity.
Li et al.
(2018) visualize “roughness” using random 2D-slices in weight space.
We follow their basic idea and consider 1D slices
[Equation]
for random directions D and t ∈ [0 , 1] .
D is initialized with entries from N 0 , 1 and normalized to || D || F = 1 .
α determines the path length.
By varying D , this samples a ball of radius α around a point W in parameter space.
For φ = id , the network f is multi-linear in W and thus p is polynomial in t ; empirically, this yields rather smooth functions (Fig. 1).
To understand general non-linearities φ better, we represent p by a complex Fourier series (Appendix B.1 discusses convergence and approximation quality):
[Equation]
Here, the sequence z : Z → C contains the Fourier coefﬁcients for p : [0 , 1] → R .
As p is real, z is symmetric in the sense of z k =
z − k .
3. F ORMALIZING R OUGHNESS
The roughness of a class of random functions can be characterized by the statistics of their powerspectrum (Musgrave, 1993):
Given a random process that generates functions p , we consider mean µ k and variance σ 2 k of the Fourier coefﬁcients z k .
For short path that we use in our experiments, the means µ k are empirically very close to zero (except from z 0 of the ﬁnal loss function). Therefore, we can focus on variance:
In general, functions where the variance of high-frequency components drop off more quickly will appear smoother.
A common model, which often ﬁts natural data well, is fractal Brownian motion (FBM-) noise, where the σ k drop off according to a power law:
[Equation]
The so-called “fractal coefﬁcient” h describes the roughness of the noise function.
In our experiments, we estimate the average power-spectrum E D ( | z k | 2 ) (by sampling D uniformly on a unit sphere) and ﬁt a power-law to these spectra in order to quantify the roughness in a single number h .
Experiments (Section 4) and analytical arguments (Appendix B.2), show that the FBM/power-law model is a realistic model of the functions computed by the lower layers of a neural network.
For higher layers the ﬁt becomes worse, a phenomenon we will explore in the next chapter using harmonic distortion analysis.
3. W HY IS THE O UTPUT F UNCTION G ETTING R OUGHER ?
Intuitively, applying ReLU to a function p is reminiscent of clipping an audio signal in amplitude, which is known to produce high-frequent ringing artifacts.
We describe the effect of a single nonlinearity φ on the spectrum of a layer-output p ; inductively, this describes the spectral shift of the whole network.
For the analysis, we assume that φ is a K -th order polynomial:
[Equation]
The effect of polynomial non-linear maps on the spectrum of a function can be understood by harmonic distortion analysis (see e.g. Feynman et al.
(1965), Ch. 50.8):
We can simply (see
Appendix B.1 for details) plug the Fourier expansion of p into the polynomial representation of φ :
[Equation]
The convolution theorem tells us that j -th power of functions corresponds to convolving the spectrum z of the function j -times with itself.
We designate by z the vector containing all Fourier coefﬁcients z k and by (cid:78) the convolution operator.
We can then write the spectrum of the output z (cid:48) as:
[Equation]
Discussion: We can make three important observations: First, each repeated auto-convolutions in Eq.
6 broadens the spectrum by adding higher-frequency terms.
We call this broadening effect blueshift .
The exact magnitude is hard to quantify and thus left to the experiments (Appendix B. gives informal arguments for a growth of O ( j 1 / 2 ) rather than the trivial upper bound of O ( j ) ).
Second and correspondingly, larger coefﬁcients a j for larger orders j , increase the blueshift.
Third, j -fold convolutions correspond to j -th powers of the z k .
Hence, larger magnitudes | z k | also increase the blueshift (the non-linearity becomes more visible with larger magnitude).
3. C OMMON N ONLINEARITIES
In practice, nonlinearities are usually not polynomial, and might not even have a globally convergent Taylor series.
It is possible to approximate any continuous function by a polynomial (Stone Weierstrass theorem).
We conjecture that a sufﬁciently close approximation will be sufﬁcient for a qualitative prediction of the blueshift effect (our theoretical model does not guarantee convergence – we therefore validate this claim experimentally). We employ a Chebyshef approximation (which is reasonably stable and non-oscillatory) on the interval [ − 5 , 5] and compare the speed at which higher order coefﬁcients drop off: Fig.
2 shows the coefﬁcient magnitudes for ReLU , tanh and softplus (which we focus on in our experiments). softplus has the strongest drop-off, followed by tanh and ReLU .
Appendix D.6 gives more details and covers several additional popular nonlinearities (Figures 18, 19). Figure 3, 17 show that this correlates with blue-shift, as expected.
3. C ONNECTION TO EXPLODING GRADIENTS
The creation of higher-frequency harmonics directly affects gradient-based training methods because the gradient operator is a high-pass ﬁlter in the spectral domain: Taking derivatives multiplies
the
Fourier coefﬁcients by 2 πik , amplifying coefﬁcients linearly with frequency k (which is trivial seen by taking the derivatives of Eq. 2).
The reciprocal fractal exponent r :=
h − 1 has exponential inﬂuence on the average gradient magnitude: With | z k | 2 ∼ Θ( k − h ) =
Θ( k r ) , we obtain gradients k · | z k | ∈ Θ( k 1+ r/ 2 ) .
Blueshift thus causes exploding gradients: Weights of lower layers are shifted more often, creating higher-order harmonics, which correspond to larger norm of the gradient function (cid:107) p [Equation] ( t ) (cid:107) = 2 π (cid:112)(cid:80)
k
k 2 | z k | 2 , which means that there must be, on average, larger gradients along the path considered (at least for some t ).
Independently of this, both exploding and vanishing gradients can be caused by other effects than blueshift: Saxe et al.
(2014) show that even deep linear networks can suffer from exploding gradients caused by stacking matrices with non-uniform singular spectrum.
The blueshift effect itself is independent of the condition number of the weight matrices: blueshift would occur even when only stacking nonlinearities.
Although it is possible to ﬁnd a locally linear region of the loss surface (e.g. CReLU + Looks Linear initialization (Balduzzi et al., 2017)) and locally achieve dynamic isometry (Xiao et al., 2018), blueshift effects become relevant again when this region is eventually left during the training process (ref.
appendix Figure 14).
In contrast, residual connections affect the entire loss surface and our model is able to capture this effect.
3. R ESIDUAL AND M ULTIPATH N ETWORKS
Residual Networks exhibit two different dampening effects on the loss surface that can be experimentally isolated:
Exponential downweighting (ED): Residual networks repeatedly compute the sum of the result of a small feedforward network with its input.
By construction, this reduces the relative weight of the non-linear output of the residual block by mixing it with the unprocessed signal.
Doing this repeatedly creates an ensemble (Huang et al., 2016) in which the weight of contributions shrink exponentially with the number of non-linear processing steps.
As detailed e.g. in (Veit et al., 2016), each non-linear block only contributes partially to the overall result.
The weight of the signal component passing through non-linearities shrinks exponentially due to vanishing gradient effects (at initialization, nonaligned non-uniform singular value spectra lead to exponential dampening; after training, a similar dampening effect is observed empirically by Veit et al.).
Frequency-dependent signal-averaging (FDSA):
According to the blueshift model, the outputs of the residual blocks will contain more high-frequency Fourier coefﬁcients (from harmonics).
The additional harmonics depend on the weights in the residual block, which are statistically independent of the input (see Appendix B.5 for details).
Therefore, we expect to see a reduction in the expected linear correlation between the main and residual branch for higher frequencies.
As an effect, when adding the main and residual branch, higher frequencies get dampened even more.
l n ( po w ) →
ReLU
TanH
SoftPlus ln( freq ) → l n ( po w ) → ln( freq ) → ln( freq ) → s h
o r t
c u t s n o - s h o r t
ReLU
TanH
SoftPlus
[Figure] [Figure]
[Figure] [Figure] [Figure] [Figure]
[Figure] [Figure] [Figure]
−
[Figure]
4[-−−   4     −−   
-]  ln( freq ) → ln( freq ) → ln( freq ) →
The second effect is weaker than the ﬁrst, governed by the law of large numbers (i.e., n − 12 decay for averaging n independent values), but is also more broadly applicable: It should generally occur when averaging over multiple computational paths with independent weights, while residual connections require an identity in one path.
E XPERIMENTAL R ESULTS : M EASURED S PECTRA
The model presented in the previous section gives us only qualitative hints on the magnitude of the blue-shift, the behavior of non-polynomial non-linearities, and the magnitude of FDSA-effects. We therefore now validate these qualitative predictions experimentally. We consider two architectures: a basic Toy-CNN model with a constant number of features in each layer for fair per-layer statistics and ResNet v1 variants for a more "realistic" model. We used Cifar10 as input data, but all ﬁgures in this section look qualitatively similar on other datasets; A more detailed network description, parameters and experiments on MNIST can be found in the appendix.
In this section, we always measure the effect of blueshift in the region around initialization (i.e., W is the initialization point);
For a visualization during training, see the Appendix (Figure 11).
4. S PECTRAL S HIFT
Our ﬁrst goal is to validate the harmonics creation and in particular measure the blueshift caused by multiple layers, i.e., the effect of deeper networks on the expected power spectrum.
We also validate the connection between the blueshift strength and the presence of larger higher-order polynomial coefﬁcients for the non-linearity.
We use a “Toy-CNN” with depth L = 50 and track the w-o surfaces of each layer while we walk in a single random direction in weight space.
We transform the resulting 1D-paths with discrete FFT and all magnitude spectra are averaged over all neurons in a layer, all input receptive ﬁelds and all batch images and then normalized by function norm.
Fig.
3a (left side) shows the resulting average magnitude spectrum |
z k | for each Fourier coefﬁcient-.
The right hand side (b) shows the same plot, but with D restricted to only changing the weights of the ﬁrst layer of the network, and taking the derivative of the resulting function (by weighting by frequency, k · | z k | ) before normalization.
Spectral shift over depth: The results in Fig.
3a conﬁrm our predictions quite well: A spectral blueshift is clearly visible and, as expected, ReLU shows a strong blueshift, tanh a slightly weaker blueshift, and softplus only a small effect.
Harmonics show up as a bump in the spectrum that travels towards higher frequencies with increasing layer.
This extends to other non-linearties as well
Frequency bin
A v e r a g e a b s . c o rr e l a ti on ReLU SoftPlus TanH (see Appendix, Fig. 17).
The plot also conﬁrms that the power spectrum of the lower layers is well described by a 1 /k -power law, as predicted.
A slight upward slant is visible at the high-frequency end of all plots.
This is an aliasing phenomenon: As we use only 100 samples without prior bandlimiting, aliasing frequencies are reﬂected back at the end of the spectrum, bending the curve slightly upward.
Scale shift of a single layer parametrization: By only changing the weights in a single layer and taking the derivative (Fig. 3b), we see that blueshift is responsible for a gradient scale mismatch between earlier and deeper layers in networks without skip-connections: the more nonlinearities are between the modiﬁed weight and the output, the higher the gradient magnitude.
Residual connections: Our model predicts that residual connections will reduce the blueshift strongly by ED and FDSA (ref.
Section 3.5).
This is also consistent with our observations: higher layers show almost the same 1 /k
-spectrum
as the initial layer.
Looking at the ﬁrst layer weights, weighted by frequency, the response is almost ﬂat, with a small emphasis on low-frequency weights, and again a bump at the upper end of the spectrum presumably due to aliasing (which is more pronounced for frequency weighting).
4. E FFECT OF NETWORK DEPTH ON NETWORK SMOOTHNESS
The spectral shift in the w-o function causes it to become rougher with network depth.
We quantify this phenomenon by ﬁtting a power law to the average power spectra | z k | 2 using non-linear leastsquares: h, b = arg min h,b ∈ R (cid:16) bt h − | z k | 2 (cid:17) 2 , t ∈ [0 , 1] .
This way we can validate that the visual results of Li et al. (2018), that consider only one random direction, are actually consistent over many random directions.
In Figure 3, we assess the average loss path smoothness of a Toy-CNN with varying depth at initialization. For averaging, we initialize each network 50 times and sample 20 paths in different random directions for each initialization.
The results conﬁrm quantitatively the visual results of Li et al.
(2018) that the loss surface becomes rougher with increasing network depth and that shortcut connections have a strong smoothing effect.
The power-law ﬁt still has some limitations: As seen in
Fig.
3, a power-law ﬁt is imperfect as the higher layers are not straight lines anymore.
This shows in curves of Fig. 4 as apparent increase of smoothness in the ﬁrst layers for the "NoShort" architectures.
Figure 6 represents what the "average loss path" looks like for a ResNet 56 at initialization with different activation functions, with and without shortcuts. For each architecture, we sample paths in 500 different random directions D and assess the roughness of every sampled path via a power-law ﬁt of the magnitude spectrum. We then select and visualize the path with median roughness value.
We clearly see that nonlinear architectures without skip-connections present most blueshift, the linear architecture has no blueshift and the architectures with skip-connections do also exhibit, albeit dampened, blueshift. [Figure]
4. R ESIDUAL NETWORKS CONTROL HARMONICS / AVERAGING NETWORKS
In Section 3.5, we described two different mechanisms that allow ResNets to dampen high-frequency modes.
We already saw that residual connections have a dampening effect on the loss surface, but we want to individuate both effects.
FDSA predicts that skip-connections have a dampening effect on high frequencies in the loss surface because the correlation between the main branch and the residual branch is decreasing with higher frequencies.
We empirically verify this prediction on a ResNet56 v2 at initialization by walking inrandom directions and measuring the respective w-o function (we only sample 10% of the outputs for computational reasons) of the two summands in each residual block just before addition.
Using
Gaussian ﬁlters in Fourier domain, we can compute the linear correlation of each pair of paths (block input/output) per frequency bin and average over all batch images, neurons and blocks in the network.
In Figure 5, we see that the absolute linear correlation is decaying for all architectures with increasing frequency, with ReLU showing the strongest effect, again followed by tanh and softplus in that order, conﬁrming our prediction.
This property seems to allow a residual network to automatically regulate the higher-frequency harmonics content of the output.
We want to individuate FDSA in a setting where ED is not present.
We modify our Toy-CNN and
ResNet "NoShort" architectures such that we replace every n -feature convolution / batch normalization / activation sequence in the network with a a · n feature sequence where the a feature groups get averaged after the activation (in reality we only use summation, since normalizing is taken care of by batch normalization).
Since high frequencies added by the nonlinearity are less correlated between features (ref.
Fig. 5), we expect that averaging feature groups smooth the w-o function; we see on the right of Figure 9 that this is indeed the case.
Finally, we check that even in a setting with ED , FDSA is still present.
Our model predicts that since
ResNet v2 blocks (He et al., 2016b) use summation after ReLU, we will see a stronger averaging effect than in ResNet v1.
Indeed, looking at their respective average power spectra in a random direction in Figure 8, we see this conﬁrmed.
4. V ISUALIZING REGIME TRANSITIONS CONTINOUSLY
We want to continuously visualize how a loss path gets rougher the more nonlinearity (and thus blueshift) is present in a network and how it can be smoothed again by the use of residual connections.
It is possible to make the transition from a linear to nonlinear network continuous by using Leaky ReLU activations and tuning its slope parameter α : 1 → 0 . Equally, it is possible to make the transition of a feedforward network to a residual network continuous by using a ResNet where the residual branch of {+each residual block is rescaled by a constant factor ν : 
+}
→ 1 before addition. In Figure 1, we observe the continuous transition of a given single loss path of a ResNet56 at initialization from a linear feedforward to a nonlinear feedforward to a nonlinear residual regime.
E MPIRICAL R ESULTS : T RAINING
In this section, we want to demonstrate a correlation between blueshift and network trainability.
Again, we use variants of the ResNet v1/v2 network, exact network parameters and training hyperparameters can be found in tabular form in Appendix A.
We use Cifar10 as training data, training results for
Cifar100 can be found in Appendix C. We hold out 1% of the training data for validation.
Controlling Blueshift with Leaky ReLU: As demonstrated in Figure 1, the amount of blueshift in a network can be controlled by tuning Leaky ReLU’s negative slope α .
We now want to show that networks with a very strong blueshift are hard to train but conversely some amount of nonlinearity is needed for expressivity.
For this, we train a ResNet56 v1/v2 with and without shortcuts forepochs once on Cifar10 for different values of α and visualize the last training accuracy (average over the last 25 values) and compare it to the blueshift of the network at initialization.
On Figure 7, we see that for the networks without skip-connections, training becomes more difﬁcult with regard to network depth, which correlates with increased blueshift.
We see that this effect can be alleviated by making the network more linear (by increasing α ) and thus reducing blueshift.
For networks with skip-connections, we see that blueshift is greatly reduced and that no deterioration of trainability with regard to network depth is noticeable. Conversely, networks that near a linear regime for high values of α tend to train worse since they lack expressivity. Interestingly, this effect is stronger for ResNet v2 blocks than for ResNet v1 blocks because the former make the loss surface smoother than v1 blocks (ref.
Figure 8). Controlling Blueshift via feature averaging: We now want to investigate the impact of the smoothing of the loss surface via FDSA (ref.
Section 3.5) on training speeds.
We choose a
ResNet56 "NoShort" as our architecture, since at this depth simple feedforward networks with batch-normalization start to become difﬁcult to train.
We experiment on the averaging network (ref. Section 4.3) with a = 3 since we experimentally determined that for a > 3 the performance stops increasing.
For fairness, we included a Wide ResNet56 "NoShort" with approximately the same parameter count than the averaging network.
We see the averaged results over ﬁve runs in Figure(shaded areas represent the standard deviation).
We observe that the averaged network outperforms both the original network and the wide network while still performing worse than the network with skip-connections. This is consistent with our theory since ED has a bigger smoothing effect than FDSA .
Frequency
P o w .
s p ec
t r u m v1Blocks
v2Blocks
[Figure]
[Figure]
−
[Figure]
−l og ( m a g ) a =    −−l og ( m a g ) a =    −−l og ( m a g ) a = log(freq)
D ISCUSSION
Our experimental ﬁndings validate the predictions of our model with respect to the roughness of the w-o function: Nonlinearities with larger higher-order polynomial coefﬁecnts create larger variances in the high-frequency part of the spectrum.
While related results on increasing complexity with depth have been given earlier (Poole et al., 2016; Schoenholz et al., 2017), the harmonics model offers a simple and more global view of the roughness of the loss surface and how it is affected by nonlinearity choice and residual connections.
Our observations on training speed are consistent with the hypothesis that spectral blueshift impedes training: Architectural choices (residual connections, ResNet v2 blocks, LReLU) that control harmonics generation are also those that are easier to train to the their full performance.
Further, the results in Figure 7 give strong hints for a correlation between high-frequency content and problems in trainability for deep but very non-linear networks (small α ).
From an analytical perspective, architectures that are prone to blueshift are prone to vanishing/exploding gradients:
We have shown that high magnitudes of high-frequencies in the w-o functions lead to an increase in the expected gradient magnitude.
Further, in a simple multi-layer network without shortcuts, we expect more high-frequency content on lower than on higher layers as soon as the non-linearity of the activation function is actually is utilized (and therefore must also creates harmonics).
A rescaling of layers can reduce these discrepancies, but the normalization will still necessarily depend on the position in parameter space.
As far as accuracy is concerned, there is an obvious trade-off between blueshift and expressivity: networks with low-degree polynomials as nonlinearities contain less high-frequent harmonics but cannot approximate complex functions (Kidger & Lyons, 2020).
Future work could consist in further exploring this trade-off.
Further, it would be interesting to examine potential architectural alternatives to residual connections with comparable performance and trainability.
C ONCLUSION AND F UTURE W ORK
In this paper, we have analyzed the effect of different architectures of non-linear networks on the smoothness of the computed function (various neural outputs and the loss landscape).
Speciﬁcally, we have linked polynomial non-linearity to harmonics creation and validated experimentally that the mean power spectrum shifts towards higher frequencies (“blueshift”).
Further, we have also described two distinct effects that explain the smoothing effect of shortcut connections, which we also conﬁrmed experimentally.
Finally, we empirically linked reduced blueshift to increased trainability and training speeds.
In future work, we hope to derive explicit closed-form equations for how architectural designs like network depth, residual connections and nonlinearity choice affect the w-o function.
It could be also interesting to further explore the roughness-expressability trade-off that we described earlier.
A N ETWORK DETAILS A. N ETWORK ARCHITECTURES
The implementation for our experiments is based on PyTorch 1.5 and will be provided on Github.
Toy-CNN: We create a simple convolutional network consisting of units of 3x3-convolutions with 16 feature channels on all layers ( 16 · a for averaging), followed by batch normalization and nonlinear activation, stacked-up L -times.
Optional short-cut connections are added (only if explicitely stated) between each unit (i.e., residual blocks of depth 1).
No pooling or striding is used.
The input layer uses a stride of 5 to reduce memory demands.
The classiﬁcation layer is composed of a fully-connected layer, softmax and cross-entropy loss.
Residual Network:
As a “real-world” example, we use a standard Residual Network v1 (BasicBlock)
He et al.
(2016a) with varying depth, with ("Short/S") and without ("NoShort/NS") skip-connections.
ResNet v2 blocks are used in Figure 1 (v2 blocks show slower transitions) and 5 (correlations are decaying faster in ResNet v2, ref. 3.5).
We use the standard number of planes for Cifar-10 training : 16/32/64 (28/56/112 for the wide network).
A. F OURIER WALK HYPERPARMETERS
We sample paths consisting of 100 samples each and with path length
α = 1 (Eq. 1) in all experiments, except for Figures 1, 6 and 5 where we double both values for higher frequency resolution and Figure 14, where we want to walk in a different radius ( α ∈ { 0 . 1 , 1 , 10 } ).
All experiments are performed in training mode and with data augmentation at initialization.
Datapoints for each architecture are measured on a different mini-batch to exclude bias by batch selection.
A. T RAINING HYPERPARMETERS
The hyper-parameters below usually reach the standard test-accuracy of approximately 92 - 93% for a
ResNet56 on Cifar10.
Dataset
Cifar10 (Cifar100 for Figure 10)
Epochs
Scheduler
Multistep ( γ = 0 . 1 )
Milestones 100,
Learning rate 0.
Batch size
Optimizer
SGD + Momentum
Momentum 0.
Weight decay 0.
Augmentation
Random Flip
B F ORMAL D ETAILS C ONCERNING H ARMONIC N ETWORK A NALYSIS B. S ERIES R EPRESENTATION AND C ONVERGENCE
In our paper, we consider feed-forward networks consisting of L layers.
Each layer l = 1 ...
L can be one of the following:
• An afﬁne map
[Equation]
that transforms the input using a linear map and an offset vector.
• A non-linearity
[Equation]
that applies a (potentially layer-speciﬁc) non-linear function to each input.
All of the discussed architectures (including convolutional networks, striding/average pooling, residual connections, parallel computations, and layer-dependent non-linearitiy) can be expressed as a composition of these two layer types.
We currently do not address max-pooling, and understand batch-normalization layers statically as a ﬁxed scaling and shifting (not analyzing their training dynamics).
The loss function itself (such as a sequence of softmax and cross-entropy) is also not included in our analysis – we restrict our consideration to layer outputs before any classiﬁcation.
As logistic regression (or similar problems) are numerically well-understood already, this is not a major obstacle.
All our experiments show layer outputs before classiﬁcation (softmax).
The main restriction of our formal model is that we assume that all non-linearities are polynomials of ﬁnite degree 1 .
Under this condition, all outputs and intermediate results of the whole network can be represented by a large multi-variate polynomial, and any linear section p ( t ) =
f ( l ) i ( W + αt D , x ) of any output i of any layer l is a univariate polynomial of ﬁnite degree.
Fourier analysis: Polynomial functions on a ﬁnite domain are Lipschitz-continuous and have ﬁnite variation; both conditions are sufﬁcient for the convergence of the Fourier series in an L 2 -sense.
Further, as the Fourier-basis forms a Schauder-basis of L 2 [0 , 1] , the series expansion is unique.
In our derivation (Equation 5), we plug the series representation into the polynomial and conclude that it can be represented in the spectral domain as a sum of convolutions (Equation 6).
To make this more rigorous, we proceed in smaller steps: Let again p : R n → R denote the linear section along a linear direction in parameter space of any layer output.
Our non-linearity expands to
[Equation]
As p ( t ) is a polynomial of ﬁnite degree, p ( t ) j for any ﬁnite j is also a polynomial of ﬁnite degree.
This implies that there is a unique Fourier series that represents p , characterized by the coefﬁcients sequence z : Z → C .
According to the convolution theorem for the complex Fourier series, the
Fourier series of p · p is given by z ⊗ z , where
[Equation]
Because the series must exist (it converges for both p and p 2 ) and it is unique, the product of the series must be represented by the auto-convolution.
Inductively, we obtain p j corresponding to the j -fold auto-convolution (cid:78) jq =1 z .
Note: We use the notation [ x ] i := x i denote the indexing of vectors and sequences that result from a computation.
Approximation quality and ﬁnite transforms: The Fourier representation is accurate in an L 2sense; it converges in L 2 -norm, which is an integral measure. This still permits any ﬁnite series expansion to have large point-wise errors (no convergence in inﬁnity-norm: large error regions rather shrink to a zero set in the limit). A common example of such problems is Gibb’s phenomenon: A discontinuous function (such as the gradient function of a ReLU) will show “overshooting” of constant magnitude near the discontinuities; only the area affected shrink with the frequency order of the Fourier series.
Further, even higher-order discontinuities lead to signiﬁcant high-frequency harmonics (i.e., show bad approximation behavior for truncated series).
While the theory is not affected by this, this might appear to be a concern for the discrete Fourier transformation (DFT) we use in our experiments.
The DFT operates at a ﬁnite frequency order, using a matching set of regular samples of the continuous function.
Here, it is important to note that the DFT, as a mapping from C d to C d will reproduce the function values faithfully at each sample point for any frequency order (the DFT is bijective and even unitary, thus even allowing numerically accurate and stable reconstructions).
Evaluating the obtained series in between sampling points, however, would reveal issues at discontinuities; as in most applications of DFT, our experiments do not perform such continuous evaluations.
An issues that does remain is aliasing: Non-linearities in general and in particular discontinuous ones (even with higher order discontinuities) broaden the spectrum (this effect is particularly strong for ﬁrst-order discontinuous functions such as ReLU).
Therefore, the sampling frequency has to be chosen sufﬁciently high to capture these effects.
Otherwise, the high-frequency harmonics will alias as lower-frequency signal components, which might lead to an underestimation of the blue-shift effect.
B. I NPUTS TO THE S ECOND L AYER N ON -L INEARITIES ARE FBM-N OISE
For the ﬁrst layer of a ReLU network, the following calculation also supports the experimental ﬁndings of an O (1 /k ) -power law from an analytical perspective.
Let p (1) be a w-o path of the output of a given neuron from the ﬁrst layer.
It follows that:
[Equation]
This is a piecewise linear function that is constant zero on one interval and linear with slope m on another interval.
With a < b ∈ [0 , 1] bounding the activation interval, the Fourier series becomes:
[Equation]
By the chain rule of derivations, this implies
[Equation]
Correspondingly, the preactivations of the second layer will be random linear combinations of FBMnoise with h = 1 , which again forms FBM-noise with that spectral variance (every Fourier coefﬁcient is a sum of independent coefﬁcients; therefore, variance is additive).
Similar ﬁndings where also previously shown in input space by Balduzzi et al.
(2017), which, for the ﬁrst layer, is just the dual to varying the weights (and therefore must yield the same result).
B. B LUESHIFT AND D EPTH
We now aim at formally understanding the blueshift effect in a multi-layer network.
For simplicity, we consider a simple stack of linear and non-linear layers with a ﬁxed non-linearity: [Equation]
[Equation]
We now consider a single output i of layer l :
[Equation]
Next, we consider a linear section t (cid:55)→ W + t D with t ∈ [0 , α ] : p ( l ) i ( t )
[Equation]
By replacing φ with ﬁnite polynomials (Equation 4), we obtain multi-variate polynomials as outputs of all of these function.
Varying parameters in a single layer: If we now select a layer l = k by setting all D ( i ) =
for all i (cid:54)
=
k , and only using the k -th matrix as direction vector, the parameter t will pass through l − k +non-linearities:
[Equation]
with a constant vector q ( i − 1) .
Correspondingly, each non-linearity will act on the spectrum of q ( k ) as a blue-shift operator (forming the sum of repeated auto-convolutions of the spectrum, weighted by the polynomial coefﬁcients).
Therefore, we see that parameters of earlier layers (small layer index k , closer to the input layer) are blueshifted more frequently than later layers.
The equation also shows that the singular value spectrum of W ( l ) affects the results as well, in addition to the spectral shifts caused by the non-linearities: the values rescale the input domain (shrink/expand space wrt.
parameter t ), thereby changing the frequency reciprocal to the scale factor.
However, a linear transformation scales the Fourier spectrum as a whole while non-linearities spread the spectrum irreversibly by applying blueshifts (weighted sums of auto-convolutions of the spectrum).
The magnitude of this non-linear effect is, nontheless, dependent on the signal magnitude and thus affected by choices of W ( l ) .
Varying parameters in layer 0 ..
l : Similar arguments hold if we vary only parameters in the ﬁrst l layers of the network.
However, we obtain a mix of spectra that have been blueshifted (and scaled by weight matrices) a different number of times.
B. S PECTRAL B ROADENING THROUGH A UTO -C ONVOLUTIONS
The exact amount of broadening of a spectrum z : Z → R by an auto-convolution is non-trivial to quantify; we therefore resort to experiments for determining the effect in practice.
In special cases, we can analyze the effect:
Band limited functions: Assuming that z is bandlimited, i.e., z k
=
z − k = 0 for all k > K , and | z k | = |
z − k | > 0 for k ≤ K it is trivial to see that an auto-convolution extends the support by k entries.
This means, after j convolutions, we obtain a non-zero spectrum within k = − jK...
jK .
The proof is obtained by considering the left-most and right-most terms in the deﬁnition of the discrete convolution.
Central limits: If we assume that z is real and positive, and has ﬁnite second moments, the central limit theorem guarantees convergence towards a Gaussian function with standard deviation O (cid:0) √ j (cid:1) .
The ﬁrst assumption is obviously unrealistic in practice.
In the general case the j -th power of a Fourier series
[Equation]
18) expands to the sum
[Equation]
If we only count the number of terms of the same frequency k = k 1 + · · · + k j , the number of occurrences of each frequency k will tend towards a normal distribution with variance j .
However, as the complex coefﬁcients can cancel out, the limit distribution is only an upper bound.
B. F REQUENCY D EPENDENCE IN A VERAGING OF M ULTIPLE C OMPUTATION P ATHS
We observe that averaging functions after non-linearities leads to some smoothing of the result by decreasing the correlation between higher-frequency Fourier coefﬁcients.
In order to understand the effect, we look at a simple 1D model case: We consider a the composite functions
[Equation]
and
[Equation]
where w 1 , w 2 are random numbers, drawn from the same normal distribution.
Let z k be the sequence of Fourier coefﬁcients of g , and u k that of the result.
In the ﬁrst case, f post , a random linear combintation of the input spectra is performed, which is then blue-shifted:
[Equation]
In the second case, u =
[Equation]
(23) we ﬁrst blue-shift the randomly-scale spectra and then form the sum.
This creates a frequency dependence because larger powers (larger values of j , corresponding to more repeated convolutions, and thus j -fold products of input Fourier coefﬁcients) behave more non-linearly and thus react more strongly to the random scaling: While a linearly weighted average will be linearly correlated with the original, averages of non-linearly transformed functions will loose linear correlation (and thus appear more random to a simple averaging operation).
The harmonic distortion analysis that expresses the powers as auto-convolutions of the spectrum shows that higher-frequency components (created by the blue-shift) are also the ones that behave more non-linearly. [Figure]
C T RAINING ON C IFAR
We repeat the experiment on averaging-networks for the Cifar100 dataset.
We leave all hyperparameters untouched.
In Figure 10, we still see the advantage of the averaging-network over the regular network, especially in validation accuracy.
D O THER SPECTRAL SHIFT FIGURES D. S PECTRAL SHIFT DURING TRAINING
To show that the blueshift effect is not limited to the initialization, we show the spectral shift of a Toy-CNN 50 at 0/20/200 epochs of training in Figure 11.
ResNets still do not show any major blue-shift at any time.
The architectures without skip-connections show blueshift at any time, albeit progressively less pronounced, indicating that blueshift is more of a problem in early phases of training (Frankle et al., 2020).
D. S PECTRAL SHIFT IN GRADIENT DIRECTION
The subspace of the loss surface where gradient descent operates has been studied (Gur-Ari et al., 2018) and attributed a speciﬁc behavior.
We want to see how blueshift behaves when slicing the network in the direction of gradient descent.
For this, we repeat the experiments of Figure 3, but compute the gradient at initialization, normalize it and use it as direction D of equation 1.
In Figure 12, we see that the effect is still visible although dampened for ReLU and tanh activations.
The effect seems a little more visible for softplus activations.
D. S PECTRAL SHIFT WITH DELTA - ORTHOGONAL INITIALIZATION
The experiments of Figure 3 with batch normalization are subject to exploding gradients.
Xiao et al.
(2018) describe an initialization scheme that ﬁxes exploding gradients for TanH activations.
Figshows that for a very small region near the orthogonal initialization point, blueshift vanishes.
As soon as we walk further from the initialization point, blueshift becomes visible again.
D. W ALK PER LAYER
To see the contribution of each layer to a loss path, we sample a random direction D , but measure the resulting loss path when only modifying weights in a single layer.
We realize this for a Toy-CNN with 50 layers at initialization in Figure 13.
We see that varying weights in higher layers (red) results in smooth paths whereas varying weights in lower layers (blue) results in rougher paths.
In the normalized power-spectra, we can see the increased blueshift of paths resulting from varying earlier layers.
[Figure] [Figure]
SoftPlus
[Figure] [Figure] [Figure]
D. V ARYING WIDTH
To see if layer width has any inﬂuence on blueshift, we repeat the experiment of Figure 3 for a
Toy-CNN 50 with number of ﬁlters w .
We can see on Figure 15 that layer width doesn’t seem to have a notable impact on blueshift strength.
D. M ORE ACTIVATION FUNCTIONS
To demonstrate that blueshift occurs with every nonlinear activation function, we repeat the experiment of Figure 3 for exponential linear units (ELU), gaussian error linear units (GELU) [Figure]
− [Figure]
−  log(freq)
l og ( m a g )
NS (w=10)  4 log(freq)
NS (w=50)  4 log(freq)
NS (w=90)
[Figure]
−  log(freq) [Figure] og ( m a g ) [Figure]
[Figure]
[Figure] log(freq) ELU + NS   −− log(freq) GELU + NS
[Figure] log(freq) HTANH + NS
[Figure] log(freq) LRELU + NS
[Figure] log(freq) SELU + NS
[Figure] log(freq) SIGMOID + NS
[Figure]
− log(freq) ELU + S   −− log(freq) GELU + S   −− log(freq)
HTANH + S   −− log(freq)
LRELU + S   −− log(freq) SELU + S   −− log(freq)
SIGMOID + S
In order to demonstrate the that spectral spread depends on the magnitude of higher-order polynomial coefﬁcients, we show the plot of the magnitude of the coefﬁcients of a polynomial approximation with Chebyshev nodes of degree 25 in Figure 19.
The ﬁt is performed for the interval [ − 5 , 5] (slightly different from Figure 2 in the main paper, which shows a ﬁt for [-1,1]).
Figure 19(a) also shows again the comparison between ReLU , tanh , and softplus , uniformely with this Chebyshef approximation; this shows the differences more clearly (please also note the logarithmic scale of the y -Axis).
The approximations are quite tight within the interval, see Figure 18 for reference.
E S MOOTHNESS MEASUREMENTS ON MNIST
To show that our smoothness measurements are mostly independent of batch size and dataset, we revisit Figure 4.
This time, we use the MNIST dataset and a batch size of 256.
All other hyperparameters are maintained.
We see that although the absolute values slightly differ, the relative behavior of the nonlinearities is qualitatively the same.
The results are shown in Figure
Polynomial Chebyshev approximation of various nonlinearities of degree 25 within [-5,5]
[Figure]
a j | (for the degree 25 approximation of Figure 18)
[Figure]
[Table]