T HE S PECTRAL B IAS OF P OLYNOMIAL N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Polynomial neural networks (PNNs) have been recently shown to be particularly effective at solving the tasks of image generation and face recognition, where high-frequency information is critical.
Previous studies have revealed that deep neural networks demonstrate a spectral bias towards low-frequency functions which yields faster learning of low-frequency components during training.
In
- spired by such studies
, we conduct a spectral analysis of the
Neural Tangent Ker- nel (
NTK
) of
PNNs
We ﬁnd that
the
Π -Net family, i e.
, a recently proposed parametrization of PNNs speeds up the learning of the higher frequencies.
In addition
, we show that such a theoretical bias is satisﬁed in practice through extensive experiments.
We expect our analysis to provide novel insights into designing architectures and learning frameworks by incorporating multiplicative interactions via polynomials.
I NTRODUCTION
Deep neural networks (DNNs) have demonstrated remarkable success in different domains [21, 10].
DNNs can approximate complex functions or even datasets with randomized labels arbitrarily well [46, 2], which makes their ability to avoid over-ﬁtting on real data surprising, since it seems to disagree with prior notions of model complexity.
This has sparked the interest in investigating the notion of an “implicit bias” in neural network training, which makes them favor low complexity solutions [41, 20, 25].
The spectral analysis of deep networks offers another perspective on this implicit bias.
Deep neural networks demonstrate a learning bias towards low frequency functions - i.e. functions that vary globally without local ﬂuctuations are learned faster when training neural networks via gradient descent [35, 45].
The phenomenon, termed as the spectral bias of neural networks [35], has also been explored from the perspective of Neural Tangent Kernel (NTK) [23] regime.
The eigenvalues of the obtained kernel inﬂuence important characteristics such as the approximation properties and rate of learning [9, 37].
For standard two-layer ReLU networks within the NTK regime, the analysis supports the idea of a spectral bias by showing faster error convergence for information in lower frequencies [11].
Apart from DNNs, multiplicative interactions have demonstrated strong empirical performance on image-based applications [42, 44, 3].
Jayakumar et al.
[24] highlight how multiplicative interactions, which construct second degree polynomials, can enlarge the hypothesis space and how this reﬂects as faster learning for certain classes of functions.
Polynomial neural networks (PNNs), that express high order polynomial expansions, have demonstrated state-of-the-art performance in the challenging tasks of image generations [27] and face recognition [16].
Both tasks rely on ﬁne-grained details, which correspond to high-frequency information.
To understand the success of PNNs, we conduct a spectral analysis inspired by the spectral analysis on DNNs.
We focus on one such instance of polynomial networks called Π -Nets [15], wherein the output is a piece-wise polynomial function of the input obtained via multiplicative layers.
The parameters of a Π -Net can be represented as high-order tensors, while polynomial expansions can offer increased representation power [16].
Our main contributions can be summarized as follows:
We analyze two-layer polynomial networks in the NTK regime.
By studying the spectral properties of the corresponding kernel, we prove a theoretical speed-up in learning higher frequencies
over standard neural networks and validate the hypothesis in the approximate NTK regime, on the task of learning spherical harmonics.
Beyond the NTK regime, we demonstrate this enhanced bias of Π -Nets towards higher frequencies in several experimental settings, beginning from synthetic learning tasks and then proceeding to state-of-art networks and inverse problems with 2D images.
Aside from improving the understanding of polynomials in neural networks, our proposed analysis also sheds new light on the effect of multiplicative interactions in neural networks, prevalent in certain domains of machine learning including vision and natural language processing [28, 6].
R ELATED W ORK
Spectral Bias : The spectral bias represents a form of implicit bias.
Basri et al.
[7] study the gradient descent training dynamics in the NTK setting for 2-layer ReLU networks with bias (ﬁxed outer layer) to demonstrate the spectral bias phenomenon.
The explicit inclusion of the bias term speeds training for odd-harmonics, but the eigenvalues of the obtained kernel show an identical asymptotic decay rate.
Basri et al.
[8] extend the work to account for non-uniform data distributions.
To overcome the issue of spectral bias towards low frequencies, Tancik et al.
[43] propose the use of hand-crafted random Fourier features from the data points and feeding it as inputs to multi-layer perceptron networks.
However, the method requires low dimensional input data and thus remain restricted to applications which use co-ordinate based MLPs.
Polynomial neural networks (PNNs) : Older works that explore polynomials in the context of neural networks is mainly divided into two categories: 1) self-organizing networks with hard coded features [22], 2) Pi-Sigma networks [39, 33].
In both cases, the polynomial features do not scale well for higher dimensional inputs, and are not used in high-dimensional signals, such as images.
More recent works have used the Hadamard product for capturing correlations between different branches of the architecture.
The goal of such methods [3, 42, 44, 16] is to construct new architectures that capture higher-order order correlations between the features.
This differs from our goal, since we do not aim to construct a new architecture, but rather to analyze the properties of polynomial neural networks through a spectral analysis.
Polynomial activation functions (PAFs) : Polynomial activation functions expand (element-wise) each feature to an r th degree, i.e., they assume a (deep) neural network where the element-wise activation functions are r th degree polynomials.
This is substantially different from capturing higher-order correlations across input (or feature) elements like PNNs.
Theoretical work on overparametrization [17], expressive power [29] and generalization of shallow nets [34] have emerged for
PAFs.
Beyond the differences between PAFs and PNNs, the aforementioned works do not conduct a spectral analysis and do not exhibit the beneﬁts of PNNs for learning high-frequency information.
A NALYSIS OF P OLYNOMIAL N EURAL N ETWORKS IN THE NTK REGIME
In this section, we conduct a careful analysis of the kernel approximation of polynomial neural networks (PNNs) to gain insight on the effect of the multiplicative interactions.
PNNs include multiplicative interactions and express high-degree polynomial expansions.
The recent parametrization of the Π -Net family [15], which we summarize below, is used as a representative PNN.
We review the tangent kernel approximation of neural networks and then we derive the tangent kernel of twolayer Π -Nets.
This allows us to conduct a spectral analysis to better understand the approximation properties as well as the spectral bias of Π -Nets.
Notation : We denote by x¨ , ¨y the standard inner-product on R d .
For two vectors x , y P R d , x ˚ y denotes the element-wise or Hadamard product.
We use ˆ m to denote the mode- m vector product 1 .
We deﬁne the asymptotic notations Ω p¨q and ˜Ω p¨q as follows: Let a n and b n be two sequences.
We write a n “ Ω p b n q if lim inf n Ñ8 | a n { b n | ą 0 .
We use ˜Ω p¨q to hide the logarithmic factors in Ω p¨q .
3. Π -N ET F ORMULATION
A polynomial expansion of the input vector z P R δ can be used to express the output x P R o as an
N th degree polynomial expansion as follows:
[Equation]
where β P R o and (cid:32) W r n s P R o ˆ ś nm “ 1 ˆ m d ( N n “ 1 are the learnable parameters, and ˆ m is the modem vector product.
Since the number of tensor parameters W r n s grow exponentially with the degree of the polynomial, a coupled tensor decomposition with factor sharing can be used.
The idea in Π -Nets is to propose such decompositions that can capture higher order correlations, and can be implemented in standard deep learning frameworks.
One such decomposition expresses the output x using the following recursive formulation:
[Equation]
The second term is exactly the recursive form of a standard neural network (without activations) and therefore in practice, Π -Nets simply augment standard neural network with multiplicative interactions via the Hadamard product.
While Π -Nets can approximate without activations, it is important to note that Π -Nets achieve state-of-art performance only with activation functions , wherein the output is a piece-wise polynomial.
We include more details in the Appendix.
[Equation]
Consider the following two-layer ReLU neural network (without bias parameters) with width m that assumes the following form (deﬁned as in [9]):
[Equation]
where W 1 P R m ˆp d ` 1 q , W 2 P R 1 ˆ m and we assume inputs t x u ni “ 1 follow some distribution τ on the unit sphere S d P R d ` 1 ; σ p¨q denotes the element-wise ReLU operator.
As the width of the network m goes to inﬁnity, if the weights at initialization W p 0 q are independent and each follow the standard normal distribution, the inner product of the network gradient at initialization gives rise to a limiting kernel, namely the Neural Tangent Kernel (NTK) [23] κ deﬁned as :
[Equation]
This kernel has been used to characterize the behavior of sufﬁciently wide networks f W during training.
For instance, if the network is trained to minimize the (cid:96) 2 loss, then its training dynamics closely track those of kernel regression under κ .
This holds in a particular training regime, referred to as “lazy training” [13], where the parameters hardly vary after initialization and the network can be approximated by its ﬁrst-order Taylor expansion at initialization as:
[Equation]
In practice however, the conditions of lazy training are often violated (notably, the requirement that the weights do not move) within the ﬁrst few steps of gradient descent.
Nevertheless, the NTK remains a useful theoretical tool for analyzing the neural network behavior as some of its predictions have been shown to hold in practice [11, 31].
The NTK for the two-layer ReLU network (deﬁned in Eq.
(3)) takes the following form [9, 13, 18]
[Equation] [Equation]
where the kernels κ 1 and κ 2 are deﬁned as follows:
[Equation]
where σ p¨q denotes the ReLU operator and σ 1 p¨q denotes the indicator function 1 r¨ ě 0 s .
The kernels κ 1 , κ 2 have been explicitly computed for the ReLU activation in [36, 14, 9] where they were shown to be positive semi-deﬁnite dot-product kernels, the function value depends only on the value of the dot-product of the arguments or more formally, κ 1 p x , x
1 q “ g 1 px x , x 1 yq and κ 2 p x , x 1 q “ g 2 px x , x 1 yq for some functions g 1 , g 2 : R Ñ R .
These kernels will serve as building blocks for the tangent kernel of the Π -Net architecture studied in the next section.
3. Π -K ERNEL
Following [7, 9, 11] that consider shallow two-layer ReLU networks, we derive the tangent kernel for a two-layer Π -Net by supplementing the standard network with a multiplicative interaction layer:
[Equation]
where W 1 , W 2 P R m ˆp d ` 1 q , W 3 P R 1 ˆ m , ˚ denotes the Hadamard product and we again assume inputs t x u ni “ 1 that follow some distribution τ on the unit sphere S d Ă R d ` 1 .
Remark
The reader may note that the considered formulation yields a piece-wise quadratic polynomial.
Additionally, it is important to consider the network with ReLU activations since otherwise, the network output simply becomes a quadratic polynomial of the input and can no longer approximate functions of higher degree polynomials, even with inﬁnite width.
We now present our ﬁrst main result on the Π -kernel.
Theorem 1 The limiting kernel for the two-layer Π -Net, denoted by κ π p x , x 1 q , takes the following form:
[Equation]
where κ 1 , κ 2 are deﬁned as in Eq.
The proof follows the standard NTK calculations and is presented in the Appendix.
Importantly, the addition of a single multiplicative interaction induces a product of kernels form.
Remark
The required width that ensures that the two-layer Π -Net does not deviate from the network at initialization (and thus the corresponding NTK is close to the limiting κ π ) is slightly higher than standard networks, by a factor of about Ω p?
log m q and in practice is quite reasonable.
We provide a rough sketch of the proof in the Appendix (end of A.3).
Remark
The advantage of using the 2-layer Π -Net is that it allows us to directly contrast against prior results on standard 2-layer feed-forward network, thereby gauging the effect of the extra multiplicative layer.
However, unlike feed-forward networks, the theory does not easily extend to polynomials of higher degree or depth, since even with a ﬁxed degree and depth, the NTK varies with the placement of the multiplicative connections.
In this work, we choose to focus primarily on the effect of the multiplicative layer and leave the extension to general polynomials for future work.
Since the Π -kernel κ π can be expressed as a sum of products of continuous positive deﬁnite dotproduct kernels κ 1 and κ 2 , it inherits their regularity properties and is itself a Mercer kernel, as a consequence of the Schur Product theorem.
3. S PECTRAL ANALYSIS
To properly characterize the approximation properties and the behavior during training of this newly derived kernel, we study its Mercer decomposition.
Indeed, this approach was used in [9] to show that the 2-layer NTK Eq.
(5) has better approximation properties than a ﬁxed ﬁrst layer network, and in [11] to study the spectral bias of feed forward ReLU networks.
The Mercer decomposition of a kernel κ is derived from the eigenvalues and eigenfunction of the integral operator associated to the kernel (see [38] Theorem 2.10 and references therein).
Taking X to be some compact set in R d , recall that for any continuous kernel function κ : X ˆ X Ñ R and
Borel measure τ on X , we can deﬁne an integral operator L κ that “convolves” any square integrable function
f P L 2 τ p X q with κ :
[Equation]
For a Mercer kernel, this linear operator admits countable, real, non-negative eigenvalues t µ 1 , µ 2 , . . .
u and the associated eigenfunctions form an orthonormal basis of L 2 τ p X q .
If the data is uniform on the sphere S d , and κ is a dot-product kernel, then these eigenfunctions of L κ are the spherical harmonics ([40], Lemma 4).
Consequently, by applying Mercer’s Theorem, we can obtain the following decomposition:
[Equation]
where Y k,j for j “ 1 , 2 ...N p d, k q represent the spherical harmonics of degree k in d ` 1 variables (whose explicit formula is given in Appendix D.1 in [4]) , and N p d, k q : “ 2 k
[Equation]
d ´ 1 k
[Equation]
From this, we can characterize the RKHS H associated to κ as follows:
[Equation]
The beneﬁt of determining the decay rate of the eigenvalues t µ 1 , µ 2 , . . .
u can be easily deduced from the previous set as the eigenvalues determine the size of H .
Indeed, the slower the decay of t µ 1 , µ 2 , . . .
u , the more sequences p a k,j q k,j will verify
[Equation]
Moreover, the results of [11] rely on this decay rate to show that gradient descent on wide networks picks up low-frequency information ﬁrst 2 (Theorem 4.2, [11]).
This decay rate is therefore of central importance.
Next, we present the prior results on characterizing this decay rate for the kernel obtained for the two-layer feed forward ReLU network (Eq.
From there, we derive the same for the Π -kernel.
Proposition 1 (Theorem 4.3 in Cao et al.
[11], Proposition 5 in Bietti & Mairal [9])
For the tangent kernel corresponding to two-layer feed forward ReLU network, the eigenvalues p µ k q k satisfy the following:
[Equation]
The decay is exponentially fast in the input dimension d .
For the Π -kernel, we show the following improvement.
Theorem 2 Let t µ π, 1 , µ π, 2 , . . .
u denote the eigenvalues of the linear operator L κ π associated to the kernel κ π .
Then for k " d p k ‰ 2 mod 4 q , it holds that
[Equation]
The proof of the theorem is provided in the Appendix.
The main idea is to plug in the Mercer decomposition of each kernel, leading to a product of polynomials form.
The expansion of this product then allows for isolating the dominant terms contributing to the eigenvalues for each frequency.
The key take-away is the much slower rate of decay the k th eigenvalue (an order almost Ω p k
p
d { 2 q q ) when compared to the standard two-layer NTK.
An immediate consequence is a “larger” RKHS, which lends the Π -kernel superior approximation properties in the higher frequencies.
Further, when combined with the ﬁndings of [11], it yields a speed-up in learning higher frequency harmonics.
Remark
On the point of a “larger” RKHS, it is important to note the two perspectives at play.
Firstly, as we already commented, is the idea that a slower decay implies more functions contained in the RKHS, leading to better approximation properties.
However, from the classical statistical learning point of view, while a larger RKHS makes the optimization problem easier, it can lead to a prediction performance that is sub-optimal [5].
Thus, it eventually relates to the traditional question about ﬁnding the optimal trade-off.
We will note through our experiments that it is the perspective relating to better approximation properties and optimization for the Π -kernel that dominates.
Remark 4 The k " d setting may not reﬂect the case for image-based applications.
For instance, generation tasks wherein the output is an image of resolution d ˆ d , the higher frequencies of interest roughly correspond to the order of Ω p d q .
Consequently, the exponential improvement in the decay rate for the k " d setting, and by extension in the approximation properties of the RKHS as well as the speed of learning higher frequency information, may not be as dramatic elsewhere.
Nevertheless, our analysis provides sufﬁcient intuition to expect some degree of improvement in realistic settings.
In the next section, we put this prediction to the test by extensively examining whether or not Π -nets exhibit a different spectral bias than standard neural networks.
We offer a brief characterization of the result in the Appendix
N UMERICAL EVIDENCE
The analysis in Section 3 reveals that polynomial neural networks in the NTK regime learn higher frequency information faster.
In practice however, neural networks signiﬁcantly deviate from the near-initialization NTK conditions within just one or two iterations of gradient descent.
Therefore, to verify the analysis on the spectral bias of Π -Nets, we conduct a series of experiments that increasingly deviate from the assumptions of the NTK regime, including image-based datasets which further verify our theoretical analysis.
We initially describe the experiments on synthetic data (Section 4.1), then move onto realistic settings.
For all our experiments, we will use Π -Nets based on the product of polynomials formulation (Appendix), in the same vein as [15].
4. E XPERIMENTS ON SYNTHETIC DATA
Our ﬁrst experiment relates to learning spherical harmonics in approximate inﬁnite width NTK regime [11].
More precisely, the task comprises of learning linear combinations of spherical harmonics with data inputs uniformly distributed on the unit sphere.
For this, we consider large-width two-layer networks to simulate the inﬁnite-width settings and we compare the performance of ΠNets against standard neural networks.
The optimization method is vanilla gradient descent with batch size equal to the number of training samples, to avoid stochastic effects.
Our observations align with our theoretical ﬁndings, the results are deferred to Appendix due to space constraints.
Next, we assess whether the spectral bias manifests beyond the NTK regime.
We consider the task of learning sinusoidal signals with smaller width networks and a larger learning rate, so as to expressly violate the NTK assumptions of staying close to initialization, and we observe the spectral bias corresponding to different frequencies.
on left
Learning Sinusoids
The goal is to learn sinusoidal signals [35].
Given frequencies k i P K , with corresponding amplitudes
A i P α and phases φ i P Φ , the target map f ˚ : r 0 , 1 s Ñ R is deﬁned as f ˚ p x q “ A i sin
p 2 πk i x ` φ i q .
We approximate this map with two methods: i) a fully connected neural network and ii) a Π -Net.
In the ﬁrst setting, we compare a 256-unit wide, six-layer deep neural network against a 256-unit wide, six-layer deep Π -Net (which has ﬁve multiplicative layers).
In each case, the network f θ (parametrized by θ ) regresses over f ˚ ( K = p 5 , 10 , ..., 45 , 50 q , φ i „ U p 0 , 2 π q and A i “ 1 @ i ), using N “ 200 evenly spaced input samples over r 0 , 1 s and with a ﬁxed learning rate (same for both networks); the spectrum of the network ˜ f
θ p k q is monitored as training progresses and we track the magnitude of learned components for each frequency of interest, averaged it over 5 runs (Fig.1).
Remark 4 The experimental setup for training the two networks replicates the setup of Rahaman et al.
[35], without any special initialization or hyper-parameter tuning for either network.
As aforementioned, the Π -Net architecture is implemented exactly as the product of polynomials formulation speciﬁed in the Appendix.
We observe that Π -Nets do speed up training of higher frequencies.
To better substantiate our claim, we conduct two additional experiments, the resuls of which are included in the Appendix.
Firstly, due to additional multiplicative layers, Π -Nets have more learnable parameters than standard networks of the same depth.
We conduct a second experiment, where we compare a nine-layer feedforward network with a six-layer Π -Net with only three multiplicative layers, such that the feedforward network includes more parameters.
We observe that while depth helps, it is not as effective as the multiplicative layers in Π -Nets (Fig. 11).
Skip connections (additive) in deeper networks are known to speed-up training.
To disentangle this effect from multiplicative skip connections, we compare a Π -Net with standard network with the same architecture, but with additive skip connections instead of multiplicative.
We verify that the improvement in the speed of learning higher frequencies can not be achieved using trivial additive skip connections (Fig. 10).
Robustness to Perturbations : Motivated by the observations of Rahaman et al.
[35], we evaluate the susceptibility of higher frequencies to random perturbations for standard networks and Π -Nets.
The setup relies on the learning task in the previous experiment.
More precisely, following convergence of the network to a low-error approximation of target f ˚ (denoted by f θ ˚ ), random isotropic perturbations are introduced to the network parameters: θ “ θ ˚ ` δ ˆ θ .
We monitor the effect of increasing the perturbation magnitude δ on the frequencies of interest.
In the ﬁrst setting (Fig. 2), we compare the effects of perturbations on the six-layer standard network to the six-layer Π -Net (with ﬁve multiplicative injections).
In the second setting, we directly compare the two variants of 6-layer deep Π -Nets i.e. one with ﬁve injections and the other with three injections, the results are deferred to the Appendix.
y networks, information is lost quickly as increases, much more effective at higher frequency information, even
Discussion :
In the task of learning sinusoids, multiplicative interactions speed up the learning of higher frequency sinusoids.
Additionally, they are more effective at doing so than simply increasing the depth of the network.
Through the robustness experiment, we see that Π -Nets are more robust to perturbations in the higher frequencies.
Remarkably, the Π -Net with more interactions is noticeably more robust than the lower degree polynomial network (Fig.
12 in the appendix).
We hypothesize that Π -Nets enhance the representational space for higher frequencies, relating to our observations on the RKHS in the NTK regime, which makes them less susceptible than standard neural networks.
This could also explain why more multiplicative ‘injections’ lead to more robustness.
4. L EARNING I MAGES
We observed that multiplicative injections allowed fully connected networks to pick up higher frequency information faster.
On the other hand, Deep Convolutional Neural Networks (DCNNs [30]) are more ubiquitous in computer vision, partly due to the DCNN structure imposing a suitable prior for tasks with natural images, termed as the Deep Image Prior (DIP) [32].
Therefore, we check how this prior changes with multiplicative interactions in Π -Nets.
We adapt the denoising setup from the
DIP framework.
Experiment - Denoising : We consider an image x
P R
ˆ H ˆ W and obtain its noisy version x 0 “ x ` δ , perturbed with Gaussian noise.
For the input, we sample a random tensor z P R N ˆ H ˆ W (N = 32 in our setup).
We consider a neural network f θ (making the parametrization by θ explicit), and optimize the following objective:
[Equation] [Equation]
We expect the DCNN structure to ﬁrst learn the (“natural”) features corresponding to the true image x and pick up the noise only in the latter stages, which can then be avoided using early-stopping.
Remark 5 Our goal in this experiment is not to look at the denoising performance.
Instead, our primary goal is to validate the experimental observation on sinusoids and assess whether Π -Nets can speed up learning of high frequencies in real-world applications.
If the speed up is conﬁrmed, Π -Nets can be early-stopped, i.e., require less iterations for achieving the target result.
As in [35], we experiment with the U-net architecture, adapting it suitably for Π -Nets with an image of resolution 480 x 600.
The implementation details are included in the Appendix.
We train both networks for 2500 iterations, with the same learning rate and identical input tensor z .
We monitor the
Peak Signal-to-Noise Ratio (PSNR) curves while training, in peak
PSNR with the true image for both standard and Π -Nets are roughly the same, indicating similar denoising performance.
However, Π -net achieves this peak PSNR in approximately half the number of iterations.
Beyond this point, the Π -net PSNR with the true image decreases as it starts to pick up the high-frequency noise.
We conﬁrm this in the visual snapshot of the output of the two networks (Fig. 15 in the appendix).
We conclude that Π -Nets do indeed speed up learning in images, by showing a reduced impedance towards high-frequency information.
Next, we check how this bias affects learning high frequency information pertaining to natural images in the absence of noise.
Experiment - Power Spectrum Analysis : We consider the identical setup as the denoising experiment but without the noise perturbation, setting δ “ 0 .
Therefore, the task for the network f θ learn θ ˚ such that f θ ˚ p
z q “ x .
We allow 600 iterations of gradient descent and we monitor the radial power spectral density (p.s.d.) of the network output image against the ground truth p.s.d.
It allows us to explicitly track the learning progress for each the frequency magnitude.
We use the same U-net structure and suitably modify it for Π -Nets as before.
Since the multiplicative interactions introduce more parameters, we reduce the depth/scale (by scale, we mean number of down/up-sampling operations in the U-net) of Π -Net to ensure roughly the same parameters ( „ 1. million).
The learning speeds in different frequencies can be observed in Fig.
In the Appendix, we repeat the same experiment for equal depth networks.
Discussion :
From the preceding two experiments, we establish that Π -Nets retain their superiority in picking up high frequency information for larger DCNNs.
In the Appendix, we note the effect of multiplicative interactions on the deep image prior of the network.
We also verify that this altered spectral bias remains relevant for standard learning tasks with natural images.
Consequently, for tasks where capturing the ﬁner details of the image is important (such as the denoising instance above), this potentially leads to a reduced training time for Π -Nets as compared to standard networks.
We expect the insights from our work to be useful for guiding network design with multiplicative interactions in large-scale experiments.
D ISCUSSION
In this work, we focus on the spectral of polynomial neural networks.
Our theoretical results in the NTK regime utilize a two-layer polynomial network and demonstrate a speed-up in the learning of higher frequencies over standard neural networks.
We experimentally verify these properties, even outside of the NTK training regime.
Additionally, the results offer intuition behind the success of networks that use multiplicative interactions, such as StyleGAN [27], whose connections to polynomials have been noted previously [16].
As a future direction, we aim to design controlled settings to study how polynomial and multiplicative interactions affect performance in state-of-the-art conditional generative models for image generation or deblurring, where high-frequency information is critical for photo-realistic outcomes.
It should be noted that the spectral bias of standard neural networks towards low frequency (complexity) functions is believed to help in generalization.
Therefore, another important direction is to explore whether this enhanced spectral bias of polynomials towards higher complexity functions translates to differences in their generalization properties.
A A PPENDIX A. A P RIMER ON P OLYNOMIAL N EURAL N ETWORKS ( Π -N ETS )
Polynomial neural networks ( Π -Nets)
[15] were recently introduced with the output being a high degree polynomial expansion of the input.
The parameters of Π -Nets can be represented as higher order tensors.
These networks are able to serve as function approximators even without the use of non-linear activations.
When used in conjunction with activations, the models demonstrate better expressivity and achieve state-of-art results on a variety of learning tasks.
We next describe the construction of Π -Nets.
A.1. M ETHOD
Mode-m vector product : Consider an M th order tensor X , with each of its element addressed by
M indices, i.e., p X q i 1 ,i 2 ,...,i M “ x i 1 ,i 2 ,...,i M .
An M th -order real-valued tensor X is deﬁned over the tensor space R
I
ˆ I 2 ˆ ...
ˆ I M , where I m P Z for m “ 1 , 2 , ....
M .
The mode-m unfolding of a tensor X P R I
ˆ I 2 ˆ ...
ˆ I M maps X to a matrix
X p m q P R
I m ˆ ˆ
I m with ˆ I m “ ś Mi “ 1 ,i ‰ m
I i .
such that the tensor element
X i 1 ,i 2 ,...,i M is mapped to the matrix element X
i m ,j where j “ 1 ` ř
Mk “ 1 ,k ‰ m p
i k ´ 1 q J k , where J k “ ś Mn “ 1 ,n ‰ m I n .
The mode-m vector product of X with a vector u P R I m , denoted by X ˆ m u P R
I
ˆ
I 2 ˆ .....
ˆ I m ´
1 ˆ I m ` 1 ....I M , results in a tensor of order M ´ 1 :
[Equation]
x i
[Equation]
i
[Equation]
...,i M u i m .
Furthermore, we denote X ˆ 1 u 1 ˆ 2 u 2 .....
ˆ M u M “ X
[Equation]
Π -Net learns a function G : R d Ñ R o , such that each element of the output x j can be expressed as a polynomial of all the input elements
z i , with i P r 1 , d s as follows:
[Equation] [Equation]
where β j P R and (cid:32)
[Equation]
r n s j
P R ś nm “ 1 ˆ m d ( N n “ 1 are parameters for approximating the output x j .
The correlations (of the input elements z i ) up to N th order emerge in Eq.
More compactly:
[Equation] [Equation]
where β P R o and (cid:32) W r n s P R o ˆ ś nm “ 1 ˆ m d ( N n “ 1 are the learnable parameters.
This form allows us to approximate any smooth function as per an extension of the Weierstrass Theorem.
To prevent an exponential number of parameters, the authors propose using coupled tensor decompositions.
A.1. T ENSOR D ECOMPOSITION FOR S INGLE P OLYNOMIAL
An appropriate tensor decomposition on the parameters in Eq.
(14) allows for implementation with a neural network.
Here, we brieﬂy describe one such decomposition:
Model: NCP : Next, we consider a joint hierarchical decomposition on the polynomial parameters.
A Nested coupled CP decomposition (NCP) results in the following recursive relationship for N th order approximation:
[Equation]
for n “ 2 , . . .
, N with x 1 “ ´
A T r n s z ¯ ˚ ´
B T r n s b r n s ¯ and x “ Cx N ` β .
The parameters C P
R o ˆ k , A r n s P R d ˆ k , S r n s P R k
ˆ k , B
r n s P R ω
ˆ k , b r n s P R ω for n “ 1 , . .
, N , are learnable.
(cid:32) b r n s P R ω ( N n “ 1 act as a scaling factor for each parameter tensor, whose role is illustrated in case of the third order approximation in eq.16:
[Equation]
Further, the joint factorization of the parameter tensors (in matrix form) for the third order NCP polynomial may be summarized as follows:
• First order parameters :
W r 1 s p 1 q “ C p
A r 3 s d B r 3 s q T .
• Second order parametes: W r 2 s p 1 q “ C
[Equation]
A r 3 s d
[Equation]
A r 2 s d B r 2 s
[Equation]
S r 3 s * T .
• Third order parameters: W r 3 s p 1 q “ C " A r 3 s d „ˆ A r
s d !
´ A r 1 s d B r 1 s ¯ S r 2 s )˙
S r 3 s * T A.1.
[Equation]
P RODUCT OF P OLYNOMIALS
The second scheme of implementation approximates the target function using a product of polynomials form, wherein the output of ﬁrst polynomial is fed to the next and so on.
The concept is visually depicted in 6; for instance, if each polynomial is degree two, then stacking N such polynomials results in an overall order of 2 N .
Remark :
In practice, each matrix operation in the recursive formulation of Π -Nets represents an afﬁne transform on a vector.
Therefore, the implementation in standard libraries may be as simple as using a convolutional or a fully-connected layer.
The other difference arises due to the multiplicative layers, which may be seen as special ’skip’ connections which are combined with the network output via the element-wise vector product (Hadamard product), as opposed to addition.
Therefore, it is easy to see why these networks scale as well as standard deep networks.
A. E IGENVALUES DETERMINE THE SPEED OF LEARNING
As noted previously, the integral operator with respect to kernel function is deﬁned as follows:
[Equation]
The key idea in [11] is to establish guarantees on the speed of convergence of gradient descent along different directions of L κ , as deﬁned by its eigenfunctions.
Consider t
λ i u
i ě 1 with λ 1 ě λ 2 ě .. be the strictly positive eigenvalues of L κ with t φ i u i ě 1 the respective eigenfunctions and deﬁne v i “ n ´ 1 p φ p x 1 q , φ p x 2 q .......φ p x n qq .
Since the eigenvalues may not be distinct, deﬁne r k as the sum of multiplicities of the ﬁrst k distinct eigenvalues of L κ .
Deﬁne V r k “ p v 1 ..., v r k q .
By deﬁnition, v ii ď r k are rescaled restrictions of orthonormal functions on the training examples.
Therefore, they form a set of almost orthonormal bases in the vector space R n .
Finally, denote y “ p y 1 , y 2 .....
y n q T and ˆ y p t q “ p f W p t q p x 1 q , .....
f W p t q p x n qq T as the ground truth and predictions at time t , respectively, where f W p t q denotes the 2-layer ReLU network.
Then the following result holds:
Theorem 4.2 [11]: Suppose | φ j p x q| ď M for j P r r k s and x P S d
`
For any (cid:15), δ ěand integer k , if n ě Ω p (cid:15) ´ 2 .
max tp λ r k
´
λ
r k ` 1 q ´ 2 , M 4 r 2 k uq , m ě ˜ Ω p poly p T, λ
[Equation]
1 r k ,
[Equation]
´ 1 qq , then with probability at least 1 ´ δ , gradient descent with step size η “ ˜ O p m ´ 1 q satisﬁes:
[Equation]
which is to say that the convergence rate of || V Tr k p y ´ ˆy p t q q|| 2 , or alternatively the projection of the residual error on the space spanned by the ﬁrst r k eigenvalues is controlled by the r th k eigenvalue λ r k .
With some additional work, this result can be extended to the 2-layer Π -Netby accounting for the extra width factor of Ω p?
log m q , as noted previously.
The key takeaway in that with a wide enough network and large enough sample size, gradient descent ﬁrst learns the target function along the eigen-directions with larger eigenvalues.
Since the decay of eigenvalues is slower for the 2-layer Π -Net, it leads to a speed-up in learning higher frequencies.
A. P ROOF OF T HEOREM 1: D ERIVING THE Π - KERNEL
In this section, we prove our ﬁrst main result for the Π -kernel corresponding to theorem 1.
Consider a two-layer Π -Net f W parametrized as follows :
[Equation]
(19) where the weights W 1 P
R m ˆ d , W 2 P R m ˆ d , W 3 P R 1 ˆ m are initialized with independent identically distributed N p 0 , 1 q coordinates.
Recall that the neural tangent kernel κ π corresponds to the limit of the following inner product
[Equation]
We can compute the inner-product Eq.
(20) by computing the derivatives with respect to W 1 , W 2 ,
W 3 of f W separately and sum up the inner products to obtain κ π , since the gradient can be split into three blocks.
We denote by ˜ α : “ W 1 x , and by ˜ β : “ W 2 x the pre-activation vectors; and by α , β the postactivation vectors where the element-wise activation σ is applied to ˜ α and ˜ β respectively.
First consider the derivative w.r.t W 1 denoted B W 1 :
[Equation]
The contribution to the NTK (Eq.
(20)) therefore corresponds to xB W 1 f W p x q , B W
1 f W p x 1 qy .
Note that since the network is symmetric in t W 1 , W 2 u
[Equation]
we need only look at W 1 to obtain the contribution for W 2 as well.
We ﬁnd that
[Equation]
As lim m Ñ
[Equation]
the above quantity converges to its expectation by the law of large numbers because the terms are independent and identically distributed with ﬁnite expectation.
Consequently,
[Equation] [Equation]
By symmetry
[Equation]
we obtain with a similar term for W 2 and their total contribution to Eq.
(20) adds up to twice the term obtained for W 1 .
Characterizing the kernel w.r.t W 3 is more straightforward:
[Equation]
Therefore
[Equation]
its contribution to the NTK is
[Equation]
Applying the law of large numbers again, as lim m Ñ 8 , this quantity tends to:
[Equation]
The theorem follows by summing up 2 ˆ Eq.
(23) and Eq.
Width Requirements
As noted previously, two-layer Π -Netneeds an the extra factor Ω p?
log m q in terms of width, to stay close to initialization The proof is largely derived from Lemma A.6 in [26], based on ideas ﬁrst noted in [1] and therefore, we just provide a rough sketch here.
For simplicity, consider the set of weights ( W 2 , W 3 ) ﬁxed at initialization, i.e. at W p 0 q 2 , W p 0 q 3 and note the ﬁrst-order Taylor expansion for the Π
-Net w.r.t W 1 as follows:
[Equation]
Here f 0 W denotes the ﬁrst order Taylor expansion of f at W 0 .
We can then expand the RHS as:
[Equation]
Consequently, we can bound the approximation error as:
[Equation]
Notice that aside from the | σ p x T W p 0 q 2 ,j q| , the error term is exactly the same as that for a standard network.
Also note that P p x T W 2 ,j ě τ q “ P p σ p x T W 2 ,j q ą τ q ď e ´ τ 2 { 2 , for τ ą 0 by the sub-gaussian tail bound.
To ensure | σ p x T W p 0 q 2 ,j q| ď τ @ j with probability 1 ´ δ , we use the union bound to obtain the condition on τ as m.e ´ τ 2 { 2 ă δ .
Consequently, we have that τ „ Ω p?
log m q , which essentially becomes an additional constant factor in the error term, over the usual error terms from the standard feed-forward network.
A. P ROOF OF T HEOREM 2: C HARACTERIZING THE Π - KERNEL E IGENDECAY
Here, we prove the Π -kernel eigenvalue decay rate stated in theorem 2.
We ﬁrst recall some connections between spherical harmonics and Gegenbauer polynomials.
Deﬁnition 1 For a given α P R , Gegenbauer (or ultraspherical) polynomials denoted C αk : r´ 1 , 1 s Ñ R are a family of orthogonal polynomials with respect to the weight function x ÞÑ p 1 ´ x 2 q α ´ 12 , i.e,
[Equation]
for k ‰ (cid:96) .
Remark: Gegenbauer polynomials are a generalization of Legendre polynomials which can be recovered by taking α “ 12 .
The following addition formula expresses Gegenbauer polynomials in terms of spherical harmonics.
Lemma 1 (Theorem 4.11 in [19]) (Addition formula) Let t Y k,j u N p d,k
q j “denote spherical harmonics of degree k in d ` 1 variables.
It holds that, for any x , x 1 P S d ,
[Equation]
By making use of this Lemma, we can rewrite the Mercer decomposition of any admissible dot product kernel K when the data is uniform on the unit sphere.
Indeed, by simplifying the Mercer’s decomposition in terms of spherical harmonics we have :
[Equation]
where p µ k q 8 k “ 0 are its corresponding eigenvalues, and the second equality follows from Lemma 1.
Now, since κ 1 and κ 2 are dot-product Mercer kernels, we can, akin to Eq.
(27), obtain their respective decompositions in terms of Gegenbauer polynomials.
There exist p µ 1 ,k q 8 k “ 0 and p µ 2 ,k q 8 k “ 0 , both sequences of eigenvalues, such that
[Equation]
The decay rate of the eigenvalues p µ 1 ,k q 8 k “ 0 and p µ 2 ,k q 8 k “ 0 is known [4, 9, 11] and is stated in the following Lemma.
Lemma 2 (Appendix D.2 of [4], Lemma 17 of [9])
For k " d , k even, we have that µ 1 ,k „
A p d q k ´ d ´ 1 and µ 2 ,k „ B p d q k ´ d ´ 2 ´ 1 { 2 , where A p d q and B p d q are constants only depending on the dimension d .
Our goal is to establish the decay rate of the eigenvalues p µ π,k q 8 k “ 0 of the kernel
[Equation]
[Equation]
By plugging in the representations in Eq.
(28) into the Π -kernel expression (Eq.
, we ﬁnd that
[Equation]
Moreover, since we can also write
[Equation]
an appropriate factorisation of the product Eq.
(30) will allow us to deduce the order of µ π,k by equating the coefﬁcients appearing in front of the Gegenbauer polynomials.
Developing the product in Eq.
(30) leads to the appearance of products of Gegenbauer polynomials.
The product of two Gegenbauer polynomials turns out to be a linear combination of other Gegenbauer polynomials.
Lemma 3 (Equation 8 in [12]) Let α P R , for any m, n P N , there exists positive coefﬁcients p λ p m,n q s q
[Equation]
m,n
[Equation]
such that
[Equation]
By using this expansion, coefﬁcients for each Gegenbauer polynomial may be identiﬁed.
Take k P N to be an even number.
We know from plugging Lemma 3 into Eq.
(30) that the coefﬁcient µ π, 2 k N p d, 2 k q in front of the polynomial C p d ´ 12 q 2 k can be lower bounded as follows.
[Equation]
We obtain this lower bound by considering the contribution of a single term in Eq.
(31) where m “ n “ k and s “ 0 to the coefﬁcient in front of C p d ´ 12 q 2 k .
This contribution is a lower bound because all coefﬁcients involved in the product are non-negative.
Consequently, in order to establish a decay rate for p µ π,k q it sufﬁces to study the decay rate of λ p k,k q.
This rate is given by the following Lemma.
Lemma 4 Let α “ d ´ 12 be an integer, the coefﬁcient λ p k,k qdeﬁned in Lemma 3 admits the following expression λ
[Equation]
Moreover, for k " d , it holds that
[Equation]
See Appendix A.4.1 for a proof.
We now dispose of all the necessary results to prove Theorem 2.
Starting from Eq.
(32), we ﬁnd that for k " d , we have that
[Equation]
Lemma 4)
[Equation] [Equation]
This concludes the proof that, for k divisible by 4 , we have µ π,k “ Ω p k ´ d { 2 ´ 2 q .
For k “ 1 mod 4 , we conduct the same reasoning taking the coefﬁcient λ p 2 k ` 1 , 2 k q. For k “ 3 mod 4 , the coefﬁcient to consider is
The derivation Lemma 4 proceeds exactly manner for these
A.4. P ROOF OF L EMMA
Let us denote p α q k : “ α p α ` 1 qp α ` 2 q .... p α ` k ´ 1 q and p α q 0 : “ 1 .
From Equation 8 in [12]
[Equation]
we have that
[Equation]
We can apply Stirling’s approximation stating that n ! „ ?
πn p ne q n to ﬁnd that:
[Equation]
Considering the case when k " α or equivalently
[Equation]
k " d since α “ p d ´ 1 q{
[Equation]
we obtain the following simpliﬁcation:
[Equation]
A. L EARNING S PHERICAL H ARMONICS WITH Π - KERNEL
Following the setup in [11], we perform a similar experiment on learning combinations of spherical harmonics in the NTK regime.
We deﬁne and initialize the Π -Net exactly as speciﬁed, with a width of 32768 neurons and using vanilla gradient descent, to approximate the kernel learning in the inﬁnite width limit.
We take n “ 1000 samples, t x u ni “ 1 from the uniform distribution on the unit sphere S 10 .
We deﬁne our target function with integral k P K as follows:
[Equation]
[Equation]
where the P k p t q is the Gegenbauer polynomial with degree k , ζ k are ﬁxed vectors that are independently generated from uniform distribution on unit sphere in R 10 , and N p K q is a suitable normalizing constant to keep the order of magnitude of the target function approximately the same for different choices of K .
As noted previously, f ˚ may be seen as a linear combination of spherical harmonics and we compare the error residuals during the learning process in standard vs Π -Nets in the NTK regime, for varying K .
We use a moving average of range 20 on these curves to smoothen out the heavy oscillations in the latter stages.
Figure
We observe (Fig.
8) that the rate of error convergence for the Π -kernel is much faster than the standard two-layer NTK and especially for higher harmonics (clearest increase for the highest k “ 4 ), which is exactly what is expected from theoretical results.
Next, we consider K “ t 1 , 3 , 4 , 5 , 8 , 12 u i.e. we consider higher frequency harmonics and also introduce odd harmonics, with the respective A k assigned the equal weights relative to each other.
Recall that the eigenvalues corresponding to odd harmonics vanish for the standard kernel, meaning that we expect a much slower convergence rate for them.
As before, we look at the rate of convergence for individual harmonics for the two kernels.
The convergence curves are presented in Fig.
We verify the speed-up in higher frequencies, and an especially notable gap for odd harmonics other than k “ 1 .
Discussion : The empirical results strongly support our hypothesis that the Π -kernel can speed up learning higher harmonics faster than the standard two-layer kernel, even for settings outside of k " d .
A. S YNTHETIC E XPERIMENTS WITH S INUSOIDS A.6. A DDITIONAL E XPERIMENTS
A. E XPERIMENTS WITH I MAGES A.7. I MPLEMENTATION D ETAILS
U-net type hourglass architectures provide the ideal inductive bias for a host of image restoration tasks in the DIP framework and for our experiments, we use the same architectural design as [32] for the standard networks.
Since we require the output and input to have the same spatial dimensions, the number of downsampling blocks is equal to the upsampling blocks.
We refer to this number as the ‘scale’ of the U-net.
We provide a schematic illustration in Fig. 13.
[Figure]
Note that the each down/upsampling block within itself contains convolutions, normalization and pooling but we abstract those details from the schematic for clarity.
For the Π -network architecture, we consider a product of two polynomials model which modiﬁes the standard network by introducing multiplicative connections.
An illustrative schematic is presented in Fig.
[Figure]
Remark :
In addition to the architecture, it is important to note another important detail about using Π -Nets.
In practice, we recommend the use of two separate learning rates while training Π -Nets, wherein the learning rate for multiplicative connection parameters speciﬁcally is lower than the learning rate corresponding the parameters in the feed-forward part of the network.
It leads to more stability while training.
A.7. A DDITIONAL E XPERIMENTS
[Figure]