Neural Networks with Hadamard Product: Separation on Extrapolation and Spectral Bias
Anonymous Author(s)
AffiliationAddress email
Abstract
Neural tangent kernel (NTK) is a powerful tool to analyze training dynamics of neural networks and their generalization bounds.
The study on NTK has been devotedto typical neural network architectures, but is incomplete for neural networks with Hadamard products (NNs-Hp), e.g., StyleGAN and polynomial neural networks.
In this work, we derive the finite-width NTK formulation for NNs-Hp, and provetheir equivalence to the kernel regression predictor with the associated NTK, whichexpands the application scope of NTK.
Based on our results, we elucidate thedifference of NNs-Hp over standard neural networks with respect to spectral biasand extrapolation.
Our two key insights are that when compared to standard neuralnetworks, a) NNs-Hp admit a slower eigenvalue decay of the respective NTK;and b) are able to fit more complicated functions in the extrapolation regime.
Ourempirical results validate those two separations, which provide a good justificationfor a deeper understanding of neural architectures.
Introduction
In deep learning theory, neural tangent kernel (NTK) [Jacot et al., 2018] is a powerful analysis tool thatbridges the gap among the training dynamics of neural networks (NNs) trained by gradient descentand kernel regression [Jacot et al., 2018, Arora et al., 2019].
NTK provides a tractable analysis forseveral phenomena in deep learning, e g., the global convergence of gradient descent [Allen-Zhu et al.
,2019, Chizat et al., 2019, Du et al., 2019a, 2018], the inductive bias behind NNs [Bietti and Mairal,2019], the spectral bias toward different frequency components [Cao et al., 2019, Choraria et al.
,2022], the extrapolation behavior [Xu et al., 2021] and the generalization ability [Huang et al., 2020].
The study on NTK has been devoted to typical NNs architectures, e.g., fully-connected NNs [Jacotet al., 2018], residual NNs [Tirer et al., 2020, Huang et al., 2020], convolutional NNs [Arora et al.
,2019], graph NNs [Du et al., 2019b] and recurrent NNs [Alemohammad et al., 2021].
Recently, NNs with Hadamard products (NNs-Hp), e.g., StyleGAN [Karras et al., 2019] and ΠNets [Chrysos et al., 2020], have received increasing attention due to their expressivity and efficiencyover traditional NNs [Chrysos et al., 2021a, Campbell and Broun, 2000, Su et al., 2020].
There havebeen several works attempting to demystify the success of NNs-Hp.
Fan et al.
[2021] prove thatsecond-degree multiplicative interactions allow NNs-Hp to enlarge the set of functions that can berepresented exactly with zero error.
Choraria et al.
[2022] reveal that NNs-Hp with second-degreemultiplicative interactions can yield faster learning of high-frequency information during training in
NTK regime.
Yet, the theoretical analysis of NNs-Hp with high-degree multiplicative interactionsis still unclear.
When using NTK for analysis, the complete and rigorous proof includes showingthe stability of empirical NTK during training and giving the width requirement, which is crucial toallow NTK-based analysis of typical NNs Arora et al.
[2019], Tirer et al.
but is still missingfor NNs-Hp.
In this work, we derive the NTK for NNs-Hp with high-degree multiplicative interactions andpresent a rigorous proof for the equivalence to the kernel regression predictor.
This analysis enablesus to further examine NNs-Hp in a theoretical perspective.
This NTK analysis can be useful forinvestigating properties, such as the extrapolation.
StyleGAN and Π -Nets have demonstrated stellarin-distribution performance, however implementing networks for real-world applications requires theability to extrapolate to unseen data.
As humans, we have the capacity to extrapolate beyond what wehave learned [Lake et al., 2017, Webb et al., 2020].
On the contrary, NNs usually lack such abilityeven in solving simple arithmetic problems [Saxton et al., 2019] or learning simple functions [Haleyand Soloway, 1992, Sahoo et al., 2018].
Early works demonstrate that fully-connected NNs are unableto achieve exact extrapolation when fitting low-degree polynomial functions [Haley and Soloway,1992].
Xu et al.
[2021] theoretically and empirically point out that 2-layer fully-connected NNscan only extrapolate to linear function.
However, data in real-world scenarios usually lie in highlynonlinear manifolds [Van Der Maaten et al., 2009].
Our findings extend these result and highlightthat NNs-Hp can extrapolate to unseen data in a non-linear way.
What’s more, studying the NTK of NNs- Hp also allows us to investigate its spectral bias.
Overall, our main contributions can be summarized as follows:
• We derive the NTK formulation for NNs-Hp with high-degree multiplicative interactions,and gives a concrete bound of the widths requirement for convergence to the NTK atinitialization, and stability during training, which allows us to bridge the gap among NNsHp trained via gradient descent and kernel regression predictor.•
We provably present the extrapolation behavior of NNs-Hp and empirically show its advantage over standard NNs in learning common used functions and performing arithmeticextrapolation in real-world dataset.•
We provide a detailed analysis for the decay rate of eigenvalues
of the NTK of NNs-Hpwith high-degree multiplicative interactions in the task of learning spherical harmonics andscrutiny that increasing the multiplicative interactions results in a faster learning towardshigh-frequency information.
Background
In this section, we establish the notations, provide an overview of NTK, and summarize the mostclosely related work in NNs-Hpas well as extrapolation.
2.1 Notation
The core operators and symbols are summarized in Table 2 at Appendix A. Vectors (matrices) aresymbolized by lowercase (uppercase) boldface letters, e.g., a , A .
We use { x i } |X| i =1 and { y i } |X| i =
1 topresent the input features and their labels of the training set ( X , Y ) , where |X| denotes the cardinality.
We symbolize by K ( x , x ′ )
the neural tangent kernel with input x and x ′ , K ∈ R |X|×|X| , where
K ij = K
( x i , x i ) .
Next, we denote by θ t the parameter vector, ∥
e ( θ t ) ∥
22 the training loss, and ˆ K tthe empirical NTK Gram matrix at time step t .
The following notation is used:
[Equation]
2.2 Neural tangent kernel
Neural networks (NNs) are relevant to kernel method, under proper initialization [Daniely et al.
, 2016,de G. Matthews et al., 2018].
Jacot et al.
[2018] provably demonstrate the equivalence between thetraining dynamics by gradient descent and kernel regression induced by NTK when employing theℓ 2 loss.
Below, we recall the exact formula regarding the NTK of N -layer ( N > 2 ) fully-connected
NNs with ReLU activation functions σ .
The corresponding NTK K ( x , x ′ )
=
K N ( x , x ′ ) could becomputed recursively by:
[Equation]
where the covariance Σ n and its derivative ˙Σ n are defined as:
[Equation]
for i = 1 , . . .
, N .
Furthermore, the aforementioned NTK is extended to residual NNs [Tirer et al.
,2020, Huang et al., 2020], convolutional NNs [Arora et al., 2019], graph NNs [Du et al., 2019b], andrecurrent NNs [Alemohammad et al., 2021].
One of the roles of such kernel is to analyze the trainingbehavior of the neural network [Allen-Zhu et al., 2019, Chizat et al., 2019, Du et al., 2019a, 2018].
For instance, Lee et al.
[2019] showcase that through NTK parameterization NNs trained via gradientdescent of any depth evolve to linear models.
Meanwhile, the inductive bias of convolutional networks,e.g., deformation stability of the images, has been studied in the NTK regime
[Bietti and Mairal, 2019].
2.3 Neural networks with Hadamard product
The ideas of augmenting NNs with Hadamard products to allow multiplicative interactions can betraced back to at least to Ivakhnenko [1971] that investigate the learnable polynomial relationships.
Most of early work e.g., Group Method of Data Handling [Ivakhnenko, 1971], pi-sigma network [Shinand Ghosh, 1991] do not scale well for high-dimensional signals.
Chrysos et al.
[2021c] factorizethe weight of NNs-Hp based on tensor decompositions to reduce the number of parameters.
Theyexhibit how to convert popular networks, such as residual networks, and convolutional NNs to theform of NNs-Hp.
StyleGAN can be also considered as a special type of NN-Hp [Chrysos et al.
New efforts have recently emerged to improve the architecture of the network with Hadamardproducts [Chrysos et al., 2021b, Babiloni et al., 2021, Chrysos et al., 2021a].
In this work, we adoptcomplementary approach and focus on the extrapolation as well as the spectral bias from a theoreticalperspective.
2.4 Extrapolation
The study of extrapolation properties of NNs dates at least back to the 90’s [Barnard and Wessels,1992, Kramer and Leonard, 1990].
Experimental results show poor performance of NNs in caseof learning simple functions [Barnard and Wessels, 1992].
Browne [2002] also suggest that fullyconnected NNs cannot extrapolate well and then illustrate how the representation of the input impactthe extrapolation.
Xu et al.
[2021] provably present the extrapolation behavior of fully-connected NNsand Graph neural networks [Jacot et al., 2018].
Specifically, they show that 2-layer fully-connected NNs with ReLU activation function extrapolate to linear function in out-of-distribution (OOD) region.
Our work exhibits that NNs-Hp can learn high degree nonlinear function.
Apart from fully-connected
NNs, Martius and Lampert [2016], Sahoo et al.
[2018] showcase a novel family of functions withlinear mapping and a non-linear transformation, which allows to use sine and cosine as nonlinearities,enabling such networks to learn well in analytical expressions.
Note that there exist multiplicationunits in EQL, which is similar to the multiplicative interactions in NNs-Hp.
Lastly, there are othertype of OOD problems with specific setting among machine learning community [Shen et al., 2021].
Domain adaption assumes the source and the target domains lie in the same feature space but withdifferent distributions [Kouw and Loog, 2019], which differs from extrapolation.
Another category ofmethodology to solve the OOD generalization problem, called invariant learning, aims to discoverhigh-level invariance feature from low-level observations through latent causal mechanisms [Arjovskyet al., 2019, Rosenfeld et al., 2021].
We believe our analysis can also encourage the usage of NNs-Hpin these OOD problems.
Analysis of NNs-Hp
Our analysis admits the following structure: we firstly study the NTK of NNs-Hp in Section 3.1, whichallows us to conduct analysis towards extrapolation in Section 3.2, and spectral bias in Section 3.3.
3.1 Neural tangent kernel of NNs-Hp
We now develop the NTK for NNs-Hp, then we bridge the gap between the NNs-Hp trained bygradient descent with respect to squared loss and the kernel regression predictor involving the
NTK. Our analysis is conducted on a specific type of NNs-Hp called polynomial neural networks(PNNs) while we still use the term NN-Hp for consistence.
The goal of such networks is to learnan N -degree
( N ≥ 2 ) polynomial expansion that outputs f ( x ) ∈ R with respect to the inputx ∈ R d .
For simplifying the proof, we consider the following formulation of NNs-Hp, which is areparameterization version of PNNs [Zhu et al., 2022].
The output is given by:
[Equation]
where each element in W N +1 ∈ R 1 × m and W n ∈ R m × d , for n = 1 , . .
, N is independentlysampled from N (0 , 1) .
Two remarks are in place: a) We multiply by the scaling factor (cid:113) 2 m after eachdegree to ensure that the norm of the network output is preserved at initialization with infinite-widthsetting.
b) ReLU is usually required to increase the performance of NNs-Hp in experiments [Chrysoset al., 2021c].
Lemma 1.
The NTK of N -degree NNs-Hp, denoted by K ( x , x ′ ) , can be derived as:
[Equation]
where κ 1 and κ 2 are defined by taking the random Gaussian vector w ∈ R d
[Equation]
The proof, which provided in Appendix B.1, is based on the standard NTK calculations.
Differentlyfrom fully-connected NNs, the existence of multiplicative interaction in NNs-Hp induces the productform of multiple kernel.
Next, we provide the following theorem that gives a concrete requirement for the width of thenetworks that is sufficient for nonasymptotic convergence to the NTK at initialization,
Theorem 1.
(Convergence to the NTK at initializatoin).
Consider N -degree NNs-Hp, and assume that the width m ≥ 2 4 N − 2 log 2 N − 1 (2 N/δ ) for any δ ∈ (0 , 1) .
Then, given two inputs x , x ′ on the unit sphere, with probability at least 1 − δ over the randomness of initialization, we have that
[Equation]
where ρ = √ 2 2 N − 1 √ 8 e 3 (2 π ) 1 / 4 e 1 / 24 (cid:0) e 2 /e
(2 N − 1) / 2 (cid:1) (2 N − 1) / 2 .
Remark:
The above result corresponds to the networks where θ = ( W 1 , · · · , W N ) but not W N +1 .
It should be noted that the term ρ and the width are exponential with respect to the degree N , but thedegree N is not large in practice, e.g., at most 15 in Chrysos et al.
[2021a].
Hence the bound is fairand reasonable.
The technical key issue of the proof is to provide probability estimates for the multiplication ofseveral sub-exponential random variables.
To this end, we rely on the concentration of sub-Weibullrandom variables [Zhang and Chen, 2020] to complete the proof, which is deferred to Appendix B.2.
Below, we show that under certain conditions, the limiting NTK of NNs-Hp stays constant whentraining with gradient descent using squared loss.
Theorem 2 (Stability of the NTK during training) .
Consider the NNs-Hp in Eq.
Assumeλ min ( K ) > 0 and the training data ( X , Y ) is contained in some compact set x ̸ =
˜ x for allx , ˜ x ∈ X .
Then, there exist some constant R 0 > 0 , M > 1 , and Q > 1 such that for everym > M , when minimizing the squared loss with gradient descent and sufficient small learning rateη 0 < 2( λ min ( K ) + λ max ( K )) − 1 , the following inequality holds with high probability over therandom initialization of θ :
[Equation]
Regarding the proof, we firstly introduce the norm control of the Gaussian weight matrices and thenderive the local boundness and local Lipschitzness.
The last step is to apply the induction rules overdifferent time steps.
Details are presented in the Appendix B.3.
Remark: Eq.
(5) bounds the training loss and Eq.
(6) implies small perturbation of the weights.
Eq.
(7) shows that ˆ K t n −→ ∞
−−−−→ ˆ K 0 .
Combining this with Theorem 1 that states ˆ K 0 n −→ ∞
−−−−→ K ,we have ˆ K t n −→ ∞ −−−−→ K .
Thus, the equivalence to the kernel regression is established.
Note that
Theorem 2 is an extension from the corresponding theorem of NNs with residual connection [Tireret al., 2020] to NNs with Hadamard product.
This property allows us to characterize the trainingprocess as kernel regression.
3.2 Extrapolation behavior
The derived kernel above enables us to study how NNs-Hp with ReLU activation trained by gradientdescent extrapolates in OOD regime.
Theorem 3 ( γ -degree ( γ ≤ N ) extrapolation of N -degree NNs-Hp) .
Suppose we train N -degree
( N ≥ 2 ) NNs-Hp f : R d → R with Eq.
(2) with infinite-width on { ( x i , y i ) } |X| i =1 , and the networkis optimized with squared loss in the NTK regime.
For any direction v ∈ R d that satisfies ∥ v ∥ 2 =max {∥ x ∥ 2 i } , let x 0 = t v and x = x
+ h v with t > 1 and h > 0 be the OOD data points, theoutput f ( x 0 + h v ) follows a γ -degree ( γ ≤ N ) function with respect to h .
Apart from NNs-Hp, we also consummate Xu et al.
[2021] that considers the extrapolation of fullyconnected NNs with only 2 -layer.
We provide the following generalized theorem for N -layer ( N > 2 )fully-connected NNs.
Theorem 4 (Linear extrapolation of N -layer fully-connected NNs) .
Suppose we train N -layer( N ≥ 2 ) fully-connected NNs f : R d → R on { ( x i , y i ) } |X| i =1 .
For any direction v ∈ R d that satisfies∥ v ∥ 2 = max {∥ x ∥ 2 i } , x 0 = t v and x = x 0 + h v with t >
and h > 0 are OOD data points , theoutput f ( x 0 + h v ) follows a linear function with respect to h .
We have already shown that NNs-Hp extrapolate to a function with specific degree and is more flexiblethan fully-connected NNs.
However, only knowing the information of degree of the extrapolationfunction is not enough.
Naturally, we might ask under which condition NNs-Hp can achieve successfulextrapolation.
Below, we build our analysis in NTK regime and show how the geometry of the trainingset affects the behavior of NNs-Hp.
Theorem 5 (Condition for exact extrapolation of NNs-Hp) .
Let f ρ ( x ) =
x ⊤ βx be the targetfunction with x ∈ R d and β ∈ R d × d .
Suppose that { x i } |X| i =1 contains orthogonal basis { e i } di =
and{− e i } di =1 , then if we train 2-degree NNs-Hp f with ReLU activation function on { ( x i , f ρ ( x i )) } |X| i
=1with
the squared loss in the NTK regime, then we have f ( x ) =
x ⊤ βx for all x ∈ R d .
Due to the constrained space, the proof of the aforementioned theorems can be found in Appendix C.1to C.3.
3.3 Spectral analysis
In this section, we characterize the approximation properties of N -degree NNs-Hp in the indistribution regime.
By studying the spectral analysis in the form of a Mercer decomposition,we explicitly show the eigenvalues and eigenfunctions of NTK.
We firstly introduce some notation.
Denote by { Y k,j } N ( d,k ) j
= the spherical harmonics of degree k ind + 1 variables.
G ( γ ) k represents the Gegenbauer polynomials with respect to the weight functionx (cid:55)→ (1 − x 2 )
γ − 12 and degree k .
Finally, denote by F ( d, k ) :
= 2 k + d − 1 k (cid:0) k
+ d − 2 d − 1 (cid:1) .
The following lemma enables to connect spherical harmonics to Gegenbauer polynomials.
Lemma 2 (Eq.
(24) in Choraria et al.
For any dot product Mercer kernel K , when the datax and x ′ are uniform on the unit sphere, we can apply the following Mercer’s decomposition in terms of spherical harmonics
[Equation]
Theorem 6.
NNs-Hp with N -degree N ≥ 2 ) multiplicative interactions is considered.
Denote by{ µ 1 , µ 2 , . . .
} the eigenvalues associated to the NTK.
Then for k ≫ d , we have µ k = Ω(( k ) − d/ 2 ) .
The proof can be found in Appendix D.1. As a comparison, the decay rate for both deep fullyconnected NNs and residual NNs is Ω(( k ) − d ) [Belfer et al., 2021].
Thus, we can see a slower decayrate when inserting Hadamard product into standard NNs, which leads to a faster learning towardshigh-frequency information.
Experiments
Our experiment is organized as follows: We firstly showcase the extrapolation of NNs-Hp in learningsome common used functions in Section 4.1.
Next, we access the extrapolation performance on nonsynthetic datasets in Section 4.2. Lastly, we conduct the experiment in learning spherical harmonicsto study the effect of Hadamard product in Section 4.3.
4.1 Extrapolation in learning simple functions
These experiments aim to examine the extrapolation behavior of NNs-Hp in regression tasks andvalidate Theorem 3 and Theorem 4.
Our first experiment includes training the networks via thesquared loss to fit several well-known and simple underlying functions.
Unless specified otherwise,
N -layer NNs is compared against NN-Hp with N − 1 degree multiplicative interactions in eachexperiment, the reason is explained in Appendix E.1.
During prediction, we sample data points beyondthe training regime and observe the extrapolation performance.
More details on implementation canbe found in Appendix E.1.
We set the target function as f ρ ( x ) =
x 3 + x 2 − 10 x + 5 and use 4-layer fully-connected NN.
Aspresented in Figure 1(a) and Figure 1(b) fully-connected NN extrapolates linearly while NN-Hpapproximates better the out-of-distribution part of the underlying non-linear function, which areconsistent with Theorem 3 and Theorem 4.
Learning f ρ ( x ) = cos (2 x ) .
We choose 11 -layer fully-connected NN.
The training set and testing setare the same as in the previous experiment.
Observing Figure 1(c) and Figure 1(d), we find that NNHp is more flexible to learn the non-linear function outside the training region while fully-connected
NN still extrapolates linearly.
Learning f ρ ( x ) = x 2(1) + x 2(2) , where x (1) and x (2) is the first and second dimension of x ∈ R 2 .
In this task, we choose 3 -layer NNs.
Each model is trained with different data distribution, i.e., thetraining set contains support in all directions in the first case while two dimension of the trainingset are fixed to be positive in the second case.
The result is visually depicted in Figure 2, whichshows that NN-Hp can achieve exact extrapolation if the training set contains support in all directionsand thereby validates Theorem 5.
On the other hand, NN without the Hadamard product fails toextrapolate to the underlying function due to its linear extrapolation.
4.2 Extrapolation in real-world dataset
Previous experiments focus exclusively on learning simple functions and show that augmenting the
Hadamard product makes NN becomes more flexible during extrapolation In this section, we accessthe extrapolation performance beyond synthetic datasets.
Variation of brightness.
This experiment is preformed on two well-known grayscale image datasets:
MNIST dataset [LeCun et al., 1998] and Fashion-MNIST dataset Xiao et al.
For these twodatasets, the original range of the pixel of each image is [0 , 1] , we divided it by 10 for the raw trainingset to construct the new one where the pixels range from 0 to 0 .
During extrapolation, we limit therange of the original testing set to [0 , r max ] through division, where r max ∈ { 0 . 1 , 0 . 2 , 0 .
3 , ..., 1 . 0 } , asillustrated in the top two panels in Figure 3.
Then we feed these images into the trained network andevaluate the accuracy.
We access 5 -layer NNs in this task.
More details on the implementation can befound in Appendix E.2.
The accuracy is summarized in the two bottom plots of Figure 3.
We findthat both networks achieve similar accuracy in the case r max = 0 .
1 while inserting Hadamard product(Hp) into NN improves the performance during extrapolation.
Arithmetic extrapolation.
Now we turn to a more challenging task.
As human we can usuallyextrapolate to arbitrarily large numbers in arithmetic.
How do the neural networks perform duringextrapolation?
Following the setup of Bloice et al.
[2020], we use MNIST dataset, where there are100 different 2-image combinations of the digits 0 ∼ 9 .
We randomly pick up 90 combinations asthe training set and the remaining 10 combinations as the extrapolation set.
This problem is treatedas regression instead of classification for higher error tolerance following Bloice et al.
Inaddition, if we design the network as a classifier, the number of the class will vary as the change ofthe splitting for the training set and testing set.
The network only outputs one single discrete value.
However, we still measure the accuracy by rounding the network output.
5 -layer fully-connected NNsand convolution NNs are chosen as the baselines.
For comparison, we implement NN-Hp with dense layers and NN-Hp with convolution layers, respectively.
More details on the implementation can befound in the Appendix E.3.
The results obtained by a three-fold cross validation are summarizedin Table 1, where we can see NN-Hp has a better extrapolation behavior in such more difficult task.
[Figure]
[Figure]
4.3 Spectral bias in learning spherical harmonics.
This experiment is to learn the linear combinations of spherical Harmonics where inputs issampled from the uniform distribution on the unit sphere.
The target function is defined by:f
[Equation]
[Equation]
=
[Equation]
[Equation]
[Equation]
K = { 1 , 3 , 4 , 5 , 8 , 12 } , where P k ( t ) denotes the k -degree
Gegenbauer polynomial, ζ k are fixed vectors that are independently sampled from uniform distribution on unit sphere, and N ( K ) is the normalizing constant.
The error residuals with different K
[Table]
      5 H V L GXD O  S U R M H F W L RQ  O
HQJ W K   O
RJ  VF D O H  GHJUHH        LWHUDWLRQV 5 H V L GXD O  S U R M H F W L RQ  O
HQJ W K   O
RJ  VF D O H  GHJUHH        LWHUDWLRQV 5 H V L GXD O  S U R M H F W L RQ  O
HQJ W K   O
RJ  VF D O H  GHJUHH N  N  N
 N  N
 N 
We can see the improvement for NNs-Hp with high-degree multiplicative interactions in the rate of convergence of error.
considers NNs with only 2-order multiplicative interactions.
In this experiment, we turn to show thatincreasing the number of multiplicative interactions can improve the rate of convergence of error, aspresented in Figure 5.
Conclusion
This paper examines a specific type of neural network with Hadamard product, called, NNs-Hp, froma theoretical perspective.
The analysis of the NTK paves the way for knowing interesting proportiesof the networks, such as extrapolation behavior and spectral bias.
Experimental results in learningsimple functions validate our hypothesis.
We further conduct several experiments in real-worlddatasets and demonstrate the advantage of inserting Hadamard product into standard neural networks.
We believe not only our framework provides a good justification for a deeper understanding of neuralarchitecture, but it also lays the foundations to investigate other more complicated OOD problemssuch as domain adaption and invariant learning in future work.