Neural Networks with Hadamard Product: Separation on Extrapolation and Spectral Bias
Anonymous Author(s)
AffiliationAddress email
Abstract
Neural tangent kernel (NTK) is a powerful tool to analyze training dynamics of neural networks and their generalization bounds.
The study on NTK has been devotedto typical neural network architectures, but is incomplete for neural networks with
Hadamard products (NNs-Hp), e.g., StyleGAN and polynomial neural networks.
In this work, we derive the finite-width NTK formulation for NNs-Hp, and provetheir equivalence to the kernel regression predictor with the associated NTK, whichexpands the application scope of NTK.
Based on our results, we elucidate thedifference of NNs-Hp over standard neural networks with respect to spectral biasand extrapolation.
Our two key insights are that when compared to standard neuralnetworks, a) NNs-Hp admit a slower eigenvalue decay of the respective NTK;and b) are able to fit more complicated functions in the extrapolation regime.
Ourempirical results validate those two separations, which provide a good justificationfor a deeper understanding of neural architectures.
Introduction
In deep learning theory, neural tangent kernel (NTK)
[Jacot et al., 2018] is a powerful analysis tool thatbridges the gap among the training dynamics of neural networks (NNs) trained by gradient descentand kernel regression [Jacot et al., 2018, Arora et al., 2019].
NTK provides a tractable analysis forseveral phenomena in deep learning, e g., the global convergence of gradient descent [Allen-Zhu et al.
,2019, Chizat et al., 2019, Du et al., 2019a, 2018], the inductive bias behind NNs [Bietti and Mairal,2019], the spectral bias toward different frequency components [Cao et al., 2019, Choraria et al.
,2022], the extrapolation behavior [Xu et al., 2021] and the generalization ability [Huang et al., 2020].
The study on NTK has been devoted to typical NNs architectures, e.g., fully-connected NNs [Jacotet al., 2018], residual NNs [Tirer et al., 2020, Huang et al., 2020], convolutional NNs [Arora et al.
,2019], graph NNs [Du et al., 2019b] and recurrent NNs [Alemohammad et al., 2021].
Recently, NNs with Hadamard products (NNs-Hp), e.g., StyleGAN [Karras et al., 2019] and ΠNets [Chrysos et al., 2020], have received increasing attention due to their expressivity and efficiencyover traditional NNs [Chrysos et al., 2021a, Campbell and Broun, 2000, Su et al., 2020].
There havebeen several works attempting to demystify the success of NNs-Hp.
Fan et al.
[2021] prove thatsecond-degree multiplicative interactions allow NNs-Hp to enlarge the set of functions that can berepresented exactly with zero error.
Choraria et al.
[2022] reveal that NNs-Hp with second-degreemultiplicative interactions can yield faster learning of high-frequency information during training in
NTK regime.
Yet, the theoretical analysis of NNs-Hp with high-degree multiplicative interactionsis still unclear.
When using NTK for analysis, the complete and rigorous proof includes showingthe stability of empirical NTK during training and giving the width requirement, which is crucial toallow NTK-based analysis of typical NNs Arora et al.
[2019], Tirer et al.
but is still missingfor NNs-Hp.
In this work, we derive the NTK for NNs-Hp with high-degree multiplicative interactions and present arigorous proof for the equivalence to the kernel regression predictor.
This analysis enables us to furtherexamine NNs-Hp in a theoretical perspective.
This NTK analysis can be useful for investigatingproperties, such as the extrapolation.
StyleGAN and Π -Nets have demonstrated stellar in-distributionperformance, however implementing networks for real-world applications requires the ability toextrapolate to unseen data, e.g., for medical imaging.
Neural networks have demonstrated weaknessesin extrapolating simple arithmetic problems [Saxton et al., 2019] or learning simple functions [Haleyand Soloway, 1992, Sahoo et al., 2018].
Early works demonstrate that fully-connected NNs are unableto achieve exact extrapolation when fitting low-degree polynomial functions [Haley and Soloway,1992].
Xu et al.
[2021] theoretically and empirically point out that 2-layer fully-connected NNs with ReLU can only extrapolate to linear functions.
However, data in real-world scenarios usually liein highly nonlinear manifolds [Van Der Maaten et al., 2009].
Our findings extend these result andhighlight that NNs-Hp can extrapolate to unseen data in a non-linear way.
What’s more, studying the NTK of NNs- Hp also allows us to investigate its spectral bias.
Overall, our main contributions can be summarized as follows:
• We derive the NTK formulation for NNs-Hp with high-degree multiplicative interactions, andgive a concrete bound of the widths requirement for convergence to the NTK at initialization,
and stability during training, which allows us to bridge the gap among NNs-Hp trained via
gradient descent and kernel regression predictor.
• We provably present the extrapolation behavior of NNs-Hp and empirically show its advantage over standard NNs in learning common used functions and performing arithmeticextrapolation in real-world dataset.
• We provide a detailed analysis for the decay rate of eigenvalues of the NTK of NNs-Hpwith high-degree multiplicative interactions in the task of learning spherical harmonics
andscrutiny that increasing the multiplicative interactions results in a faster learning towardshigh-frequency information.
Background
In this section, we establish the notations, provide an overview of NTK, and summarize the mostclosely related work in NNs-Hpas well as extrapolation.
2.1 Notation
The core operators and symbols are summarized in Table 2 at Appendix A. Vectors (matrices) aresymbolized by lowercase (uppercase) boldface letters, e.g., a , A .
We use { x i } |X| i =1 and { y i } |X| i =
1 topresent the input features and their labels of the training set ( X , Y ) , where |X| denotes the cardinality.
We symbolize by K ( x , x ′ )
the neural tangent kernel with input x and x ′ , K ∈ R |X|×|X| , where
K ij = K
( x i , x i ) .
Next, we denote by θ t the parameter vector, ∥
e ( θ t ) ∥
22 the training loss, and ˆ K tthe empirical NTK Gram matrix at time step t .
The following notation is used:
[Equation]
2.2 Neural tangent kernel
Neural networks (NNs) are relevant to kernel method, under proper initialization [Daniely et al.
, 2016,de G. Matthews et al., 2018].
Jacot et al.
[2018] provably demonstrate the equivalence between thetraining dynamics by gradient descent and kernel regression induced by NTK when employing theℓ 2 loss.
Below, we recall the exact formula regarding the NTK of N -layer ( N > 2 ) fully-connected
NNs with ReLU activation functions σ .
The corresponding NTK K ( x , x ′ )
=
K N ( x , x ′ ) could becomputed recursively by:
[Equation]
where the covariance Σ n and its derivative ˙Σ n are defined as:
[Equation]
for i = 1 , . . .
, N .
Furthermore, the aforementioned NTK is extended to residual NNs [Tirer et al.
,2020, Huang et al., 2020], convolutional NNs [Arora et al., 2019], graph NNs [Du et al., 2019b], andrecurrent NNs [Alemohammad et al., 2021].
One of the roles of such kernel is to analyze the trainingbehavior of the neural network [Allen-Zhu et al., 2019, Chizat et al., 2019, Du et al., 2019a, 2018].
For instance, Lee et al.
[2019] showcase that through NTK parameterization NNs trained via gradientdescent of any depth evolve to linear models.
Meanwhile, the inductive bias of convolutional networks,e.g., deformation stability of the images, has been studied in the NTK regime
[Bietti and Mairal, 2019].
2.3 Neural networks with Hadamard product
The ideas of augmenting NNs with Hadamard products to allow multiplicative interactions can betraced back to at least to Ivakhnenko [1971] that investigate the learnable polynomial relationships.
Most of early work e.g., Group Method of Data Handling [Ivakhnenko, 1971], pi-sigma network [Shinand Ghosh, 1991] do not scale well for high-dimensional signals.
Chrysos et al.
[2021c] factorizethe weight of NNs-Hp based on tensor decompositions to reduce the number of parameters.
Theyexhibit how to convert popular networks, such as residual networks, and convolutional NNs to theform of NNs-Hp.
StyleGAN can be also considered as a special type of NN-Hp [Chrysos et al.
New efforts have recently emerged to improve the architecture of the network with Hadamardproducts [Chrysos et al., 2021b, Babiloni et al., 2021, Chrysos et al., 2021a].
In this work, we adoptcomplementary approach and focus on the extrapolation as well as the spectral bias from a theoreticalperspective.
2.4 Extrapolation
The study of extrapolation properties of NNs dates at least back to the 90’s [Barnard and Wessels, 1992,
Kramer and Leonard, 1990].
Experimental results show poor performance of NNs in case of learningsimple functions [Barnard and Wessels, 1992].
Browne [2002] also suggest that fully-connected NNscannot extrapolate well and then illustrate how the representation of the input impact the extrapolation.
Xu et al.
[2021] provably present the extrapolation behavior of fully-connected NNs and Graph neuralnetworks [Jacot et al., 2018].
Specifically, they show that 2-layer fully-connected NNs with ReLUactivation function extrapolate to linear function in extrapolation region.
Our work exhibits that
NNs-Hp can learn high degree nonlinear function.
Apart from fully-connected NNs, Martius and
Lampert [2016], Sahoo et al.
[2018] showcase a novel family of functions with linear mapping anda non-linear transformation, which allows to use sine and cosine as nonlinearities, enabling suchnetworks to learn well in analytical expressions.
Note that there exist multiplication units in EQL,which is similar to the multiplicative interactions in NNs-Hp.
Lastly, extrapolation is often consideredin the context of out-of-distribution (OOD).
There are other types of OOD problems with specificsetting among machine learning community [Shen et al., 2021].
Domain adaption assumes the sourceand the target domains lie in the same feature space but with different distributions [Kouw and
Loog, 2019], which differs from extrapolation.
Another category of methodology to solve the OODgeneralization problem, called invariant learning, aims to discover high-level invariance feature fromlow-level observations through latent causal mechanisms [Arjovsky et al., 2019, Rosenfeld et al.
We believe our analysis can also encourage the usage of NNs-Hp in these OOD problems.
Analysis of NNs-Hp
Our analysis admits the following structure: we firstly study the NTK of NNs-Hp in Section 3.1, whichallows us to conduct analysis towards extrapolation in Section 3.2, and spectral bias in Section 3.3.
3.1 Neural tangent kernel of NNs-Hp
We now develop the NTK for NNs-Hp, then we bridge the gap between the NNs-Hp trained bygradient descent with respect to squared loss and the kernel regression predictor involving the
NTK.
Our analysis is conducted on a specific type of NNs-Hp called polynomial neural networks(PNNs) while we still use the term NN-Hp for consistency.
The goal of such networks is to learnan N -degree
( N ≥ 2 ) polynomial expansion that outputs f ( x ) ∈ R with respect to the inputx ∈ R d .
For simplifying the proof, we consider the following formulation of NNs-Hp, which is areparameterization version of PNNs [Zhu et al., 2022].
The output is given by:
[Equation]
where each element in W N +1 ∈ R 1 × m and W n ∈ R m × d , for n = 1 , . .
, N is independentlysampled from N (0 , 1) .
Two remarks are in place: a) We multiply by the scaling factor (cid:113) 2 m after eachdegree to ensure that the norm of the network output is preserved at initialization with infinite-widthsetting.
b) ReLU is usually required to increase the performance of NNs-Hp in experiments [Chrysoset al., 2021c].
Lemma 1.
The NTK of N -degree NNs-Hp, denoted by K ( x , x ′ ) , can be derived as:
[Equation]
where κ 1 and κ 2 are defined by taking the random Gaussian vector w ∈ R d
[Equation]
The proof, which provided in Appendix B.1, is based on the standard NTK calculations.
Differentlyfrom fully-connected NNs, the existence of multiplicative interaction in NNs-Hp induces the productform of multiple kernel.
Next, we provide the following theorem that gives a concrete requirement for the width of thenetworks that is sufficient for nonasymptotic convergence to the NTK at initialization,
Theorem 1.
(Convergence to the NTK).
Consider N -degree NNs-Hp, and assume that the width m ≥ 2 4 N − 2 log 2 N − 1 (2 N/δ ) for any δ ∈ (0 , 1) .
Then, given two inputs x , x ′ on the unit sphere, with probability at least 1 − δ over the randomness of initialization, we have that
[Equation]
where ρ = √ 2 2 N − 1 √ 8 e 3 (2 π ) 1 / 4 e 1 / 24 (cid:0) e 2 /e
(2 N − 1) / 2 (cid:1) (2 N − 1) / 2 .
Remark:
This result exhibits that the inner product of the Jacobian converges to the NTK atinitialization, which has not been studied before for NNs-Hp.
This theorem allows us to furtheranalyze the extrapolation of networks from the perspective of NTK.
It should be noted that the term ρand the width are exponential with respect to the degree N , but the degree N is not large in practice,e.g., at most 15 in Chrysos et al.
[2021a].
Hence the bound is fair and reasonable.
The technical key issue of the proof is to provide probability estimates for the multiplication ofseveral sub-exponential random variables.
To this end, we rely on the concentration of sub-Weibullrandom variables [Zhang and Chen, 2020] to complete the proof, which is deferred to Appendix B.2.
Below, we show that under certain conditions, the limiting NTK of NNs-Hp stays constant whentraining with gradient descent using squared loss.
Theorem 2 (Stability of the NTK during training) .
Consider the NNs-Hp in Eq.
Assumeλ min ( K ) > 0 and the training data ( X , Y ) is contained in some compact set x ̸ =
˜ x for allx , ˜ x ∈ X .
Then, there exist some constant R 0 > 0 , M > 1 , and Q > 1 such that for everym > M , when minimizing the squared loss with gradient descent and sufficient small learning rateη 0 < 2( λ min ( K ) + λ max ( K )) − 1 , the following inequality holds with high probability over the random initialization of θ :
[Equation]
Regarding the proof, we firstly introduce the norm control of the Gaussian weight matrices and thenderive the local boundness and local Lipschitzness.
The last step is to apply the induction rules overdifferent time steps.
Details are presented in the Appendix B.3.
Remark: Eq.
(5) bounds the training loss and Eq.
(6) implies small perturbation of the weights.
Eq.
(7) shows that ˆ K t m −→ ∞
−−−−→ ˆ K 0 .
Combining this with Theorem 1 that states ˆ K 0 m −→ ∞
−−−−→ K ,we have ˆ K t m −→ ∞ −−−−→ K .
Thus, the equivalence to the kernel regression is established.
Note that
Theorem 2 is an extension from the corresponding theorem of NNs with residual connection [Tireret al., 2020] to NNs with Hadamard product.
This property allows us to characterize the trainingprocess as kernel regression.
3.2 Extrapolation behavior
The derived kernel above enables us to study how NNs-Hp with ReLU activation trained by gradientdescent extrapolates in OOD regime.
Theorem 3 ( γ -degree ( γ ≤ N ) extrapolation of N -degree NNs-Hp) .
Suppose we train N -degree
( N ≥ 2 ) NNs-Hp f : R d → R with Eq.
(2) with infinite-width on { ( x i , y i ) } |X| i =1 , and the networkis optimized with squared loss in the NTK regime.
For any direction v ∈ R d that satisfies ∥ v ∥ 2 =max {∥ x ∥ 2 i } , let x 0 =
t v and x = x
+ h v with t > 1 and h > 0 be the OOD data points, theoutput f ( x 0 + h v ) follows a γ -degree ( γ ≤ N ) function with respect to h .
Apart from NNs-Hp, we also consummate Xu et al.
[2021] that considers the extrapolation of fullyconnected NNs with only 2 -layer.
We provide the following generalized theorem for N -layer ( N > 2 )fully-connected NNs.
Theorem 4 (Linear extrapolation of N -layer fully-connected NNs) .
Suppose we train N -layer( N ≥ 2 ) fully-connected NNs f : R d → R on { ( x i , y i ) } |X| i =1 .
For any direction v ∈ R d that satisfies∥ v ∥ 2 = max {∥ x ∥ 2 i } , x 0 = t v and x = x 0 + h v with t >
and h > 0 are OOD data points , theoutput f ( x 0 + h v ) follows a linear function with respect to h .
We have already shown that NNs-Hp extrapolate to a function with specific degree and is more flexiblethan fully-connected NNs.
However, only knowing the information of degree of the extrapolationfunction is not enough.
Naturally, we might ask under which condition NNs-Hp can achieve successfulextrapolation.
Below, we build our analysis in NTK regime and show how the geometry of the trainingset affects the behavior of NNs-Hp.
Theorem 5 (Condition for exact extrapolation of NNs-Hp) .
Let f ρ ( x ) =
x
⊤ βx be the targetfunction with x ∈ R d and β ∈ R d × d .
Suppose that { x i } |X| i =1 contains orthogonal basis { e i } di =
and{− e i } di =1 , then if we train 2-degree NNs-Hp f with ReLU activation function on { ( x i , f ρ ( x i )) } |X| i
=1with
the squared loss in the NTK regime, then we have f ( x ) =
x ⊤ βx for all x ∈ R d .
Remark:
This result only considers quadratic functions as our proof heavily relies on the constructionof the feature map of NTK, which is harder for high-degree case.
Due to constrained space, the proof of aforementioned theorems can be found in Appendix C.1 to C.3.
3.3 Spectral analysis
In this section, we characterize the approximation properties of N -degree NNs-Hp in the indistribution regime.
By studying the spectral analysis in the form of a Mercer decomposition,we explicitly show the eigenvalues and eigenfunctions of NTK.
[Figure]
We firstly introduce some notation.
Denote by { Y k,j } N ( d,k ) j
= the spherical harmonics of degree k ind + 1 variables.
G ( γ ) k represents the Gegenbauer polynomials with respect to the weight functionx (cid:55)→ (1 − x 2 )
γ − 12 and degree k .
Finally, denote by F ( d, k ) :
= 2 k + d − 1 k (cid:0) k
+ d − 2 d − 1 (cid:1) .
The following lemma enables to connect spherical harmonics to Gegenbauer polynomials.
Lemma 2 (Eq.
(24) in Choraria et al.
For any dot product Mercer kernel K , when the datax and x ′ are uniform on the unit sphere, we can apply the following Mercer’s decomposition in termsof spherical harmonics
[Equation]
Theorem 6.
NNs-Hp with N -degree
N ≥ 2 ) multiplicative interactions is considered.
Denote by{ µ 1 , µ 2 , . . .
} the eigenvalues associated to the NTK.
Then for k ≫ d , we have µ k = Ω(( N 2 k ) − d/ 2 ) .
The proof can be found in Appendix D. As a comparison, the decay rate for both deep fully-connected
NNs and residual NNs is Ω(( k ) − d ) [Belfer et al., 2021].
Thus, we can see a slower decay ratewhen inserting Hadamard product into standard NNs, which leads to a faster learning towardshigh-frequency information.
Experiments
Our experiment is organized as follows: We firstly showcase the extrapolation of NNs-Hp in learningsome common functions in Section 4.1.
Next, we assess the extrapolation performance on nonsynthetic dataset in Section 4.2 and conduct the experiment in learning spherical harmonics in
Section 4.3.
Due to the constrained space, the extrapolation in visual analogy task and the spectralbias in image classification task are deferred to Appendix E.5 and Appendix E.6, respectively.
4.1 Extrapolation in learning simple functions
These experiments aim to examine the extrapolation behavior of NNs-Hp in regression tasks andvalidate Theorem 3 and Theorem 4.
Our first experiment includes training the networks via thesquared loss to fit several well-known and simple underlying functions.
Unless specified otherwise,
N -layer NNs is compared against NN-Hp with N − 1 degree multiplicative interactions in eachexperiment, the reason is explained in Appendix E.1.
During prediction, we sample data points beyondthe training regime and observe the extrapolation performance.
More details on implementation canbe found in Appendix E.1.
[Figure] [Figure]
We set the target function as f ρ ( x ) =
x 3 + x 2 − 10 x + 5 and use 4-layer fully-connected NN.
Aspresented in Figure 1(a) and Figure 1(b) fully-connected NN extrapolates linearly while NN-Hpapproximates better the out-of-distribution part of the underlying non-linear function, which areconsistent with Theorem 3 and Theorem 4.
Learning f ρ ( x ) = cos (2 x ) .
We choose 11 -layer fully-connected NN.
The training set and testing setare the same as in the previous experiment.
Observing Figure 1(c) and Figure 1(d), we find that NNHp is more flexible to learn the non-linear function outside the training region while fully-connected
NN still extrapolates linearly.
Learning f ρ ( x ) = x 2(1) + x 2(2) , where x (1) and x (2) is the first and second dimension of x ∈ R 2 .
In this task, we choose 3 -layer NNs.
Each model is trained with different data distribution, i.e., thetraining set contains support in all directions in the first case while two dimension of the trainingset are fixed to be positive in the second case.
The result is visually depicted in Figure 2, whichshows that NN-Hp can achieve exact extrapolation if the training set contains support in all directionsand thereby validates Theorem 5.
On the other hand, NN without the Hadamard product fails toextrapolate to the underlying function due to its linear extrapolation.
4.2 Extrapolation in real-world dataset
In this section, we assess the extrapolation performance beyond synthetic datasets.
Variation of brightness.
This experiment is conducted on two well-known grayscale image datasets:
MNIST dataset [LeCun et al., 1998] and Fashion-MNIST dataset Xiao et al.
For these twodatasets, the original range of the pixel of each image is [0 , 1] , we divided it by 10 for the raw trainingset to construct the new one where the pixels range from 0 to 0 .
During extrapolation, we limit therange of the original testing set to [0 , r max ] through division, where r max ∈ { 0 . 1 , 0 . 2 , 0 .
3 , ..., 1 . 0 } , asillustrated in the top two panels in Figure 3.
Then we feed these images into the trained networkand evaluate the accuracy.
More details on the implementation can be found in Appendix E.2.
Theaccuracy is summarized in the two bottom plots of Figure 3.
We find that both networks achievesimilar accuracy in the case r max = 0 .
1 while inserting Hadamard product (Hp) into NN improvesthe performance during extrapolation.
Arithmetic extrapolation.
Now we turn to a more challenging task.
As human we can usuallyextrapolate to arbitrarily large numbers in arithmetic.
How do the neural networks perform duringextrapolation?
Following the setup of Bloice et al.
[2020], we use MNIST dataset, where there are100 different 2-image combinations of the digits 0 ∼ 9 .
We randomly pick up 90 combinations asthe training set and the remaining 10 combinations as the extrapolation set.
This problem is treatedas regression instead of classification for higher error tolerance following Bloice et al.
Inaddition, if we design the network as a classifier, the number of the class will vary as the change ofthe splitting for the training set and testing set.
The network only outputs one single discrete value.
However, we still measure the accuracy by rounding the network output.
5 -layer fully-connected NNsand convolution NNs are chosen as the baselines.
For comparison, we implement NN-Hp with denselayers and NN-Hp with convolution layers, respectively.
More details on the implementation can befound in the Appendix E.3.
The results obtained by a three-fold cross validation are summarizedin Table 1, where we can see NN-Hp has a better extrapolation behavior in such more difficult task.
[Figure]
Seen pair
Unseen
[Figure]
4.3 Spectral bias in learning spherical harmonics
This experiment is to learn the linear combinations of spherical Harmonics where inputs aresampled from the uniform distribution on the unit sphere.
The target function is defined by:f ∗
[Equation]
x
[Equation]
=
[Equation]
= { 1 , 3 , 4 , 5 , 8 , 12 } , where P k ( t ) denotes the k -degree
[Table]
      5 H V L GXD O  S U R M H F W L RQ  O
HQJ W K   O
RJ  VF D O H  GHJUHH        LWHUDWLRQV 5 H V L GXD O  S U R M H F W L RQ  O
HQJ W K   O
RJ  VF D O H  GHJUHH        LWHUDWLRQV 5 H V L GXD O  S U R M H F W L RQ  O
HQJ W K   O
RJ  VF D O H  GHJUHH N  N  N
 N  N
 N 
We can see the improvement for high-degree interactions in the rate of convergence of error.
Gegenbauer polynomial, ζ k are fixed vectors that are independently sampled from uniform distribution on unit sphere, and N ( K ) is the normalizing constant.
The error residuals with different K arecompared during the training process.
Implementation details are deferred to Appendix E.4.
In thisexperiment, we show that increasing the number of multiplicative interactions can improve the rate ofconvergence of error, as presented in Figure 5.
Conclusion
This paper examines a specific type of neural network with Hadamard product, called, NNs-Hp, froma theoretical perspective.
The analysis of the NTK paves the way for knowing interesting propertiesof the networks, such as extrapolation behavior and spectral bias.
Experimental results in learningsimple functions validate our hypothesis.
We further conduct several experiments in real-worlddatasets and demonstrate the advantage of inserting Hadamard product into standard neural networks.
We believe not only our framework provides a good justification for a deeper understanding of neuralarchitecture, but it also lays the foundations to investigate other more complicated OOD problemssuch as domain adaption and invariant learning in future work.