Extrapolation and Spectral Bias of Neural Nets with Hadamard Product: a Polynomial Net Study
Abstract
Neural tangent kernel (NTK) is a powerful tool to analyze training dynamics ofneural networks and their generalization bounds.
The study on NTK has beendevoted to typical neural network architectures, but is incomplete for neural networks with Hadamard products (NNs-Hp), e.g., StyleGAN and polynomial neuralnetworks.
In this work, we derive the finite-width NTK formulation for a specialclass of NNs-Hp, i.e., polynomial neural networks. We prove their equivalenceto the kernel regression predictor with the associated NTK, which expands theapplication scope of NTK.
Based on our results, we elucidate the separation of PNNs over standard neural networks with respect to extrapolation and spectral bias.
Our two key insights are that when compared to standard neural networks, PNNsare able to fit more complicated functions in the extrapolation regime and admita slower eigenvalue decay of the respective NTK.
Besides, our theoretical resultscan be extended to other types of NNs-Hp, which expand the scope of our work.
Our empirical results validate the separations in broader classes of NNs-Hp, whichprovide a good justification for a deeper understanding of neural architectures.
Introduction
In deep learning theory neural tangent kernel (NTK) [Jacot et al., 2018] is a powerful analysis tool thatbridges the gap among the training dynamics of neural networks (NNs) trained by gradient descentand kernel regression [Jacot et al., 2018, Arora et al., 2019].
NTK provides a tractable analysis forseveral phenomena in deep learning, e g., the global convergence of gradient descent [Allen-Zhu et al.
,2019, Chizat et al., 2019, Du et al., 2019a, 2018], the inductive bias behind NNs [Bietti and Mairal,2019], the spectral bias toward different frequency components [Cao et al., 2019, Choraria et al.
,2022], the extrapolation behavior [Xu et al., 2021], and the generalization ability [Huang et al., 2020].
The study on NTK has been devoted to typical NNs architectures, e.g., fully-connected NNs [Jacotet al., 2018], residual NNs [Tirer et al., 2020, Huang et al., 2020], convolutional NNs [Arora et al.
,2019], graph NNs [Du et al., 2019b] and recurrent NNs [Alemohammad et al., 2021].
Recently, NNs with Hadamard products (NNs-Hp), e.g., StyleGAN [Karras et al., 2019] and ΠNets [Chrysos et al., 2020], have received increasing attention due to their expressivity and efficiencyover traditional NNs [Chrysos et al., 2021a, Campbell and Broun, 2000, Su et al., 2020].
There have been several works attempting to demystify the success of NNs-Hp.
Fan et al.
[2021] prove thatsecond-degree multiplicative interactions allow NNs-Hp to enlarge the set of functions that can berepresented exactly with zero error.
Choraria et al.
[2022] reveal that NNs-Hp with second-degreemultiplicative interactions can yield faster learning of high-frequency information during training in
NTK regime.
Yet, the theoretical analysis of NNs-Hp with high-degree multiplicative interactionsis still unclear.
When using NTK for analysis, the complete and rigorous proof includes showingthe stability of empirical NTK during training and giving the width requirement, which is crucial to allow NTK-based analysis of typical NNs Arora et al.
[2019], Tirer et al.
but is still missingfor NNs-Hp.
Polynomial neural networks [Chrysos et al., 2021a, Babiloni et al., 2021], a special class of NNsHp [Jayakumar et al., 2020], have showcased remarkable performance on a broad range of applications.
As a step for analyzing NNs-Hp, in this work, we derive the NTK for PNNs with high-degreemultiplicative interactions and present a rigorous proof for the equivalence to the kernel regressionpredictor.
This analysis enables us to further examine PNNs in a theoretical perspective.
This NTK analysis can be useful for investigating properties, such as the extrapolation.
StyleGAN andΠ -Nets have demonstrated stellar in-distribution performance, however implementing networks forreal-world applications requires the ability to extrapolate to unseen data, e.g., for medical imaging.
Neural networks have demonstrated weaknesses in extrapolating simple arithmetic problems [Saxtonet al., 2019] or learning simple functions [Haley and Soloway, 1992, Sahoo et al., 2018].
Earlyworks demonstrate that fully-connected NNs are unable to achieve exact extrapolation when fittinglow-degree polynomial functions [Haley and Soloway, 1992].
Xu et al.
[2021] theoretically andempirically point out that 2-layer fully-connected NNs with ReLU can only extrapolate to linearfunctions.
However, data in real-world scenarios usually lie in highly nonlinear manifolds [Van Der Maaten et al., 2009].
Our findings extend these results and highlight that PNNs can extrapolateto unseen data in a non-linear way.
Besides, studying the NTK of PNNs also allows us to investigateits spectral bias.
Overall, our main contributions can be summarized as follows:
• We derive the NTK formulation for PNNs with high-degree multiplicative interactions, andgive a concrete bound of the widths requirement for convergence to the NTK at initialization,and stability during training, which allows us to bridge the gap among PNNs trained viagradient descent and kernel regression predictor.
• We provably demonstrate the extrapolation behavior of PNNs as well as other NNs-Hp,including multiplicate filter networks and non-local multiplicate networks. Besides, thespectral analysis of NTK of PNNs is also given for better understanding.• We empirically show the advantage of NNs-Hp over standard NNs in learning commonlyused functions, performing arithmetic extrapolation in real-world dataset, and conductingvisual analogy extrapolation task.
We scrutiny that increasing the multiplicative interactionsresults in a faster learning towards high-frequency information.
Background
In this section, we establish the notations, provide an overview of NTK, and summarize the mostclosely related work in NNs-Hp as well as extrapolation.
2.1 Notation
The core operators and symbols are summarized in Table 2 at Appendix A. Vectors (matrices) aresymbolized by lowercase (uppercase) boldface letters, e.g., a , A .
We use { x i } |X| i =1 and { y i } |X| i =
1 topresent the input features and their labels of the training set ( X , Y ) , where |X| denotes the cardinality.
We symbolize by K ( x , x ′ )
the neural tangent kernel with input x and x ′ , K ∈ R |X|×|X| , where
K ij = K
( x i , x i ) .
Next, we denote by θ t the parameter vector, ∥
e ( θ t ) ∥
22 the training loss, and ˆ K tthe empirical NTK Gram matrix at time step t .
The following notation is used:
[Equation]
|X|×|X| .
2.2 Neural tangent kernel
Neural networks (NNs) are relevant to kernel method, under proper initialization [Daniely et al.
, 2016,de G. Matthews et al., 2018].
Jacot et al.
[2018] provably demonstrate the equivalence between thetraining dynamics by gradient descent and kernel regression induced by NTK when employing the ℓ 2 loss.
Below, we recall the exact formula regarding the NTK of N -layer ( N > 2 ) fully-connected
NNs with ReLU activation functions σ .
The corresponding NTK K ( x , x ′ )
=
K N ( x , x ′ ) could becomputed recursively by:
[Equation]
where the covariance Σ n and its derivative ˙Σ n are defined as:
[Equation]
for i = 1 , . . .
, N .
Furthermore, the aforementioned NTK is extended to residual NNs [Tirer et al.
,2020, Huang et al., 2020], convolutional NNs [Arora et al., 2019], graph NNs [Du et al., 2019b], andrecurrent NNs [Alemohammad et al., 2021].
One of the roles of such kernel is to analyze the trainingbehavior of the neural network [Allen-Zhu et al., 2019, Chizat et al., 2019, Du et al., 2019a, 2018].
For instance, Lee et al.
[2019] showcase that through NTK parameterization NNs trained via gradientdescent of any depth evolve to linear models.
Meanwhile, the inductive bias of convolutional networks,e.g., deformation stability of the images, has been studied in the NTK regime
[Bietti and Mairal, 2019].
.3 Neural networks with Hadamard product
The ideas of augmenting NNs with Hadamard products to allow multiplicative interactions can betraced back to at least to Ivakhnenko [1971] that investigate the learnable polynomial relationships.
Most of early work e.g., Group Method of Data Handling [Ivakhnenko, 1971], pi-sigma network [Shinand Ghosh, 1991] do not scale well for high-dimensional signals.
Chrysos et al.
[2021c] factorizethe weight of NNs-Hp based on tensor decompositions to reduce the number of parameters.
Theyexhibit how to convert popular networks, such as residual networks, and convolutional NNs to theform of NNs-Hp.
StyleGAN can be also considered as a special type of NN-Hp [Chrysos et al.
New efforts have recently emerged to improve the architecture of the network with Hadamardproducts [Chrysos et al., 2021b, Babiloni et al., 2021, Chrysos et al., 2021a].
In this work, we adoptcomplementary approach and focus on the extrapolation as well as the spectral bias from a theoreticalperspective.
2.4 Extrapolation
The study of extrapolation properties of NNs dates at least back to the 90’s [Barnard and Wessels, 1992,
Kramer and Leonard, 1990].
Experimental results show poor performance of NNs in case of learningsimple functions [Barnard and Wessels, 1992].
Browne [2002] also suggest that fully-connected NNscannot extrapolate well and then illustrate how the representation of the input impact the extrapolation.
Xu et al.
[2021] provably present the extrapolation behavior of fully-connected NNs and Graph neuralnetworks [Jacot et al., 2018].
Specifically, they show that 2-layer fully-connected NNs with ReLUactivation function extrapolate to linear function in extrapolation region.
Our work exhibits that
NNs-Hp can learn high degree nonlinear function.
Apart from fully-connected NNs, Martius and
Lampert [2016], Sahoo et al.
[2018] showcase a novel family of functions with linear mapping anda non-linear transformation, which allows to use sine and cosine as nonlinearities, enabling suchnetworks to learn well in analytical expressions.
Note that there exist multiplication units in EQL,which is similar to the multiplicative interactions in NNs-Hp.
Lastly, extrapolation is often consideredin the context of out-of-distribution (OOD).
There are other types of OOD problems with specificsetting among machine learning community [Shen et al., 2021].
Domain adaption assumes the sourceand the target domains lie in the same feature space but with different distributions [Kouw and
Loog, 2019], which differs from extrapolation.
Another category of methodology to solve the OODgeneralization problem, called invariant learning, aims to discover high-level invariance feature fromlow-level observations through latent causal mechanisms [Arjovsky et al., 2019, Rosenfeld et al.
We believe our analysis can also encourage the usage of NNs-Hp in these OOD problems.
Analysis of PNNs
Our analysis admits the following structure: we firstly study the NTK of PNNs in Section 3.1, whichallows us to conduct analysis towards extrapolation in Section 3.2, and spectral bias in Section 3.3.
In Appendix F and G, of the supplementary, we consider extensions beyond PNNs to other familiesof NNs-Hp, e.g. multiplicative filter networks and non-local networks with Hadamard product.
3.1 Neural tangent kernel
We now develop the NTK for PNNs, then we bridge the gap between the NNs-Hp trained by gradientdescent with respect to squared loss and the kernel regression predictor involving the NTK.
Ouranalysis is conducted on a specific type of NNs-Hp called PNNs .
The goal of such networks is to learnan N -degree
( N ≥ 2 ) polynomial expansion that outputs f ( x ) ∈ R with respect to the input x ∈ R d .
For simplifying the proof, we consider the following formulation, which is a reparameterizationversion of PNNs [Zhu et al., 2022].
The output is given by:
[Equation]
where each element in W N +1 ∈ R 1 × m and W n ∈ R m × d , for n = 1 , . .
, N is independentlysampled from N (0 , 1) .
Two remarks are in place: a) We multiply by the scaling factor (cid:113) 2 m after eachdegree to ensure that the norm of the network output is preserved at initialization with infinite-widthsetting.
b) ReLU is usually required to increase the performance of NNs-Hp in experiments [Chrysoset al., 2021c].
Lemma 1.
The NTK of N -degree PNNs , denoted by K ( x , x ′ ) , can be derived as:
[Equation]
where κ 1 and κ 2 are defined by taking the random Gaussian vector w ∈ R d
[Equation]
The proof, which provided in Appendix B.1, is based on the standard NTK calculations.
Differentlyfrom fully-connected NNs, the existence of multiplicative interaction in PNNs induces the productform of multiple kernel.
Next, we provide the following theorem that gives a concrete requirement for the width of thenetworks that is sufficient for nonasymptotic convergence to the NTK at initialization,
Theorem 1.
(Convergence to the NTK).
Consider N -degree PNNs , and assume that the width m ≥ 2 4 N − 2 log 2 N − 1 (2 N/δ ) for any δ ∈ (0 , 1) .
Then, given two inputs x , x ′ on the unit sphere, with probability at least 1 − δ over the randomness of initialization, we have that
[Equation]
where ρ = √ 2 2 N − 1 √ 8 e 3 (2 π [Equation] [Equation] e 1 /
[Equation]
Remark:
This result exhibits that the inner product of the Jacobian converges to the NTK atinitialization, which has not been studied before for PNNs.
This theorem allows us to further analyzethe extrapolation of networks from the perspective of NTK.
It should be noted that the term ρ and thewidth are exponential with respect to the degree N , but the degree N is not large in practice, e.g., atmost 15 in Chrysos et al.
[2021a].
Hence the bound is fair and reasonable.
The technical key issue of the proof is to provide probability estimates for the multiplication ofseveral sub-exponential random variables.
To this end, we rely on the concentration of sub-Weibullrandom variables [Zhang and Chen, 2020] to complete the proof, which is deferred to Appendix B.2.
Below, we show that under certain conditions, the limiting NTK of PNNs stays constant when trainingwith gradient descent using squared loss.
Theorem 2 (Stability of the NTK during training) .
Consider the PNNs in Eq.
Assume λ min ( K ) >0 and the training data ( X , Y ) is contained in some compact set x ̸ =
˜ x for all x , ˜ x ∈ X .
Then, thereexist some constant R 0 > 0 , M > 1 , and Q > 1 such that for every m > M , when minimizing the squared loss with gradient descent and sufficient small learning rate η 0 < 2( λ min ( K )+ λ max ( K )) − 1 ,the following inequality holds with high probability over the random initialization of θ :
[Equation]
Regarding the proof, we firstly introduce the norm control of the Gaussian weight matrices and thenderive the local boundness and local Lipschitzness.
The last step is to apply the induction rules overdifferent time steps.
Details are presented in the Appendix B.3.
Remark: Eq.
(5) bounds the training loss and Eq.
(6) implies small perturbation of the weights.
Eq.
(7) shows that ˆ K t m −→ ∞
−−−−→ ˆ K 0 .
Combining this with Theorem 1 that states ˆ K 0 m −→ ∞
−−−−→ K ,we have ˆ K t m −→ ∞ −−−−→ K .
Thus, the equivalence to the kernel regression is established.
Note that
Theorem 2 is an extension from the corresponding theorem of NNs with residual connection [Tireret al., 2020] to PNNs.
This property allows us to characterize the training process as kernel regression.
3.2 Extrapolation behavior
The derived kernel above enables us to study how PNNs with ReLU activation trained by gradientdescent extrapolates.
Theorem 3 ( γ -degree ( γ ≤ N ) extrapolation of N -degree NNs-Hp) .
Suppose we train N -degree
N ≥ 2 ) PNNs f : R d → R with Eq.
(2) with infinite-width on { ( x i , y i ) } |X| i =1 , and the network isoptimized with squared loss in the NTK regime.
For any direction v ∈ R d that satisfies ∥ v ∥ 2 =max {∥ x ∥ 2 i } , let x 0 =
t v and x = x
+ h v with t > 1 and h > 0 be the extrapolation data points,the output f ( x 0 + h v ) follows a γ -degree ( γ ≤ N ) function with respect to h .
Apart from PNNs, we also consummate Xu et al.
[2021] that considers the extrapolation of fullyconnected NNs with only 2 -layer.
We provide the following generalized theorem for N -layer ( N > 2 )fully-connected NNs.
Theorem 4 (Linear extrapolation of N -layer fully-connected NNs) .
Suppose we train N -layer( N ≥ 2 ) fully-connected NNs f : R d → R on { ( x i , y i ) } |X| i =1 .
For any direction v ∈ R d that satisfies∥ v ∥ 2 = max {∥ x ∥ 2 i } , x 0 = t v and x = x 0 + h v with t >
and h > 0 are extrapolation data points, the output f ( x 0 + h v ) follows a linear function with respect to h .
We have already shown that PNNs extrapolate to a function with specific degree and is more flexiblethan fully-connected NNs.
However, only knowing the information of degree of the extrapolationfunction is not enough.
Naturally, we might ask under which condition PNNs can achieve successfulextrapolation.
Below, we build our analysis in NTK regime and show how the geometry of thetraining set affects the behavior of PNNs .
Theorem 5 (Condition for exact extrapolation of PNNs ) .
Let f ρ ( x ) =
x ⊤ βx be the target functionwith x ∈ R d and β ∈ R d × d .
Suppose that { x i } |X| i =1 contains orthogonal basis { e i } di =
and{− e i } di =1 , then if we train 2-degree PNNs f with ReLU activation function on { ( x i , f ρ ( x i )) } |X| i
=1with
the squared loss in the NTK regime, then we have f ( x ) =
x ⊤ βx for all x ∈ R d .
Remark:
This result only considers quadratic functions as our proof heavily relies on the constructionof the feature map of NTK, which is harder for high-degree case.
Due to constrained space, the proof of aforementioned theorems can be found in Appendix C.1 to C.3.
3.3 Spectral analysis
In this section, we characterize the approximation properties of N -degree PNNs in the in-distributionregime.
By studying the spectral analysis in the form of a Mercer decomposition, we explicitly showthe eigenvalues and eigenfunctions of NTK.
[Figure]
We firstly introduce some notation.
Denote by { Y k,j } N ( d,k ) j
= the spherical harmonics of degree k ind + 1 variables.
G ( γ ) k represents the Gegenbauer polynomials with respect to the weight functionx (cid:55)→ (1 − x 2 )
γ − 12 and degree k .
Finally, denote by F ( d, k ) :
= 2 k + d − 1 k (cid:0) k
+ d − 2 d − 1 (cid:1) .
The following lemma enables to connect spherical harmonics to Gegenbauer polynomials.
Lemma 2 (Eq.
(24) in Choraria et al.
For any dot product Mercer kernel K , when the datax and x ′ are uniform on the unit sphere, we can apply the following Mercer’s decomposition in termsof spherical harmonics
[Equation]
Theorem 6.
PNNs with N -degree ( N ≥ 2 ) multiplicative interactions is considered.
Denote by{ µ 1 , µ 2 , . . .
} the eigenvalues associated to the NTK.
Then for k ≫ d , we have µ k = Ω(( N 2 k ) − d/ 2 ) .
The proof can be found in Appendix D. As a comparison, the decay rate for both deep fully-connected
NNs and residual NNs is Ω(( k ) − d ) [Belfer et al., 2021].
Thus, we can see a slower decay ratewhen inserting Hadamard product into standard NNs, which leads to a faster learning towardshigh-frequency information.
Experiments
Our experiment is organized as follows: We firstly showcase the extrapolation of NNs-Hp in learningsome common functions in Section 4.1.
Next, we assess the extrapolation performance on nonsynthetic dataset in Section 4.2 and conduct the experiment in learning spherical harmonics in
Section 4.3.
Due to the constrained space, the extrapolation in visual analogy task and the spectralbias in image classification task are deferred to Appendix E.5 and Appendix E.6, respectively.
4.1 Extrapolation in learning simple functions
These experiments aim to examine the extrapolation behavior of NNs-Hp in regression tasks andvalidate Theorem 3 and Theorem 4.
Our first experiment includes training the networks via thesquared loss to fit several well-known and simple underlying functions.
Unless specified otherwise,
N -layer NNs is compared against NN-Hp with N − 1 degree multiplicative interactions in eachexperiment, the reason is explained in Appendix E.1.
During prediction, we sample data points beyondthe training regime and observe the extrapolation performance.
More details on implementation canbe found in Appendix E.1.
[Figure] [Figure]
We set the target function as f ρ ( x ) =
x 3 + x 2 − 10 x + 5 and use 4-layer fully-connected NN.
Aspresented in Figure 1(a) and Figure 1(b) fully-connected NN extrapolates linearly while NN-Hpapproximates better the extrapolation part of the underlying non-linear function, which are consistentwith Theorem 3 and Theorem 4.
Learning f ρ ( x ) = cos (2 x ) .
We choose 11 -layer fully-connected NN.
The training set and testing setare the same as in the previous experiment.
Observing Figure 1(c) and Figure 1(d), we find that NNHp is more flexible to learn the non-linear function outside the training region while fully-connected
NN still extrapolates linearly.
Learning f ρ ( x ) = x 2(1) + x 2(2) , where x (1) and x (2) is the first and second dimension of x ∈ R 2 .
In this task, we choose 3 -layer NNs.
Each model is trained with different data distribution, i.e., thetraining set contains support in all directions in the first case while two dimension of the trainingset are fixed to be positive in the second case.
The result is visually depicted in Figure 2, whichshows that NN-Hp can achieve exact extrapolation if the training set contains support in all directionsand thereby validates Theorem 5.
On the other hand, NN without the Hadamard product fails toextrapolate to the underlying function due to its linear extrapolation.
4.2 Extrapolation in real-world dataset
In this section, we assess the extrapolation performance beyond synthetic datasets.
Variation of brightness.
This experiment is conducted on two well-known grayscale image datasets:
MNIST dataset [LeCun et al., 1998] and Fashion-MNIST dataset Xiao et al.
For these twodatasets, the original range of the pixel of each image is [0 , 1] , we divided it by 10 for the raw trainingset to construct the new one where the pixels range from 0 to 0 .
During extrapolation, we limit therange of the original testing set to [0 , r max ] through division, where r max ∈ { 0 . 1 , 0 . 2 , 0 .
3 , ..., 1 . 0 } , asillustrated in the top two panels in Figure 3.
Then we feed these images into the trained networkand evaluate the accuracy.
More details on the implementation can be found in Appendix E.2.
Theaccuracy is summarized in the two bottom plots of Figure 3.
We find that both networks achievesimilar accuracy in the case r max = 0 .
1 while inserting Hadamard product (Hp) into NN improvesthe performance during extrapolation.
Arithmetic extrapolation.
Now we turn to a more challenging task.
As human we can usuallyextrapolate to arbitrarily large numbers in arithmetic.
How do the neural networks perform duringextrapolation?
Following the setup of Bloice et al.
[2020], we use MNIST dataset, where there are100 different 2-image combinations of the digits 0 ∼ 9 .
We randomly pick up 90 combinations asthe training set and the remaining 10 combinations as the extrapolation set.
This problem is treatedas regression instead of classification for higher error tolerance following Bloice et al.
Inaddition, if we design the network as a classifier, the number of the class will vary as the change ofthe splitting for the training set and testing set.
The network only outputs one single discrete value.
However, we still measure the accuracy by rounding the network output.
5 -layer fully-connected NNsand convolution NNs are chosen as the baselines.
For comparison, we implement NN-Hp with denselayers and NN-Hp with convolution layers, respectively.
More details on the implementation can befound in the Appendix E.3.
The results obtained by a three-fold cross validation are summarizedin Table 1, where we can see NN-Hp has a better extrapolation behavior in such more difficult task.
[Figure]
Seen pair
Unseen
[Figure]
4.3 Spectral bias in learning spherical harmonics
This experiment is to learn the
[2022], which only considers NNs-Hp with one Hadamard product.
The target function is defined
[Table]
      5 H V L GXD O  S U R M H F W L RQ  O
HQJ W K   O
RJ  VF D O H  GHJUHH        LWHUDWLRQV 5 H V L GXD O  S U R M H F W L RQ  O
HQJ W K   O
RJ  VF D O H  GHJUHH        LWHUDWLRQV 5 H V L GXD O  S U R M H F W L RQ  O
HQJ W K   O
RJ  VF D O H  GHJUHH N  N  N
 N  N
 N 
We can see the improvement for high-degree interactions in the rate of convergence of error by:
[Equation]
K = { 1 , 3 , 4 , 5 , 8 , 12 } , where P k ( t ) denotes the k-degree Gegenbauer polynomial, ζ k are fixed vectors that are independently sampled from uniformdistribution on unit sphere, and N ( K ) is the normalizing constant.
The error residuals with different
K are compared during the training process.
Implementation details are deferred to Appendix E.4.
Inthis experiment, we show that increasing the number of multiplicative interactions can improve therate of convergence of error, as presented in Figure 5.
Conclusion
This paper examines neural network with Hadamard product with a particular focus on polynomialneural networks from a theoretical perspective. The analysis of the NTK paves the way for knowinginteresting properties of the networks, such as extrapolation behavior and spectral bias.
Experimentalresults in learning simple functions validate our hypothesis. We further conduct several experiments inreal-world datasets and demonstrate the advantage of inserting Hadamard product into standard neuralnetworks.
We believe not only our framework provides a good justification for a deeper understandingof neural architecture, but it also lays the foundations to investigate other more complicated OODproblems such as domain adaption and invariant learning in future work.
Checklist
For all authors...
(a)
Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes
(b) Did you describe the limitations of your work?
[Yes] See Appendix
I
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Appendix H
(d) Have you read the ethics
review guidelines
and ensured
that your
paper conforms to
them? [Yes] If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] We provide detailed derivations
in the appendix.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No] The datasets
we use are all publicly available and commonly used in image-related tasks. We intend
to release the source code upon the acceptance of the paper, which should enable
interested practitioners to further improve our framework and verify our results.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Appendix E
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] We conduct three runs in our quantitative experiments,
e.g., Table 1.
(d) Did you include the total amount of compute and the type of resources used (e
g., type
of GPUs, internal cluster, or cloud provider)? [Yes] All our experiments are conducted
on a single GPU, which is part of our cluster.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] (b) Did you mention the license of the assets?
[N/A]
This is a theoretical work; we either
use synthetic experiments, e.g. analytic functions as the target, or standard image-based datasets.
All of the public datasets are publicly available and we cite their respecting
papers, where the licenses are mentioned.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[Table]
information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table] Board (IRB) approvals, if applicable? [N/A]
[Table]