Towards Understanding the Condensation of Neural Networks at Initial Training
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Empirical works show that for ReLU neural networks (NNs) with small initialization, input weights of hidden neurons (the input weight of a hidden neuronconsists of the weight from its input layer to the hidden neuron and its bias term)condense on isolated orientations.
The condensation dynamics implies that thetraining implicitly regularizes a NN towards one with a much smaller effective size.
In this work, we illustrate the formation of the condensation in multi-layer fullyconnected NNs and show that the maximal number of condensed orientations inthe initial training stage is twice the multiplicity of the activation function, where“multiplicity” indicates the multiple roots of activation function at origin.
Ourtheoretical analysis conﬁrms experiments for two cases, one is for the activationfunction of multiplicity one with arbitrary dimension input, which contains manycommon activation functions, and the other is for the layer with one-dimensionalinput and arbitrary multiplicity.
This work makes a step towards understandinghow small initialization leads NNs to condensation at the initial training stage.
Introduction
The question why over-parameterized neural networks (NNs) often show good generalization attractsmuch attention (Breiman, 1995; Zhang et al., 2021).
Luo et al.
(2021) found that when initializationis small the input weights of hidden neurons in two-layer ReLU NNs (the input weight or the featureof a hidden neuron consists of the weight from its input layer to the hidden neuron and its bias term)condense on isolated orientations during the training.
As illustrated in the cartoon example in Fig.
1,the condensation transforms a large network with one of only a few effective neurons, leading to anoutput function with low complexity.
Since the complexity bounds the generalization error (Bartlettand Mendelson, 2002), the study of condensation could provide insight to how over-parameterized NNs are implicitly regularized to achieve good generalization performance in practice.
Small initialization leads NNs to rich non-linearity during the training (Mei et al., 2019; Rotskoff and
Vanden-Eijnden, 2018; Chizat and Bach, 2018; Sirignano and Spiliopoulos, 2020).
For example, inover-parameterized regime, small initialization can achieve low generalization error (Advani et al.,2020).
Irrespective of network width, small initialization can make two-layer ReLU NNs convergeto a solution with maximum margin (Phuong and Lampert, 2020).
Small initialization also enablesneural networks to learn features actively (Lyu et al., 2021;
Luo et al., 2021).
Condensation isan important phenomenon that reﬂects the feature learning process.
Therefore, it is important tounderstand how condensation emerges during the training with small initialization.
The dynamic behavior of the training at the initial state is important for the whole training process,because it largely determines the training dynamics of a neural network and the region it ends upin (Fort et al., 2020;
Hu et al., 2020), which impacts the characteristics of the neural networks inthe ﬁnal stage of training (Luo et al., 2021; Jiang et al., 2019;
Li et al., 2018).
For two-layer ReLU
NNs, several works have studied the mechanism underlying the condensation at the initial trainingstage when the initialization of parameters goes to zero (Maennel et al., 2018; Pellegrini and Biroli,2020).
However, it still remains unclear that for NNs of more general activation functions, howthe condensation emerges at the initial training stage.
In this work, we show that the condensation at the initial stage is closely related to the multiplicityp at x = 0 , which means the derivative of activation at x = 0 is zero up to the ( p − 1)
th -orderand is non-zero for the p -th order.
Many common activation functions, e.g., tanh( x ) , sigmoid( x ) ,softplus( x ) , Gelu( x ) , Swich( x ) , etc, are all multiplicity p = 1 , and x tanh( x ) and x 2 tanh( x ) havemultiplicity two and three, respectively.
Our contribution is summarized as follows:
• Our extensive experiments suggest that the maximal number of condensed orientations in the initial training is twice the multiplicity of the activation function used in general NNs.
• We present a theory for the initial condensation with small initialization for two cases,one is for the activation function of multiplicity one with arbitrary dimension input, andthe other is for the layer with one-dimensional input and arbitrary multiplicity.
As many common activation functions are multiplicity p = 1 , our theory would be of interest to general readers.
Related works
Luo et al.
(2021) systematically study the effect of initialization for two-layer ReLU NN with inﬁnitewidth by establishing a phase diagram, which shows three distinct regimes, i.e., linear regime (similarto the lazy regime)
(Jacot et al., 2018; Arora et al., 2019; Zhang et al., 2020; E et al., 2020; Chizatand Bach, 2019), critical regime (Mei et al., 2019; Rotskoff and Vanden-Eijnden, 2018; Chizat and
Bach, 2018; Sirignano and Spiliopoulos, 2020) and condensed regime (non-linear regime), based onthe relative change of input weights as the width approaches inﬁnity, which tends to 0 , O (1) and + ∞ ,respectively.
As shown in Luo et al.
(2021), two-layer ReLU NNs with inﬁnite width do not condensein the neural tangent kernel (NTK) regime , slightly condense in the mean-ﬁeld regime, and clearlycondense in the non-linear regime.
However, in Luo et al.
(2021), it is not clear how general thecondensation phenomenon is when other activation functions are used and why there is condensation.
Zhang et al.
(2021a,b) propose a general Embedding Principle of loss landscape of DNNs thatunravels a hierarchical structure of the loss landscape of NNs, i.e., loss landscape of an DNN containsall critical points of all the narrower DNNs.
The embedding principle shows that a large DNN canexperience critical points where the DNN condenses and its output is the same as that of a much smaller DNN.
However, embedding principle does not explain how the training can take the DNN tosuch critical points.
The condensation is consistent with previous works that suggest that NNs may learn data from simpleto complex patterns (Arpit et al., 2017; Xu et al., 2019; Rahaman et al., 2019; Xu et al., 2020; Jin et al.
,2020; Kalimeris et al., 2019).
For example, an implicit bias of frequency principle is widely observedthat NNs often learn the target function from low to high frequency (Xu et al., 2019; Rahaman et al.
Xu et al., 2020), which has been utilized to understand various phenomena (Ma et al., 2020;
Xu and Zhou, 2021) and inspired algorithm design (Liu et al., 2020; Cai et al., 2020; Tancik et al.
,2020; Li et al., 2020, 2021).
Preliminary: Neural networks and initial stage
A two-layer NN is
[Equation]
where σ ( · ) is the activation function, w j = ( ¯ w j , b j ) ∈ R d
+1 is the neuron feature including theinput weight and bias terms, and
x = (¯ x , 1) ∈ R
d +1 is combination of the input sample and scalar1 , θ is the set of all parameters, i.e., { a j , w j } mj =1 .
For simplicity, we call w j as input weight orweight and x as input sample.
A L -layer NN can be recursively deﬁned by feeding the output of the previous layer as the input tothe current hidden layer, i.e.,x [0] =
( x , 1) , x [1] = ( σ ( W [1] · x [0] ) , 1) , x [ l ] = ( σ ( W [ l ] · x [ l − 1] ) , 1) , for l ∈ { 2 , 3 , ..., L } f ( θ
[Equation]
where W [ l ] =
( ¯ W [ l ] , b
[ l ] ) ∈ R m l × ( m l − 1 +1) , and m l represents the dimension of the l -th hiddenlayer.
For simplicity, we also call each row of W [ l ] as input weight or weight and x [ l − 1] as inputto the l -th hidden layer.
The target function is denoted as f ∗ ( x ) .
The training loss function is meansquared error
[Equation]
Without loss of generality, we assume that the output is one-dimensional for theoretical analysis,because, for high-dimensional cases, we only need to sum up the components directly.
For summation,it does not affect the results of our theories.
We consider the gradient ﬂow training
[Equation]
To ensure that the training is close to the gradient ﬂow, all the learning rates used in this paper arerelatively small.
We characterize the activation function by the following deﬁnition.
Deﬁnition 1 (multiplicity p ) .
Suppose that σ ( x ) satisﬁes the following condition, there exists a p ∈
Nand p ≥ 1 , such that the s -th order derivative σ ( s ) (0)
=
for s = 1 , 2 , · · · , p − 1 , and σ ( p ) (0) (cid:54) = 0 ,then we say σ has multiplicity p .
Remark 3.1 .
Here are some examples.
tanh( x ) , sigmoid( x ) and softplus( x ) have multiplicity p = 1 .x
tanh( x ) has multiplicity p = 2 .
We illustrate small initialization and initial stage as follows
Small initialization : W [ l ] ∼ o (1) and W [ l ] · x [ l − 1] ∼ o (1) for all l ’s and x [ l − 1] .
We want to remark that it dose not make sense to deﬁne the initial stage by the number of trainingsteps, because the training is affected by many factors, such as the learning rate.
Therefore, in ourexperiments, the epochs we use to show phenomena can across a wide range.
Alternatively, weconsider the initial stage as follows.
Initial stage : the period when the leading-order Taylor expansion w.r.t activated neurons is still validfor theoretical analysis of Sec. 5.
As the parameters of a neural network evolve, the loss value will also decay accordingly, which iseasy to be directly observed.
Therefore, we propose an intuitive deﬁnition of the initial stage oftraining by the size of loss in this article, that is the stage before the value of loss function decays to70 % of its initial value.
Such a deﬁnition is reasonable, for generally a loss could decay to 1 % of itsinitial value or even lower.
The loss of the all experiments in the article can be found in Appendix
A.3, and they do meet the deﬁnition of the initial stage here.
Cosine similarity: The cosine similarity for two vectors u and v is deﬁned as
[Equation]
Initial condensation of input weights
In this section, we would empirically show how the condensation differs among NNs with activationfunction of different multiplicities in the order of a practical example, multidimensional syntheticdata, and 1-d input synthetic data, followed by theoretical analysis in the next section.
4.1 Experimental setup
For Synthetic dataset: Throughout this work, we use fully-connected neural network with size,d - m - · · · - m - d out .
The input dimension d is determined by the training data.
The output dimensionis d out = 1 .
The number of hidden neurons m is speciﬁed in each experiment.
All parameters areinitialized by a Gaussian distribution N (0 , var ) .
The total data size is n .
The training method is
Adam with full batch, learning rate lr and MSE loss.
We sample the training data uniformly froma sub-domain of R d .
The sampling range and the target function are chosen randomly to show thegenerality of our experiments.
For CIFAR10 dataset: We use Resnet18-like neural network, which has been described in Fig.
2,and the input dimension is d = 32 ∗ 32 ∗ 3 .
The output dimension is d out = 10 .
All parameters areinitialized by a Gaussian distribution N (0 , var ) .
The total data size is n .
The training method is
Adam with batch size 128, learning rate lr and cross-entropy loss.
4.2 A practical example
The condensation of the weights of between the fully-connected (FC) layers of a Resnet18-likeneural network on CIFAR10 is shown in Fig.
2, whose activation functions for FC layers aretanh( x ) , sigmoid( x ) , softplus( x ) and x tanh( x ) , indicated by the corresponding sub-captions,respectively.
As shown in Fig.
2(a), for activation function tanh( x ) , the color indicates cosinesimilarity D ( u, v ) of two hidden neurons’ weights, whose indexes are indicated by the abscissa andthe ordinate, respectively.
If the neurons are in the same beige block, D ( u, v ) ∼ 1 (navy-blue block,
D ( u, v ) ∼ − 1 ), their input weights have the same (opposite) direction.
Input weights of hiddenneurons in Fig.
2(a) condense at two opposite directions, i.e., one line.
Similarly, weights of hiddenneurons for NNs with sigmoid( x ) and softplus( x ) (Fig.
2(b, c)), which are frequently used and havemultiplicity one, condense at one direction.
As the multiplicity increases, NNs with x tanh x (Fig.2(d)) condense at two different lines.
These experiments suggest that the condensation is closelyrelated to the multiplicity of the activation function.
In these experiments, we ﬁnd that the performance of the Resnet18-like network with tanh FC layerswith small initialization is similar to the one with common initialization in Appendix A.4.
4.3 Multidimensional synthetic data
For convenience of experiments, we then use synthetic data to perform extensive experiments tostudy the relation between the condensation and the multiplicity of the activation function.
We use two-layer fully-connected NNs with size 5 - 50 - 1 to ﬁt n = 80 training data sampled from a5-dimensional function (cid:80) 5 k =1 3 .
5 sin(5 x k + 1) , where x = ( x 1 , x 2 , · · · , x 5 ) (cid:124) ∈ R 5 and each x kis uniformly sampled from [ − 4 , 2] .
As shown in Fig. 3(a), for activation function tanh( x ) , inputweights of hidden neurons condense at two opposite directions, i.e., one line.
As the multiplicityincreases, NNs with x tanh( x ) (Fig. 3(b)) and x 2 tanh x (Fig. 3(c)) condense at two and three
[Figure]
[Figure]
If neurons are in the same beige block, D ( u, v ) ∼ 1 ( -blue block, D ( u, v ) ∼ − 1 ), their input weights have the same (opposite) direction.
The activation functions are indicated by the sub-captions.
The training data is 80 points sampled from (cid:80) 5 k =1 3 .
5 sin(5 x k + 1) , where each k is uniformly sampled different lines, respectively.
For activation function sigmoid( x ) in Fig. 3(d) and softplus( x ) in Fig.3(e), NNs also condense at two opposite directions.
For multi-layer NNs with different activation functions, we show that the condensation for allhidden layers is similar to the two-layer NNs.
In deep networks, residual connection is oftenintroduced to overcome the vanishing of gradient.
To show the generality of condensation, weperform an experiment of six-layer NNs with residual connections.
To show the difference of variousactivation functions, we set the activation functions for hidden layer 1 to hidden layer 5 as x 2 tanh( x ) ,x tanh( x ) , sigmoid( x ) , tanh( x ) and softplus( x ) , respectively.
The structure of the residual ish l +1 ( x ) =
σ ( W l h l ( x ) + b l ) + h l ( x ) , where h l ( x ) is the output of the l -th layer.
As shown in Fig.4, input weights condense at three, two, one, one and one lines for hidden layer 1 to hidden layer 5,respectively.
Note that residual connections are not necessary.
We show an experiment of the samestructure as in Fig.
4 but without residual connections in Appendix A.5.
Through these experiments, we conjecture that the maximal number of condensed orientations atinitial training is twice the multiplicity of the activation function used.
To understand the mechanismof the initial condensation, we turn to experiments of 1-d input and two-layer NNs, which can beclearly visualized in the next subsection.
4.4 1-d input and two-layer NN
For 1-d data, we visualize the evolution of the two-layer NN output and each weight, which conﬁrmsthe connection between the condensation and the multiplicity of the activation function.
[Figure]
[Figure]
We display the outputs at initial training in Fig.
Due to the small magnitude of parameters, anactivation function with multiplicity p can be well approximated by a p -th order polynominal, thus,the NN output can also be approximated by a p -th order polynominal.
As shown in Fig.
5, the NNoutputs with activation function tanh( x ) , x tanh( x ) and x 2 tanh( x ) overlap well with the auxiliaryof a linear, a quadratic and a cubic polynominal curve, respectively in the beginning.
This experiment,although simple, but convincingly shows that NN does not always learn a linear function at the initialtraining stage and the complexity of such learning depends on the activation function.
[Figure]
All settings are the same as Fig.
Around the original point, the ﬁeld has one, two, three stables lines, on which an input weight would keep its direction, for tanh( x ) , x tanh( x ) , and x 2 tanh( x ) , respectively.
We also display the value of each weight by the green dots and the corresponding directions by the orange arrows.
We visualize the direction ﬁeld for input weight w j := ( w j , b j ) , following the gradient ﬂow,
[Equation]
where e i := f θ ( x i ) − f ∗ ( x i ) .
Since we only care about the direction of w j and a j is a scalar at each epoch, we can visualize ˙ w j by ˙
w j /a j .
For simplicity, we do not distinguish ˙ w j /a j and ˙ w j if there is no ambiguity.
When we compute ˙ w j for different j ’s, e i x i for ( i = 1 , · · · , n ) is independent with j .
Then, at each epoch, for a set of { e i , x i } ni =1 , we can consider the following direction ﬁeld
[Equation]
When ω is set as w j , we can obtain ˙ w j .
As shown in Fig.
6, around the original point, the ﬁeld hasone, two, three stables lines, on which a neuron would keep its direction, for tanh( x ) , x tanh( x ) ,and x 2 tanh( x ) , respectively.
We also display the input weight of each neuron on the ﬁeld bythe green dots and their corresponding velocity directions by the orange arrows.
Similarly to thehigh-dimensional cases, NNs with multiplicity p activation functions condense at p different linesfor p = 1 , 2 , 3 .
Therefore, It is reasonable to conjecture that the maximal number of condensedorientations is twice the multiplicity of the activation function used.
Taken together, we have empirically shown that the multiplicity of the activation function is akey factor that determines the complexity of the initial output and condensation.
To facilitate theunderstanding of the evolution of condensation in the initial stage, we show several steps during theinitial stage of each example in Appendix A.6.
Analysis of the initial condensation of input weights
In this section, we would present a preliminary analysis to understand how the multiplicity of theactivation function affects the initial condensation.
At each training step, we consider the velocityﬁeld of weights in each hidden layer of a neural networks.
Considering a network with L hidden layers, we use row vector W [ k ] j to represent the weight fromthe ( k - 1) -th layer to the j -th neuron in the k -th layer.
Since condensation is always accompanywith small initialization, together with the initial stage deﬁned in Sec. 3, we make the followingassumptions,
Assumption 1 .
Small initialization infers that W [ k ] j · x [ k − 1] ∼ o (1) applies for all k ’s and j ’s.
Assumption 2 .
During the initial stage of condensation, Taylor expansion of each correspondingactivated neurons holds, i.e.,
[Equation]
Suppose the activation function has multiplicity p , i.e., σ ( s ) (0)
=
for s = 1 , 2 , · · · , p − 1 , andσ ( p ) (0) (cid:54) = 0 .
Then, together with Assumption 2, we have
[Equation]
For each k and j , W [ k ] j satisﬁes the following dynamics, (see Appendix A.2)
[Equation]
where w can represent W [ k ] j (cid:124) for all k ’s and j ’s, r = (cid:107) w (cid:107) 2 is the amplitude, and u = w /r .
For convenience, we deﬁne an operator P satisfying P w := ˙
w − u ( ˙ w · u ) .
To specify the condensationfor theoretical analysis, we make the following deﬁnition,
Condensation : the weight evolves towards a direction which will not change in the direction ﬁeldand is deﬁned as follows,
[Equation]
Since ˙ w · u is a scalar, ˙ w is parallel with u .
u is a unit vector, therefore, we have u = ± ˙ w / (cid:107) ˙ w (cid:107) 2 .
Remark 1 (An intuitive explanation for condensation) .
In this work, we consider NNs with sufﬁcientlysmall parameters.
Suppose r = (cid:107) w (cid:107) 2 ∼ O
(cid:15) ) , where (cid:15) is a small quantity, then dynamics (8) willshow that O ( ˙ r ) ∼ O ( ˙ w ) and O ( ˙ u ) ∼ O ( ˙ r ) /O
( (cid:15) ) .
Here O ( ˙ r ) ∼ O ( ˙ w ) refers that the evolution of ˙ ris at the same order as every component of ˙ w , and it is the same for O ( ˙ u ) ∼ O ( ˙ r ) /O
( (cid:15) ) .
Therefore,the orientation u would move much more quickly than the amplitude r .
By the dynamics for w (Equ.10) and the Taylor approximation with multiplicity p (Equ. 7), it is easy to ﬁnd that the solutionsfor Equ. 9 are ﬁnite.
Then, taken together, the orientation u would converge rapidly into certaindirections, leading to condensation.
In the following, we study the case of (i) p = 1 and (ii) m k − 1 = 1 (the dimension of input of thek -th layer equals one), and reach the following theorem,
Theorem 5.1.
Under Assumption 1 and 2, suppose we only consider the leading-order Taylorexpansion of Equ. 9, then the maximal number of roots for Equ. 9 is twice the multiplicity of theactivation function used as initialization towards zero for two cases: (i) p = 1 and (ii) m k − 1 = 1 .
Proof.
Case 1: p =
By gradient ﬂow, we can obtain the dynamics for W [ k ] j (see Appendix A.2),
[Equation]
where we use E l = W [ l ] (cid:124) diag { σ (cid:48)
( W [ l ] · x [ l − 1] ) } , for l ∈ { 2 , 3 , ..., L } , E [ q : p ] =
E q E q +1 ...
E p ,and x [ k ] i represents the neurons of the k -th layer generated by the i -th sample.
For a ﬁxed step, we only consider the gradient of loss w.r.t.
W [ k ] j .
According to our assumptionp = 1 , we have σ (cid:48) (0) (cid:54) = 0 .
Suppose that parameters are small and denote e i :=
( f ( θ , x i ) − y i ) .
By
Taylor expansion,
[Equation]
where operator Q is the leading-order approximation of operator P , and here E [ k +1:
L ] is independent with i , because diag { σ (cid:48)
[Equation]
Since diag { σ (cid:48) ( 0 ) } =
c I , c (cid:54) = 0 by assumption, and, WLOG, we assume a (cid:54) =
0 , then
[Equation]
We have
[Equation]
This calculation shows that for layer k , the input weights for any hidden neuron j have the sametwo stable directions.
Together with the analysis before, i.e., when parameters are sufﬁciently small,the orientation u would move much more quickly than the amplitude r , all input weights wouldmove towards the same direction or the opposite direction, i.e., condensation on a line, under smallinitialization.
Case 2: the k -th layer with one-dimensional input, i.e., m k − 1 =
By the deﬁnition of the multiplicity p , we have
[Equation]
where ( · ) p − 1 and σ ( p ) ( · ) operate on component here.
Then up to the leading order in terms of themagnitude of θ , we have (see Appendix A.2)
[Equation]
WLOG, we also assume a (cid:54) = 0 .
And by deﬁnition, w = r u , we have
[Equation]
Since m k − 1 + 1 = 2 (for x [ k − 1] = ( σ ( W [ k − 1] x [ k − 2] ) , 1) and m k − 1 = 1 ), we denote u
= ( u 1 , u 2 ) (cid:124) ∈ R 2 and x [ k − 1] i = (( x [ k − 1] i ) 1 , ( x [ k − 1] i
[Equation] [Equation] (cid:124) ∈ [Equation]
2 , then,
[Equation]
We obtain the equation for ˆ u , n (cid:88)
[Equation]
Since it is an univariate p -th order equation, ˆ u =
u 1 u 2 has at most p complex roots.
Because u is a unitvector, u at most has p pairs of values, in which each pair are opposite.
Taken together, our theoretical analysis is consistent with our experiments, that is, the maximalnumber of condensed orientations is twice the multiplicity of the activation function used whenparameters are small.
As many commonly used activation functions are either multiplicity p = 1 or
ReLU-like, our theoretical analysis is widely applied and sheds light on practical training.
Discussion
In this work, we have shown that the characteristic of the activation function, i.e., multiplicity, isa key factor to understanding the complexity of NN output and the weight condensation at initialtraining.
The condensation restricts the NN to be effectively low-capacity at the initial training stage,even for ﬁnite-width NNs.
During the training, the NN increases its capacity to better ﬁt the data,leading to a potential explanation for their good generalization in practical problems.
This work alsoserves as a starting point for further studying the condensation for multiple-layer neural networksthroughout the training process.
For general multiplicity with high-dimensional input data, the theoretical analysis for the initialcondensation is a very difﬁcult problem, which is equivalent to counting the number of the roots ofa high-order high-dimensional polynomial with a special structure originated from NNs.
Trainingdata can also affect the condensation but not the maximal number of condensed orientations.
Whendata is simple, such as low frequency, the number of the condensed orientations can be less, someexperiments of MNIST and CIFAR100 can be found in Appendix A.7.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes] In Sec. 1 and Abstract.(b)
Did you describe the limitations of your work?
[N/A]
(c) Did you discuss any potential negative societal impacts of your work?
[No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] In Sec. 3 and5.(b) Did you include complete proofs of all theoretical results?
[Yes] In Sec. 5 andAppendix A.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] In thesupplemental material. (b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they were chosen)?
[Yes] In Sec. 4 and the captions of each ﬁgure.
(c) Did you report error bars (e
g., with respect to the random seed after running experi- ments multiple times)?
[No] (d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
[No]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]