Stochastic Adaptive Activation Function
Anonymous Author(s)
AffiliationAddress email
Abstract
Human neurons and neurotransmission mechanisms have been realized in a deepneural network by the theoretical implementations of activation functions.
However, recent studies have reported that the threshold potential of neurons exhibiteddifferent values by locations and types of individual neurons, and the activationfunctions have limitations in representing it.
Therefore, this study proposes a simple yet effective activation function that exhibits different thresholds and adaptiveactivations according to the positions of units and the contexts of inputs, and wedenoted it as ASH activation function.
ASH highlights informative features, whichexhibit large values in the top percentiles in an input, whereas it rectifies low values.
Most importantly, ASH exhibits trainable, adaptive, and context-aware propertiescompared to other activation functions.
Furthermore, ASH represents generalformula of the previously studied activation function and provides a reasonablemathematical background for the superior performances.
To validate the effectiveness and robustness of ASH, we implemented ASH into many deep learningmodels for various tasks, including classification, detection, segmentation, andimage generation.
The experimental analysis demonstrates that our activationfunction exhibits outstanding performance in any deep learning applications.
Introduction
Searching for the optimal activation function has been challenging in artificial intelligence (Maaset al., 2013; Ramachandran et al., 2017; Clevert et al., 2015).
Early activation functions have beenstudied to compensate for the non-linearity of the artificial neural networks or ameliorate the gradientvanishing problem (Hertz et al., 1997; Hochreiter, 1998).
Recently, novel activation functions havebeen suggested for the zero-centered or parametric properties that improve the training efficiencyof deep neural networks (Maas et al., 2013; Clevert et al., 2015).
Currently, the activation functionsregarding the stability of deep neural networks or probabilistic distribution of inputs have beenproposed (Hendrycks, Gimpel, 2016; Misra, 2019).
The advances in the activation functions haveopened the era of deep neural networks in the creative tasks of detecting/segmenting target objectsin sophisticated images or generating images beyond the simple classifiers (Simonyan, Zisserman,2014; Zhao et al., 2019; Badrinarayanan et al., 2017; Goodfellow et al., 2014).
The development of activation functions has evolved to behave more like a human neuron (Sharmaet al., 2017; Lee et al., 2017).
However, Izhikevich (2003); Evans et al.
(2018) reported that theneurotransmission mechanism, including a membrane, action, and threshold potential of humanneurons, is subject to the location or the connection type of the neurons.
In addition, humans perceiveobjects with surrounding contexts using N : N mapping of visions to human neurons rather than 1:1mapping of a pixel to an input node in neural networks (Liu et al., 2018).
The connections betweenneurons can be realized by the linear combinations of layers in a deep neural network, but deep neural networks have limitations in realizing contextual perception.
It implies that there is roomfor improvement in deep neural networks and convolutional neural networks despite the superiorperformance in analyzing images (Jinsakul et al., 2019; Misra, 2019).
Therefore, the development ofa deep neural network is leaned to mimic human perception by realizing the mechanism of humanneurons (Aggarwal, others, 2018; Lindsay, 2021).
Currently, the primary issue is that activation functions exhibit passivity, in this paper, indicating thatthe activation function determines the output only concerning the value of one element rather thanthe entire contexts.
For instance, the Rectified Linear Unit (ReLU), defined as f ( x ) =
max ( x, 0) ,determines the output values related to x (Fukushima, Miyake, 1982), whereas the softmax functiongenerates output values as the ratio of the input value to the totals (Goodfellow et al., 2017); ReLUexhibits passivity, whereas softmax does not.
Suppose an image can be classified by considering80% of the total portion.
The current activation functions exhibit the limitation in classifyingthis image since only elements (pixels) of the image are used to rectify the image rather thanthe ratio.
Additionally, another issue is that the activation function is invariant.
Even though theparametric activation function trains its parameters during training, the values of the parametricactivation functions are invariant in the inference phase (Xu et al., 2015; Bingham, Miikkulainen,2022).
Therefore, the limited rectification could be realized by the invariant parameters or thresholdsregardless of new inputs from different domains (e.g., test set).
Contribution To realize the mechanism of human neurons that rectify inputs considering theircontexts, we propose ASH activation function.
The main contributions of this study are to suggest asimple yet effective activation function, ASH, and to implement ASH in a mathematically effectiveform.
Going beyond the passive activation functions, ASH activation function is designed as (1) anactive activation function that provides outputs regarding the context of inputs and (2) a conditionalactivation function that employs an adaptive threshold.
Unlike ReLU or Leaky ReLU, the thresholdvalue of
ASH is adaptively changing
considering the contextual information
[Equation]
In particular, the threshold value ( θ ) is adaptively changing according to the input distribution withoutheavy calculation, and thus ASH provides outputs considering the contexts of inputs adaptively.
Byapplying ASH, we obtained the following theoretical and experimental results:
• We conducted mathematical modeling for ASH in an effective form to be trainable and parametric, and thus ASH exhibits parametric and adaptive properties.
The baseline threshold ofASH is trainable during the training phase, and the threshold value is adaptively changingaccording to the contexts of inputs without heavy calculation.
• We theoretically verified ASH adaptively changes its threshold alongside the stochastic
distribution of inputs.
This implies ASH provides outputs regarding the entire contexts of inputs, as well as leads to better feature extraction.
• We theoretically verified that ASH exhibited general formula for Swish activation functionand provided the mathematical explanations for the superior performance of Swish, which
was empirically searched in the previous works.• We experimentally showed that ASH improves the performance of deep learning models forvarious tasks as well as shortens training time.
Related works Activation functions affect the performance of the training process to determinefunctional subspace in a deep neural network (DNN) (Hayou et al., 2019).
In particular, the nonlinearity using activation functions has been introduced to avoid the linear transform causing simplefeature extractions in the DNN (Misra, 2019; Jarrett et al., 2009).
Therefore, the DNN with nonlinearity has realized any complex tasks in the current (Leshno et al., 1993).
In the early era, Rectified
Linear Unit (ReLU) replaced the classical activation functions such as sigmoid and tanh (Nair,
Hinton, 2010), due to its simple and computational efficiency compared to other activation functions.
During decades, many activation functions have been proposed such as Leaky ReLU (Maas et al.
,2013), Exponential Linear Unit (ELU) (Clevert et al., 2015), Gaussian Error Linear Unit (GELU)(Hendrycks, Gimpel, 2016), Scaled Exponential Linear Unit (SELU) (Klambauer et al., 2017), and
Swish (Ramachandran et al., 2017) for better performance and stability of learning parameters in
DNNS.
Those activation functions have solved the dying ReLU problem, which exhibits zero valuein the negative region and improved DNNs more smoothly for stable optimization.
In particular,
ELU and SELU have realized internal normalization in the layer using the zero-mean property.
Furthermore, GELU exploited a Gaussian error that could implement adaptive dropout to apply ahigher probabilistic intuition.
Problem statement For adaptive thresholding, ASH exploits stochastic selection methodology likequick selection (Hoare, 1961).
In particular, for better feature extraction, the informative elements,which exhibit large values, should be identified like an attention mechanism.
In contrast, someelements, which exhibit low relevance, are required to be rectified.
To this end, ASH is designed toidentify informative elements but rectify others as 0.
Let X ∈ R
H
× W × C be a tensor (here, feature-map) disregarding batch, but with height ( H ), width( W ), and channel ( C ), and let ¯ X ∋
X be a set of feature-maps.
Let A be an activation function suchthat A : ¯ X → ¯ X .
Then, we can extract i th element from X , and denote it as x ( i ) .
Here, the goal ofthis study is to design an activation function represented as follows:
[Equation]
Note that, a novel activation function A is represented similar to ReLU, whereas the threshold isnot invariant compared to ReLU, but is subjected to the distribution of input X .
To simplify, let
C ( x ( i ) , k ; X ) be a condition whether x ( i ) is ranked in top k % percentile of X , and the negation of
C is denoted as ¬ C .
In particular, the elements that satisfy C ( x ( i ) , k ; X ) are from the first largestelement to the ( 0 . 01 kN )-th largest element in X , where N indicates the number of elements in X .
Here, the elements can be extracted using a simple algorithm like a quick selection.
Suppose a setˆ X that includes all elements in X such that
ˆ X
=
{ x (1) , x (2) , ..., x ( N ) } .
Then, we can constructsubsets of ˆ X as ˆ X C =
{ x ( i ) ∈
X | C ( x ( i ) , k ; X ) } and ( ˆ X C ) c
= { x ( i ) ∈
X |¬ C
( x ( i ) , k ; X ) } .
Then,
Equation (2) can be simplified as follows:
[Equation]
To summarize, the activation function is designed to sample the top- k percentile from input, wherethe criteria is the value (less or greater) of the element.
The sampling examples are illustrated in Fig.1 compared to ReLU.
[Figure]
Method
This study aims at designing an activation function for a stochastic sampling of the top- k percentileelements from inputs.
However, sorting or sampling methods like quick selection requires highcomputational cost.
In contrast, sampling the top- k percentile could be realized by a z-score-basedmethod in a simple calculation despite the prerequisites of normal distribution (also known as
Gaussian distribution).
Since we reviewed that the outputs of neurons are normally distributed, weemploy the stochastic sampling to design ASH activation function.
In this section, we (1) reviewed that the outputs of neurons are distributed in a normal distribution, (2)conducted modeling for stochastic sampling using a z-score, (3) formulated ASH activation function,(4) verified ASH is parametric and trainable, and (5) searched general application of ASH activationfunction.
2.1 Background
[Figure] Gaussian distribution Currently, many deep learning models employ a normalization method dueto its stability in training deep learning models (Ioffe, Szegedy, 2015; Ulyanov et al., 2016; Wu, He,2018).
In the previous study, Ioffe, Szegedy (2015) reported that the output of the convolutional layer,x = Wu + b , is more likely to have a symmetric, non-sparse distribution, that is "more Gaussian".
Since most deep learning models are based on convolutional operations, the outputs of neurons aresupposed to be normally distributed (Gaussian distribution).
Therefore, it is concluded that the inputsof the activation functions are normally distributed when activation functions follow convolutionallayers, such that x ∼ N ( µ x , σ 2 x ) , where x is an input feature-map of an activation function, µ x andσ x are mean and standard deviation of x , respectively.
Therefore, we obtained the following.
Proposition 1.
The outputs of neurons in convolutional neural networks are normally distributed.
Sampling from normal distribution This study aims at sampling the top- k % elements, whichexhibit greater values, from an input feature-map.
Since the area under normal distribution indicatesthe percentile, sampling the top- k % percentile from a normal distribution is theoretically identical tothe statement calculating the area under curve shown in Fig.
Let F be a tensor, a multi-dimensionalarray or a matrix.
Then, F is supposed to be normally distributed, and thus we could sample theelements ( f ( i ) ) in the top- k % percentile from F using the following equation:
[Equation]
However, heavy computational costs are required to find the trivial solution from the probabilitydensity function of normal distribution, defined as
[Equation]
Therefore, we leaned toprobability theory to simplify the computation rather than calculus .
To this end, we employed thestandard normal distribution (Z-score normalization) for Equation (4), and we obtained the following:
[Equation]
where z k =
( z ′ k − µ F ) /σ F , which is the Z-normalized value from z ′ k , and thus z k is subjected to k ,indicating percentile to sample, in terms of Z-Table.
Then, we go Z-table and easily find the propervalue for z k , intuitively.
Therefore, the condition, Z ( i ) =
( f ( i ) − µ F ) /σ
F ≥ z k ⇔ f ( i ) ≥ µ F + z k σ F ,is mathematically identical to sample the top- k % elements from F .
To summarize, we obtained thefollowing.
Proposition 2.
Element x ( i ) in X , which is normally distributed, is in the top- k % percentile of X ifx ≥ µ X
+ z k σ X , where z k is a z-value subjected to k in Z-Table.
Differentiation
The mechanism of convolutional neural network (CNN) has been studied in manyprevious works (Rumelhart et al., 1986; Bottou, 2010; Zhang, 2016;
Hu et al., 2018).
Training CNNmodels is subjected to the backpropagation derived from the partial derivatives of loss function bythe individual convolutional parameter.
Let L be a loss function for deep learning model M and let
W be one of the variables in M .
Then, the derivative of L in terms of W is represented as ∂L∂W , and
W is updated as W ← W − η ∂L∂W with learning rate η .
On the other hand, suppose variable θ be thethreshold and the function f ( x ) is αx if x ≥ θ , otherwise 0, such that the partial derivative of f isrepresented as following:
[Equation]
Here, α is arithmetically combined with f ( x ) , whereas θ does not.
Therefore, α is trainable, and θ isnot trainable in this context.
In basic calculus , it is trivial that if the loss function is not dependenton the variable, the partial derivative is zero, and thus the variable cannot be trained or optimized;invariant.
Therefore, we obtain the following:
Lemma 1.
The derivative of a variable in the conditional statement is zero, and thus that variablecannot be optimized.
2.2 ASH Activation Function
Let X be an input of ASH activation function ( Æ ) and be a tensor of which elements are normallydistributed.
Besides, let x ( i ) ∈
X be the i -th element of X .
Then, by Proposition 2 , ASH activationfunction, which samples the top- k % elements from input, is represented as following:
[Equation]
where µ X and σ X are the mean and the standard deviation of all elements in X , and z k is the z-scoreconcerning percentile ( k ) to sample (i.g. z = 1 .
if k = 2 .
5 %, See Z-Table).
Equation (7) exhibitsthat ASH activation function is represented in a simple yet effective form with low computationalcosts.
Intuitively, we assumed that the activation level (percentile) is supposed to be different by each neuronand the tasks of the deep learning model like human neurons.
However, in Equation (7), the condition( x ( i ) ≥ µ X + z k σ X ) is invariant by Lemma 1 .
Note that, µ X + z k σ X is variable and changeableconcerning X , but the sampled portion ( k % percentile related to z k ) from X is invariant, and thus
ASH in Equation (7) is not parametric and trainable.
To make ASH be trainable and parametric, let
Equation (7) be substituted using a proxy function as Æ ( x ( i ) )
=
x ( i ) f
( x ( i ) ) , such that the proxyfunction f ( x ( i ) ) is represented as the following:
[Equation]
To simplify, suppose Heaviside step function (Weisstein, 2002), defined as H ( x ) =
ddx max (0 , x ) .
Then, f ( x ( i ) ) =
H ( x ( i ) − µ X
−
z k σ X ) .
Therefore, we obtain the arithmetical form to formulate
ASH activation function as the following:
[Equation]
Despite the arithmetic formula, ASH activation function in Equation (9) is still independent of z k ,and thus the z k is still not trainable.
However, it is well known that the Heaviside step function isanalytically approximated as 2 H ( x )
= 1 + 1 tanh( αx ) with a large value of α (Iliev et al., 2017),and thus ASH activation function is approximated using the smooth Heaviside step function as the following:
[Equation]
Since z k is arithmetically placed, z k indicating sampling percentiles is trainable, and thus ASHactivation function is also trainable and parametric.
By optimizing z k , ASH activation functionsexhibit different thresholds for sampling percentiles.
Therefore, it is concluded that ASH exhibitsdifferent activation levels by a stochastic sampling of inputs and different thresholds like humanneurons, synapses, and their potential.
Like human neurons, the mechanism of ASH is the following:(1)
In the training phase, each ASH activation function optimizes its z k and fine-tunes the thresholdfor the percentile sampling of inputs.
Thus, it implies that ASH activation function realizes thearbitrary threshold potentials in human neurons (Clevert et al., 2015; Evans et al., 2018).
Someexamples of z k related to Equation (7) are:
Example 1.
Small value of z k , even negative, implies the dense activation, and the dying ReLUproblem could be improved.
Example 2.
Large value of z k implies the sparse activation, and the sparsity could be employed.
Furthermore, (2) in the training or inference phase, ASH activation function rectifies the inputsusing the learnt threshold value and contexts of inputs.
In particular, to sample the top- k % percentile,
ASH employs the mean and the standard deviation of input, representing the contexts of the entireinput.
Therefore, it implies that ASH realizes the adaptive activation considering the contexts ofinputs.
Some examples of the adaptive activation related to Equation (7) are:
Example 3.
Small threshold value ( θ s ) is employed to calculate input X that exhibits large mean andstandard deviation ( X > θ s ).
Example 4.
Large threshold value ( θ l ) is employed to calculate input X ′ that exhibits large meanand standard deviation ( X ′ >
θ l ≫ θ s ).
2.3 Generalized Activation Function
We found the innovation while representing Equation (10) using the sigmoid function S ( x ) =
[Equation]
x 212 as the following:
[Equation]
In the previous work, Ramachandran et al.
(2017) introduced the leverage of automatic searchtechniques to discover the best performance activation function.
The experiments empiricallydiscovered that the Swish activation function is the best performance activation function, defined asxS ( x ) .
Intuitively, the definition of the Swish activation function is the same with Equation (11), and Equation (11) represents more generalized formula.
Therefore, ASH (Adaptive SwisH) activationfunction provides the theoretical explanations for why Swish was the best performance activationfunction in the empirical evaluations.
Therefore, we obtain the following.
Lemma 3.
ASH activation function exhibits general formula for Swish activation function.
Interestingly, the activation function designed for stochastic adaptive sampling is converged to thegeneralized Swish activation function.
The extreme impression is that the stochastic percentilesampling by the activation function that mimics real neurons expresses the general formula of theswish activation function formerly known as state-of-the-art.
Therefore, the stochastic percentilesampling could partially be applied to the Swish activation function.
In addition, it could be supposedthat the Swish activation function achieved superior performance in the previous study due to theutilization of stochastic percentile sampling.
This paper initially addressed the activation function that enables stochastic percentile sampling in amathematically effective manner, but we found that the mathematical expression of ASH supports thetheoretical background of the Swish activation function.
Therefore, this paper provides the theoreticalbackgrounds and rationales for the Swish activation function, which was empirically searched.
It isa significant innovation to provide the mathematical theorem that the Swish activation function isderived from a stochastically designed activation function.
Main Result
Like the previous study (Ramachandran et al., 2017), we compare ASH against several additionalbaseline activation functions on various models for a variety of tasks using various datasets.
Sincemany activation functions have been introduced, we employ commonly used activation functions:
ReLU, Leaky ReLU (LReLU) (Maas et al., 2013), Parametric ReLU (PReLU) (He et al., 2015), Softplus (Nair, Hinton, 2010), Exponential Linear Unit (ELU) (Clevert et al., 2015), Scaled Exponential
Linear Unit (SELU) (Klambauer et al., 2017), Gaussian Error Linear Unit (GELU) (Hendrycks, Gimpel, 2016).
In the experiments, every hyper-parameter between ASH and other activation functions isrestricted to be the same to exhibit the contribution of ASH compared to other activation functions.
In tables, the highest accuracies are highlighted as bold .
3.1 Classification Task
We first compare ASH to all baseline activation functions on the datasets of CIFAR-10 (Krizhevskyet al., 2009), CIFAR-100 (Krizhevsky et al., 2009), and ImageNet (Russakovsky et al., 2015).
Weemploy the environments from the previous study (Ramachandran et al., 2017), and re-implement thebaseline models of the ResNet-164 (He et al., 2016),
Wide ResNet28-10 (Zagoruyko, Komodakis,2016), and DenseNet-100-12 (Huang et al., 2017).
Due to different environments, small differencesexist from the previous paper, but we believe that accuracy trends are the same.
We first evaluate ASHactivation function against other activation functions on the ImageNet 2012 classification dataset.
Since ImageNet is the widely utilized dataset in the classification tasks, we employ the ImageNet2012dataset to measure the performances of ASH and other activation functions.
We then evaluate allactivation functions using CIFAR-10 and CIFAR-100 datasets widely utilized in the benchmark.
[Table]
[Table]
[Table] [Table]
Here, the ImageNet dataset evaluations are averaged from the resultant accuracy of three deep learningmodels.
The results in Tables 1-3 exhibit the outstanding performance of ASH activation function inimproving predictive accuracy.
Since the deep learning models for the classification tasks demandssparsity, it is expected that ASH activation function could improve the accuracy compared to otheractivation functions.
3.2 Detection Task
We then compare ASH to all baseline activation functions on the datasets of COCO (Lin et al.
, 2014)and PASCAL VOC (Everingham et al., 2010) for the detection task.
Both datasets are widely utilizedin the benchmark of the detection task.
We employ the same environments from the classificationtask and implement the baseline models of the Mask-R-CNN (He et al., 2017), SSD (Liu et al., 2016),and YOLOv
(Bochkovskiy et al., 2020).
In the detection task, the deep learning models output thebounding boxes for the location of the target objects.
We exploit mAP@50 for the evaluation metricdue to its popular utilization in the detection tasks.
[Table]
[Table]
The quantitative results in Tables 4 and 5 exhibit the outstanding performance of ASH activationfunction compared to other activation functions.
Higher mAP indicates the predictive boundingboxes are significantly correct with the annotations, ASH activation function could provide superiorperformance in detecting target objects in various datasets while combining many deep learningmodels.
Since the deep learning models for the detection task demands the locality to generate thebounding boxes, it is expected that the z k could be small, and
thus relatively much activation couldbe realized in ASH activation function compared to the models for the classification tasks.
3.3 Segmentation Task
Finally, we compare ASH to all baseline activation functions on the datasets of ADE20K (Zhouet al., 2017) and PASCAL VOC (Everingham et al., 2010) for the segmentation task.
Both datasetsinclude many target objects in one scene, and thus they are widely utilized in the benchmark of thesegmentation task.
We employ the same environments from the classification and detection tasks, andimplement the baseline models of the U-Net (Ronneberger et al., 2015), DeepLabV3+(DLV3+) (Chenet al., 2018), and EfficientNet (Tan, Le, 2019).
Like other general benchmarks, we exploit Intersection Model [Table] [Table]
[Table]
over Union (IoU) and mean IoU (mIoU) values as evaluation metrics due to their popular utilizationin segmentation tasks.
Like previous tasks, the quantitative results in Tables 6 and 7 highlight the outstanding performanceof ASH activation function compared to other activation functions.
Since the locality is significantlyimportant to segment target objects from the background in the segmentation task, we could intuitivelyunderstand that ASH activation function improves the locality during the feature extraction.
Theexperimental results show that superior segmentation performance improvement could be realizedusing ASH activation function, which significantly helps to localize the target objects.
[Figure]
We empirically explore the effectiveness of ASH activation function on the training time by monitoringthe validation loss values for all of the activation functions.
Fig.
3 demonstrates that the loss values of ASH activation function exhibit a more steep slope compared to those of other activation functions.
Therefore, since the ASH activation function reaches a convergence value significantly faster thanother activation functions, we could empirically achieve the results that ASH has a superior effect inreducing the training time.
Through the experiments, we explore the outstanding performance of ASH activation functioncompared to other activation functions, including the improvement in accuracy, sparsity, training timeand localization property.
Despite the lack of proof of those properties mathematically, the experimental results demonstrate the outstanding performance of ASH activation function.
Additionally,the supporting experiments and the results of other tasks of image generation or language translationare exhibited in the Supplementary Material .
Conclusion
In this paper, we devised a novel activation function to rectify inputs using an adaptive thresholdconsidering entire contexts of inputs more like human neurons.
To this end, we designed the activationfunction to extract elements in top- k % percentile from the input feature-map.
Since sorting algorithmbased selections or quick selection algorithm demands a heavy computational cost, we employed thestochastic technique utilizing normal distribution to realize stochastic percentile sampling.
From themathematical derivation, we implemented ASH activation function in simple yet effective formula( f ( x ) = x · sigmoid ( ax + b ) ) with low computational cost in sampling top-
k % percentile from theinput.
In addition, we implemented ASH activation function, realizing (1) the adaptive threshold byemploying the z-score-based trainable variables and (2) the perception of entire contexts in rectifyingan input by utilizing the mean and standard deviation of the input.
Meanwhile, ASH activationfunction represented the generalized form of the Swish activation function that was empiricallysearched in the previous study.
Therefore, this paper exhibited the additional novel contributionof the mathematical proofs for the state-of-the-art performance of the Swish activation function.
The experiments using many deep learning models for various tasks (classification, detection, andsegmentation) exhibited superior improvements in the accuracy, the localization, and training time tobe converged while employing ASH activation function.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[N/A]• Did you include the license to the code and datasets?
[N/A]
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes] See Introduction
(b) Did you describe the limitations of your work?
[Yes] See Section 2.1, Appendix(c)
Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 2. (b) Did you include complete proofs of all theoretical results?
[Yes] See Section
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See Abstract(b)
Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Appendix(c)
Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[N/A](d)
Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix4.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]