Stochastic Adaptive Activation Function
Abstract
The simulation of human neurons and neurotransmission mechanisms has been realized in deep neural networks based on the theoretical implementations of activation functions.
However, recent studies have reported that the threshold potential of neurons exhibits different values according to the locations and types of individual neurons, and that the activation functions have limitations in terms of representing this variability.
Therefore, this study proposes a simple yet effective activation function that facilitates different thresholds and adaptive activations according to the positions of units and the contexts of inputs.
Furthermore, the proposed activation function mathematically exhibits a more generalized form of Swish activation function, and thus we denoted it as Adaptive SwisH (ASH).
ASH highlights informative features that exhibit large values in the top percentiles in an input, whereas it rectifies low values.
Most importantly, ASH exhibits trainable, adaptive, and context-aware properties compared to other activation functions.
Furthermore, ASH represents general formula of the previously studied activation function and provides a reasonable mathematical background for the superior performance.
To validate the effectiveness and robustness of ASH, we implemented ASH into many deep learning models for various tasks, including classification, detection, segmentation, and image generation.
Experimental analysis demonstrates that our activation function can provide the benefits of more accurate prediction and earlier convergence in many deep learning applications.
Introduction
Searching for the optimal activation functions has been a challenge in the field of artificial intelligence (Maas et al., 2013; Ramachandran et al., 2017; Clevert et al., 2015).
Early activation functions have been studied to compensate for the non-linearity of the artificial neural networks or ameliorate the gradient vanishing problem (Hertz et al., 1997; Hochreiter, 1998).
Recently, novel activation functions have been suggested with the zero-centered or parametric properties that improve the training efficiency of deep neural networks (DNNs) (Maas et al., 2013; Clevert et al., 2015).
Currently, the activation functions focusing on the stability of DNNs or probabilistic distribution of inputs have been proposed (Hendrycks, Gimpel, 2016; Misra, 2019).
Advances in the activation functions have allowed DNNs to perform various tasks such as detecting or segmenting target objects in sophisticated images or even generating new images beyond the simple classifiers (Simonyan, Zisserman, 2014; Zhao et al., 2019; Badrinarayanan et al., 2017; Goodfellow et al., 2014).
The activation functions has evolved to behave more like a human neuron (Sharma et al., 2017; Lee et al., 2017).
However, Izhikevich (2003); Evans et al.
(2018) reported that the neurotransmission mechanism, including the membrane, action, and threshold potentials of human neurons, is subject to the location or the connection type of the neurons.
Additionally, humans perceive objects with surrounding contexts using the N : N mapping of visions to human neurons rather than the 1: mapping of a pixel to an input node in a neural network (Liu et al., 2018).
The connections between neurons can be realized through the linear combinations of layers in DNNs. However, DNNs have limitations in terms of realizing contextual perception.
This implies that the further improvement in deep neural networks and convolutional neural networks (CNNs) can be realized despite the impressive performance on image analysis (Jinsakul et al., 2019; Misra, 2019).
Therefore, the development of DNNs is leaned to mimic human perception by realizing the mechanism of human neurons (Aggarwal, others, 2018; Lindsay, 2021).
tCurrently, the primary issue is that many activation functions exhibit passivity, in this paper, indicating that they determine outputs only concerning the value of one element rather than entire contexts.
For instance, the Rectified Linear Unit (ReLU), defined as f ( x ) =
max ( x, 0) , determines the output values related to x (Fukushima, Miyake, 1982), whereas the softmax function generates output values as the ratio of the input value to the totals (Goodfellow et al., 2017). Particularly, ReLU exhibits passivity, whereas softmax does not.
Suppose an image can be classified by considering 80% of the total portion.
Current activation functions are limited in terms of classifying such an image since only elements (pixels) of the image are used to rectify the image rather than a ratio.
Another limitation is that the activation functions are invariant.
Although the parametric activation functions update their parameters during training, the resulting parameters are invariant during the inference phase (Xu et al., 2015; Bingham, Miikkulainen, 2022).
Therefore, the limited rectification can be realized by the invariant parameters or thresholds regardless of new inputs from different domains (e.g., test set).
Contribution To realize the mechanism of human neurons that rectify inputs considering their contexts, we propose a novel ASH activation function.
The main contributions of this study are to suggest a simple yet effective activation function, ASH, and to implement ASH in a mathematically effective form.
Going beyond the passive activation functions, ASH activation function is designed as (1) an active activation function that provides outputs regarding the context of inputs and (2) a conditional activation function that employs an adaptive threshold.
Unlike ReLU or Leaky ReLU, the threshold value of ASH is adaptively changed by considering the contextual information.
[Equation]
In particular, the threshold value ( θ ) is adaptively changed according to the input distribution without heavy calculation, and thus ASH provides outputs considering the contexts of inputs adaptively.
By applying ASH, we obtained the following theoretical and experimental results
: • We conducted mathematical modeling on ASH in an effective form to ensure trainable and parametric properties, and thus ASH exhibits parametric and adaptive properties.
The baseline threshold of ASH is initially trained during the training phase, and the threshold value is adaptively fine-tuned according to the contexts of inputs without heavy calculations.
• We theoretically verified that ASH adaptively changes its threshold alongside the stochastic distribution of inputs.
This implies that ASH provides outputs regarding the entire contexts of inputs, thus leading to enhanced feature extraction.
• We theoretically verified that ASH exhibits a generalized formula of Swish activation function and provided the mathematical explanations for the superior performance of Swish, which was empirically searched in the previous work.
• We experimentally showed that ASH improves the performance of deep learning models on various tasks and shortens the convergence epoch.
Related works Activation functions affect the performance of the training process to determine a functional subspace of a DNN (Hayou et al., 2019).
In particular, the non-linearity using activation functions have been introduced to prevent the issue of the linear transformation causing simple feature extractions in the DNNs (Misra, 2019; Jarrett et al., 2009).
DNNs with non-linearity have been employed to perform complex tasks (Leshno et al., 1993).
In the early era, Rectified Linear
Unit (ReLU) replaced the classical activation functions such as sigmoid and tanh (Nair, Hinton, 2010) due to its simple and computational efficiency compared to other activation functions.
During decades, many activation functions have been proposed , including Leaky ReLU (Maas et al., 2013), Exponential Linear Unit (ELU) (Clevert et al., 2015), Gaussian Error Linear Unit (GELU) (Hendrycks, Gimpel, 2016), Scaled Exponential Linear Unit (SELU) (Klambauer et al., 2017), and
Swish (Ramachandran et al., 2017) to improve the performance and stability of learning parameters in
DNNs.
Those activation functions have solved the dying ReLU problem, which exhibits a zero value in the negative region, and improved DNNs more smoothly for stable optimization.
In particular, ELU and SELU have realized internal normalization in the layer using the zero-mean property.
GELU exploited a Gaussian error and could implement an adaptive dropout to apply a higher probabilistic intuition.
Problem statement For adaptive thresholding, ASH exploits a stochastic selection methodology such as a quick selection (Hoare, 1961).
In particular, for enhanced feature extraction, the informative elements, which exhibit large values, should be identified as an attention mechanism.
In contrast, some elements, which exhibit low relevance, should be required to be rectified.
To this end, ASH is designed to identify informative elements but rectify others as 0.
Let X ∈ R
H
× W × C be a tensor (i.e., feature-map) disregarding the batch, but with a height ( H ), width ( W ), and channel ( C ), and let ¯ X ∋
X be a set of feature-maps.
Let A be an activation function such that A : ¯ X → ¯ X .
We can then extract the i th element from X , and denote it as x ( i ) .
Here, the goal of this study is to design the activation function represented as follows:
[Equation]
Note that, the novel activation function A is represented similar to ReLU, whereas its threshold is not invariant in contrast to ReLU, but it is subjected to the distribution of the input X .
To simplify, let
C ( x ( i ) , k ; X ) be a condition whether x ( i ) is ranked in the top k percentile of X , and the negation of
C is denoted as ¬ C .
In particular, the elements that satisfy C ( x ( i ) , k ; X ) are from the first largest element to the ( 0 . 01 kN )-th largest element in X , where N indicates the number of elements in X .
Here, the elements can be extracted using a simple algorithm as a quick selection.
Suppose a set ˆ X that includes all elements in X such that
ˆ X
=
{ x (1) , x (2) , ..., x ( N ) } .
We can then construct subsets of ˆ X as ˆ X C
= { x ( i ) ∈
X | C ( x ( i ) , k ; X ) } and ( ˆ X C ) c
= { x ( i ) ∈
X |¬ C
( x ( i ) , k ; X ) } .
Equation (2) can be then simplified as follows:
[Equation]
In summary, the activation function is designed to sample the top- k percentile from the input, where criteria are the values of elements.
Sampling examples are presented in Fig. 1, compared to ReLU.
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation] [Figure] [Equation]  Method
This study aims to design an activation function for a stochastic sampling of the top- k percentile elements from inputs.
However, sorting or sampling methods such as a quick selection requires high computational costs.
In contrast, sampling the top- k percentile can be realized using a Zscore-based method in a simple calculation despite the prerequisites of a normal distribution (also known as Gaussian distribution).
As discussed below, the outputs of neurons are normally distributed. Therefore, we employed the stochastic sampling to design ASH activation function.
In this section, we (1) demonstrate that the outputs of neurons are normally distributed, (2) construct a model for stochastic sampling using a Z-score, (3) formulate ASH activation function, (4) verify that ASH is parametric and trainable, and (5) search for general applications of ASH activation function.
2.1 Background
Gaussian distribution Many deep learning models employ normalization methods to improve their stability in training (Ioffe, Szegedy, 2015; Ulyanov et al., 2016; Wu, He, 2018).
In a previous study, Ioffe, Szegedy (2015) reported that the output of the convolutional layer, x = Wu + b , is more likely to have a symmetric, non-sparse distribution, that is "more Gaussian".
Since most deep learning models are based on convolutional operations, the outputs of neurons are supposed to be normally distributed (Gaussian distribution).
Therefore, it is concluded that the inputs of the activation functions are normally distributed when activation functions follow convolutional layers, such that x ∼ N ( µ x , σ 2 x ) , where x is an input feature-map of an activation function, µ x and σ x are mean and standard deviation of x , respectively.
Therefore, we can obtain the following proposition.
Proposition 1.
The outputs of neurons in convolutional neural networks are normally distributed.
Sampling from a normal distribution This study aimed to sample the top- k % elements that exhibit large values in an input feature-map.
Since the area under the normal distribution indicates the percentile, sampling the top- k % from a normal distribution is theoretically identical to the statement calculating the area under the curve presented in Fig.
Let F be a tensor, a multi-dimensional array or a matrix.
F should be then normally distributed, and thus we can sample the elements ( f ( i ) ) in the top- k % from F using the following equation:
[Equation]
However, heavy computational costs are incurred to obtain a trivial solution from the probability density function of the normal distribution, defined as
[Equation]
F .
Therefore, we leaned to probability theory to simplify the computation rather than calculus .
To this end, we employed the standard normal distribution (Z-score normalization) for Equation (4), and we obtained the following equation:
[Equation]
where z k =
( z ′ k − µ F ) /σ F , which is the Z-normalized value from z ′ k , and thus z k is subjected to k , indicating percentile to sample, in terms of Z-table (Larsen, Marx, 2005).
Then, we go Z-table and easily find the proper value for z k , intuitively.
Therefore, the condition, Z ( i ) =
( f ( i ) − µ F ) /σ
F ≥ z k ⇔ f ( i ) ≥ µ F + z k σ F , is mathematically identical to sample the top- k % elements from F .
To summarize, we obtained the following proposition.
Proposition 2.
Element x ( i ) ∈
X , which is normally distributed, is in the top- k % of X if x ( i ) ≥ µ X
+ z k σ X , where z k is a z-value subjected to k in Z-table (Larsen, Marx, 2005).
Differentiation
The mechanisms of convolutional neural networks (CNNs) have been studied in many previous works (Rumelhart et al., 1986; Bottou, 2010; Zhang, 2016;
Hu et al., 2018).
Training CNN models is subjected to the backpropagation derived from the partial derivatives of loss functions by the individual convolutional parameters.
Let L be a loss function for the deep learning model
M and let W be one of the variables in M .
Then, the derivative of L in terms of W is represented as ∂L∂W , and W is updated as W ← W − η ∂L∂W with a learning rate η .
On the other hand, suppose variable θ be the threshold, the function f ( x ) is αx if x ≥ θ , otherwise 0. Thus, the partial derivative of f is represented as follows:
[Equation]
Here, α is arithmetically combined with f ( x ) , whereas θ does not.
Therefore, α is trainable, but θ is not trainable in this context.
In basic calculus , it is trivial that if the loss function is not dependent on the variable, the partial derivative is zero, and thus the variable cannot be trained or optimized; In other words, it is invariant.
Therefore, we obtain the following lemma:
Lemma 1.
The derivative of a variable in the conditional statement is zero, and thus that the variable cannot be optimized.
2.2 ASH Activation Function
Let X be an input of ASH activation function ( Æ ) and be a tensor of which elements are normally distributed.
Furthermore, let x ( i ) ∈
X be the i -th element in X .
Then, by Proposition 2 , ASH activation function, which samples the top- k % elements from the input, is represented as follows:
[Equation]
where µ X and σ X are the mean and the standard deviation of all elements in X , respectively, and z k is the Z-score concerning percentile ( k ) to sample (i.e., z = 1 . {+
+}
if k = 2 .
5 %, see Z-table).
Equation (7) exhibits that ASH activation function is represented in a simple yet effective form with low computational costs.
Intuitively, we assumed that the activation level (percentile) is supposed to be different by each neuron and the tasks of the deep learning model, similar to human neurons.
However, in Equation (7), the condition ( x ( i ) ≥ µ X + z k σ X ) is invariant by Lemma 1 .
Note that, µ X + z k σ X is variable and changeable with respect to X , but the sampled portion ( k % related to z k ) from X is invariant.
Therefore, ASH in Equation (7) is not parametric and trainable.
To make ASH be trainable and parametric, let Equation (7) be substituted using a proxy function as Æ ( x ( i ) )
=
x ( i ) f
( x ( i ) ) , such that the proxy function f ( x ( i ) ) is represented as follows:
[Equation]
For simplicity, suppose that a Heaviside step function (Weisstein, 2002) is defined as H ( x )
= ddx max (0 , x ) , and thus f ( x ( i ) ) =
H ( x ( i ) − µ X
−
z k σ X ) .
Then, we obtain the arithmetical form to formulate ASH activation function as follows:
[Equation]
Even with the arithmetic formula, ASH activation function in Equation (9) is still independent of z k , and thus the z k is still not trainable.
However, it is well known that the Heaviside step function is analytically approximated as 2 H ( x )
= 1+1 tanh( αx ) with a large value of α (Iliev et al., 2017), and thus ASH activation function is approximated using the smooth Heaviside step function as follows:
[Equation]
Since z k is arithmetically placed, z k representing a sampling percentile is trainable, and thus ASH activation function is also trainable and parametric.
By optimizing z k , ASH activation functions exhibit different thresholds.
Therefore, it is concluded that ASH exhibits different activation levels based on the stochastic sampling of inputs and different thresholds, similar to human neurons, synapses, and their potentials.
As human neurons, the mechanism of ASH can be summarized as follows: (1) In the training phase, each ASH activation function optimizes its z k and fine-tunes the threshold for the sampling percentile of inputs.
Thus, it implies that ASH activation function realizes the arbitrary threshold potentials as human neurons (Clevert et al., 2015; Evans et al., 2018).
Some examples of z k related to Equation (7) are:
Example 1.
A small value of z k , even a small negative value, implies the dense activation, and the dying ReLU problem can be solved.
Example 2.
A large value of z k implies the sparse activation, and the sparsity can be leveraged.
In the training or inference phase, ASH activation function rectifies the inputs using the learned threshold value and contexts of inputs.
In particular, to sample the top- k percentile, ASH employs the mean and the standard deviation of inputs, representing the contexts of the entire inputs.
Therefore, it implies that ASH realizes the adaptive activation considering the contexts of inputs.
Some examples of the adaptive activation related to Equation (7) are:
Example 3.
A small threshold value ( θ s ) is employed to calculate the input X that exhibits large mean and standard deviation values ( X > θ s ).
Example 4.
A large threshold value ( θ l ) is employed to calculate the input X ′ that exhibits large mean and standard deviation values ( X ′ >
θ l ≫ θ s ).
2.3 Generalized Activation Function
We found the innovation while representing Equation (10) using the sigmoid function S ( x ) =
[Equation]
as follows:
[Equation]
In a previous work, Ramachandran et al.
(2017) introduced the leverage of automatic search techniques to discover the best performance activation function.
The experiments empirically discovered that the Swish activation function is the best performance activation function, defined as xS ( x ) (Ramachandran et al., 2017).
Intuitively, the definition of the Swish activation function is the same with Equation (11), and Equation (11) represents more generalized formula.
Therefore, ASH (Adaptive SwisH) activation function provides the theoretical explanations for why Swish was the best performance activation function in the empirical evaluations.
Therefore, we obtain the following.
Lemma 3.
ASH activation function exhibits general formula for the Swish activation function.
Interestingly, the activation function designed for stochastic adaptive sampling is converged to the generalized Swish activation function.
The extreme impression is that the stochastic percentile sampling by the activation function that mimics real neurons expresses the general formula of the swish activation function formerly known as state-of-the-art.
Therefore, the stochastic percentile sampling can partially be applied to the Swish activation function.
Additionally, it can be supposed that the Swish activation function achieved superior performance in the previous studies based on the utilization of stochastic percentile sampling.
This paper initially considered an activation function that enables stochastic percentile sampling in a mathematically effective manner. However, we found that the mathematical expression of ASH supports the theoretical background of the Swish activation function.
Therefore, this paper provides the theoretical backgrounds and rationales for the Swish activation function, which was empirically investigated.
It is a significant innovation to provide the mathematical theorem that the Swish activation function is derived from a stochastically designed activation function.
Main Result
Similar to a previous study (Ramachandran et al., 2017), we compared ASH to several baseline activation functions on various models for different tasks using public datasets.
Because many activation functions have been developed, we employed some of the most commonly used activation functions, namely ReLU, leaky ReLU (LReLU) (Maas et al., 2013), parametric ReLU (PReLU) (He et al., 2015), Softplus (Nair, Hinton, 2010), ELU (Clevert et al., 2015), SELU (Klambauer et al., 2017), and GELU (Hendrycks, Gimpel, 2016).
In our experiments, every hyper-arameter in ASH and the other activation functions was set to be the same to demonstrate the advantages of ASH compared to other activation functions.
In the tables, the highest accuracy values are highlighted in bold .
3.1 Classification Task
We first compared ASH to all the baseline activation functions on the CIFAR-10 (Krizhevsky et al., 2009), CIFAR-100 (Krizhevsky et al., 2009) datasets, and ImageNet (Russakovsky et al., 2015) datasets.
We employed environments from a previous study (Ramachandran et al., 2017) and reimplemented the baseline models of ResNet-164 (He et al., 2016), wide ResNet28-10 (Zagoruyko,
Komodakis, 2016), and DenseNet-100-12 (Huang et al., 2017).
Based on these different environments, small differences were reported previously, but we believe that the accuracy trends are similar.
We first evaluated ASH activation function against other activation functions using the ImageNetclassification dataset because ImageNet is a widely utilized dataset in classification tasks.
We then evaluated all activation functions using the CIFAR-10 and CIFAR-100 datasets, which have been widely utilized as benchmarks
[Table]
[Table] [Table]
The ImageNet dataset evaluations were averaged based on the accuracy values of the three deep learning models.
The results in Tables 1 to 3 highlight the outstanding performance of ASH activation function in terms of improving predictive accuracy.
Because deep learning models for classification tasks demand sparsity [Table] it is intuitive that ASH activation function improves accuracy compared to other activation functions [Table]
3.2 Detection Task
We compared ASH to all the baseline activation functions on the COCO (Lin et al., 2014) and
PASCAL VOC (Everingham et al., 2010) datasets for the detection task.
Both of these datasets are widely used as benchmarks for detection tasks.
We employed the same environments as the classification task and implemented the baseline models of Mask-R-CNN (He et al., 2017), SSD (Liu et al., [-2016),and YOLOv
-]2016) [Table]
and YOLOv4 (Bochkovskiy et al., 2020).
For detection tasks, deep learning models output bounding boxes representing the locations of target objects.
We exploited mAP@50 as an evaluation metric based on its popularity for detection tasks
[Table]
[Table]
The quantitative results in Tables 4 and 5 highlight the outstanding performance of ASH activation function compared to other activation functions.
A higher mAP indicates that the predicted bounding boxes are closer to the annotations. ASH activation function provides superior performance for detecting target objects in various datasets for various deep learning models.
Because the deep learning models used for detection tasks demand locality to generate bounding boxes, it is expected that z k will be small, demonstrating that greater activation can be realized using ASH activation function compared to the models used for the classification task.
3.3 Segmentation Task
We compared ASH to all of the baseline activation functions on the ADE20K (Zhou et al., 2017) and PASCAL VOC (Everingham et al., 2010) datasets for the segmentation task.
Both datasets include many target objects in one scene. Therefore, they are widely utilized as benchmarks for segmentation tasks.
We employed the same environments as the classification and detection tasks and implemented the baseline models of U-Net (Ronneberger et al., 2015), DeepLabV3+(DLV3+) (Chen et al., 2018), and EfficientNet (Tan, Le, 2019).
Similar to other general benchmarks, we adopted intersection over union (IoU) and mean IoU (mIoU) values as evaluation metrics based on their popularity for segmentation tasks.
Similar to the previous tasks, the quantitative results in Tables 6 and 7 highlight the outstanding performance of ASH activation function compared to the other activation functions.
Because locality is important for segmenting target objects from the background in segmentation tasks, it is intuitive that ASH activation function improves locality during feature extraction.
The experimental results demonstrate that superior segmentation performance can be realized by using ASH activation function, which aids significantly in localizing target objects
[Table] [Table]
3.4 Training Time
We empirically explored the effectiveness of ASH activation function in terms of training time by monitoring the validation loss values for all activation functions.
Fig.
3 reveals that the loss values of ASH activation function exhibit a steeper slope than those of the other activation functions.
Therefore, because ASH activation function reaches convergence significantly faster than the other activation functions, we can empirically conclude that ASH has a superior effect in terms of reducing training time.
Through our experiments, we explored the outstanding performance of ASH activation function compared to other activation functions, including improvements in accuracy, sparsity, training time, and localization.
In this study, the experimental results demonstrate the outstanding performance of ASH activation function.
Additionally, supporting experiments and the results of other tasks such as image generation is presented in the Supplementary Material .
Conclusions
In this paper, we proposed a novel activation function to rectify inputs using an adaptive threshold considering the entire contexts of inputs more like human neurons.
To this end, we designed an activation function to extract elements in the top- k percentile from the input feature-map.
Since sorting algorithm-based selections or quick selection algorithm demands a heavy computational cost, we employed the stochastic technique utilizing normal distribution to realize stochastic percentile sampling.
Based on the mathematical derivations, we implemented ASH activation function in simple yet effective formula ( f ( x ) = x · sigmoid ( ax + b ) ) with low computational cost for sampling the top- k percentile from the input.
In addition, we implemented ASH activation function, realizing (1) the adaptive threshold by employing the Z-score-based trainable variables and (2) the perception of entire contexts in rectifying an input by utilizing the mean and standard deviation of the input.
Meanwhile, ASH activation function represented the generalized form of the Swish activation function that was empirically searched in the previous study.
Therefore, this study also exhibited a novel contribution of the mathematical proofs for the state-of-the-art performance of the Swish activation function.
Experiments using various deep learning models on different tasks (classification, detection, and segmentation) demonstrated superior performance for ASH activation function, in terms of accuracy, localization, and training time. Acknowledgment
This work was supported in part by the National Research Foundation of Korea (NRF) under
Grant NRF-2020R1A2B5B01002786 and in part by the Bio & Medical Technology Development
Program of the National Research Foundation (NRF) funded by the Korean government (MSIT) (No.2017M3A9G8084463).
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[N/A] • Did you include the license to the code and datasets? [N/A] • Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] See Introduction (b) Did you describe the limitations of your work?
[Yes] See Section 2.1, Appendix (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 2. (b) Did you include complete proofs of all theoretical results?
[Yes] See Section3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See Abstract (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Appendix (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] statistical analysis were provided using confidence interval (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]