Original: S MOOTH A CTIVATIONS AND R EPRODUCIBILITY IN
Revised: S MOOTH A CTIVATIONS AND R EPRODUCIBILITY IN

Original: D EEP N ETWORKS
Revised: D EEP N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The amazing success of deep networks is penetrating almost every domain in our lives.
Revised: Deep networks are gradually penetrating almost every domain in our lives due to their amazing success.

Original: However, with substantive performance accuracy improvements comes the price of irreproducibility .
Revised: However, with substantive performance accuracy improvements comes the price of irreproducibility .

Original: Two identical models, trained on the exact same training dataseton highly distributed parallel systems, while exhibiting similar average accuracy may exhibit large differences in predictions on individual examples.
Revised: Two identical models, trained on the exact same training dataset may exhibit large differences in predictions on individual examples even when average accuracy is similar, especially when trained on highly distributed parallel systems.

Original: The popular Rectiﬁed Linear Unit (ReLU) activation has been key to recent success of deep networks
Revised: The popular Rectiﬁed Linear Unit (ReLU) activation has been key to recent success of deep networks.

Original: We demonstrate, however that ReLU is also a catalyzer to irre - producibility in deep networks.
Revised: We demonstrate, however, that ReLU is also a catalyzer to irreproducibility in deep networks.

Original: We show that not only can activations smoother than ReLU provide better accuracy but they can also provide better accuracy - reproducibility tradeoffs
Revised: We show that not only can activations smoother than ReLU provide better accuracy, but they can also provide better accuracy-reproducibility tradeoffs.

Original: We propose a new family of activations ; Smooth ReLU ( SmeLU  designed to give such better tradeoffs, while also keeping the mathe- matical expression simple.
Revised: We propose a new family of activations; Smooth ReLU ( SmeLU ), designed to give such better tradeoffs, while also keeping the mathematical expression simple, and thus implementation cheap.

Original: SmeLU is monotonic, mimics ReLU while providing continuous gradients yielding better reproducibility.
Revised: SmeLU is monotonic, mimics ReLU, while providing continuous gradients, yielding better reproducibility.

Original: We generalize SmeLU to give even more ﬂexibility and then demonstrate that SmeLU and its generalized form are special cases of a more general methodology of REctiﬁed Smooth Con- tinuous Unit (RESCU) activations.
Revised: We generalize SmeLU to give even more ﬂexibility and then demonstrate that SmeLU and its generalized form are special cases of a more general methodology of REctiﬁed Smooth Continuous Unit (RESCU) activations.

Original: Empirical results demonstrate the superior accuracy-reproducibility tradeoffs with smooth activations, SmeLU in particular
Revised: Empirical results demonstrate the superior accuracy-reproducibility tradeoffs with smooth activations, SmeLU in particular.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recent developments in deep learning leave no question about the advantages of deep networks over classical methods, that relied heavily on linear convex optimization solutions.
Revised: Recent developments in deep learning leave no question about the advantages of deep networks over classical methods, which relied heavily on linear convex optimization solutions.

Original: With their astonishing unprecedented success, deep models are providing solutions to a gradually increasing number of domains in our lives.
Revised: With their astonishing unprecedented success, deep models are providing solutions to a continuously increasing number of domains in our lives.

Original: These solutions, however, while much more accurate than their convex counterparts, are usually irreproducible in the predictions they provide.
Revised: These solutions, however, while much more accurate than their convex counterparts, are usually irreproducible in the predictions they provide.

Original: While average accuracy of deep models on some validation dataset is usually much higher than that of linear convex models, predictions on individual examples of two models, that were trained to be identical, may diverge substantially, exhibiting Prediction Differences that may be as high as non-negligible fractions of the actual predictions (see, e.g.,  Dusenberry et al.
Revised: While average accuracy of deep models on some validation dataset is usually much higher than that of linear convex models, predictions on individual examples of two models, that were trained to be identical, may diverge substantially, exhibiting Prediction Differences that may be as high as non-negligible fractions of the actual predictions (see, e.g., Chen et al. Dusenberry et al.

Original: Like humans, deep networks express  what they learned (Achille et al., 2017; Bengio et al., 2009).
Revised: Deep networks express (only) what they learned. Like humans, they may establish different beliefs as function of the order in which they had seen training data (Achille et al., 2017; Bengio et al., 2009).

Original: Due to the huge amounts of data required to train such models, enforcing determinism (Nagarajan et al., 2018) may not be an option.
Revised: Due to the huge amounts of data required to train such models, enforcing determinism (Nagarajan et al., 2018) may not be an option.

Original: Deep networks may be trained on highly distributed, parallelized systems.
Revised: Deep networks may be trained on highly distributed, parallelized systems.

Original: Thus two supposedly identical models, with the same architecture, parameters, training algorithm and training hyper-parameters that are trained on the same training dataset, even if they are initialized identically, will exhibit some randomness in the order in which they see the training set and apply updates.
Revised: Thus two supposedly identical models, with the same architecture, parameters, training algorithm and training hyper-parameters that are trained on the same training dataset, even if they are initialized identically, will exhibit some randomness in the order in which they see the training set and apply updates.

Original: Due to the highly nonconvex objective, such models may converge to different optima, which may exhibit equal average objective, but can provide very different predictions to individual examples.
Revised: Due to the highly non-convex objective, such models may converge to different optima, which may exhibit equal average objective, but can provide very different predictions to individual examples.

Revised: Irreproducibility in deep models is not the classical type of epistemic uncertainty, widely studied in the literature, nor is it overﬁtting.

Revised: It differs from these phenomena in several ways: It does not diminish with more training examples like classical epistemic uncertainty, and it does not cause degradation to test accuracy by overﬁtting unseen data to the training examples.

Original: While this behavior may be acceptable for some applications, it can be very detrimental in applications, such as medical ones, where two different diagnoses to the same symptoms may be unacceptable.
Revised: While irreproducibilty may be acceptable for some applications, it can be very detrimental in applications, such as medical ones, where two different diagnoses to the same symptoms may be unacceptable.

Original: Furthermore, in online and/or re-enforcement systems, which rely on their predictions to determine actions, that in turn, determine the remaining training datasets, even small initial irreproducibility can cause large divergence of models that are supposed to be identical.
Revised: Furthermore, in online and/or re-enforcement systems, which rely on their predictions to determine actions, that in turn, determine the remaining training examples, even small initial irreproducibility can cause large divergence of models that are supposed to be identical.

Original: One exampleof such an application isthat of sponsored advertisement online Click-Through-Rate (CTR) prediction (McMahan et al., 2013).
Revised: One example is sponsored advertisement online Click-Through-Rate (CTR) prediction (McMahan et al., 2013).

Revised: The effect of irreproducibility in CTR prediction can go far beyond changing the predicted CTR of an example, as it may affect actions that take place downstream in a complex system.

Revised: Reproducibility is a problem even if one trains only a single model, as it may be impossible to determine whether the trained model provides acceptable solutions for applications that cannot tolerate unacceptable ones.

Original: A major factor to the unprecedented success of deep networks in recent years has been the Rectiﬁed Linear Unit (ReLU) activation, (Nair & Hinton, 2010).
Revised: A major factor to the unprecedented success of deep networks in recent years has been the Rectiﬁed Linear Unit (ReLU) activation, (Nair & Hinton, 2010).

Original: ReLU nonlinearity together with back-propagation give simple updates, accompanied with superior accuracy.
Revised: ReLU nonlinearity together with back-propagation give simple updates, accompanied with superior accuracy.

Original: ReLU thus became the undisputed activation used in deep learning.
Revised: ReLU thus became the undisputed activation used in deep learning.

Original: However, is ReLU really the best to use?
Revised: However, is ReLU really the best to use?

Original: While it gives better optima than those achieved with simple convex models, it imposes an extremely nonconvex objective surface with many such optima.
Revised: While it gives better optima than those achieved with simple convex models, it imposes an extremely nonconvex objective surface with many such optima.

Original: The direction of a gradient update with a gradient based optimizer is determined by the speciﬁc example which generates the update.
Revised: The direction of a gradient update with a gradient based optimizer is determined by the speciﬁc example which generates the update.

Original: Thus the order of seeing examples or applying updates can determine which optimum is reached.
Revised: Thus the order of seeing examples or applying updates can determine which optimum is reached.

Revised: Many such optima, as imposed by ReLU, thus provide a recipe for irreproducibility.

Original: In recent years, different works started challenging the ReLU’s dominance, exploring alternatives.
Revised: In recent years, different works started challenging the dominance of ReLU, exploring alternatives.

Original: Overviews of various activations were reported in Nwankpa et al.
Revised: Overviews of various activations were reported in Nwankpa et al.

Original: ; Pedamonti (2018).
Revised: ; Pedamonti (2018).

Original: Variations on ReLU were studied in Jin et al.
Revised: Variations on ReLU were studied in Jin et al.

Original: Activations like SoftPlus (Zheng et al., 2015), Exponential Linear Unit (ELU) (Clevert et al., 2015),or Scaled Exponential Linear Unit (SELU) (Klambauer et al., 2017; Sakketou & Ampazis, 2019; Wang et al.,  2017) were proposed, as well as the Gaussian Error Linear Unit (GELU) (Hendrycks & Gimpel, 2016).
Revised: Activations like SoftPlus (Zheng et al., 2015), Exponential Linear Unit (ELU) (Clevert et al., 2015), Scaled Exponential Linear Unit (SELU) (Klambauer et al., 2017; Sakketou & Ampazis, 2019; Wang et al., 2017), or Continuously differentiable Exponential Linear Unit (CELU) (Barron, 2017) were proposed, as well as the Gaussian Error Linear Unit (GELU) (Hendrycks & Gimpel, 2016).

Original: Speciﬁcally, the Swish activation (Ramachandran et al., 2017) (that can approximate GELU) was found through automated search to achieve superior accuracy to ReLU.
Revised: Speciﬁcally, the Swish activation (Ramachandran et al., 2017) (that can approximate GELU) was found through automated search to achieve superior accuracy to ReLU.

Original: Further activations with similarity to GELU were proposed recently; Mish (Misra, 2019), and TanhExp (Liu & Di, 2020).
Revised: Further activations with similarity to GELU were proposed recently; Mish (Misra, 2019), and TanhExp (Liu & Di, 2020).

Original: Unlike ReLU, many of these activations are smooth with continuous gradient.
Revised: Unlike ReLU, many of these activations are smooth with continuous gradient.

Original: Good properties of smooth activations were studied as early as Mhaskar (1997) (see also Du (2019)).
Revised: Good properties of smooth activations were studied as early as Mhaskar (1997) (see also Du (2019); Lokhande et al.

Original: These series of papers started suggesting that smooth activations, if conﬁgured properly, may be superior to ReLU in accuracy.
Revised: These series of papers started suggesting that smooth activations, if conﬁgured properly, may be superior to ReLU in accuracy.

Original: Recent work by Xie et al.
Revised: Recent work by Xie et al.

Original: (2020), that was done subsequently to our work we report in this paper,  demonstrated also the advantage of smooth activations for adversarial training (speciﬁcally focusing on Swish and SoftPlus).
Revised: (2020), that was done subsequently to our work, and was inspired from our results that we report in this paper, (Lin & Shamir, 2019), demonstrated also the advantage of smooth activations for adversarial training.

Original: Our Contributions: In this paper, we ﬁrst demonstrate the advantages of smooth activations to reproducibility in deep networks.
Revised: Our Contributions: In this paper, we ﬁrst demonstrate the advantages of smooth activations to reproducibility in deep networks.

Original: We show that not only can smooth activations improve accuracy of deep networks, they can also achieve superior tradeoffs between reproducibility and accuracy, by attaining a lower average Prediction Difference (PD) for the same or better accuracy.
Revised: We show that not only can smooth activations improve accuracy of deep networks, they can also achieve superior tradeoffs between reproducibility and accuracy, by attaining a lower average Prediction Difference (PD) for the same or better accuracy.

Original: Smooth activations, like Swish, GELU, Mish, and TanhExp, all have a very similar non-monotonic form, that does not provide a clear stop region (with strict 0 ; not only approaching 0 ), and slope 1 region.
Revised: Smooth activations, like Swish, GELU, Mish, and TanhExp, all have a very similar non-monotonic form, that does not provide a clear stop region (with strict 0 ; not only approaching 0 ), and sloperegion.

Original: While these activations approximate the mathematical form of ReLU, they lack these properties of ReLU.
Revised: While these activations approximate the mathematical form of ReLU, they lack these properties of ReLU.

Original: All these activations including SoftPlus also require morecomplex and expensive mathematical expressions, involving exponents, and, in some, logarithms, or even numerically computed values (e.g., GELU).
Revised: All these activations including SoftPlus also require more expensive mathematical expressions, involving exponents, and, in some, logarithms, or even numerically computed values (e.g., GELU).

Revised: This can make deployment harder, especially with certain simpliﬁed hardware that supports only a limited number of operations, as well as can slow down training due to the heavier computations.

Original: Unlike ReLU, which can be transformed into Leaky ReLU, the smooth activations described cannot be easily transformed into more general forms.
Revised: Unlike ReLU, which can be transformed into Leaky ReLU, the smooth activations described cannot be easily transformed into more general forms.

Original: In this work, we propose Smooth ReLU (SmeLU) , which is mathematically simple and based only on linear and quadratic expressions, can provide  a clearly deﬁned 0 activation region, as well as a slope 1 region, is monotonic, and is also extendable to a leaky or more general form.
Revised: In this work, we propose Smooth ReLU (SmeLU) , which is mathematically simple and based only on linear and quadratic expressions. It can be more easily deployed with limited hardware, and can provide faster training when hardware is limited. SmeLU provides a clearly deﬁned 0 activation region, as well as a slope 1 region, is monotonic, and is also extendable to a leaky or more general form.

Original: SmeLU gives the good properties of smooth activations providing better reproducibility as well as better accuracy-reproducibility tradeoffs.
Revised: SmeLU gives the good properties of smooth activations providing better reproducibility as well as better accuracy-reproducibility tradeoffs.

Original: Its generalized form allows even further accuracy improvements, and the methodology to construct SmeLU is shown to be even more general, allowing for more complex smooth activations, which are all clustered under the category of Rectiﬁed Smooth  Units (RESCUs) .
Revised: Its generalized form allows even further accuracy improvements. The methodology to construct SmeLU is shown to be even more general, allowing for more complex smooth activations, which are all clustered under the category of Rectiﬁed Smooth Continuous Units (RESCUs) .

Original: Related Work: Ensembles (Dietterich, 2000) have been used to reduce uncertainty (Lakshminarayanan et al., 2017).
Revised: Related Work: Ensembles (Dietterich, 2000) have been used to reduce uncertainty (Lakshminarayanan et al., 2017).

Original: They are useful also  reducing irreproducibility.
Revised: They are useful also for reducing irreproducibility.

Original: However, they make models more complex, and  trade off accuracy in favor of reproducibility (Shamir & Coviello, 2020).
Revised: However, they make models more complex, and can trade off accuracy in favor of reproducibility if one attempts to keep constant computation costs (which require reducing capacity of each component of the ensemble).

Original: Compression of deep networks into smaller networks that attempt to describe the same information is the emerging area of distillation (Hinton et al., 2015).
Revised: Compression of deep networks into smaller networks that attempt to describe the same information is the emerging area of distillation (Hinton et al., 2015).

Original: Predictions of a strong teacher train a weaker student model.
Revised: Predictions of a strong teacher train a weaker student model.

Original: The student is then deployed.
Revised: The student is then deployed.

Original: This approach is very common if there are ample training resources, but deployment is limited, as for mobile network devices.
Revised: This approach is very common if there are ample training resources, but deployment is limited, as for mobile network devices.

Original: Co-distillation , proposed by Anil et al.
Revised: Co-distillation , proposed by Anil et al.

Original: (2018) (see also Zhang et al.
Revised: (2018) (see also Zhang et al.

Original: (2018)), took advantage of distillation to address irreproducibility.
Revised: (2018)), took advantage of distillation to address irreproducibility.

Original: Instead of unidirectional transfer of knowledge, several models distill information between each other.
Revised: Instead of unidirectional transfer of knowledge, several models distill information between each other.

Original: They attempt to converge to the same solution.
Revised: They attempt to converge to the same solution.

Original: The method requires more training resources to co-train models, but deployment only requires a single model.
Revised: The method requires more training resources to co-train models, but deployment only requires a single model.

Original: A somewhat opposite approach; Anti-Distillation , to address irreproducibility was proposed by Shamir & Coviello (2020), embracing ensembles, with an additional loss that forces their components away from each other.
Revised: A somewhat opposite approach; Anti-Distillation , to address irreproducibility was proposed by Shamir & Coviello (2020), embracing ensembles, with an additional loss that forces their components away from one another.

Original: Each component is forced to capture a (more) different part of the objective space, and as a whole, the predictions of the ensemble are more reproducible.
Revised: Each component is forced to capture a (more) different part of the objective space, and as a whole, the predictions of the ensemble are more reproducible.

Original: To the best of our knowledge, all previously reported techniques to address irreproducibility in deep networks required some form of ensembles.
Revised: To the best of our knowledge, all previously reported techniques to address irreproducibility in deep networks required some form of ensembles.

Original: In this work, we leverage smooth activations, and do not require ensembles.
Revised: In this work, we leverage smooth activations, and do not require ensembles.

Original: Outline: Section 2 proposes several PD metrics, which we use to measure irreproducibility.
Revised: Outline: Section 2 proposes several PD metrics, which we use to measure irreproducibility.

Original: Next, we overview our setup and smooth activations in Section 3, and describe SmeLU and its generalization in Section 4.
Revised: Next, we overview our setup and smooth activations in Section 3, and describe SmeLU and its generalization in Section 4.

Original: Experimental results are shown in Section 5.
Revised: Experimental results are shown in Section 5.

Original:  P REDICTION D IFFERENCE
Revised:  P REDICTION D IFFERENCE

Original: Average individual per-example Prediction Difference (PD) over a set of models that are conﬁgured, trained, and initialized identically over some validation dataset can be deﬁned in various ways.
Revised: Average individual per-example Prediction Difference (PD) over a set of models that are conﬁgured, trained, and initialized identically over some validation dataset can be deﬁned in various ways.

Original: We refer to Shamir & Coviello (2020) for a more detailed discussion.
Revised: We refer to Shamir & Coviello (2020) for a more detailed discussion.

Original: We opt to the same deﬁnitions they used, where we describe the classiﬁcation case, in which we measure the PD using some L p norm on the actual label predictions.
Revised: We opt to the same deﬁnitions they used, where we describe the classiﬁcation case, in which we measure the PD using some L p norm on the actual label predictions.

Original: Following their notation, denote the number of models by M , and the number of validation examples by N .
Revised: Following their notation, denote the number of models by M , and the number of validation examples by N .

Original: Let P n,m be the distribution over labels predicted by model m for example n .
Revised: Let P n,m be the distribution over labels predicted by model m for example n .

Original: ( P n,m ( (cid:96) ) is the probability predicted for label (cid:96) .)
Revised: ( P n,m ( (cid:96) ) is the probability predicted for label (cid:96) .)

Original: Let P n (cid:52) =
Revised: Let P n (cid:52) =

Original: (cid:80) m P n,m /M be the expected distribution over labels over all M models.
Revised: (cid:80) m P n,m /M be the expected distribution over labels over all M models.

Original: Then, the p th norm PD, ∆ p , is given by 
Revised: Then, the p th norm PD, ∆ p , is given by ∆ p =

Original: [Equation]
Revised: [Equation]

Original: With practical large scale systems, with very high training costs, we can use M = 2 , where for binary labels, ∆ 1 = 1 /N · (cid:80) n | P n, 1 (1) − P n, 2 (1) | , where 1 is the positive label.
Revised: With practical large scale systems, with very high training costs, we can use M = 2 , where for binary labels, ∆ 1 = 1 /N · (cid:80) n | P n, 1 (1) − P n, 2 (1) | , where 1 is the positive label.

Original: As in Shamir & Coviello (2020), we can consider relative PD , ∆ r 1 , normalizing the  summand in (1) by P n ( (cid:96) ) , which for binary problems can be tweaked into ˜ ∆ r 1 , normalizing by P n (1) instead.
Revised: As in Shamir & Coviello (2020), we can consider relative PD , ∆ r 1 , normalizing the innermost summand in (1) by P n ( (cid:96) ) , which for binary problems can be tweaked into ˜∆ r 1 , normalizing by P n (1) instead.

Original: PD, ∆ L 1 , can be computed only on the observed true label, by replacing the  sum on (cid:96) in (1) by the absolute difference for the true label, normalizing by P n ( (cid:96) true ) .
Revised: PD, ∆ L 1 , can be computed only on the observed true label, by replacing the innermost sum on (cid:96) in (1) by | P n,m ( (cid:96) true ) − P n ( (cid:96) true ) | for the true label, normalizing by P n ( (cid:96) true ) .

Revised: Its computation, however, requires knowledge of the true label.

Original: Finally, in classiﬁcation, one can use Hamming PD ; ∆ H , specifying the average  ofthe labels predicted differently between model pairs.
Revised: Finally, in classiﬁcation, one can use Hamming PD ; ∆ H , specifying the average fraction of labels predicted differently between model pairs.

Original:  S MOOTH A CTIVATIONS
Revised:  S MOOTH A CTIVATIONS

Original: We consider standard Deep Neural Networks.
Revised: We consider standard Deep Neural Networks.

Original: Inputs can also be embedding vectors representing unmeasurable features, learned together with weights and biases of hidden layers.
Revised: Inputs can also be embedding vectors representing unmeasurable features, learned together with weights and biases of hidden layers.

Original: Networks can be deep or shallow (a few or even a single hidden layer).
Revised: Networks can be deep or shallow (a few or even a single hidden layer).

Original: The output of the matrix multiplication in any of the networks’ hidden layers is activated by a nonlinear activation.
Revised: The output of the matrix multiplication in any of the networks’ hidden layers is activated by a nonlinear activation.

Original: Our results apply in a standard setup, where training commences over a training dataset for multiple epochs, allowing example revisits.
Revised: Our results apply in a standard setup, where training passes over a training dataset for multiple epochs, allowing example revisits.

Original: We also consider the online setting, where training commences in a single pass over the data.
Revised: We also consider the online setting, where training consists of a single pass over the data.

Original: Inference is applied to unseen examples before updates.
Revised: Inference is applied to unseen examples before updates.

Original: Normally, updates are applied in mini-batches.
Revised: Normally, updates are applied in minibatches.

Original: Some form of normalization may be applied in the hidden layers to limit signal range, guarding from vanishing or exploding gradients, and providing some scale invariance.
Revised: Some form of normalization may be applied in the hidden layers to limit signal range, guarding from vanishing or exploding gradients, and providing some scale invariance.

Original: Without normalization, activations can start drifting to become larger.
Revised: Without normalization, activations can start drifting to become larger.

Original: As we show, smooth activations are controlled by parameters that dictate their behavior.
Revised: As we show, smooth activations are controlled by parameters that dictate their behavior.

Original: Inputs that scale up generate an effect of scaling down widths of the activation regions.
Revised: Inputs that scale up generate an effect of scaling down widths of the activation regions.

Original: As we will observe, slower smoother gradient changes, manifested as wider input regions for the same output change, are more reproducible.
Revised: As we will observe, slower smoother gradient changes, manifested as wider input regions for the same output change, are more reproducible.

Original: With such drifts, they gradually appear to be narrower, slowly losing reproducibility beneﬁts.
Revised: With such drifts, they gradually appear to be narrower, slowly losing reproducibility beneﬁts.

Original: Normalizations, as weight normalization (Salimans & Kingma, 2016), layer normalization (Ba et al., 2016), or batch normalization (Ioffe & Szegedy, 2015) can be considered.
Revised: Normalizations, as weight normalization (Salimans & Kingma, 2016), layer normalization (Ba et al., 2016), or batch normalization (Ioffe & Szegedy, 2015) can be considered.

Original: In this work, however, we use a slightly different form of weight normalization, where some L p norm normalizes all matrix link weights incoming to an output neuron of a layer.
Revised: In this work, however, we use a slightly different form of weight normalization, where some L p norm normalizes all matrix link weights incoming to an output neuron of a layer.

Original: We speciﬁcally normalize the L 2 norm of the incoming matrix weights into the neuron to some norm v without centering around the mean.
Revised: We speciﬁcally normalize the L 2 norm of the incoming matrix weights into the neuron to some norm v without centering around the mean.

Original: Let W (cid:96) be the weight matrix incoming into layer (cid:96) from layer (cid:96) − 1 , use w (cid:96)j to denote the j th row of W (cid:96) .
Revised: Let W (cid:96) be the weight matrix incoming into layer (cid:96) from layer (cid:96) − 1 , use w (cid:96)j to denote the j th row of W (cid:96) .

Original: Then, the pre-activation neuron vector of layer (cid:96) is given by a (cid:96) =
Revised: Then, the pre-activation neuron vector of layer (cid:96) is given by a (cid:96) =

Original: W (cid:96) · f ( a (cid:96) − 1 ) , where f ( · ) denotes the activation non-linearity (except for the input layer (cid:96) = 0 , where it is the identity).
Revised: W (cid:96) · f ( a (cid:96) − 1 ) , where f ( · ) denotes the activation non-linearity (except for the input layer (cid:96) = 0 , where it is the identity).

Original: Then, with L 2 weight normalization, we replace w (cid:96)j by ˜ w (cid:96)j (cid:52) =  w (cid:96)j / (cid:107) w (cid:96)j (cid:107) 2· v .
Revised: Then, with L 2 weight normalization, we replace w (cid:96)j by ˜ w (cid:96)j (cid:52) = v · w (cid:96)j / (cid:107) w (cid:96)j (cid:107) 2 .

Original: Norms, other than L 2 , as well as other normalization regimes can also be used.
Revised: Norms, other than L 2 , as well as other normalization regimes can also be used.

Original: As we demonstrate below, for smooth activations, the norm v trades off with the activation parameters, tuning the accuracy/reproducibility tradeoffs, but not changing the good effects of smooth activations.
Revised: As we demonstrate below, for smooth activations, the norm v can trade off with the activation parameters, tuning the accuracy/reproducibility tradeoffs, but not changing the good effects of smooth activations.

Original: As demonstrated in Appendix A, lack of normalization with nonlinearity in the form of clipping values  diminish the beneﬁts of smooth activations.
Revised: As demonstrated in Appendix A, lack of normalization with nonlinearity in the form of clipping values can diminish the beneﬁts of smooth activations.

Original: Note that recent work by Liu et al.
Revised: Note that recent work by Liu et al.

Original: (2020) combined normalization with activations.
Revised: (2020) combined normalization with activations.

Original: As described, several forms of smooth activations have been recently proposed in the literature.
Revised: As described, several forms of smooth activations have been recently proposed in the literature.

Original: They all attempt to mimic the shape of ReLU but with a smoother function.
Revised: They all attempt to mimic the shape of ReLU but with a smoother function.

Original: They also try to avoid similarity to the smooth Sigmoid, σ ( x )
Revised: They also try to avoid similarity to the smooth Sigmoid, σ ( x )

Original: = 1 / [1 + exp( − x )] , which was considered several decades ago, but was limited  due to diminishing gradients.
Revised: = 1 / [1 + exp( − x )] , which was considered several decades ago, but had limited success due to diminishing gradients.

Original: Many of them can be parameterized by one parameter or more.
Revised: Many of them can be parameterized by one parameter or more.

Original: While some are smooth for all parameter values, others, like SELU, are only smooth with speciﬁc values of the parameters.
Revised: While some are smooth for all parameter values, others, like SELU, are only smooth with speciﬁc values of the parameters.

Original: Let β be the main activation parameter (SELU will also have another parameter λ ).
Revised: Let β be the main activation parameter (SELU will also have another parameter λ ).

Original: Let erf ( · ) denote the standard error function and Φ( · ) the standard normal CDF.
Revised: Let erf ( · ) denote the standard error function and Φ( · ) the standard normal CDF.

Original: Then, activation functions for some smooth activations are given by
Revised: Then, activation functions for some smooth activations are given by

Original: [Equation]
Revised: [Equation]

Original: ELU is SELU with λ = 1 . We generalized SoftPlus, GELU, Mish, and TanhExp by adding the β parameter.
Revised: ELU is SELU with λ = 1 , and CELU is a differentially continuous variant of SELU, shown in Appendix C. We generalized SoftPlus, GELU, Mish, and TanhExp by adding the β parameter.

Original: GELU can be approximated by Swish with GELU β ( x ) ≈ xσ ( (cid:112) 8 /πβx ) .
Revised: GELU can be approximated by Swish with GELU β ( x ) ≈ xσ ( (cid:112) 8 /πβx ) .

Original: Fig.
Revised: Fig.

Original: 1 shows the different activations and their gradients for different values of the parameter β .
Revised: 1 shows the different activations and their gradients for different values of the parameter β .

Original: (More detailed ﬁgures are shown in Appendix C.)
Revised: (More detailed ﬁgures are shown in Appendix C.)

Original: All the activations described thus far share a region in which the activation function limits changes in the signal with a gradient close to 0 (on the left), and a region in which the gradient approaches 1 on the right.
Revised: All the activations described thus far share a region in which the activation function limits changes in the signal with a gradient close to 0 (on the left), and a region in which the gradient approaches 1 on the right.

Original: The width of the middle transition region is a function of the parameter β , where it is wider with smaller β and narrower with greater β .
Revised: The width of the middle transition region is a function of the parameter β , where it is wider with smaller β and narrower with greater β .

Original: For β → ∞ , the activation resembles ReLU, and for β → 0 , it becomes closer to linear.
Revised: For β → ∞ , the activation resembles ReLU, and for β → 0 , it becomes closer to linear.

Original: ) for different β parameter values.

Original: The discontinuous gradient of the ReLU partitions the parameter space into regions, each of which has its own local optimum.
Revised: The discontinuous gradient of the ReLU partitions the parameter space into regions, each of which has its own local optimum.

Original: Many of the local optima may be identical in the overall total objective, but not on how they predict for individual examples.
Revised: Many of the local optima may be identical in the overall total objective, but not on how they predict for individual examples.

Original: Randomness in training(order of examples and updates) leads the model into one of the regions, and locks it in a region of some local optima.
Revised: Randomness in training leads the model into one of the regions, and locks it in a region of some local optima.

Original: Speciﬁcally, sudden jumps in the gradient, like those of ReLU, may lock some hidden units at some state, requiring other units to compensate and move towards their nearest optimum.
Revised: Speciﬁcally, sudden jumps in the gradient, like those of ReLU, may lock some hidden units at some state, requiring other units to compensate and move towards their nearest optima.

Original: If this happens early in training (see, e.g., Achille et al.
Revised: If this happens early in training (see, e.g., Achille et al.

Original: , it determines the trajectory of the optimization based on the examples seen and the updates applied.
Revised: , it determines the trajectory of the optimization based on the examples seen and the updates applied.

Original: Different example and update orders thus lead to different trajectories even for trained instances of identical models.
Revised: Different example and update orders thus lead to different trajectories.

Original: Smoother activations give a training model less opportunities to diverge.
Revised: Smoother activations give a training model less opportunities to diverge.

Original: Wider transition regions allow gradients to change more slowly, and units in the network to gradually adjust to updates and changes.
Revised: Wider transition regions allow gradients to change more slowly, and units in the network to gradually adjust to updates and changes.

Original: Wider transition regions are closer in shape to more reproducible linear functions.
Revised: Wider transition regions are closer in shape to more reproducible linear functions.

Original: However, a good activation must also retain good accuracy.
Revised: However, a good activation must also retain good accuracy.

Original: The Sigmoid was unable to do that.
Revised: The Sigmoid was unable to do that.

Original: Unfortunately, while reproducibility mostly tends to improve with widening the transition region of the activation, accuracy moves mostly in an opposite direction.
Revised: Unfortunately, while reproducibility mostly tends to improve with widening the transition region of the activation, accuracy moves mostly in an opposite direction.

Original: As we lower β and widen the region, we generally trade accuracy for reproducibility.
Revised: As we lower β and widen the region, we generally trade accuracy for reproducibility.

Original: This, however, is not the full picture.
Revised: This, however, is not the full picture.

Original: While moving towards ReLU improves accuracy on datasets we observed, the optimal point is not at the
Revised: While moving towards ReLU improves accuracy on datasets we observed, the optimal point is not at the

Original: ReLU point of β → ∞ but for some ﬁnite β , after which accuracy degrades.
Revised: ReLU point of β → ∞ but for some ﬁnite β , after which accuracy degrades.

Original: This shows that properly designed smooth activations for the speciﬁc dataset can be actually superior to ReLU in both accuracy and reproducibility.
Revised: This shows that properly designed smooth activations for the speciﬁc dataset can be actually superior to ReLU in both accuracy and reproducibility.

Original: They also provide a knob to trade off between the two.
Revised: They also provide a knob to trade off between the two.

Original: Similar behavior is also observed in the opposite direction, where the best reproducibility for some datasets may be achieved for some β > 0 , worsening for smaller values.
Revised: Similar behavior is also observed in the opposite direction, where the best reproducibility for some datasets may be achieved for some β > 0 , worsening for smaller values.

Original: The overall tradeoffs are dataset, model architecture and model conﬁguration dependent.
Revised: The overall tradeoffs are dataset, model architecture and model conﬁguration dependent.

Original:  S ME LU TO THE RESCU
Revised:  S ME LU TO THE RESCU

Original: Swish, GELU, Mish, and TanhExp, all have a non-monotonic region on the left of the transition region, neighboring the signal suppression region.
Revised: Swish, GELU, Mish, and TanhExp, all have a non-monotonic region on the left of the transition region, neighboring the signal suppression region.

Original: This may be vulnerable to input irregularities, possibly less interpretable, and perhaps suboptimal.
Revised: This may be vulnerable to input irregularities, possibly less interpretable, and perhaps suboptimal.

Original: SoftPlus is monotonic, but the wider the transition region, the farther it is from the origin, degrading its accuracy.
Revised: SoftPlus is monotonic, but the wider the transition region, the farther it is from the origin, degrading its accuracy.

Original: In addition, for all these activations, there are no clean stop and pass regions like in ReLU, except for very large β .
Revised: In addition, for all these activations, there are no clean stop and pass regions like in ReLU, except for very large β .

Original: Asymptotically, for β → ∞ , all tend to look like ReLU, but for wider transition regions, which are good for reproducibility, they do not reach the ReLU extremes on the left and right.
Revised: Asymptotically, for β → ∞ , all tend to look like ReLU, but for wider transition regions, which are good for reproducibility, they do not reach the ReLU extremes on the left and right.

Original: Furthermore, all the activations considered require complex mathematical functions, such as exponents, some of which may require more complex hardware implementations. Such implementations could be  costly in huge scale systems. 
Revised: Furthermore, all the activations considered require complex mathematical functions, such as exponents. Such functions require more complex hardware implementations, that could be very costly in huge scale systems. The computationally heavy functions could also slow down training and inference.

Original: We now show a very simple smooth ReLU activation, SmeLU , that beneﬁts from smoothness, monotonicity, clean stop and pass regions, a very simple mathematical implementation, and comparable or better accuracy-reproducibility tradeoffs.
Revised: We now show a very simple smooth ReLU activation, SmeLU , that beneﬁts from smoothness, monotonicity, clean stop and pass regions, a very simple mathematical implementation, and comparable or better accuracy-reproducibility tradeoffs.

Original: We show that the same design is extendible to a generalization of this activation as well as to more sophisticated extensions.
Revised: We show that the same design is extendible to a generalization of this activation as well as to more sophisticated extensions.

Original: SmeLU: In some literature, SoftPlus is referred to as Smoothed ReLU (see, e.g. Xie et al.
Revised: SmeLU: In some literature, SoftPlus is referred to as Smoothed ReLU.

Original: Recent work (Bresler & Nagaraj, 2020) also considered smoothing ReLU using initial coefﬁcients of its Fourier series.
Revised: Recent work (Bresler & Nagaraj, 2020) also considered smoothing ReLU using initial coefﬁcients of its Fourier series.

Original: ReLU is piecewise continuous linear, but with a gradient discontinuity.
Revised: ReLU is piecewise continuous linear, but with a gradient discontinuity.

Original: As shown in Chen & Ho (2018); Montufar et al.
Revised: As shown in Chen & Ho (2018) Montufar et al.

Original: (2014), the concept of piecewise linear can be extended to multiple pieces.
Revised: (2014), the concept of piecewise linear can be extended to multiple pieces.

Original: We take a further step here and construct a piecewise continuous activation but also with a continuous gradient, by simply enforcing these conditions.
Revised: We take a further step here and construct a piecewise continuous activation but also with a continuous gradient, by simply enforcing these conditions.

Original: The general idea is to deﬁne the activation as a piecewise linear and quadratic (resembling Huber loss (Huber, 1992) on one side).
Revised: The general idea is to deﬁne the activation as a piecewise linear and quadratic (resembling Huber loss (Huber, 1992) on one side).

Original: On both sides, we match ReLU and its gradients, and we ﬁt a quadratic middle region between.
Revised: On both sides, we match ReLU and its gradients, and we ﬁt a quadratic middle region between.

Original: We deﬁne β to be the half-width of a symmetric transition region around x = 0 .
Revised: We deﬁne β to be the half-width of a symmetric transition region around x = 0 .

Original: Note that this deﬁnition of SmeLU uses a parameter β that is reciprocal to the β in Swish and other activations like it in the sense that larger β here gives a wider transition region, and a smaller β a narrower one.
Revised: Note that this deﬁnition of SmeLU uses a parameter β that is reciprocal to the β in Swish and other activations like it in the sense that larger β here gives a wider transition region, and a smaller β a narrower one.

Original: To the left of the transition region, y = 0 , to its right, y = x .
Revised: To the left of the transition region, y = 0 . To its right, y = x .

Original: Then, we enforce the gradients to be continuous by
Revised: Then, we enforce the gradients to be continuous by

Original: [Equation]
Revised: [Equation]

Original: Applying the conditions gives the SmeLU activation
Revised: Applying the conditions gives the SmeLU activation

Original: [Equation]
Revised: [Equation]

Original: Fig.
Revised: Fig.

Original: 1 (also Fig. 12) shows SmeLU and its continuous gradient as function of β .
Revised: 1 (also Fig. 12) shows SmeLU and its continuous gradient as function of β .

Original: It matches ReLU on both sides, but gives a gentle  transition region in the middle, which is wider with greatervalues of β and narrower with smaller β . 
Revised: It matches ReLU on both sides, giving a gentle middle transition region, wider with greater β and narrower with smaller β . SmeLU is a convolution of a ReLU with a box of magnitude 1 / (2 β ) in [ − β, β ] .

Revised: Its gradient is a hard Sigmoid.

Original: As with other smooth activations, we observe that with greater β , reproducibility is improved but at the expense of accuracy.
Revised: As with other smooth activations, we observe that with greater β , reproducibility is improved possibly at the expense of accuracy.

Original: Again, the optimal accuracy on datasets we experimented with is for some β > 0 .
Revised: Optimal accuracy on datasets we experimented with is for some β > 0 .

Original: At the limit β → 0 , SmeLU becomes ReLU.
Revised: For β → 0 , SmeLU becomes ReLU.

Original: Fig.demonstrates the surface of the output of a network with two inputs as function of the values of these inputs, with ReLU, SmeLU with different β , and without an activation (top).
Revised: Fig. 2 demonstrates the surface of the output of a network with two inputs as function of the values of these inputs, with ReLU, SmeLU with different β , and no activation (top).

Original: Surfaces are obtained with weight normalization.
Revised: Surfaces are obtained with weight normalization.

Original: The bottom row shows SmeLU with β = 0 .
Revised: The bottom row shows SmeLU with β = 0 .

Original: 5 for different L 2 weight norms.
Revised: 5 for different L 2 weight norms.

Original: Examples of effects of normalization, clipping, and other activations on this surface are shown in Figs.
Revised: Examples of effects of normalization, clipping, and other activations on this surface are shown in Figs.

Original: 5-7 in Appendix A. Fig.
Revised: 5-7 in Appendix A. Fig.

Original: 5 demonstrates the importance of normalization if activations are clipped to some cap.
Revised: 5 demonstrates the importance of normalization with clipping activations.

Original: Fig.
Revised: Fig.

Original: 6 shows similarbehavior of surfaces of other smooth activations.
Revised: 6 shows similar surfaces of other smooth activations.

Original: Fig.shows effects of (other) normalizations on no activations, ReLU and SmeLU.
Revised: Fig. 7 shows effects of (other) normalizations.

Original: In Fig 2, we observe that ReLU exhibits multiple valleys (that lead to multiple optima in higher dimensions).
Revised: In Fig 2, we observe that ReLU exhibits multiple valleys (that lead to multiple optima in higher dimensions).

Original: Smoothing the activation by increasing β smoothes these valleys.
Revised: Smoothing the activation by increasing β smoothes these valleys.

Original: Thus with a smooth activation, there are fewer optima that the model can ﬁnd while training, each spanning a larger region around it.
Revised: Thus with a smooth activation, there are fewer optima that the model can ﬁnd while training, each spanning a larger region around it.

Original: However, activations that are too smooth resemble linear models with worse accuracy.
Revised: However, activations that are too smooth resemble linear models with worse accuracy.

Original: The parameter of SmeLU (as well as of other smooth activations) interacts with the normalization applied, as demonstrated in the second row of Fig.
Revised: The parameter of SmeLU (as well as of other smooth activations) interacts with the normalization applied, as demonstrated in the second row of Fig.

Original: Smoothing of the surface can be achieved by either tuning β or the norm of the normalization applied.
Revised: Smoothing of the surface can be achieved by either tuning β or the norm of the normalization applied.

Original: This, however, is only possible with smooth activations.
Revised: This, however, is only possible with smooth activations.

Original: Tuning weight norm with ReLU does not smoothen the surface.
Revised: Tuning weight norm with ReLU does not smoothen the surface.

Original: Thus the smooth activation allows for smoother surfaces whose smoothness (and good reproducibility behavior) can be tuned by either the parameters of the activation or by normalization.
Revised: Thus the smooth activation allows for smoother surfaces whose smoothness (and good reproducibility behavior) can be tuned by either the parameters of the activation or by normalization.

Original: For consistency, one can keep the normalization to a norm of v = 1 , and study the tradeoff as function of β , or apply SmeLU with no normalization (which may affect accuracy and reproducibility) and no clipping.
Revised: For consistency, one can keep the normalization to a norm of v = 1 , and study the tradeoff as function of β , or apply SmeLU with no normalization (which may affect accuracy and reproducibility) and no clipping.

Original: An approach that may improve the overall PD/accuracy tradeoff is to use wider values of β for layers closer to the inputs and narrower β values for layers closer to the output.
Revised: An approach that may improve the overall PD/accuracy tradeoff is to use wider values of β for layers closer to the inputs and narrower β values for layers closer to the output.

Original: Reproducibility seems to be dominated by rich parameter sets, and applying wider β for layers that control such sets could improve it.
Revised: Reproducibility seems to be dominated by rich parameter sets, and applying wider β for layers that control such sets could improve it.

Original: Then, layers closer to the output can focus on better prediction accuracy.
Revised: Then, layers closer to the output can focus on better prediction accuracy.

Original: Alternatively, one can learn the values of β for the whole model, per layer, or even for individual units, as part of the network training.
Revised: Alternatively, one can learn the values of β for the whole model, per layer, or even for individual units, as part of the network training.

Original: Some insights on how this can be done are given in Appendix B.
Revised: Some insights on how this can be done are given in Appendix B.

Original: Generalized SmeLU: The constraints in (8) can be generalized to a wider family of activations, including ones such as a leaky SmeLU, by allowing different gradients g − (cid:54)
Revised: Generalized SmeLU: The constraints in (8) can be generalized to a wider family of activations, including ones such as a leaky SmeLU, by allowing different gradients g − (cid:54)

Original: = 

Revised: =

Original: to the left and g + > g − to the right of the quadratic transition region, and allowing for an asymmetric transition region, and for a bias shift.
Revised: to the left and g + > g − to the right of the quadratic transition region, and allowing for an asymmetric transition region, and for a bias shift.

Original: The conditions are then
Revised: The conditions are then

Original: [Equation]
Revised: [Equation]

Original:  The ﬁve parameters { α, β, g − , g + , t } deﬁne a generalized SmeLU .
Revised: [Equation] The ﬁve parameters { α, β, g − , g + , t } deﬁne a generalized SmeLU .

Original: Typically (but not necessarily), α, β > 0 , and t ≤ 0 possibly to allow the activation to cross through the origin.
Revised: Typically (but not necessarily), α, β > 0 , and t ≤ 0 possibly to allow the activation to cross through the origin.

Original: For a leaky activation, g − > 0 , but we can also have a negative slope on the left (where then we relax the monotonicity requirement).
Revised: For a leaky activation, g − > 0 , but we can also have a negative slope on the left (where then we relax the monotonicity requirement).

Original: Enforcing (10) and continuity of the function at x = − α and x = β , we come to the deﬁnition of the generalized SmeLU activation
Revised: Enforcing (10) and continuity of the function at x = − α and x = β , we come to the deﬁnition of the generalized SmeLU activation

Original: [Equation]
Revised: [Equation]

Original: wherea = g
Revised: where

Original: [Equation]
Revised: [Equation]

Original: The ﬁfth pair of graphs in Fig.
Revised: The ﬁfth pair of graphs in Fig.

Original: 1 (also Fig. 12) shows examples of generalized SmeLU with t =and asymmetric transition regions, including leaky versions with g − > 0 and also versions with g − ≤ 0 .
Revised: 1 (also Fig. 12) shows examples of generalized SmeLU with t =and asymmetric transition regions, including leaky versions with g − > 0 and also versions with g − ≤ 0 .

Original: SmeLU is a special case of the generalized version, with g − = 0 , g + = 1 , α = β , and t = 0 .
Revised: SmeLU is a special case of the generalized version, with g − = 0 , g + = 1 , α = β , and t = 0 .

Original: Some additional special cases are reviewed in Appendix D.
Revised: Some additional special cases are reviewed in Appendix D.

Original: The hyper-parameters of generalized SmeLU can be learned in training of a deep network together with the model parameters, either for the whole model, per layer, and even per neuron, as noted for SmeLU and discussed in Appendix B. Experiments demonstrate that for some datasets, layers closer to inputs may learn a negative g − , implying that they learn some sparsiﬁcation of the input, for better model accuracy.
Revised: The hyper-parameters of generalized SmeLU can be learned in training of a deep network together with the model parameters, either for the whole model, per layer, and even per neuron, as noted for SmeLU and discussed in Appendix B. Experiments demonstrate that for some datasets, layers closer to inputs may learn a negative g − , implying that they learn some sparsiﬁcation of the input, for better model accuracy.

Original: Layers closer to the output learned, in similar cases, slopes closer to 0 .
Revised: Layers closer to the output learned, in similar cases, slopes closer to 0 .

Original: Further Generalization - RESCU:
Revised: Further Generalization - RESCU:

Original: The general idea to derive the generalized SmeLU can be extended in various ways.
Revised: The general idea to derive the generalized SmeLU can be extended in various ways.

Original: Generally, we deﬁne REctiﬁed Smooth Continuous Unit (RESCU) as any function that consists of multiple smooth pieces, and is continuously differentiable.
Revised: Generally, we deﬁne REctiﬁed Smooth Continuous Unit (RESCU) as any function that consists of multiple smooth pieces, and is continuously differentiable.

Original: At any point, α , y ( α − ) =
Revised: At any point, α , y ( α − ) =

Original: y ( α + ) , as well as ( dy/dx ) | α − =
Revised: y ( α + ) , as well as ( dy/dx ) | α − =

Original: ( dy/dx )
Revised: ( dy/dx )

Original: | α + .
Revised: | α + .

Original: We can extend the generalized SmeLU by additional linear pieces and quadratic pieces connecting between any pair of linear pieces with different slopes.
Revised: We can extend the generalized SmeLU by additional linear pieces and quadratic pieces connecting between any pair of linear pieces with different slopes.

Original: For example, the non-monotonicity of Swish (and alike) can be achieved by adding a concave piece to the left of the convex quadratic piece.
Revised: For example, the non-monotonicity of Swish (and alike) can be achieved by adding a concave piece to the left of the convex quadratic piece.

Original: At the point the gradient becomes 0 it can be joined by a linear piece to its left with 0 gradient.
Revised: At the point the gradient becomes 0 it can be joined by a linear piece to its left with 0 gradient.

Original: Alternatively, the same concept can be used by joining pieces of more complex mathematical functions, ensuring smoothness at the connections between the pieces.
Revised: Alternatively, the same concept can be used by joining pieces of more complex mathematical functions, ensuring smoothness at the connection points between the pieces.

Original: This makes activations like Swish special cases of RESCU with a single piece.
Revised: This makes activations like Swish special cases of RESCU with a single piece.

Original: SELU with β = 1 is another example with two pieces.
Revised: SELU with β = 1 is another example with two pieces.

Original: Another example is the REctiﬁed Sigmoid Continuous Unit , which consists of a Sigmoid on the left, but linear on the right
Revised: Another example is the REctiﬁed Sigmoid Continuous Unit , which consists of a Sigmoid on the left, but linear on the right

Original: [Equation]
Revised: [Equation]

Original: We can apply the concept with polynomials of higher degree as well.
Revised: We can apply the concept with polynomials of higher degree as well.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Real Data: We tested various smooth activations on a large private dataset for ad Click-ThroughRate (CTR) prediction for sponsored advertisement.
Revised: Criteo: We evaluated SmeLU and Swish on the benchmark Criteo Display Advertising Challenge dataset 1 . The dataset provides a binary classiﬁcation task with 45 M examples, which are split to 37 M training examples, and validation and test sets of 4 M examples each. Each example has 13 integer features and 26 categorical features. Models have 3 fully connected hidden layers of dimension 2572, 1454, 1596, respectively.

Revised: Each categorical feature x k is hashed into N k buckets.

Revised: If N k < 110 the hash bin is encoded as a one-hot vector, otherwise it is embedded as a vector of dimension d k .

Revised: Values of N k and d k are taken from Ovadia et al.

Revised: (see Appendix E).

Revised: Each model is trained for one epoch using the Adam optimizer.

Revised: Unlike Ovadia et al.

Revised: (2019), input features are not fed into a batch-norm layer.

Revised: Integer features are log-square-transformed.

Revised: We trained models with ReLU, SmeLU and Swish activations.

Revised: Each experiment consisted of 40 independent runs.

Revised: For each run, models are initialized to different random values, and a different random shufﬂe is applied to the data.

Revised: Training mini-batches of 1024 examples were used.

Revised: Table 1 shows AUC and PDs; ∆

Revised: [Table]

Revised: (absolute) and ∆ r 1 (relative), on the test data.

Revised: ReLU exhibits huge PDs of over 36% .

Revised: While only slightly improving AUC, around 40% decrease in both PD measurements as well as substantial order of magnitude reductions in AUC and PD standard deviations, are observed with SmeLU.

Revised: Tablesand 3 in Appendix E show that similar less extreme improvements are attained when training data is not shufﬂed, or models are identically initialized.

Revised: Baseline ReLU PDs are lower but still over 20% .

Revised: SmeLU still improves PD (about 12% without shufﬂing) and AUC/PD standard deviations over ReLU.

Revised: With Swish, on this dataset, we were unable to obtain comparable AUC and PD values.

Revised: Real Data: We tested various smooth activations on a large private dataset for ad Click-ThroughRate (CTR) prediction for sponsored advertisement.

Original: We constructed a simple 6 layer neural network with dimensions [1024 , 512 , 256 , 128 , 64 , 16] with 5 informative features that are used as learned embedding vectors of different dimensions, providing a total of 240 inputs.
Revised: We constructed a simple 6 layer fully connected network with dimensions [1024 , 512 , 256 , 128 , 64 , 16] with 5 informative features that are used as learned embedding vectors of different dimensions, providing a total of 240 inputs.

Original: Models are  trained  on hundreds of billions of examples, using online mini-batch, single pass over the data, optimizing with the Adagrad optimizer (Duchi et al., 2011).
Revised: Models are identically initialized, trained with data-shufﬂing on hundreds of billions of examples, using online mini-batch, single pass over the data, optimizing with the Adagrad optimizer (Duchi et al., 2011).

Original: In such cases, resources limit the quantity of models trained, so we use M = 2 .
Revised: Resources limit the quantity of models trained, so we use M = 2 .

Original: Inference is applied to examples prior to training on them, and progressive validation (Blum et al., 1999), as commonly used for this problem (McMahan et al., 2013), is applied to measure validation logarithmic and ranking loss on predictions.
Revised: Inference is applied to examples prior to training on them, and progressive validation (Blum et al., 1999), as commonly used for this problem (McMahan et al., 2013), is applied to measure validation logarithmic and ranking losses on predictions.

Original: Fig.
Revised: Fig.

Original: 3 reports relative PD; ˜∆ r 1 , (as deﬁned for binary labels) as function of both average validation logarithmic (cross-entropy) loss (left) and ranking loss (right).
Revised: 3 reports relative PD; ˜∆ r 1 , (as deﬁned for binary labels) as function of both average validation logarithmic (cross-entropy) loss (left) and ranking loss (right).

Original: Losses are expressed as percentage of the baseline ReLU model.
Revised: Losses are expressed as percentage of the baseline ReLU model.

Original: Relative PD is expressed as percentage of the actual positive label prediction.
Revised: Relative PD is expressed as percentage of the actual positive label prediction.

Original: On the graph, moving left implies better accuracy, and moving down implies better reproducibility.
Revised: On the graph, moving left implies better accuracy, and moving down implies better reproducibility.

Original: For smooth activations, the different points are the result of different β .
Revised: For smooth activations, the different points are the result of different β .

Original: Moving down at least before reaching an optimum implies larger β for SmeLU and smaller β for the other activations.
Revised: Moving down at least before reaching an optimum implies larger β for SmeLU and smaller β for the other activations.

Original: In such a large scale system, we observe major relative PD values of over 10% with ReLU.
Revised: In such a large scale system, we observe major relative PD values of over 10% with ReLU ( 12% when models initialized equally, and 13% when initialized differently).

Original: Smooth activations are substantially better on PD and better on the tradeoffs than ReLU.
Revised: Smooth activations are substantially better on PD and better on the tradeoffs than ReLU.

Original: Among themselves, the tradeoffs are more comparable, where SmeLU, GELU and Swish appear to be slightly better on this data than Mish and Tanhexp.
Revised: Among themselves, the tradeoffs are more comparable, where SmeLU, GELU and Swish appear to be slightly better on this data than Mish and Tanhexp.

Original: SoftPlus appears to push towards better PD, but at the expense of accuracy with unacceptable trade-offs.
Revised: SoftPlus appears to push towards better PD, but at the expense of accuracy with unacceptable trade-offs (in such a system even a fraction of percent can make a difference).

Revised: Continuously differential SELU appears inferior in accuracy and PD.

Original: SmeLU also seems to be stronger on PD, whereas Swish on accuracy.
Revised: SmeLU also seems to be stronger on PD, whereas Swish on accuracy.

Original: Results shown use weight normalization with norm v = 1 without clipping.
Revised: Results shown use weight normalization with norm v = 1 without clipping.

Original: We observe similar relative behavior without normalization (no clipping).
Revised: We observe similar relative behavior without normalization (no clipping).

Original: However, all PD’s, including baseline, increase.
Revised: However, PD’s of smooth activations increase, but are still better than the ReLU baseline.

Original: Benchmark MNIST Dataset: We tested PD and accuracywith ReLU, Swish, SoftPlus, and SmeLU on the MNIST dataset (LeCun, 1998).
Revised: Benchmark MNIST Dataset: We tested PD and accuracy on the MNIST dataset (LeCun, 1998).

Original: Models were implemented with Tensorﬂow Keras.
Revised: Models were implemented with Tensorﬂow Keras.

Original: Networks were simple 2 layer networks, with layers of width 1200 .
Revised: Networks were simple 2 layer networks, with layers of width 1200 .

Original: We regularized with dropout (input 0 .
Revised: We regularized with dropout (input 0 .

Original: 2 , layers 0 .
Revised: 2 , layers 0 .

Original: 5 ) and image augmentation in training (starting from epoch 2 , two dimensional independent random shifts by {− 3 , − 2 , − 1 , 0 , 1 , 2 , 3 } pixels take place with probability 0 .
Revised: 5 ) and image augmentation in training (starting from epoch 2 , two dimensional independent random shifts by {− 3 , − 2 , − 1 , 0 , 1 , 2 , 3 } pixels take place with probability 0 .

Original: Hidden weights were  initialized with the default Keras uniform random, and biases to 0 .
Revised: Hidden weights were identically initialized with the default Keras uniform random, and biases to 0 .

Original: For AdaGrad, we used learning rate 0 .
Revised: For AdaGrad, we used learning rate 0 .

Original: 02 , accumulator initialization 0 .
Revised: 02 , accumulator initialization 0 .

Original: 1 , and 150 epochs without weight normalization.
Revised: 1 , and 150 epochs without weight normalization.

Original: For SGD, we used learning rate 0 .
Revised: For SGD, we used learning rate 0 .

Original: 01 , momentum 0 .
Revised: 01 , momentum 0 .

Original: 9 , and 50 epochs.
Revised: 9 , and 50 epochs.

Revised: Batch size is 32 .

Original: For each experiment we trained M = 12 models.
Revised: For each experiment, we trained M = 12 models.

Original: Fig.
Revised: Fig.

Original: 4 shows PD ∆ 1 as function of error rate for both conﬁgurations, AdaGrad (left) and SGD (right).
Revised: 4 shows PD ∆ 1 as function of error rate for both conﬁgurations, AdaGrad (left) and SGD (right).

Original: Additional graphs are in Appendix E. While in smaller scale, PD values may not be as large as they are in large scale data,  we still see PD change substantially relative to its lowest values (over 10% for AdaGrad, over 30% for SGD).
Revised: Graphs for other PD metrics, with similar behavior, are in Appendix F. PD values may not be as large as they are in large scale data, but we still see PD change substantially relative to its lowest values (over 10% for AdaGrad, over 30% for SGD).

Original: As in large scale, ReLU’s PD is much worse than those achievable by smooth activations, and among themselves, smooth activations are comparable.
Revised: As in large scale, ReLU’s PD is much worse than those achievable by smooth activations, and among themselves, smooth activations are comparable.

Original: With SGD, however, Swish appears inferior on PD to SmeLU and SoftPlus.
Revised: With SGD, however, Swish appears inferior on PD.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We studied irreproducibility in deep networks, showing that the popular ReLU activation is a major factor in exacerbating prediction differences.
Revised: We studied irreproducibility in deep networks, showing that the popular ReLU activation is a major factor in exacerbating prediction differences.

Original: Smooth activations allow substantial improvement, giving better accuracy-reproducibility tradeoffs.
Revised: Smooth activations allow substantial improvement, giving better accuracy-reproducibility tradeoffs.

Original: We introduced SmeLU, and its generalizations, speciﬁcally demonstrating that superior accuracy-reproducibility tradeoffs are achievable even with very simple inexpensive to implement activations, whose performance on both accuracy and PD do not fall from and can be better than those of more complex expensive to implement, and nonmonotonic, smooth activations.
Revised: We introduced SmeLU, and its generalizations, speciﬁcally demonstrating that superior accuracy-reproducibility tradeoffs are achievable even with very simple inexpensive to implement activations, whose performance on both accuracy and PD do not fall from and can be better than those of more complex expensive to implement, and nonmonotonic, smooth activations.

Original: We empircially demonstrated these tradeoffs, showing that proper smooth activations are superior to ReLU.
Revised: We empircially demonstrated these tradeoffs, showing that proper smooth activations are superior to ReLU.

Original: For a speciﬁc dataset, model architecture, and model conﬁguration, one smooth activation may be better than the other.
Revised: For a speciﬁc dataset, model architecture, and model conﬁguration, one smooth activation may be better than the other.

Original: Although, we have shown that the much simpler SmeLU can be comparable or superior to the more complex activations on some examples.
Revised: We have shown, however, that the much simpler SmeLU can be comparable or superior to the more complex activations.

Original: A S MOOTH A CTIVATIONS AND N ORMALIZATION I NDUCED S URFACES
Revised: A S MOOTH A CTIVATIONS AND N ORMALIZATION I NDUCED S URFACES

Original: In this appendix, we demonstrate effects of smooth activations and normalization on the objective surfaces, enhancing on the description in Sections 3-4, and on the illustrations of Fig.
Revised: In this appendix we demonstrate effects of smooth activations and normalization on the objective surfaces, enhancing on the description in Sections 3- and on the illustrations of Fig.

Original: Fig.
Revised: Fig.

Original: 5 expands on Fig.
Revised: 5 expands on Fig.

Original: 2, showing objective surfaces for different activations, parameters and normalizations.
Revised: 2, showing objective surfaces for different activations, parameters and normalizations.

Original: It shows the transition from a non-smooth ReLU to a linear activation through different parameters of SmeLU.
Revised: It shows the transition from a non-smooth ReLU to a linear activation through different parameters of SmeLU.

Original: As we increase β , the objective surface becomes smoother.
Revised: As we increase β , the objective surface becomes smoother.

Original: It also demonstrates the importance of normalization, especially when activations are clipped to some cap.
Revised: It also demonstrates the importance of normalization, especially when activations are clipped to some cap.

Original: The ﬁrst row duplicates that of Fig.
Revised: The ﬁrst row duplicates that of Fig.

Original: 2 for completeness.
Revised: 2 for completeness.

Original: The second row shows the transition of a weight normalized clipped activation from ReLU to linear through widening SmeLUs.
Revised: The second row shows the transition of a weight normalized clipped activation from ReLU to linear through widening SmeLUs.

Original: With weight normalization there are slight deviations from the top row, but the behavior appears rather similar.
Revised: With weight normalization there are slight deviations from the top row, but the behavior appears rather similar.

Original: The third row shows the same transition without weight normalization and clipping.
Revised: The third row shows the same transition without weight normalization and clipping.

Original: In order to be as smooth, a larger β must be used than in the ﬁrst row.
Revised: In order to be as smooth, a larger β must be used than in the ﬁrst row.

Original: This result is in line with the experiments reported in
Revised: This result is in line with the experiments reported in

Original: Section 5 on real data, where PD values increased in all cases with the same β parameters if weight normalization was not applied.
Revised: Section 5 on real data, where PD values increased in all cases with the same β parameters if weight normalization was not applied.

Original: Furthermore, the objective variance is now much larger than in the ﬁrst row.
Revised: Furthermore, the objective variance is now much larger than in the ﬁrst row.

Original: When clipping is added, as shown in the last row, the effect of the smoothness disappears, where numerous peaks appear in all activations.
Revised: When clipping is added, as shown in the last row, the effect of the smoothness disappears, where numerous peaks appear in all activations.

Original: With large β , the effectiveness of the activation altogether also appears to diminish.
Revised: With large β , the effectiveness of the activation altogether also appears to diminish.

Original: This happens even with the linear activation.
Revised: This happens even with the linear activation.

Original: As observed in the second row, weight normalization diminishes the effect of clipping.
Revised: As observed in the second row, weight normalization diminishes the effect of clipping.

Original: Fig.
Revised: Fig.

Original: 6 shows surfaces for Swish, SoftPlus, GELU and SELU with weight normalization, without clipping.
Revised: 6 shows surfaces for Swish, SoftPlus, GELU and SELU with weight normalization, without clipping.

Original: For this simple two dimensional input example, the smooth activations appear rather similar to the surfaces of SmeLU.
Revised: For this simple two dimensional input example, the smooth activations appear rather similar to the surfaces of SmeLU.

Original: As mentioned earlier, the smoothness is reversed as a function of β from SmeLU, with the larger β values being less smooth.
Revised: As mentioned earlier, the smoothness is reversed as a function of β from SmeLU, with the larger β values being less smooth.

Original: Since GELU can be approximated by
Revised: Since GELU can be approximated by

Original: Swish with a larger parameter, for the same β , Swish is smoother.
Revised: Swish with a larger parameter, for the same β , Swish is smoother.

Original: Very similar surfaces are observed for Mish and Tanhexp.
Revised: Very similar surfaces are observed for Mish and Tanhexp.

Original: SELU is smooth for β = 1 , but is clearly not smooth for other values of β .
Revised: SELU is smooth for β = 1 , but is clearly not smooth for other values of β .

Original: This is observed in Fig.
Revised: This is observed in Fig.

Original: 6 for β = 10 .
Revised: 6 for β = 10 .

Original: Fig.
Revised: Fig.

Original: 7 demonstrates the effect of normalization (weight and layer) on no activation and on ReLU, and the effects of layer normalization (Ba et al., 2016) on SmeLU.
Revised: 7 demonstrates the effect of normalization (weight and layer) on no activation and on ReLU, and the effects of layer normalization (Ba et al., 2016) on SmeLU.

Original: Unlike weight normalization, layer normalization changes the shape of the surface from the shape obtained when there is no normalization.
Revised: Unlike weight normalization, layer normalization changes the shape of the surface from the shape obtained when there is no normalization.

Original: While the type of norm changes the shape of the surface obtained, unlike smooth activations, the norm value v has no effect on the shape of the surface observed for either no activation or ReLU.
Revised: While the type of norm changes the shape of the surface obtained, unlike smooth activations, the norm value v has no effect on the shape of the surface observed for either no activation or ReLU.

Original: (It does affect the magnitude of the objective, though.)
Revised: (It does affect the magnitude of the objective, though.)

Original: For smooth activations, on the other hand, as observed in Fig. 2, the magnitude v of the norm does affect the smoothness of the surface also with 
Revised: For smooth activations, on the other hand, as observed in Fig. 2, the magnitude v of the norm does affect the smoothness of the surface also with layer normalization, as demonstrated for SmeLU on the bottom row, with smoother objective as the norm is smaller.

Original: B L EARNING S ME LU P ARAMETERS
Revised: B L EARNING S ME LU P ARAMETERS

Original: As we observed, β in smooth activations roughly serves as a tuning knob between accuracy and reproducibility.
Revised: As we observed, β in smooth activations roughly serves as a tuning knob between accuracy and reproducibility.

Original: However, the optimal accuracy is attained away from the end point ( 0 for SmeLU, ∞ for Swish and other smooth activations).
Revised: However, the optimal accuracy is attained away from the end point ( 0 for SmeLU, ∞ for Swish and other smooth activations).

Original: Similarly, there are no guarantees that the wider is the activation the more reproducible it is, at the other extreme.
Revised: Similarly, there are no guarantees that the wider is the activation the more reproducible it is, at the other extreme.

Original: It is expected that the optimal points will be properties of both the actual dataset and the model architecture.
Revised: It is expected that the optimal points will be properties of both the actual dataset and the model architecture.

Original: For the generalized SmeLU and wider RESCU generalization, more parameters should be tuned.
Revised: For the generalized SmeLU and wider RESCU generalization, more parameters should be tuned.

Original: While tuning can optimize various tasks, we focus here on tuning for accuracy and reproducibility.
Revised: While tuning can optimize various tasks, we focus here on tuning for accuracy and reproducibility.

Original: The parameter β or the parameters of the generalized SmeLU ( α, β, g − , g + , t ) can be trained and learned as part of the training process of a deep network.
Revised: The parameter β or the parameters of the generalized SmeLU ( α, β, g − , g + , t ) can be trained and learned as part of the training process of a deep network.

Original: Learning can optimize accuracy, reproducibility or any other objective.
Revised: Learning can optimize accuracy, reproducibility or any other objective.

Original: Training can learn a single parameter set for the whole network, a single parameter set per layer, or a single parameter set per neuron.
Revised: Training can learn a single parameter set for the whole network, a single parameter set per layer, or a single parameter set per neuron.

Original: Depending on the task and number of updates, learning rates can be tuned for each of the learned hyper-parameters.
Revised: Depending on the task and number of updates, learning rates can be tuned for each of the learned hyper-parameters.

Original: Initialization can initialize to known good parameters.
Revised: Initialization can initialize to known good parameters.

Original: For the generalized SmeLU, we can initialize values to those of a reasonable SmeLU, e.g., with g − = 0 , g + = 1 , t = 0 , and α = β = 0 .
Revised: For the generalized SmeLU, we can initialize values to those of a reasonable SmeLU, e.g., with g − = 0 , g + = 1 , t = 0 , and α = β = 0 .

Original: Empirically, we observe, when optimizing for accuracy, that each layer learns a different set of parameters.
Revised: Empirically, we observe, when optimizing for accuracy, that each layer learns a different set of parameters.

Original: Speciﬁcally, we observed that layers closer to the input tend to learn a negative g − .
Revised: Speciﬁcally, we observed that layers closer to the input tend to learn a negative g − .

Original: The ﬁrst hidden layer that is connected to the input (especially in systems where the input consists of learned embeddings) feeds input that is usually more symmetric around the origin, while layers farther from the input see biased inputs.
Revised: The ﬁrst hidden layer that is connected to the input (especially in systems where the input consists of learned embeddings) feeds input that is usually more symmetric around the origin, while layers farther from the input see biased inputs.

Original: With the learned g −  it appears that the layer closest to the input attempts to learn sparsiﬁcation, forcing its inputs either to an extreme or to the origin.
Revised: With the learned g − , it appears that the layer closest to the input attempts to learn sparsiﬁcation, forcing its inputs either to an extreme or to the origin.

Original: Learning hyper-parameters for accuracy can be done simultaneously with training the parameters of the networks.
Revised: Learning hyper-parameters for accuracy can be done simultaneously with training the parameters of the networks.

Original: Alternatively, this can be done in one training session.
Revised: Alternatively, this can be done in one training session.

Original: Then, the hyper-parameters can be ﬁxed, and used for training the parameters.
Revised: Then, the hyper-parameters can be ﬁxed, and used for training the parameters.

Original: The procedure can be further applied to learning functional forms of the activation as done by Ramachandran et al.
Revised: The procedure can be further applied to learning functional forms of the activation as done by Ramachandran et al.

Original: One can apply such a methodology on a RESCU activation, where the pieces are learned under the constraints of continuous function and gradient between the pieces.
Revised: One can apply such a methodology on a RESCU activation, where the pieces are learned under the constraints of continuous function and gradient between the pieces.

Original: Optimization of the parameters can combine an objective for both accuracy and reproducibility, as a mean or weighted sum of both objectives.
Revised: Optimization of the parameters can combine an objective for both accuracy and reproducibility, as a mean or weighted sum of both objectives.

Original: However, this must be done carefully, as these objectives may be conﬂicting.
Revised: However, this must be done carefully, as these objectives may be conﬂicting.

Original: Optimizing the activation for reproducibility may be more involved, as gradient methods with the objective loss used for training neural networks tend to try to improve the model’s accuracy.
Revised: Optimizing the activation for reproducibility may be more involved, as gradient methods with the objective loss used for training neural networks tend to try to improve the model’s accuracy.

Original: There are several options for training the activation parameters to improve reproducibility.
Revised: There are several options for training the activation parameters to improve reproducibility.

Original: We can include reproducibility optimization of the parameters in training the full network, or train the model ofﬂine while learning the parameters that are better for reproducibility ﬁrst, and then repeat training to learn the model itself using these parameters.
Revised: We can include reproducibility optimization of the parameters in training the full network, or train the model ofﬂine while learning the parameters that are better for reproducibility ﬁrst, and then repeat training to learn the model itself using these parameters.

Original: In either case, ensembles can be used as a proxy for reproducibility, where we impose a loss that minimizes the prediction difference (either in probability or on log-odds score) between components of the ensemble.
Revised: In either case, ensembles can be used as a proxy for reproducibility, where we impose a loss that minimizes the prediction difference (either in probability or on log-odds score) between components of the ensemble.

Original: This approach essentially applies co-distillation (Anil et al., 2018) but for the purpose of learning the hyper-parameters that improve reproducibility instead of training the actual model.
Revised: This approach essentially applies co-distillation (Anil et al., 2018) but for the purpose of learning the hyper-parameters that improve reproducibility instead of training the actual model.

Original: Co-trained ensemble may not mimic the full effect causing divergence of models as they may have less randomness in the order of examples they see and updates they apply, but empirical results show that even with that components of ensemble tend to diverge from one another, even when initialized identically, and especially when initialized differently.
Revised: Co-trained ensemble may not mimic the full effect causing divergence of models as they may have less randomness in the order of examples they see and updates they apply, but empirical results show that even with that components of ensemble tend to diverge from one another, even when initialized identically, and especially when initialized differently.

Original: Unlike co-distillation, to learn the hyper-parameters for better reproducibility, the objective loss in training the ensemble is applied only on the parameters, but the similarity loss is applied only on the hyper-parameters.
Revised: Unlike co-distillation, to learn the hyper-parameters for better reproducibility, the objective loss in training the ensemble is applied only on the parameters, but the similarity loss is applied only on the hyper-parameters.

Original: There are disadvantages to learning the parameters while training.
Revised: There are disadvantages to learning the hyper-parameters while training.

Original: This is because of potentially conﬂicting objectives to accuracy and reproducibility.
Revised: This is because of potentially conﬂicting objectives to accuracy and reproducibility.

Original: Furthermore, in many cases, the preferred deployed model should not be an ensemble.
Revised: Furthermore, in many cases, the preferred deployed model should not be an ensemble.

Original: We can use an ensemble to optimize the hyper-parameters for reproducibility ofﬂine, and then train and deploy a model that retrains with the reproducibility optimized activation hyper-parameters.
Revised: We can use an ensemble to optimize the hyper-parameters for reproducibility ofﬂine, and then train and deploy a model that retrains with the reproducibility optimized activation hyper-parameters.

Original: If we deploy an ensemble, training with identically initialized components may be valuable for optimizing activation hyper-parameters for better reproducibility, but will be suboptimal for deployment, at least for better reproducibility.
Revised: If we deploy an ensemble, training with identically initialized components may be valuable for optimizing activation hyper-parameters for better reproducibility, but will be suboptimal for deployment, at least for better reproducibility.

Original: As shown by Shamir & Coviello (2020), ensembles are advantageous for reproducibility by leveraging and encouraging the differences between their components, so that as a whole, the ensemble captures the objective space better.
Revised: As shown by Shamir & Coviello (2020), ensembles are advantageous for reproducibility by leveraging and encouraging the differences between their components, so that as a whole, the ensemble captures the objective space better.

Original: Initializing components identically for optimizing the hyper-parameters defeats this idea, and thus it is more desirable to apply such initialization only when learning the activation hyper-parameters, but then retrain the models with these parameters with different initializations of the components, for better deployed reproducibility.
Revised: Initializing components identically for optimizing the hyper-parameters defeats this idea, and thus it is more desirable to apply such initialization only when learning the activation hyper-parameters, but then retrain the models with these parameters with different initializations of the components, for better deployed reproducibility.

Original: C G RAPHS OF D IFFERENT S MOOTH A CTIVATIONS
Revised: C G RAPHS OF D IFFERENT S MOOTH A CTIVATIONS

Original: In this appendix, we provide activation function and gradient curves for the different (smooth) activations discussed in this paper as function of their parameter.
Revised: In this appendix, we provide activation function and gradient curves for the different (smooth) activations discussed in this paper as function of their parameter.

Original: This set of curves completes the ones shown in Fig.
Revised: This set of curves completes the ones shown in Fig.

Original: Fig 8 shows curves for the different activations with β = 1 .
Revised: Fig 8 shows curves for the different activations with β = 1 .

Original: Figs.
Revised: Figs.

Original: 9, 10, 11 and 12 show SELU, SoftPlus, Swish, GELU, Mish, Tanhexp, SmeLU, generalized SmeLU, and Sigmoid-RESCU, respectively.
Revised: 9, 10, 11 and 12 show SELU and CELU, SoftPlus, Swish, GELU, Mish, Tanhexp, SmeLU, generalized SmeLU, and Sigmoid-RESCU, respectively.

Original: SELU is shown to be smooth only for β = 1 . 
Revised: SELU is shown to be smooth only for β = 1 . The equations for

Original:  We observe the close similarity between Swish, GELU, Mish, and Tanhexp, and the relation between Sigmoid-RESCU and SmeLU.
Revised: CELU are given by [Equation] We observe the close similarity between Swish, GELU, Mish, and Tanhexp, and the relation between Sigmoid-RESCU and SmeLU.

Revised: Subsequently to our work, and inspired by SmeLU, (Lin & Shamir, 2019), a different SmoothRelu activation was given in Xie et al.

Revised: [Equation]

Revised: This activation is also a RESCU activation.

Original:  D G ENERALIZED S ME LU - R EFORMULATIONS AND S OME S PECIAL C ASES
Revised: Graphs for this activation are included in Fig. As observed, this activation does not achieve full slope 1 , unless β (cid:29) 1 , in which case, it already approaches ReLU. Thus it may lack the smoothness tuning ﬂexibilities of SmeLU. D G ENERALIZED S ME LU - R EFORMULATIONS AND S OME S PECIAL C ASES

Original: In this appendix we show some special cases of the generalized SmeLU deﬁned in (11), and also address the asymptotic relation between SmeLU and SoftPlus.
Revised: In this appendix we show some special cases of the generalized SmeLU deﬁned in (11), and also address the asymptotic relation between SmeLU and SoftPlus.

Original: D.L EAKY S ME LU
Revised: D.

Revised: A SYMMETRIC S ME LU

Revised: SmeLU is a special case of the generalized SmeLU.

Revised: The deﬁnition of the generalized SmeLU in (11) can be used to obtain a version of SmeLU that is asymmetric around the origin.

Revised: Like SmeLU, this activation can guarantee 0 output on the left, and slope 1 on the right, but a transition region in [ − α, β ] , where α (cid:54) =

Revised: β ,

Original:  Setting g − > 0 , but g + = 1 > g − ,  and t = 0  gives the SmeLU version of a Leaky ReLU , referred to as Leaky SmeLU , and given by
Revised: [Equation] Setting α = β gives SmeLU. D. L EAKY S ME LU Setting g − > 0 , but g + = 1 > g − , α = β and t = 0 , gives the SmeLU version of a Leaky ReLU , referred to as Leaky SmeLU , and given by

Original: [Equation]
Revised: [Equation]

Original: Setting α = β gives a symmetric version of this activation, that apart for the middle regions appears like Leaky ReLU.
Revised: Apart for the middle region, this activation resembles Leaky ReLU.

Original: D. S HIFTED S ME LU
Revised: D. S HIFTED S ME LU

Original: The constraint used to generate the generalized SmeLU constrains the g − slope region to end at the point x = − α , y = t .
Revised: The constraint used to generate the generalized SmeLU constrains the g − slope region to end at the point x = − α , y = t .

Original: Constraints can be speciﬁed differently, where this region ends at the origin, and then shifted horizontally and/or vertically.
Revised: Constraints can be speciﬁed differently, where this region ends at the origin, and then shifted horizontally and/or vertically.

Original: Vertical shift is given by t , whereas we can deﬁne the activations z ( x ) =
Revised: Vertical shift is given by t , whereas we can deﬁne the activation z ( x ) =

Original: y ( x − s ) for a horizontal shift by s .
Revised: y ( x − s ) for a horizontal shift by s .

Original: D. O RIGIN C ROSSING S ME LU
Revised: D. O RIGIN C ROSSING S ME LU

Original: SmeLU as deﬁned in (9) does not cross the origin.
Revised: SmeLU as deﬁned in (9) does not cross the origin.

Original: We can similarly deﬁne a version of generalized
Revised: We can similarly deﬁne a version of generalized

Original: SmeLU that is constrained to cross the origin instead of a point ( − α, t ) .
Revised: SmeLU that is constrained to cross the origin instead of a point ( − α, t ) .

Original: We can apply the generalized SmeLU with the desired α , β and gradients, and then shift it vertically to cross the origin (or we can solve directly with the new constraints).
Revised: We can apply the generalized SmeLU with the desired α , β and gradients, and then shift it vertically to cross the origin (or we can solve directly with the new constraints).

Original: A zero crossing version keeps the sign of the input on one hand, but does not suppress negative values to 0 .
Revised: A zero crossing version keeps the sign of the input on one hand [Equation] but does not suppress negative values to 0 .

Original: The form is given by
Revised: The form is given by

Original: [Equation]
Revised: [Equation]

Original: D. A SYMPTOTIC R ELATION OF S ME LU AND S OFT P LUS
Revised: D. A SYMPTOTIC R ELATION OF S ME LU AND S OFT P LUS

Revised: The gradient of the SmeLU function is a hard Sigmoid.

Revised: This already shows a connection to SoftPlus, whose gradient is a Sigmoid.

Original: To show the relation between SmeLU and SoftPlus, we express SoftPlus using a reciprocal parameter γ to the original deﬁnition in (3), 
Revised: To further demonstrate the relation between SmeLU and SoftPlus, we express SoftPlus using a reciprocal parameter γ to the original deﬁnition in (3), y SoftPlus = [Equation] ln (

Original: [Equation]
Revised: [Equation]

Original:  Asymptotically, for x → −∞ ,  y → 0 .
Revised: + [Equation] [Equation] Asymptotically, for x → −∞ , we have y → 0 .

Original: For x (cid:29) 0 , y ≈ x .
Revised: For x (cid:29) 0 , y ≈ x .

Original: For | x | (cid:28) 1 , we can approximate the SoftPlus using Taylor series expansion by
Revised: For | x | (cid:28) 1 , we can approximate the SoftPlus using Taylor series expansion by

Original: [Equation]
Revised: [Equation]

Original: Assigning the SmeLU parameter β = 2 γ , we obtain for | x | (cid:28) 1 ,
Revised: Assigning the SmeLU parameter β = 2 γ , we obtain for | x | (cid:28) 1 ,

Original: [Equation]
Revised: [Equation]

Original: Thus the middle region around x = 0 of SoftPlus is a vertical shift by β/ 4 ln(4 /e ) of the middle region of SmeLU with a factor of 2 wider parameter, but the extremes on both sides meet those of
Revised: Thus the middle region around x = 0 of SoftPlus is a vertical shift by β/ 4 ln(4 /e ) of the middle region of SmeLU with a factor of 2 wider parameter, but the extremes on both sides meet those of

Original: SmeLU.
Revised: SmeLU.

Original: Therefore, SoftPlus opens wider, away from the origin, making it even smoother for the same parameter, on one hand, but also potentially less accurate on the other, due to the distance of the middle region from the origin.
Revised: Therefore, SoftPlus opens wider, away from the origin, making it even smoother for the same parameter, on one hand, but also potentially less accurate on the other, due to the distance of the middle region from the origin.

Original: E A DDITIONAL RESULTS ON MNIST
Revised: E E XPERIMENTS ON THE C RITEO D ATASET

Revised: We describe the set up of the experiments on the Criteo benchmark dataset and provide additional results.

Revised: Model architecture: Models have 3 fully connected hidden layers of dimension 2572, 1454,respectively.

Revised: Each categorical feature x k is hashed into N k buckets.

Revised: If N k < 110 the hash bin is encoded as a one-hot vector, otherwise it is embedded as a vector of dimension d k .

Revised: Values of N k and d k are taken from Ovadia et al.

Revised: and are the following (where d k = 0 means that the hash bin is encoded as a one-hot vector):

Revised: [Equation]

Revised: Unlike the set up in Ovadia et al.

Revised: (2019), input features are not fed into a batch-norm layer, while integer features are log-square-transformed.

Revised: Additional results: Table 2 shows results for ReLU, SmeLU and Swish when the training data is not shufﬂed.

Revised: As a consequence, all models visit the data in the same order of batches, but are initialized differently.

Revised: (Note, however, that due to randomness in the platform, within a batch, there could still be some randomness in the order examples are being processed.)

Revised: Table 3 shows results for ReLU,

Revised: SmeLU and Swish when all models have the same initialization of the hidden layer weights.

Revised: In both cases, models exhibit irreproducibility.

Revised: Without shufﬂing, as a result of initialization (and random effect in the platform that applies the updates).

Revised: With identical initialization, irreproducibility is due to the training data shufﬂing.

Revised: In both case, SmeLU improves ∆ 1 and ∆ r 1 by about 12% , and exhibits a factor of 4 reduction of the standard deviations of both AUC and ∆ 1 .

Revised: F A DDITIONAL RESULTS ON MNIST

Original: In this appendix we show matching graphs to those in Fig.
Revised: In this appendix, we show matching graphs to those in Fig.

Original: 4 for the other PD metrics; ∆ 2 , ∆ L 1 , and ∆ H , in Figs 13, 14, and 15, respectively.
Revised: 4 for the other PD metrics; ∆ 2 , ∆ L 1 , and ∆ H , in Figs 13, 14, and 15, respectively.

Original: The behavior of the different activations shown in these ﬁgures for all the metrics is very similar to the behavior w.r.t.
Revised: The behavior of the different activations shown in these ﬁgures for all the metrics is very similar to the behavior w.r.t.

Original: ∆ 1 .
Revised: ∆ 1 .

Revised: [Table]

Revised: [Table]

Revised: Model

Revised: [Table]

Revised: ReLU

Revised: [Table]

Revised: [Table]

Revised: SmeLU

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Swish

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]