P ADÉ A CTIVATION U NITS : E ND - TO - END L EARNING OF F LEXIBLE A CTIVATION F UNCTIONS IN D EEP N ETWORKS
A BSTRACT
The performance of deep network learning strongly depends on the choice of the non-linear activation function associated with each neuron.
However, deciding on the best activation is non-trivial, and the choice depends on the architecture, hyper-parameters, and even on the dataset.
Typically these activations are ﬁxed by hand before training.
Here, we demonstrate how to eliminate the reliance on ﬁrst picking ﬁxed activation functions by using ﬂexible parametric rational functions instead.
The resulting Padé Activation Units (PAUs) can both approximate common activation functions and also learn new ones while providing compact representations.
Our empirical evidence shows that end-to-end learning deep networks with
PAUs can increase the predictive performance.
Moreover, PAUs pave the way to approximations with provable robustness.
https://github.com/ml-research/pau
I NTRODUCTION
An important building block of deep learning is the non-linearities introduced by the activation functions f ( x ) .
They play a major role in the success of training deep neural networks, both in terms of training time and predictive performance.
Consider e.g. Rectiﬁed Linear Unit (ReLU) due to Nair and Hinton (2010).
The demonstrated beneﬁts in training deep networks, see e.g. (Glorot et al., 2011), brought renewed attention to the development of new activation functions.
Since then, several ReLU variations with different properties have been introduced such as LeakyReLUs (Maas et al., 2013),
ELUs (Clevert et al., 2016), RReLUs (Xu et al., 2015), among others.
Another line of research, such as (Ramachandran et al., 2018) automatically searches for activation functions.
It identiﬁed the Swish unit empirically as a good candidate.
However, for a given dataset, there are no guarantees that Swish unit behaves well and the proposed search algorithm is computationally quite demanding.
The activation functions are traditionally ﬁxed and, in turn, impose a set of inductive biases on the network.
One attempt to relax this bias are for instance PReLUs (He et al., 2015), where the negative slope is subject to optimization allowing for more ﬂexibility than other ReLU variants.
Learnable activation functions generalize this idea.
They exploit parameterizations of the activation functions, adapted in an end-to-end fashion to different network architectures and datasets during training.
For instance, Maxout (Goodfellow et al., 2013) and Mixout (Zhao et al., 2017) use a ﬁxed set of piecewise linear components and optimized their (hyper-)parameters.
Although they are theoretically universal function approximators, they heavily increase the number of parameters of the network and strongly depend on hyper-parameters such as the number of components to realize this potential.
Vercellino and Wang (2017) used a meta-learning approach for learning task-speciﬁc activation functions (hyperactivations).
However, as Vercellino and Wang admit, the implementation of hyperactivations, while easy to express notationally, can be frustrating to implement for generalizability over any given activation network.
Recently, Goyal et al.
(2019) proposed a learnable activation function based on
Taylor approximation and suggested a transformation strategy to avoid exploding gradients.
However, relying on polynomials suffers from well-known limitations such as exploding values and a tendency to oscillate (Trefethen, 2012).
[Figure] [Figure]
[Figure]
ReLU
Sigmoid
Tanh
Swish
Leaky ReLU
ReLU ∗
Sigmoid ∗
Tanh ∗
Swish ∗
Leaky ReLU ∗
As an alternative, we here introduce a learnable activation function based on the Padé approximation, i.e., rational functions.
In contrast to approximations for high accuracy hardware implementation of the hyperbolic tangent and the sigmoid activation functions (Hajduk, 2018), we do not assume ﬁxed coefﬁcients.
The resulting Padé Activation Units (PAU) can be learned using standard stochastic gradient and, hence, seamlessly integrated into the deep learning stack.
PAUs provide more ﬂexibility and increase the predictive performance of deep neural networks, as we demonstrate.
We proceed as follows.
We start off by introducing PAUs.
Then introduce Padé networks and show that they are universal approximators.
Before concluding, we present our empirical evaluation.
P ADÉ A CTIVATION U NITS (PAU)
Our starting point is the set of intuitive assumptions activation functions should ultimately fulﬁll shown in Tab. 1.
The assumptions (i,v) concern the ability of neural networks to approximate functions.
Rational functions can fulﬁll assumptions (i,iv), and our experimental evaluation demonstrates that assumptions (ii,iii,v) also hold.
(i)
They must allow the networks to be universal function approximators.
(ii) They should ameliorate gradient vanishing.
(iii)
They should be stable.
(iv) They should be parsimonious on the number of parameters.
(v) They should provide networks with high predictive performance.
Assumptions Activation
Learnable i ii iii iv
[Equation]
ReLU
[Equation]
ReLU
[Equation] PReLU [Equation] Maxout
[Equation] Mixture [Equation] PAU
[Equation]
2. P ADÉ A PPROXIMATION OF A CTIVATION F UNCTIONS
Let us now formally introduce PAUs.
Assume for the moment that we start with a ﬁxed activation function f ( x ) .
The Padé approximant (Brezinski and Van Iseghem, 1994) is the “best” approximation of f ( x ) by a rational function of given orders m and n .
Applied to typical actication functions, Fig.shows that they can be approximated well using rational functions.
More precisely, given f ( x ) , the Padé approximant is the rational function F ( x ) over polynomials
P ( x ) , Q ( x ) of order m , n of the form
[Equation]
which agrees with f ( x ) the best.
The Padé approximant often gives a better approximation of a function f ( x ) than truncating its Taylor series, and it may still work where the Taylor series does not converge.
For these reasons, it has been used before in the context of graph convolutional networks (Chen et al., 2018).
However, they have not been considered so far for general deep networks.
Padé
Activation Units (PAUs) go one step further, instead of ﬁxing the coefﬁcients a j , b k to approximate a particular activation function, we allow them to be free parameters that can be optimized end-to-end with the rest of the neural network.
This allows the optimization process to ﬁnd the activation function needed at each layer automatically.
The ﬂexibility of Padé is not only a blessing but might also be a curse: it can model processes that contain poles.
For a learnable activation function, however, a pole may produce undeﬁned values depending on the input as well as instabilities at learning and inference time.
Therefore we consider a restriction, called safe PAU , that guarantees that the polynomial Q ( x ) is not 0, i.e., we avoid poles.
In general, restricting Q ( x ) implies that either Q ( x ) (cid:55)→ R > 0 or Q ( x ) (cid:55)→ R < 0 , but as P ( x ) (cid:55)→ R we can focus on Q ( x ) (cid:55)→ R > 0 wlog.
However, as lim Q ( x ) → 0 + F ( x ) → ∞ learning and inference become unstable.
To ﬁx this, we impose a stronger constraint, namely Q ( x ) ≥ q (cid:29) 0 .
In this work, q = 1 , i.e., ∀ x : Q ( x ) ≥ 1 , preventing poles and allowing for safe computation on R :
[Equation]
Other values for q ∈ (0 , 1) might still be interesting, as they could provide gradient ampliﬁcation due to the partial derivatives having Q ( X ) in the denominator.
However, we leave this for future work.
L EARNING S AFE P ADÉ A PPROXIMATIONS USING B ACKPROPAGATION
In contrast to the standard way of ﬁtting Padé approximants where the coefﬁcients are found via derivatives and algebraic manipulation against a given function, we optimize their polynomials via backpropagation and (stochastic) gradient descent.
To do this, we have to compute the gradients with respect to the parameters ∂F ∂a j , ∂F ∂b k as well as the gradient for the input
∂F ∂x .
A simple alternative is to implement the forward pass as described in Eq. 2, and let automatic differentiation do the job.
To be more efﬁcient, however, we can also implement PAUs directly in CUDA (Nickolls et al.
(2008)), and
for this we need to compute the gradients ourselves: ∂F ∂x = ∂P ( x ) ∂x
Q ( x ) − ∂Q ( x ) ∂x
P ( x )
Q ( x ) 2 , ∂F ∂a j = x j
Q ( x ) and ∂F ∂b k = − x k
A ( X ) | A ( X ) |
P ( X )
Q ( x ) 2 , where ∂P ( x ) ∂x = [Equation]
[Equation]
A
[Equation]
X ) =
[Equation]
x +
[Equation]
2 x
[Equation]
+ · · · + [Equation] n x n , and Q ( x ) = [-
-]1 + | A ( X ) | .
Here we reuse the expressions to reduce computations.
To avoid divisions by zero when computing the gradients, we deﬁne z | z | as the sign of z .
With the gradients at hand, PAUs can seamlessly be placed together with other modules onto the differentiable programming stack.
P ADÉ N ETWORKS
Having PAUs at hand, one can deﬁne Padé networks as follows:
Padé networks are feedforward networks with PAU activation functions that may include convolutional and residual architectures with pooling layers.
To use Padé networks effectively, one simply replaces the standard activation functions in a neural network by PAUs and then proceed to optimize all the parameters and use the network as usual.
However, even if every PAU contains a low number of parameters (coefﬁcients a j , b k ), in the extreme case, learning one PAU per neuron may considerably increase the complexity of the networks and in turn the learning time.
To ameliorate this and inspired by the idea of weightsharing as introduced by Teh and Hinton (2001), we propose to learn one PAU per layer.
Therefore we only add φ many parameters, where φ = L ∗ ( m + n ) and L is the number of activation layers in the network.
In our experiments we set φ = 10 L , a rather small number of parameters (iv).
The last step missing before we can start the optimization process is to initialize the coefﬁcients of the
PAUs.
Surely, one can do random initialization of the coefﬁcients and allow the optimizer to train the network end-to-end.
However, we obtained better results after initializing all PAUs with coefﬁcients that approximate standard activation functions.
For a discussion on how to obtain different PAU coefﬁcients, we refer to Sec.
A.1.
Before evaluating Padé Networks empirically, let us touch upon their expressivity and how to sparsify them.
3. P ADÉ N ETWORKS ARE U NIVERSAL F UNCTION A PPROXIMATORS
A standard multi-layer perceptron (MLP) with enough hidden units and non-polynomial activation functions is a universal approximator, see e.g. (Hornik et al., 1989; Leshno et al., 1993).
Padé
Networks are also universal approximators.
Theorem 1.
Let ρ : R → R be a PAU activation function.
Let N ρ represent the class of neural networks with activation function ρ .
Let K ⊆ R n be compact.
Then N ρ is dense in C ( K ) .
The proof holds for both PAUs and safe PAUs as it makes no assumptions on the form of the denominator Q ( x ) , and is a direct application of the following propositions:
Proposition 1.
(From Theorem 1.1 in (Kidger and Lyons, 2019)) Let ρ : R → R be any continuous function.
Let N ρn represent the class of neural networks with activation function ρ , with n neurons in the input layer, one neuron in the output layer, and one hidden layer with an arbitrary number of neurons.
Let K ⊆ R n be compact.
Then N ρn is dense in C ( K ) if and only if ρ is non-polynomial.
Proposition 2.
(From Theorem 3.2 in (Kidger and Lyons, 2019)) Let ρ : R → R be any continuous function which is continuously differentiable at at least one point, with nonzero derivative at that point.
Let K ⊆ R n be compact.
Then NN ρn,m,n + m +2 is dense in C ( K ; R m ) .
Proof.
Let ρ ( x ) =
P ( x ) /Q
( x ) , we have to consider the following two cases:
Case 1: Q ( x ) (cid:54) = 1 , by deﬁnition, ρ ( x ) is non-polynomial.
Then by proposition 1, we get that N ρn is dense in C ( K ) .
Case 2: Q ( x ) = 1 , here ρ ( x ) =
(cid:80) m j =
a j x j , is polynomial, continuous and continuously differentiable in R .
Let any a j> 0 > 0 , then there exists a point α ∈ R such that ρ (cid:48)
( α ) (cid:54) =
and then by proposition 2, we get that NN ρn,m,n + m +
is dense in C ( K ; R m ) .
3. S PARSE P ADÉ N ETWORKS AND R ANDOMIZED PAU S
Padé Networks can (cid:15) -approximate neural networks with ReLU activations (Telgarsky, 2017).
This implies that by using PAUs, we are embedding a virtual network into the networks we want to use.
This, in turn, is the operating assumption of the lottery ticket hypothesis due to Frankle and Carbin (2019).
Thus, we expect that lottery ticket pruning can ﬁnd well-performing Padé networks that are smaller than their original counterparts while reducing inference time and potentially improving the predictive performance.
Generally, overﬁtting is an important practical concern when training neural networks.
Usually, we can apply regularization techniques such as Dropout (Srivastava et al., 2014).
Unfortunately, although each PAU approximates a small ReLU network section, we do not have access to the internal representation of this virtual network.
Therefore, we can not regularize the activation function via
[Figure]
standard Dropout.
An alternative for regularizing activation functions was introduced in Randomized
Leaky ReLUs (RReLUs, Xu et al.
(2015)), where the negative slope parameter is sampled uniformly on a range.
This makes the activation function behave differently at training time for every input x , forwarding and backpropagating according to x and the sampled noise.
We can employ a similar technique to make PAUs resistant to overﬁtting.
Consider a PAU with coefﬁcients C
= { a 0 , · · · , a m , b 0 , · · · , b n } .
We can introduce additive noise during training into each coefﬁcient c i ∈ C for every input x j via c i,j = c
i
+ z i,j
where z i,j ∼ U ( l i , u i ) , l i = (1 − α %) ∗ c i and reciprocally u i =
(1 + α %)
∗ c i .
This results in Randomized PAU (RPAU):
[Equation]
We compute the gradients as before and simply replace the coefﬁcients by their noisy counterparts.
E XPERIMENTAL E VALUATION
Our intention here is to investigate the behavior and performance of PAUs as well as to compare them to other activation functions using standard deep neural networks.
All our experiments are implemented in PyTorch with PAU implemented in CUDA, and were executed on an NVIDIA DGXsystem.
In all experiments, we initialized PAUs with coefﬁcients that approximate LeakyReLUs for a rational function of order m = 5 , n = 4 .
In all experiments except for ImageNet, we report the mean of 5 runs initialized with different seeds for the accuracy on the test-set after training.
And, we compared PAU to the following activation functions: ReLU , ReLU6 , Leaky ReLU (LReLU),
Random ReLU (RReLU), ELU , CELU , Swish , Parametric ReLU (PReLU), Maxout , Mixture of activations (Mixture), SLAF , APL and SReLU .
For details on all the activation functions, we refer to Appendix A.2.
As datasets we considered MNIST , Fahion-MNIST , CIFAR-10 and ImageNet .
4. E MPIRICAL R ESULTS ON MNIST AND F ASHION -MNIST B ENCHMARKS
First we evaluated PAUs on MNIST (LeCun et al., 2010) and Fashion-MNIST (Xiao et al., 2017) using two different architectures: LeNet (LeCun et al., 1998) and VGG-8 (Simonyan and Zisserman, 2015).
For more details on the architectures, learning settings, and results, we refer to Sec.
A.3.
As can be seen in Fig.
2 and Tab. 2, PAU outperformed on average the baseline activation functions on every network in terms of predictive performance.
Moreover, the results are stable on different runs (c.f. mean ± std).
PAUs also enable the networks to achieve a lower loss during training compared to all baselines on all networks.
Actually, PAU achieved the best results on both datasets and on
Fashion-MNIST it provides the best results for both architectures.
As expected, reducing the bias is beneﬁcial in this experiment.
Comparing the baseline activation functions on the MNIST dataset and the different architectures, there is no clear choice of activation that achieves the best performance.
However, PAU always matches or even outperforms the best performing baseline activation function.
This shows that a learnable activation function relieves the network designer of having to commit to a
[Table]
potentially suboptimal choice.
Moreover, Fig. 2 also shows that PAU is more stable than SLAF.
This is not unexpected as Taylor approximations tend to oscillate and overshoot (Trefethen, 2012).
We also observed undeﬁned values at training time for SLAF; therefore, we do not compare against it in the following experiments.
Finally
[Table]
when considering the number of parameters used by PAU
[Table]
we can see that they are very efﬁcient.
The VGG-8 network uses 9.2 million parameters, PAU here usesparameters
[Table]
and for LeNet, the network uses 0.5 million parameters while PAU uses only 40.
In summary
[Table]
this shows that PAUs are stable, parsimonious and can improve the predictive performance of deep neural networks (iii,iv,v).
4. L EARNED A CTIVATION F UNCTIONS ON MNIST AND F ASHION -MNIST
When looking at the activation functions learned from the data, we can see that the PAU family is ﬂexible yet presents similarities to standard functions.
In particular, Fig. 3 illustrates that some of the learned activations seem to be smoothed versions of Leaky ReLUs, since V-shaped activations are simply Leaky ReLUs with negative α values.
This is not surprising as we initialize PAUs with coefﬁcients that match Leaky ReLUs.
Finding different initialization and optimization parameters is left as future work.
In contrast, when learning piecewise approximations of the same activations using Maxout, we would require a high k .
This signiﬁcantly increases the number of parameters of the network.
This again provides more evidence in favor of PAUs being ﬂexible and parsimonious (iv).
SLAF produced undeﬁned values during training on all networks except Fashion-MNIST where
LeNet ﬁnished 4 runs and VGG only one run.
4. E MPIRICAL R ESULTS ON CIFAR-
After investigating PAU on MNIST and Fashion-MNIST, we considered a more challenging setting:
CIFAR-10 (Krizhevsky et al.
We also considered other learnable activation functions, namely
Maxout (k=2) and Mixture of activations (Id and ReLU) as well as another popular deep network architectures: MobileNetV2 (Sandler et al., 2018), ResNet101 (He et al., 2016) and DenseNet (Huang et al., 2017).
For more details on the learning settings and results, we refer to Sec.
A.4.
Let us start by considering the results for VGG-8 and MobileNetV2 on CIFAR-10.
These networks are the smallest of this round of experiments and, therefore, could beneﬁt more from bias reduction.
Indeed, we can see in Tab. 3 that both networks take advantage of learnable activation functions, i.e., Maxout, PAU, and RPAU.
As expected, adding more capacity to VGG-8 helps and this is what
Maxout is doing.
Moreover, even if Mixtures do not seem to provide a signiﬁcant beneﬁt on VGG-8,
[Table]
they do help in MobileNetV2.
Here, we see again that PAU and RPAU are either in the lead or close to the best when it comes to predictive performance, without having to make a choice apriori.
Now
[Table]
let us have a look at the performance of PAU and RPAU on the larger networks ResNet101 and
DenseNet121.
As these networks are so expressive
[Table]
we do not expect the ﬂexibility of the learnable activation functions to have a big impact on the performance.
Tab. 3 conﬁrms this.
Nevertheless, they are still competitive and their performance is stable as shown by the standard deviation.
On
ResNet101, PAUs actually provided the top performance.
4. F INDING S PARSE P ADÉ N ETWORKS
As discussed in Sec. 3.2, using PAUs in a network is equivalent to introducing virtual networks of
ReLUs in the middle of the network, effectively adding virtual depth to the networks.
Therefore, we also investigated whether pruning can help one to unmask smaller sub-networks whose performance is similar to the original network.
In a sense, we are removing blocks of the real network as they get replaced by the virtual network.
Here, we only do pruning on the convolutional layers.
For details about the algorithm and hyper-parameters, wee refer to Sec.
A.4.3.
Speciﬁcally, we compared PAU against the best activation functions for the different architectures.
However, we discarded Maxout, as instead of pruning it introduces many more parameters into
[Figure]
Layer
[Figure]
Layer
[Figure]
Layer [Figure] Furthermore, PAU enables the ResNet101 subnetwork, pruned by 30%, to achieve a higher compared to all and not the network defeating the original purpose.
As one can see in Fig.
4, pruning on the already sizeoptimized networks VGG-8 and MobileNetV2 has an effect on the predictive performance.
However, the performance of PAU remains above the other activation functions despite the increase in pruning pressure.
In contrast, when we look at ResNet101, we see that the performance of PAU is not inﬂuenced by pruning, showing that indeed we can ﬁnd sparse Padé network without major loss in accuracy.
And what is more, PAU enables the ResNet101 subnetwork, pruned by 30%, to achieve a higher accuracy compared to all pruned and not pruned networks.
4. E MPIRICAL R ESULTS ON I MAGE N ET
Finally, we investigated the performance on a much larger dataset, namely ImageNet (Russakovsky et al.
(2015)) used to train MobileNetV2.
As can be seen in Fig.
5 and Tab. 4, PAU and Swish clearly dominate in performance (v).
PAU leads in top-1 accuracy and Swish in top-5 accuracy.
Moreover, both PAU and Swish show faster learning compared to the other activation functions.
Furthermore, we argue that the rapid learning rate of PAU in all the experiments indicate that they do not exhibit vanishing gradient issues (ii).
[Figure]
Epoch
[Figure]
[Figure]
[Figure]
il e N e t V
A cc u r a c y
Epoch 1 .1 .1 .1 .1 .1 .1 .1 .2 .
[Figure]
PReLU
ReLU ReLU
RReLU
SReLU
Swish
[Table]
4. S UMMARIZED R ESULTS
Finally
[Table]
we compare the PAU family of activation functions to all the other activation functions and we aggregate the number of occurrences where PAU performed better or worse in comparison.
The aggregate results can be found in Tab. 5.
These results are the aggregates of all the experiments on all datasets and all architectures.
As we can see, the PAU family is very competitive.
[Table]
To summarize, PAUs satisfy all the assumptions (i-v).
They allow the network to be universal function approximators (i) as shown by theorem 1.
They present a fast and stable learning behavior (ii, iii) as shown in Figs.
The number of parameters introduced by PAUs is minimal in comparison to the size of the networks.
In our experiments, we add 10 parameters per layer, showing that PAUs are parsimonious (iv).
Finally, they allow deep neural networks to provide high predictive performance (v) as shown in Tab. 5.
C ONCLUSIONS
We have presented a novel learnable activation function, called Padé Activation Unit (PAU).
PAUs encode activation functions as rational functions, trainable in an end-to-end fashion using backpropagation.
This makes it easy for practitioners to replace standard activation functions with PAU units in any neural network.
The results of our empirical evaluation for image classiﬁcation demonstrate that PAUs can indeed learn new activation functions and in turn novel neural networks that are competitive to state-of-the-art networks with ﬁxed and learned activation functions.
Actually, across all activation functions and architectures, Padé networks are among the top performing networks.
This clearly shows that the reliance on ﬁrst picking ﬁxed, hand-engineered activation functions can be eliminated and that learning activation functions is actually beneﬁcial and simple.
Moreover, our results provide the ﬁrst empirically evidence that the open question “Can rational functions be used to design algorithms for training neural networks?” raised by Telgarsky (2017) can be answered afﬁrmatively for common deep architectures.
Our work provides several interesting avenues for future work.
One should explore more the space between safe and unsafe PAUs, in order to gain even more predictive power.
Most interestingly, since Padé networks can be reduced to ReLU networks.
one should explore globally optimal training (Arora et al., 2018) as well as provable robustness (Croce et al., 2019) of Padé approximations of general deep networks.
Acknowledgments.
PS and KK were supported by funds of the German Federal Ministry of Food and Agriculture (BMEL) based on a decision of the Parliament of the Federal Republic of Germany via the Federal Ofﬁce for Agriculture and Food (BLE) under the innovation support program, project “DePhenS” (FKZ 2818204715).
A A PPENDIX A. I NITIALIZATION COEFFICIENTS
As show in Table 6 we compute initial coefﬁcients for PAU approximations to different known activation functions.
We predeﬁned the orders to be [5,4] and for Sigmoid, Tanh and Swish
[Table]
we have computed the Padé approximant using the standard techniques.
For the different variants of PRelu,
LeakyRelu and Relu we optimized the coefﬁcients using least squares over the line range between [-3,3] in steps of 0.
[Table]
0.68720375 b  0 1.
13.70648993 b1 /4 /
b 2 / 28 4.
6.07781733 b  0 0.
12.32535229 b 4 1 /1 / 63 b 4 / 1680 0.
A. L IST OF A CTIVATION F UNCTIONS
For our experiments, we compare against the following activation functions with their respective parameters.
• ReLU (Nair and Hinton, 2010): y = max ( x, 0) • ReLU6 (Krizhevsky and Hinton, 2010):
y = min ( max ( x, 0) , 6) , a variation of ReLU with an upper bound.
• Leaky ReLU (Maas et al., 2013): y = max (0 , x ) + α ∗ min (0 , x ) with the negative slope, which is deﬁned by the parameter α .
Leaky ReLU enables a small amount of information to ﬂow when x < 0 .
• Random ReLU (Xu et al., 2015): a randomized variation of Leaky ReLU.
• ELU (Clevert et al., 2016): y = max (0 , x ) + min (0 , α ∗ ( exp ( x ) − 1)) .
• CELU (Barron, 2017): y = max (0 , x ) + min (0 , α ∗ ( exp ( x/α ) − 1)) .
• Swish
(Ramachandran et al., 2018):
y = x ∗ sigmoid ( x ) , which tends to work better than ReLU on deeper models across a number of challenging datasets.
• Parametric ReLU (PReLU) (He et al., 2015)
y = max (0 , x ) + α ∗ min (0 , x ) , where the leaky parameter α is a learn-able parameter of the network.
• Maxout
(Goodfellow et al., 2013): y = max ( z ij ) , where z ij = x T W ...
ij + b ij , and W ∈ R d
× m × k and b ∈ R m × k are learned parameters.
• Mixture of activations (Manessi and Rozza, 2018): a combination of weighted activation functions e.g. {id, ReLU}, where the weight is a learnable parameter of the network.
• SLAF (Goyal et al., 2019): a learnable activation function based on a Taylor approximation.
• APL (Agostinelli et al., 2015): a learnable piecewise linear activation function.
• SReLU (Jin et al., 2016): a learnable S-shaped rectiﬁed linear activation function.
A. D ETAILS OF THE MNIST AND F ASHION -MNIST E XPERIMENT A.3. N ETWORK A RCHITECTURES
Here we describe the architectures for the networks VGG and LeNet, along with the number of trainable parameters.
The number of parameters of the activation function is reported for using
PAU.
Common not trainable activation functions do not have trainable parameters.
PReLU has one
[Table]
A.3. L EARNING P ARAMETERS
The parameters of the networks, both the layer weights and the coefﬁcients of the PAUs, were trained over 100 epochs using Adam (Kingma and Ba, 2015) with a learning rate of 0 . 002 or SGD (Qian, 1999) with a learning rate of 0 .
01 , momentum set to 0 .
5 , and without weight decay.
In all experiments we used a batch size of 256 samples.
The weights of the networks were initialized randomly and the coefﬁcients of the PAUs were initialized with the initialization constants of Leaky ReLU, see
Tab. 6.
We report the mean of 5 different runs for both the accuracy on the test-set and the loss on the train-set after each training epoch.
[Figure] [Figure]
L
[Figure]
N
[Figure]
A. D ETAILS OF THE C IFAR 10 AND I MAGENET E XPERIMENT A.4. L EARNING P ARAMETERS
The parameters of the networks, both the layer weights and the coefﬁcients of the PAUs, were trained over 400 epochs using SGD with momentum set to 0 .
On the Cifar10 dataset we have different optimizer setups for PAU layers and the rest of the network.
For the PAU layers we use constant learning rates per networks and no weight decay.
For updating the rest of the network we use initial learning rate of 0 . 1 , and learning rate decay of 0 . 985 per epoch and set weight decay to 5 e − 4 .
In all experiments we used a batch size of 64 samples.
The weights of the networks were initialized randomly and the coefﬁcients of the PAUs were initialized with the initialization constants of Leaky
ReLU, see Tab. A.1.
The additive noise of the Randomized PAUs is set to α = 1% training the networks VGG8 and MobileNetV2, respectively α = 10% for ResNet101.
On the Imagenet dataset we use the same optimizer for PAU and the rest of the network.
We follow the default setup provided by Pytorch and use an initial learning rate of 0 . 1 , and decay the learning rate by 10% after 30, 60 and 90 epochs.
A.4. N ETWORK A RCHITECTURES
The network architectures were taken from reference implementations in PyTorch and we modiﬁed them to use PAUs.
All architectures are the same among the different activation functions except for Maxout.
The default amount of trainable parameters of VGG8 (Simonyan and Zisserman, 2015) is 3,918,858.
Using PAU 50 additional parameters are introduced.
Maxout is extending the
VGG8 network to a total number of 7,829,642 parameters.
MobileNetV2 (Sandler et al., 2018) is contains by default 2,296,922 trainable parameters.
PAU adds 360 additional parameters.
The
Maxout activation function is results in a total number of 3,524,506 parameters.
With respect to the number of parameters ResNet101 (He et al., 2016) is the largest network we train.
By default it contains 42,512,970 trainable parameters, we introduce 100 PAUs and therefore add 1000 additional parameters to the network.
If one is replacing each activation function using Maxout the resulting
ResNet101 network contains 75,454,090 trainable parameters.
The default DenseNet121 (Huang et al., 2017) network has 6,956,298 parameters.
Replacing the activation functions with PAU adds 1200 parameters to the network.
A.4. P RUNING E XPERIMENT
For the pruning experiment, we implement the "Lottery ticket hypothesis" (Frankle and Carbin (2019)) in PyTorch.
We compare PAUs against the best activation for the network architecture according to the average predictive accuracy from Tab. 3.
More precisely, we compare the predictive performance under pruning for the networks N 1 = { VGG-8 pau , MobileNetV2 pau , ResNet101 pau } against the networks N 2 = { VGG-8 LReLU , MobileNetV2 RReLU , ResNet101 pau } .
Here we avoided Maxout as it heavily increases the parameters in the model, defeating the purpose of pruning.
Unlike the original paper, we compress the convolutions using a ﬁxed pruning parameter per iteration of p % = 10 , 20 , 30 , 40 , 50 , 60 and evaluated once per network.
After each training iteration we remove p % of ﬁlters in every convolution and the ﬁlters we remove are the ones where the sum of its weights is lowest.
After pruning, we proceed to re-initialize the network and repeat the training and pruning proceedure with the next p % parameter.
A.4.
P REDICTIVE P ERFORMANCE C IFAR81 .
VGG
A cc u r a c y
L o ss 77 .
M o b il e N e t
V
A cc u r a c y
L o ss 84 .85 .87 .
R e s N e t
A cc u r a c y
L o ss
Epoch 82 .84 .85 .87 .88 .
D e n s e N e t
A cc u r a c y
Epoch
L o ss
[Figure]