Revisiting the Activation Function for Federated Image Classification
Abstract
Federated learning (FL) has become one of the most popular distributed machine learning paradigms; these paradigms enable training on a large corpus of decentralized data that resides on devices.
The recent evolution in FL research is mainly credited to the refinements in training procedures by developing the optimization methods.
However, there has been little verification of other technical improvements, especially improvements to the activation functions (e.g., ReLU), that are widely used in the conventional centralized approach (i.e., standard data-centric optimization).
In this work, we verify the effectiveness of activation functions in various federated settings.
We empirically observe that off-the-shelf activation functions that are used in centralized settings exhibit a totally different performance trend than do federated settings.
The experimental results demonstrate that HardTanh achieves the best accuracy when severe data heterogeneity or low participation rate is present.
We provide a thorough analysis to investigate why the representation powers of activation functions are changed in a federated setting by measuring the similarities in terms of weight parameters and representations.
Lastly, we deliver guidelines for selecting activation functions in both a cross-silo setting (i.e., a number of clients ≤ 20 ) and a cross-device setting (i.e., a number of clients ≥ 100 ).
We believe that our work provides benchmark data and intriguing insights for designing models FL models.
The code is available at https://github.com/Jaewoo-Shin/FL_ACT .
Introduction
Federated learning (FL) has become a common and ubiquitous paradigm for collaborative machine learning techniques [3, 4, 23, 33, 34, 44, 37, 45] because it maintains data privacy.
Each client (e g., mobile devices or the whole business) communicates with the central server by transferring the model but not the data; all local updates are aggregated in a global server-side model.
Although a centralized method enhances generalization by employing a large amount of training data, the features of the
FL methods appear to differ from those of a centralized method owing to data heterogeneity, client resource capability, and model communication [24, 37, 50].
Most FL studies focus on improving the performance of the global model.
To this end, they apply a new regularizer in the optimization algorithm
Recently, there has been an increasing demand for the personalization of models according to the client.
Jiang et al.
[22] and Fallah et al.
[9] attempt to train personalized models for each client with a few rounds of fine-tuning rather than focusing on the performance of a server model.
In consideration of system heterogeneity (i.e., clients having different computational and communication capabilities),
Avdiukhin et al.
[2] mitigate model communication problems by using asynchronous local SGD;
Horvath et al.
[17] improve accuracy in heterogeneous resource capacity by using different model sizes per client.
[Figure]
Despite the popularity of FL, some options for federated model optimization remain under-explored.
Designing FL-familiar training recipes is essential to optimizing model performance, but few studies have attempted to design a new recipe instead of using those intended for a centralized setting.
Charles et al.
[5] present an empirical analysis of the impact of hyperparameter settings for federated training dynamics from the perspective of a large cohort size.
However, FL activation functions [37, 25, 34] have rarely been studied, whereas activation functions play a crucial role in facilitating generalization and convergence.
We thus raise the seemingly doubtful question: Do activation functions that are popular in centralized settings also produce good optima in FL?
We conduct a pilot experiment to compare performance in centralized settings with performance in FL settings to answer this question.
Figure 1 (b) and (c) show the accuracy of neural networks trained under a centralized setting and a FL setting according to the changes in activation functions.
A shallow neural network that uses Tanh has better accuracy than ReLU, which is a silver bullet in the centralized deep learning field.
The problems mentioned above lead us to the intriguing question:
Do off-the-shelf activation functions intended for a centralized setting also perform appropriately in the FL setting?
In this work, we answer the question with thorough empirical evaluations: the latest developed activation functions rather degrade the performance of the server as the heterogeneity gets severe.
Several considerations (e.g., the total number of clients, client participation, heterogeneity) in selecting activation functions may improve significantly.
Combining considerations may further boost the model accuracy.
We experiment with various activation functions, including widely-used and rarelyused functions in the centralized setting, in various environments based on CIFAR-10 and CIFAR-100.
The experiments identify an interesting phenomenon in FL in which applying activation functions like
ReLU in stacked convolutional layers demonstrates low accuracy owing to the shape of the function.
We also provide an analysis of the representation power according to the changes in activation functions for federated image classification.
Our key contributions are summarized as follows:
• We provide guidelines for selecting activation functions in FL.
FL has the following special considerations: number of clients, participation ratio, and non-IIDness.
We provide guidelines for cross-silo settings (i.e., for a number of clients ≤ 20 ) and cross-device settings (i
e., for a number of clients ≥ 100 ); the suitable activation function depends on the situation.
• We provide an explanation for the performance degradetion (i.e., for the performance difference between centralized settings and FL settings) of activation functions, that are the preferred in a centralized setting.
Specifically, we measure the similarities in the manner of weight parameters and representations.
• We empirically show that the HardTanh activation function [7] leads to a better optimum do other activation functions such as ReLU [39],
Leaky ReLU [36], and GeLU [16] for severe
[Table]
non-IID setting, a low participation rate, and with a large number of clients
[Table]
and we provide benchmark data for activation functions in FL.
Related Work
This work is related to activation functions in neural networks
[Table]
and FL methods.
For details, refer to
Appendix A.
Experiments
In this section
[Table]
we compare several activation functions.
We categorize the activation functions into two groups: (1) ReLU, Leaky ReLU, Swish, Mish
[Table]
and GeLU as recent SOTA activation functions that are widely used in centralized settings; and (2) Tanh and HardTanh as Tanh-like activation functions that are not widely used in centralized settings.
3.1 Experimental Setup
Dataset and Heterogenous Settings.
We use two benchmark datasets: CIFAR-10 and CIFAR100 [29].
We provide the descriptions of the datasets in Appendix B.
To randomize the heterogeneous data, we assume that all client training data use class labels according to an independent categorical distribution for N classes parameterized by the vector q : q i ≥ 0 , i ∈
[1 , N ] and (cid:88) i ∈
[1 ,N ] q i =
For the heterogeneous distribution, the Dirichlet distribution [18, 48], q ∼ Dir ( α ) is used, where α is an N -length concentration vector having all elements α > 0 , that is, the prior distribution for N classes controls the heterogeneity of clients.
Models.
Our study focuses on compact models that are realistically possible in FL.
Therefore, we use a simple ConvNet having four convolutional layers and one classifier; ConvNet4 refers to
ConvNet with four convolutional layers.
The first convolution layer has 64 kernels, and deeper layers have a larger number of kernels [40].
For additional models, which have a shortcut and batch normalization layer, we use Resnet20 , Resnet32 , Resnet44 [15], and MobileNetv2 [43].
Training Details.
In this study, we conduct numerical experiments by varying the number of clients
N , the client participation ratio R , and the Dirichlet distribution constant α .
We mainly exhibit the training of ConvNet4 on heterogeneously distributed CIFAR-10 by modifying R and α .
In the captions, we explain each N , R , and α value.
The details are explained in Appendix B.
3.2 Comparative Experiments on the Changes in Activation Functions
Table 1 shows the result of both centralized and FL settings using CIFAR-10 and CIFAR-100 as the datasets.
In centralized settings, GeLU shows the best performance, and other recent SOTA activation functions surpass the Tanh-like activation functions.
However, in FL settings, the activation functions show a significantly different tendency.
HardTanh achieves the highest accuracy.
Furthermore, the recent SOTA activation functions show lower accuracy than Linear using CIFAR-100 as the dataset.
[Table] [Table]
[Table] [Table]
[Table]
Activation functions have different accuracy drop; only recent SOTA activation functions have an accuracy drop near 40, while HardTanh and Tanh have 26.21 and 28.42 at CIFAR-10.
As a result, we can find that the most popular activation function, ReLU (as well as recent SOTA activation functions), does not show outstanding performance in an FL setting.
3.3 Strategies for Selecting Activation Functions in FL
This section presents the experimental results and guidelines for selecting the activation functions for various FL settings.
FL settings have various environmental limitations relative to centralized settings.
It has additional components to consider, such as the number of clients, IID-ness, and the participation ratio.
Number of Clients.
In different FL strategies, the number of clients varies.
A cross-silo setting uses fewer than 20 clients, and a cross-device setting use more than 100 clients.
As the number of clients differs, the drop in accuracy caused by the activation functions varies.
The α = 0 .
1 columns of Table 2 lists the accuracy for different client numbers.
As the number of clients decreases, the overall drop in accuracy decreases.
For both numbers of clients, HardTanh achieves the highest accuracy.
However, the accuracy difference between the recent SOTA activation functions and
Tanh-like activation functions is larger for 100 clients.
Considering the observations for the number of clients, we hypothesize that as the number of clients increases, recent SOTA activation functions are increasingly affected and show a more significant accuracy drop.
IID-ness.
With the Dirichlet distribution parameter α , we can control the IID-ness: a larger value of α indicates higher IID-ness (lower heterogeneity).
Table 2 presents the accuracy for different different values of α .
In most cases, HardTanh shows the highest accuracy.
For 100 clients, at lower
IID-ness the accuracy of the Tanh-like activation functions surpasses the accuracy of the recent SOTA activation functions with larger accuracy gap.
For 20 clients, the recent SOTA activation functions surpass the Tanh-like activation functions at high IID-ness.
Activation’s shape cause severe accuracy drop of the recent SOTA activations with low IID-ness, which we discuss at section 4.
The low accuracy of the Tanh-like activation functions at α = 0 .
01 occurs due to tough training settings and the Tanh-like activation functions fail to find the optimum, such as Linear.
Participation Ratio.
The participation of clients is limited in FL depending on the environment.
In a cross-silo setting, high participation may be possible, whereas only limited participation is available in a the cross-device setting.
Table 3 shows the accuracy in FL settings for four different values of the
[Figure]
[Figure]
participation decreases the most recent activation functions client participation client drift [ reduces activation shape the recent drops and occur smaller accuracy.
Different FL settings components affect accuracy when different activation functions are used, according to the observations above.
The number of clients is the most dominant component, and it interacts with the influence of other components when a small number of clients is used.
Therefore, the Tanh-like activation functions are favorable for a large number of clients, such as in a cross-device setting.
For a cross-silo setting, with low data IID-ness, and a low ratio of client participation the
Tanh-like activation functions are preferred.
Conversely, the recent SOTA activation functions are preferred for a cross-silo setting, with high data IID-ness, and high ratio of client participation.
3.4 Additional Experiment
Additional experiments with different FL method and models showed a similar tendency, where Tanhlike activation functions surpassed recent SOTA activation functions as observed above.
We chose
FedProx as the additional FL method and Resnet20 , Resnet32 , Resnet44 , and MobileNetv2 as the additional models.
For details of the experimental results, refer to Appendix C.
Analysis
We present thorough investigations on model behavior and the changes in representation during the local training to answer this question: do recent SOTA activation functions have a disadvantage in
FL settings?
4.1 Investigation of Weight Parameters and Latent Representations
The activation function selects important features to pass through each layer, and the model is trained using these features.
The number of selected features varies according to the shape of an activation function.
In a conventional centralized setting, a single model can access all the data and select optimal training features.
However, in an FL setting, each client can only access a portion of the data, which is partitioned in the non-IID condition, and each client trains its model to select features that are important to itself.
This results in a phenomenon known as client drift .
During the FL aggregation step, a problem arises where important features for the global optimum cannot be selected due to client drift.
This phenomenon appears to be severe when the recent SOTA activation functions are used.
Due to the shape of their activation functions, the excluded features are greater in number than for Tanh-like activation functions, and a severe accuracy drop occurs.
Figure 5 (a) in Appendix C shows the feature distribution after the first convolution layer.
The recent SOTA activation functions exclude more feature and have a high density near 0.
The empirical results in section 3 show that for a large number of clients and a small client participation ratio, the client drift increases, and the accuracy drop for recent SOTA activation functions is maximal.
This can be simply summarized by saying that, the number of excluded features varies due to the different shapes of the activation functions, and as an activation function excludes more feature values, the drop in accuracy increases.
Recent SOTA activation functions tend to exclude features smaller than 0.
Tanh-like activation functions, on the other hand, exclude features that are smaller thanand larger than 1.
Tanh-like activation functions have low sensitivity about the accuracy drop in the
FL aggregation step because they exclude a much smaller number of features than do recent SOTA activation functions.
Additionally, HardTanh and Tanh show better accuracy than Linear in most situations due to their existence of non-linearity.
Figure 5 (b) in Appendix C shows the accuracy in
FL settings for different versions of ConvNet , which have different numbers of convolutional layers.
Deeper models exclude more features, and as a result, ConvNet used with recent SOTA activation functions shows a significant drop in accuracy.
To observe how the activation function affects ConvNet4 , we perform two additional experimental studies on heterogeneous local models.
For simplicity, we look at 10 clients out of a total 100.
First, we perform Centered Kernel Alignment (CKA) [28] to measure the similarity of the output features between different clients.
Each client trains the server model for 5 epochs with their own non-IID data.
In Figure 2, ReLU has a higher CKA similarity than HardTanh in every layer.
Because feature selection fails for the server model, as indicated above, ConvNet4 with ReLU has a smaller feature change than does HardTanh, which indicates a lower learning ability.
Second, we calculate the weight differential to check how similar other clients weights are.
Figure 3 reveals that ConvNet4 with ReLU has a smaller weight difference in each layer than does HardTanh, which also indicates ReLU has a limited learning ability.
4.2 Analysis of Landscape
To check if each activation functions reaches the global optimum, we visualize the 2-D landscape of
Tanh, HardTanh, ReLU, and Leaky ReLU in Appendix D. Figure 5 shows the 2-D landscape of each activation function.
It seems that ReLU and Leaky ReLU fails to approach to the global optimum.
In contrast, Tanh and HardTanh succeed in reaching the global optimum.
Furthermore, ReLU and
Leaky ReLU has steep slope in their landscape where Tanh and HardTanh does not.
These findings suggest that the sensitivity of the activation functions in the FL aggregation step, where the accuracy drop occurs, is indicated by the shape of the activation functions.
The accuracy drop increases as the number of features excluded by an activation function increases, and it reaches a maximum with severe client drift; a large number of clients, a low client participation ratio, and high heterogeneity.
Conclusion
This study clarifies that the drop in accuracy varies according to the activation function in FL.
Our key finding is that the accuracy of the recent SOTA activation functions drops in an FL setting due to the shape of the functions, and HardTanh outperforms other activation functions in most environments.
Additionally, we provide guidelines and benchmark data for selecting activation functions in various
FL settings.
We believe that further research is needed to find the silver bullet of activation function in an FL setting, which our paper did not address, and that our work will inspire others.
Acknowledgement
This work was supported by Institute of Information & communications Technology Planning &
Evaluation (IITP) grant funded by Korea government (MSIT)
[No. 2021-0-00907, Development of Adaptive and Lightweight Edge-Collaborative Analysis Technology for Enabling Proactively
Immediate Response and Rapid Learning, 90%] and [No. 2019-0-00075, Artificial Intelligence
Graduate School Program (KAIST), 10%].
Appendix A Related Work A.1 Activation Functions in Neural Networks
In deep neural networks, using activation functions is a ubiquitous technique for learning non-linear latent representations; an input signal is transformed into the non-linear output centered on zero.
Recent evolution occurs along with the enhancement of representation power and the efficiency of computational costs.
In the experiment, we use the following non-linear activation functions:
Tanh.
The equation of Tanh is:
[Equation]
It is known that it is zero-centered, but computationally expensive and causes vanishing gradient problems as neural networks become deeper.
HardTanh.
It is another variant of the hyperbolic tangent function, which represents computationally more efficient form of tanh:
[Equation]
ReLU.
The equation of ReLU [12, 20, 39] is:
[Equation]
ReLU is the abbreviation of Rectified Linear Unit , a modified linear function.
When the input value of ReLU is negative, the gradient of its output value is zero; the model does not learn.
ReLU has shown great performance with Convolutional Neural Networks (CNN) [30, 31].
Since ReLU is computationally cheap, it is still commonly used regardless of numerous attempts to replace it [36, 11, 14, 6, 27, 8].
Leaky ReLU.
The equation of Leaky ReLU [36] is:
[Equation]
This function was created to solve the Dying ReLU problem in ReLU.
It is ReLU multiplied by a tiny constant on the negative part.
Due to the small range, the graphs are drawn almost similarly with
ReLU.
Swish.
The equation of Swish [41] is:
[Equation]
This function shows better accuracy than ReLU in deep neural networks regardless of batch size.
Mish.
The equation of Mish [38] is:
[Equation]
This function has the characteristic of allowing gradients to flow better than the Relu Zero Bound because it allows some negative numbers.
GeLU.
The equation of GeLU [16] is:
[Equation]
This function is widespread on BERT, GPT, VIT models.
GeLU is derived by combining the characteristics of dropout, zoneout, and ReLU.
Activation functions have been devised for the gradient exploding/vanishing issues; the magnitude of gradients becomes either near zero or infinite during backward propagation.
A general choice for activation functions is ReLU which raises significantly cheap computational costs.
Still, it is not differentiable at zero as well as causes significant dying neurons forgetting the information during propagation.
Recently, some works [41, 38] lead to more smooth optima by designing self-regularized gradients.
On the other hand, these studies are based on centralized settings, whereas those based on
FL settings are badlands.
Details of activation functions are explained in Appendix B.
A.2 Federated Learning Methods
Federated optimization methods manage to handle multiple clients without collecting data, and they use server weights from a central server to coordinate the global model across the network.
In particular, these methods aim to minimize the following objective function:
[Equation]
where f ( k ) is the loss function based on the client k .
N is the total number of clients.
At each round
K ≪ N clients are selected from the total devices.
The selected clients run each local model using
SGD for E number of local epochs and finally aggregate the selected models at the server model.
In the FL environment, the oracle of a global model can be drifted by optimizing the local clients because the statistical data heterogeneity causes different local optimums widely apart from each other.
It is called Client drift [25, 26, 42] which indicates the inconsistency among each optimum.
Recently, some works prevent Client drift designing aggregation methods; Wang et al.
[47] present a method of normalized averaging that removes objective inconsistency, and Zhang et al.
[49] propose a training algorithm for group knowledge transfer, which allows each client to keep a personalized prediction on the server to assist the local training of others.
Federated Averaging (FedAvg [37]) uses the local server for stochastic gradient descent (SGD) locally for E number of epochs.
As a result, the selected client resulting k ’s weight is updated as w k .
At each round, to aggregate the local client models, FedAvg sums and averages for the server model parameters formulated as:
[Equation]
where w t is the server weight of the t -th round, w ti is the client k ’s model after local training using w t − 1 , and S t is the client set.
McMahan et al.
[37] empirically shows the significance of tuning the hyperparameters in FL training additionally, we present that with respect to the architectural and operational side.
A.3 Algorithms of Federated Learning Methods
We use both FedAvg and FedProx for federated learning methods.
Algorithm 1 shows the algorithm of FedAvg and Algorithm 2 shows the algorithm of FedProx.
FedProx is similar to FedAvg in that it selects a selection of clients at each round, performs local training, and then averages client’s weight to generate a global update.
However, the difference between FedAvg and FedProx is shown in line 6.
For local training, FedAvg trains each client’s model using SGD with its local data whereas FedProx, trains each client with additional proximal term, µ 2 ∥ w − w t ∥ 2 .
Using the proximal term which contributes to the method’s stability by efficiently reducing the impact of variable modifications.
1: Input: K , T , η , E , w 0 , N , k ∈ [1 , · · · , N ] 2: for t = 0 , · · · , T − 1 do 3:
Server selects a subset S t randomly which includes number of K devices 4:
Server send w t to all selected devices 5: for i = 0 , · · · , E − 1 do 6:
Selected device k ∈ S t updates their local weight w t +1 k using SGD with step-size η 7: end for 8:
Selected device k ∈ S t sends their local weight w t +1 k back to the server 9:
Server aggregates the local weights, w t +1 k , and gets new server weight w t +1 = 1 K (cid:80) k ∈ S t w tk 10: end for
1: Input: K , T , η , µ , E , w 0 , N , k ∈ [1 , · · · , N ] 2: for t = 0 , · · · , T − 1 do 3:
Server selects a subset S t randomly which includes number of K devices 4:
Server send w t to all selected devices 5: for i = 0 , · · · , E − 1 do 6:
Selected device k ∈ S t updates their local weight w t
+1 k
≈ min w h ( k ) ( w ; w t ) =
f ( k ) ( w ) + µ 2 ∥ w − w t ∥ 2 with step-size η 7: end for 8:
Selected device k ∈ S t sends their local weight w t +1 k back to the server 9:
Server aggregates the local weights, w t +1 k , and gets new server weight w t +1 = 1 K (cid:80) k ∈ S t w tk 10: end for
B Implementation Details B.1 Model Architecture
Figure 4 shows ConvNet with five different depth (3,4,5,6, and 7).
Each version of ConvNet has convolution layer corresponding to the number after the model (e.g., ConvNet3 has three convolution layers and ConvNet7 has seven convolution layers).
The details of each convolution layer is shown in
Table 4.
ConvNet with different depth use convolution layers sequentially from the top in Table 4 (e.g.,
ConvNet3 use Conv1, Conv2, Conv3 whereas, ConvNet7 use Conv1, Conv2, Conv3, Conv4, Conv5,
Conv6, and Conv7).
[Figure]
x
[Figure]
,activation function 3 x 3 conv,activation function 3 x 3 conv,activation function 3 x 3 conv,activation function max pool, /
Image fc3 x 3 conv,activation function 3 x 3 conv,activation function 3 x 3 conv,activation function 3 x 3 conv,activation function 3 x 3 conv,activation function 3 x 3 conv,activation function max pool, / Image fc3 x 3 conv,activation function 3 x 3 conv,activation function 3 x 3 conv,activation function 3 x 3 conv,activation function 3 x 3 conv,activation function max pool, /
Image fc3 x 3 conv,activation function 3 x 3 conv,activation function 3 x 3 conv,activation function max pool, / Image fc3 x 3 conv,activation function 3 x 3 conv,activation function 3 x 3 conv,activation function 3 x 3 conv,activation function max pool, /
Image fc3 x 3 conv,activation function 3 x 3 conv,activation function 3 x 3 conv,activation function
ConvNet
ConvNet ConvNet
ConvNet ConvNet
B.2 Dataset statistics
We use both CIFAR-10 and CIFAR-100 for our experiments.
As shown in Table 5, CIFAR-10 consists of 60000 images of size 32 × 32.
It is divided into 10 classes, and each class consists of 6000 images.
Also, 5000 are training data and 1000 are test data.
CIFAR-100 also consists of 60000 images with a
[Table]
size of 32 × 43.
It is classified into 100 classes, and each class consists of 600 images.
Theseclasses are divided into 20 superclasses, which we do not use in our experiments.
Also, 500 are training data and 100 are test data.
[Table]
B.3 Training Details
In this study, we conduct numerical experiments by changing the number of clients N , the client participation ratio R , and the Dirichlet distribution constant α .
We adapt FedAvg and performrounds with 5 local epochs using a learning rate of 0.01, with a learning decay of 0.1 at the 50th and 75th round, a weight decay of 1e − 4 , and a momentum of 0.9.
Limited client numbers are available in different FL settings; in a cross-silo setting, a small number of clients are available, and a large number of clients are requested in a cross-device .
For the cross-silo setting, we use N = 20 and
R = 0 . 2 in which we select 4 clients each round.
For the cross-device setting, we use N = 100 and
R = 0 .
2 , which we select 20 clients each round.
We mainly exhibit the training of ConvNet4 on CIFAR-10 heterogeneously distributed by modifying the α in the Dirichlet distribution.
In the captions, we explain each N , R , and α value.
C Additional Experiment Result C.1 ConvNet4 Result
Experiment results in section 3 fix variables such as N , R , and α .
Table 6 shows the result with all combinations of R and α with N = 100 .
Table 7 shows the result using CIFAR-100 with all combinations of R and α with N = 100 .
We implement all experiments on Single GPU NVIDIA 2080-Ti.
Figure 5 (a) shows the feature distribution in ConvNet4 with different activation functions after passing the first convolution layer.
After passing the convolution layer and activation function, we flatten the feature values and draw a distribution.
The feature densities of the recent SOTA activation functions have discreteness and a high density near 0 due to their function shape.
As the features pass deeper layers, the densities near 0 of the recent SOTA activation functions increase compared with the Tanh-like activation functions.
In a centralized setting, the recent SOTA activation functions, especially ReLU work well due to its discreteness at 0 and its function shape, which is similar to a linear function which prevents the vanishing gradient and exploding gradient problems [19].
[Table]
Mish
[Table]
GeLU
[Table]
[Figure]
Distribution
[Figure]
Linear
Tanh
HardTanh
[Table]
Leaky
[Table]
Swish
Mish
GeLU
[Table]
Linear
[Table]
C.2 Resnet Result
Table 8, Table
[Table]
and Table 10 shows the result of Resnet20 , Resnet
[Table]
and Resnet44 with all combinations of R and α with N = 100 .
For all values of R and α , Resnet20 using HardTanh shows the highest accuracy.
As model gets deeper, Resnet32 and Resnet44 , in some cases recent SOTA activation functions overcome Tanh-like activation functions.
It seems to occur due to shortcut’s existence.
A deeper layer can use features that the activation function has excluded via a shortcut [15], which helps to prevent the recent SOTA activation functions’ accuracy drop.
[Table] [Table]
[Table]
C.3 MobileNetv2 Result
Table 11 shows the result of MobileNetv2 with four different R with N = 100 and α = 0 .
Tanh-like activation functions surpass recent SOTA activation functions.
Especially recent SOTA activation functions accuracy is lower than Linear.
[Table]
C.4 FedProx Result
As mentioned in section
[Table]
studies to improve the server model in FL add proximal terms to the local object to enhance the method’s stability so that the local model rarely differs from the global model.
We choose FedProx [33] as an additional FL method because it is the most basic method that focuses on improving the server model by adding a proximal term.
The details of FedProx is shown in Appendix B. Table 12 shows the accuracy for different activation functions using FedProx as a learning algorithm.
Similar to FedAvg, HardTanh achieves the best accuracy, followed by the other Tanh-like activation functions.
Proximal terms in local training do not appear to help prevent accuracy loss.
[Table]
D Landscape Result
We visualize the affect of different activation functions; Tanh, HardTanh, ReLU, and Leaky ReLU.
The color bar beside each figure shows the loss difference.
Tanh and HardTanh has extremely small number compared to ReLU and Leaky ReLU.
Which emphasize that Tanh and HardTanh smooth out the landscape considerably comparison to ReLU and Leaky ReLU.
Additionally, Tanh and HardTanh has its significant lowest value where perturbation X and perturbation Y is both 0.
However, ReLU and Leaky ReLU does not show this phenomenon.
Which we can consider that only Tanh and
HardTanh succeed in reaching the global optimum.
[Figure]