Paper: LayerAct: Advancing CNNs with BatchNorm through Layer-direction Normalization (ICLR-2024)

Reviewer comment (Experiments):
Some existing advanced batch normalization improvements like IEBN [1] and SwitchNorm [2] have shown enhanced performance. It would be interesting to investigate whether LayerAct can further improve the performance of these normalization methods.

Author response:
### Question 2. Investigation whether LayerAct can further improve the performance of other batch-direction normalization methods. 
- We agree that evaluating performance on networks with advanced normalization methods is important. 
- Considering our main contribution in this work is introducing a new activation method to enhance the performance of CNNs with BatchNorm, we plan to conduct additional experiments with ResNets with IEBN, decorrelated batch normalization (DBN; an advanced normalization method that utilizes batch-direction normalization introduced by Huang et al (2018)), and SwitchNorm. We will report the results of the remaining networks as soon as possible.
- However, LayerAct’s effectiveness in networks using SwitchNorm is may be tempered, as SwitchNorm utilizes LayerNorm. This aligns with our earlier response, where we noted that LayerAct’s benefits might be reduced when LayerNorm precedes it.

Huang, L., Yang, D., Lang, B., & Deng, J. “Decorrelated batch normalization.” CVPR. 2018.

Reviewer comment (Writing):
I still don't quite understand the advantage of "layer-direction" activation over "element-wise" activation. Could the author please provide a concise explanation with simple examples or a summary?

Author response:
### Question 3. The advantage of layer-level activation over element-level activation. 
- Thanks to the reviewers, we recognized the lack of a detailed discussion on one-side saturation of element-level activation functions, zero-like mean activation, and the trade-off problem between them. We will enhance our manuscript with a comprehensive explanation about them in the Appendix, based on this response. 
- In this response, we aim to clarify LayerAct’s contributions as an activation function. To this end, we will detail the limitations of existing element-level activation functions and how LayerAct can address those. 

#### 3-1. Summary of this response. 
- LayerAct functions are specifically designed to overcome the limitations of existing activation functions. These limitations include i) the trade-off problem between one-side saturation and zero-like mean activation, and ii) the large variance in noise-robustness across samples. 
- Notably, addressing the trade-off problem stands as a unique contribution of LayerAct. This benefit is unachievable with other activation functions, regardless of their integration with LayerNorm. 
- LayerAct has a lower upper bound of activation fluctuation compared to element-level activation functions with the same activation input vector. This is substantiated by our experimental analysis detailed in Subsection 4.1 of the manuscript and the experimental results on noisy image datasets. 

#### 3-2. Important properties of activation functions and the trade-off problem. 
- The saturation state of an activation function contributes to robustness against shifts in activation input, as the changes in saturated activation inputs minimally affect the output. However, early activation functions like Sigmoid and Tanh, which saturate both positive and negative sides, suffer from the vanishing gradient problem.
- To overcome this while maintaining noise-robustness, one-sided saturation became an important property after the great success of ReLU. ReLU can ensure noise-robustness (as the outputs are not affected by the input shift in the saturation state) and avoids vanishing gradient problems by allowing large positive outputs.
- However, functions like ReLU, which do not allow negative output, encounter another issue of bias shift, leading to ineffective and inefficient training. After Clevert et al. (2016) demonstrated that an activation mean closer to zero can solve such a problem, “zero-like mean activation” has become an important property of activation functions.
- To incorporate both properties, one-sided saturation and zero-like mean activation, activation functions such as ELU, GELU, and SiLU were designed to allow small negative output to push the activation mean towards zero while saturating large negative output for noise-robustness.
- Nevertheless, a problem still remains: there is a trade-off between one-sided saturation and zero-like mean activation. Saturating large negative output naturally restricts the zero-like mean activation. For example, allowing larger negative outputs pushes the activation mean closer to zero but diminishes the noise-robustness of the activation function as the range of the saturation state decreases.
- Placing a LayerNorm layer parallel to an activation layer cannot address this trade-off problem. Large negative outputs will be restricted for saturation state. This is because the range of the saturation state entirely depends on the design of the activation function.

#### 3-3. LayerAct can address the trade-off problem. 
- LayerAct functions address the trade-off between saturation and zero-like mean activation by integrating layer-direction normalization **into** the activation. The layer-direction normalization only determines the input of the activation scale function, which defines the saturation state (where $s\\left(n_{i}\\right)\\approx0)$. 
- This mechanism of LayerAct cuts off the link between saturation and negative activation output. Hence, LayerAct can produce larger negative output while maintaining saturation states. For example, in a scenario where $\\mu_{y}\\ll0$, the activation output $a_{i}=\\mu_{y}/2\\ll0$ where the activation input $y_{i}=\\mu_{y}$.  
- In conclusion, despite its structural simplicity, LayerAct functions offer a significant contribution by addressing the trade-off problem of activation. 

Clevert, Djork-Arné, Thomas Unterthiner, and Sepp Hochreiter. "Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUS)." ICLR. 2016.

Reviewer comment (Experiments):
Additionally, I'd be interested in understanding in which applications LayerAct might excel or not excel. For example, we have instance norm for tasks like style transfer, batch norm for CNN-based classification tasks, and layer norm for transformer-related tasks. Can LayerAct be analyzed and discussed in a similar manner, suggesting suitable application areas?

Author response:
### Question 4. Application of LayerAct. 
- LayerAct has demonstrated its potential in image classification tasks using CNNs with BatchNorm, as evidenced by our experimental results on CIFAR10, CIFAR100, and ImageNet. Notably, the noise-robustness of LayerAct during inference is particularly advantageous for tasks where models have to be trained on clean datasets but need to classify or segment noisy images. 
- A relevant example is the classification of satellite images, which can be significantly affected by various external factors including weather conditions (Pritt, et al., 2017). Another example is task on motion blurred images in real-world due to the movement of the object or camera (Dai, 2008). 
- We would like to emphasize that the noisy datasets, which are utilized to evaluate the performance of the networks with LayerAct, include such real-world noises. 

Pritt, M., and Gary C. "Satellite image classification with deep learning.” IEEE applied imagery pattern recognition workshop (AIPR). 2017.
Dai, S., & Wu, Y. “Motion from blur.” CVPR. 2008.

Reviewer comment (Novelty):
I think the technical contribution is overall low. The proposed LayerAct can be viewed as Layer Normalization combining the (general) sigmoid-linear-units. Both of them are good methods in improving the performance of neural network. It is not surprise the proposed method can improve the performance over the original activation functions. It is good that this paper addresses the robust of the proposed method, however I have concerns in the second point.

Author response:
Thank you very much for your comments. We are glad to see that our topic and experiments meet your preference. In response to your concerns and questions, we describe our responses as follows.

### Weakness 1. I think the technical contribution is overall low. & Weakness 4-2. About the statement on one-sided saturation.
- Thanks to the reviewers, we recognized the lack of detailed discussion on one-sided saturation, zero-like mean activation, and the trade-off problem between them. We will enhance our manuscript with a comprehensive explanation about them in the Appendix, based on the following response.
- In this response, we aim to clarify LayerAct's contributions as a new activation mechanism. To this end, we will detail the limitations of existing element-level activation functions that cannot be addressed by the mere combination of LayerNorm and activation functions. This response also resolves the concerns raised about one-sided saturation in Weakness 4 (See 1-2 of this response).

#### 1-1. Summary of this response. 
- We acknowledge that the structure of LayerAct, a combination of LayerNorm and activation functions, is simple. Behind this simple mechanism, it is important to emphasize that LayerAct offers unique benefits that cannot be achieved through utilizing LayerNorm and element-level activation layers together.
- LayerAct functions specifically address the limitations of existing activation functions: i) the trade-off problem between one-sided saturation and zero-like mean activation, and ii) the large variance of noise-robustness across samples. In this response, we focus on the first benefit, as the significance of noise-robustness is already well recognized (For the response to Weakness 2 about the noise-robustness of LayerAct, please see the next response).
- Notably, addressing the trade-off problem stands as a unique contribution of LayerAct. This benefit is unachievable with other activation functions, regardless of their integration with LayerNorm.

#### 1-2. Important properties of activation functions and the trade-off problem. 
- The saturation state of an activation function contributes to robustness against shifts in activation input, as the changes in saturated activation inputs minimally affect the output. However, early activation functions like Sigmoid and Tanh, which saturate both positive and negative sides, suffer from the vanishing gradient problem.
- To overcome this while maintaining noise-robustness, one-sided saturation became an important property after the great success of ReLU. ReLU can ensure noise-robustness (as the outputs are not affected by the input shift in the saturation state) and avoids vanishing gradient problems by allowing large positive outputs.
- However, functions like ReLU, which do not allow negative output, encounter another issue of bias shift, leading to ineffective and inefficient training. After Clevert et al. (2016) demonstrated that an activation mean closer to zero can solve such a problem, “zero-like mean activation” has become an important property of activation functions.
- To incorporate both properties, one-sided saturation and zero-like mean activation, activation functions such as ELU, GELU, and SiLU were designed to allow small negative output to push the activation mean towards zero while saturating large negative output for noise-robustness.
- Nevertheless, a problem still remains: there is a trade-off between one-sided saturation and zero-like mean activation. Saturating large negative output naturally restricts the zero-like mean activation. For example, allowing larger negative outputs pushes the activation mean closer to zero but diminishes the noise-robustness of the activation function as the range of the saturation state decreases.
- Placing a LayerNorm layer parallel to an activation layer cannot address this trade-off problem. Large negative outputs will be restricted for saturation state. This is because the range of the saturation state entirely depends on the design of the activation function. 

Clevert, Djork-Arné, Thomas Unterthiner, and Sepp Hochreiter. "Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUS)." ICLR. 2016.

Reviewer comment (Theory):
I have concerns on the clarity of why LayerAct is noise-robust. The main claim is that the scale activation function is bounded (Eqn.8 and Eqn.9), thus the model is robust. But, bounding the activation function cannot bounding the $y_i$, which can still be not robust. This paper should well address this point. It is true the proposed LayerAct can obtain good empirical results on corruption datasets. However, this is not surprising, because the previous methods [1] have show that the combination of Batch-Free normalization (e.g., LayerNorm) and BatchNorm can be more robust for distribution shift (e.g., corruption). The main insights is that LayerNorm can alleviate the train-inference inconsistency problem of BatchNorm (I noted all experiments on noise-robust uses the networks with BatchNorm). I want to ask whether the proposed LayerAct can be noise-robust on the network without BatchNorm?

Author response:
#### 1-3. LayerAct can address the trade-off problem. 
- LayerAct functions address the trade-off between saturation and zero-like mean activation by integrating layer-direction normalization **into** the activation. The layer-direction normalization only determines the input of the activation scale function, which defines the saturation state (where $s\\left(n_{i}\\right)\\approx0)$. 
- This mechanism of LayerAct cuts off the link between saturation and negative activation output. Hence, LayerAct can produce larger negative output while maintaining saturation states. For example, in a scenario where $\\mu_{y}\\ll0$, the activation output $a_{i}=\\mu_{y}/2\\ll0$ where the activation input $y_{i}=\\mu_{y}$.  
- In conclusion, despite its structural simplicity, LayerAct functions offer a significant contribution by addressing the trade-off problem of activation. 


### Weakness 2. Concerns on the noise-robustness of LayerAct.
#### 2-1. The activation input $y_{i}$ is not bounded, which impact noise-robustness significantly. 
- We agree that the activation input $y_{i}$ significantly influences activation fluctuation. However, it is important to note that this challenge applies to all activation functions, not only to LayerAct functions. 
- Our objective was to demonstrate that LayerAct functions have smaller activation fluctuations **compared to** other activation functions under identical conditions, in our case, the same activation input.
- We recognize that our explanation in Subsection 3.2 of the submitted manuscript has not been sufficiently clear. Our objective was to demonstrate that LayerAct functions have smaller activation fluctuations **compared to** other activation functions under identical conditions, in our case, the same activation input.
- Comparing methods under identical conditions has been used to analyze the effects of a method. Santurkar et al. (2018) compared networks with and without BatchNorm under the same gradient conditions to assess BatchNorm’s impact on optimization. Similarly, Xu et al. (2019) conducted comparisons of different variants of LayerNorm under the same input conditions. 
- In this way, we showed that LayerAct functions can ensure more robust processing during the forward pass compared to other activation functions, given the same level of activation input. This benefit of LayerAct is supported by both mathematical analysis (in Subsection 3.2) and experimental results (in Subsection 4.1). 
- Thus, unless the training process does not lead to the generation of larger activation inputs in networks utilizing LayerAct, LayerAct is noise-robust compared to element-level activation functions. Please refer to Figure 7 in Appendix E, which demonstrates the distribution of activation input across activation functions. The figure shows that LayerAct functions typically maintain a similar or lower mean of activation inputs.
- In summary, LayerAct functions are designed to facilitate more robust processing during the forward pass, as was the primary aim of Equations 8 and 9. We acknowledge the need for clarity in our manuscript on this matter and will make revisions to eliminate any confusion.
Santurkar, S., Tsipras, D., Ilyas, A., and Madry, A. “How does batch normalization help optimization?”. Neurips. 2018
Xu, J., Sun, X., Zhang, Z., Zhao, G., and Lin, J. “Understanding and improving layer normalization.” Neurips. 2019

#### 2-2. Does the noise-robustness of LayerAct maintain without BatchNorm? 
- It is important to highlight that the activation input $y_{i}$ in Equations 8 and 9 is not exclusively derived from the output of a normalization layer. Additionally, in the experimental analysis detailed in Subsection 4.1, we did not use any normalization methods. The results from these experiments affirm that the noise-robustness of LayerAct is not contingent on the use of normalization layers. 
- To directly address your concern, we are conducting additional experiments using ResNets without any normalization layers. We have demonstrated the experimental results in our global response. In cases of networks without any normalization method, certain trials of ResNet20 and 44 with SiLU exploded during training. Consequently, we are presenting the result from networks with ReLU and LA-SiLU. 
- On clean datasets, the performance varied with the complexity of the networks: ResNet 32 and 44 with ReLU demonstrated better performance compared to those with LA-SiLU on both CIFAR10 and CIFAR100, whereas the opposite was true for ResNet. 
- Here, it is important to observe that networks with LA-SiLU outperformed those with ReLU on noisy datasets in all cases, which reconfirm the intended utility of LayerAct mechanism and contribution of our work. This experimental result shows that the noise-robustness of LayerAct is independent of the usage of normalization.

Reviewer comment (Writing):
I have concern on the title of this paper. The title addresses “ADVANCING CNNS WITH BATCHNORM THROUGH LAYER-DIRECTION NORMALIZATION”. However, I find the description of the proposed LayerAct is independent to the BatchNorm (e.g, this paper does not say how LayerAct alleviates the problem of CNN with BatchNorm). I think this paper should well clarify it.

Author response:
### Weakness 3. Concerns on the title of the paper. 
- Firstly, we wish to clarify that our intention was to present our contributions in a conservative manner. Considering that CNNs primarily utilize BatchNorm in their normalization layers, and our experimental results are based on ResNets with BatchNorm, the purpose of our title was to underscore the unique benefits of LayerAct as an activation mechanism in enhancing the performance of CNNs that utilize BatchNorm. 
- However, we acknowledge that the current title may inadvertently imply a direct association between LayerAct and BatchNorm. To eliminate any potential confusion, we have decided to revise the title to:
>- “LayerAct: Advanced activation mechanism utilizing layer-direction normalization for CNNs with BatchNorm” 
- If there are still concerns that the revised title might lead to the same confusion, another option could be to remove the specific reference to "BatchNorm.":
>- “LayerAct: Advanced activation mechanism utilizing layer-direction normalization for CNNs”

Reviewer comment (Theory):
Why “the sum of activation scale $\|s(n^{LN})\|$ will be similar across all samples”? Please clarify it in detail.

Author response:
#### 4-3. About the statement on the sum of activation scale. 
- We recognize that the statement is only true when all of the activation inputs across samples are well distributed. Therefore, we will revise the sentence “With LayerNorm, the sum of activation scale $||s\left(n^{LN}\right)||$ will be similar across all samples, which helps to reduce the variance of noise-robustness across all samples” to: 
>- “With LayerNorm, the sum of activation scale $||s\left(n^{LN}\right)||$ will have a lower upper bound across all samples, which helps to reduce the variance of noise-robustness across all samples.”

Reviewer comment (Experiments):
If LayerAct provides different effects from LayerNorm, there may be benefits the proposed function may bring for networks with LayerNorm. How does the proposed activation function behave with Transformers?

Author response:
### Future research: transformers with LayerAct. 
- Our analysis of the relationship between LayerAct and LayerNorm, particularly the observation that LayerAct may offer reduced benefits with LayerNorm compared to BatchNorm, is confirmed by additional experiments on ResNets with LayerNorm, detailed in our Meta response 2-2. 
- Consequently, we are considering utilizing PowerNorm as the normalization method for the transformer with LayerAct functions, which is a variant of BatchNorm for transformer architecture in NLP tasks. We look forward to the opportunity to present these findings to you in the future.