Paper: Generalized Activation via Multivariate Projection (ICLR-2024)

Reviewer comment (Writing):
The exposition of the paper is problematic because there is a very long motivation for the MPU (both Section 2.1 and Section 3 serve as a motivation), but very little analysis of the MPU itself. For example: (i) there is no explicit definition of the MPU or the MPU layer; (ii) there is no discussion on how to choose the cone (only specifying which cone is used in the experiments); (iii) no discussion on how to perform the projection (which is relegated to an appendix); (iv) no discussion on the theoretical computational complexity (only an empirical computation of MACS).

Author response:
**A1 / A4**: We apologize for the confusion caused. Indeed, the previous structure of Section 2 is not clear enough, and thank you for pointing this out! To address these problems, we have revised the manuscript in the following aspects: 

1. Add another Section 2.2 to **formally introduce** our method after the motivation Section 2.1 in the revised manuscript, where we explicitly define the MPU unit in Definition 1. 

2. Add the **selection of cones** in Section G, where we mainly show the selection of three factors of the cone, including the cone's vertex and axis, the dimension of the cone, and the half-apex angle $\alpha$ of  the cone. Given all these factors, the cone is well-established.

3. **Explicit calculation** of cone projections is already derived in Theorem 4. We further refer to this theorem in Section 2.2.

4. Add the analysis for **computational complexity** analysis in Section H. 

We hope these revisions help to improve this paper.

Reviewer comment (Experiments):
The results do not seem very strong. Ignoring the artificial datasets, on CIFAR-10 (Table 1) it is inside 1 standard deviation of Leaky ReLU. On ImageNet (Table 2), LeakyReLU is superior but also inside 1 standard deviation. DeiT (Appendix F) is only provided for a single run and no comparison, and ReLU is still only at 0.07 distance. In addition, there are not many important ablations and comparisons, including on the choice of the cone or its dimensionality, and many baselines are missing.

Author response:
**A2 / A5**: Thank you for the helpful comments. We agree that, based on the rather limited experiments conducted, the improvements are by no means significant. However, we would like to point out that, for the sake of fairness, the same hyperparameters were maintained  across all experiments, with the sole variation being the Activation  Function (AF). Under these conditions, the enhancement in test accuracy is **typically marginal**. To substantiate our viewpoint, Table 8 of Dubey et al. [4] shows a comprehensive benchmark of the existing Single-Input Single-Output (SISO) activation functions under the same training configuration, such as ReLU, PReLU, ELU, GELU, e.t.c. From these empirical results, we can see that usually, the modification of the activation function alone cannot drastically increase the accuracy of the network, compared to the celebrated ReLU (The best performance on ResNet50 is 94.09% by CELU, ReLU achieves 93.74% already).  

Moreover, to further illustrate the empirical improvement of the proposed MPU, we have added more experiments on both the existing AFs, such as the Winner-Takes-All (WTA)[1], the CReLU[2] and MaxOut[3], and the proposed method on a deeper ResNet101. Both results further illustrate the superiority of the proposed MPU in most settings. Notably, the new empirical results on ResNet101 show a significant increase in the accuracy of the proposed 2-dimensional Leaky MPU, whose average accuracy reaches the highest among all. Moreover, its maximum test accuracy also achieves the highest.

As for the experiments on the choice of the cone and its dimensionality, we are now performing additional experiments for further investigation. We sincerely thank you for this suggestion. However, we are still running the code, and we will report our results later before the discussion stage ends.

Reviewer comment (Novelty):
The related works section is very shallow. Some works on multi-input activation functions (e.g., Maxout, CReLU) are mentioned but not used in the comparisons. Many other works are missing (see, e.g., https://arxiv.org/pdf/2005.00817.pdf), including winner-take-all activation functions and network-in-network models. Also, complex activation functions can natively work with 2 inputs and 2 outputs and can be generalized (e.g., quaternion, octonion) for more dimensions.

Author response:
**A3 / A6**: We sincerely thank the reviewer for the suggestions. We have modified the related works section to include more representing approaches and divided them into four categories for clarity. Moreover, we have added more comparison experiments to Section 4, including winner-takes-all [1], Maxout [2], and CReLU [3]. We can observe that our MPU still beats these methods in the tested settings.

Moreover, we assume that by "complex activation functions", you are referring to the activation functions of the complex-valued neural networks. We have reviewed some related papers briefly added the related contents to the literature review section.

Reviewer comment (Writing):
There is a lack of a clear definition of the activation function and insufficient discussion of its design, including the choice of the cone and the computational complexity of the projection operation.

Author response:
**A1 / A4**: We apologize for the confusion caused. Indeed, the previous structure of Section 2 is not clear enough, and thank you for pointing this out! To address these problems, we have revised the manuscript in the following aspects: 

1. Add another Section 2.2 to **formally introduce** our method after the motivation Section 2.1 in the revised manuscript, where we explicitly define the MPU unit in Definition 1. 

2. Add the **selection of cones** in Section G, where we mainly show the selection of three factors of the cone, including the cone's vertex and axis, the dimension of the cone, and the half-apex angle $\alpha$ of  the cone. Given all these factors, the cone is well-established.

3. **Explicit calculation** of cone projections is already derived in Theorem 4. We further refer to this theorem in Section 2.2.

4. Add the analysis for **computational complexity** analysis in Section H. 

We hope these revisions help to improve this paper.

Reviewer comment (Experiments):
There is no demonstration of at least one use case where the activation function provides significant improvements, either in accuracy or time.

Author response:
**A2 / A5**: Thank you for the helpful comments. We agree that, based on the rather limited experiments conducted, the improvements are by no means significant. However, we would like to point out that, for the sake of fairness, the same hyperparameters were maintained  across all experiments, with the sole variation being the Activation  Function (AF). Under these conditions, the enhancement in test accuracy is **typically marginal**. To substantiate our viewpoint, Table 8 of Dubey et al. [4] shows a comprehensive benchmark of the existing Single-Input Single-Output (SISO) activation functions under the same training configuration, such as ReLU, PReLU, ELU, GELU, e.t.c. From these empirical results, we can see that usually, the modification of the activation function alone cannot drastically increase the accuracy of the network, compared to the celebrated ReLU (The best performance on ResNet50 is 94.09% by CELU, ReLU achieves 93.74% already).  

Moreover, to further illustrate the empirical improvement of the proposed MPU, we have added more experiments on both the existing AFs, such as the Winner-Takes-All (WTA)[1], the CReLU[2] and MaxOut[3], and the proposed method on a deeper ResNet101. Both results further illustrate the superiority of the proposed MPU in most settings. Notably, the new empirical results on ResNet101 show a significant increase in the accuracy of the proposed 2-dimensional Leaky MPU, whose average accuracy reaches the highest among all. Moreover, its maximum test accuracy also achieves the highest.

As for the experiments on the choice of the cone and its dimensionality, we are now performing additional experiments for further investigation. We sincerely thank you for this suggestion. However, we are still running the code, and we will report our results later before the discussion stage ends.

Reviewer comment (Experiments):
There is a need for more ablations and baselines, especially of activation functions closely linked to the paper.

Author response:
**A3 / A6**: We sincerely thank the reviewer for the suggestions. We have modified the related works section to include more representing approaches and divided them into four categories for clarity. Moreover, we have added more comparison experiments to Section 4, including winner-takes-all [1], Maxout [2], and CReLU [3]. We can observe that our MPU still beats these methods in the tested settings.

Moreover, we assume that by "complex activation functions", you are referring to the activation functions of the complex-valued neural networks. We have reviewed some related papers briefly added the related contents to the literature review section.

Reviewer comment (Presentation):
There are multiple references in the citations to "MS Windows NT kernel description" which may not be relevant or appropriate.

Author response:
**A7**: We apologize for the confusion, and we have corrected the citations in the revised version.

Reviewer comment (Writing):
On page 4, the statement "the set S is a certain polyhedron" is unclear; it is necessary to clarify what polyhedron is meant or point to a specific definition in the paper.

Author response:
**A8**: We apologize for the confusion. After thorough thinking, we found that the corresponding example is too complicated to explain and is not rigorous enough, so we removed this example for the simplicity of the presentation.

Reviewer comment (Writing):
A definition for the cone \( C_n \) should be added.

Author response:
**A9**: Thank you for the comment, and we apologize for the confusion. In the revised manuscript, we have added a formal definition in Section 2.1 to enhance the clarity of the paper.

Reviewer comment (Writing):
The organization and writing of the paper need to be improved.

Author response:
We apologize for not organizing the paper properly, which led to some confusion. We have made some modifications to the previous manuscript. Some significant changes in relation to the organization and clarity of the paper are listed below:

1. We have added Section 2.2 to **formally introduce** the proposed method. We believe that this division allows the reader to quickly locate the method that we propose and better follow the discussion.

2. We have added the **selection of cones** in Section G, where we mainly show the selection of three factors of the cone, including the cone's vertex and axis, the dimension of the cone, and the half-apex angle $\alpha$ of  the cone. Given all these factors, the cone is well-established.

3. To increase the readability of the paper, we have restructured Appendix Sections E and G to illustrate the three aspects of the MPU's implementation clearly.

4. We have introduced the **rigorous definition** of the $n$-dimensional cones, the MPU, and the projection cones $C_\alpha^{(m)}$.

Reviewer comment (Theory):
The proposed Theorem 1 is not so rigorous.

Author response:
Thank you for the careful reading and the valuable comment. We apologize for not writing the theorem rigorously. Is the confusion due to comparing a single PGD iteration with a shallow FNN featuring a linear-activation-linear structure? This comparison may be misleading, as the two do not necessarily correlate directly. We have refined Theorem 1 and all the other related illustrations to connect a single PGD iteration with a single FNN layer. Furthermore, we have also explicitly written the expression of the FNN for better clarity.

We are not sure if this resolves your concern. If not, we would be happy to answer more questions!

Reviewer comment (Experiments):
The experiments are not enough.

Author response:
Thank you. We have performed more experiments suggested by the reviewers, such as benchmarking more existing activation functions, including Winner-takes-all [1], Maxout [2], and CReLU [3], and investigating the performance of the proposed MPU on deeper network architectures, such as ResNet101. The results are reported in the revised manuscript in Sections 4 and F.

Moreover, we have also performed additional experiments on the MPU with higher dimensional cones, i.e., $m=4, 5, 6$, to show the reason why we chose small $m$, and the result is reported in Table 7. The result shows the superiority of the 2-dimensional MPU over those in higher dimensions. We sincerely thank you for this suggestion.

Reviewer comment (Theory):
The projected gradient descent (PGD) algorithm in proposition 1 has many iterative steps. However, the Theorem 1 has a single layer, which corresponds to a single iteration of the PGD algorithm. The structural connection is weak.

Author response:
Thank you for your valuable comment. First, we want to stress that this structural connection only serves as the **motivation** for our derived activation function. Moreover, we apologize for not making it clear in the paper, and we want to briefly explain the connection here. First, the connection between a single layer of the PGD and a single FNN layer is established in Theorem 1. Moreover, for the neural network with multiple layers, an RNN (or, equivalently, an FNN where the weights and biases of each layer are equivalent) exactly corresponds to the PGD iteration. To further clarify the relationship for multiple iterations, we have also added Corollary 1 in the revised manuscript.

Moreover, this connection has also been noticed by some other literature, especially in the field of Learning to Optimize (L2O); see [4] for a comprehensive review. For example, Gregor and Lecun [5] leverage this connection and construct a neural network to implement a truncated form of the Iterative Shrinkage and thresholding Algorithm (ISTA), a special case of the proximal gradient descent algorithm introduced in Proposition 2, where the function $g$in equation 6 is chosen as the vector 1-norm $\|\cdot\|_1$.

Reviewer comment (Experiments):
How to set \(\alpha\) in Theorem 1 and Proposition 2 in the network training? It is especially so for the convolutional network. It is better to give some discussion on the setting of \(\alpha\).

Author response:
Actually, the $\alpha$ in Theorem 1 and Theorem 2 do not have the same meaning, and we are deeply sorry for the confusion caused. The $\alpha$ in Theorem 1 means the step size of the PGD process, and the $\alpha$ in Theorem 2 denotes the half-apex angle of the second-order cone. In the revised manuscript, we have changed the $\alpha$ in Theorem 1 and Proposition 2 to $\gamma$ for clarity. 

For the parameter $\gamma$ in Theorem 1 and Proposition 2, we solely leverage this expression to discuss the correspondence between a single layer of FNN and a single iteration of PGD and show that these two terms are exactly the same if the weights and biases of the FNN are chosen as the expression in Theorem 1. However, this $\gamma$ is not considered in the training process. We still directly learn the weights and biases of the FNN during training.

Reviewer comment (Theory):
The paper only gives the derivations with fully-connected feedforward networks. It does not present the skip-connection. Since the experiments use ResNets, it is better to discuss this issue.

Author response:
The reason why we consider FNN in Section 2 is merely for theoretical analysis. To the best of our knowledge, most existing results on the expressive capability of neural networks also focus on shallow FNN or CNN, such as multiple versions of the universal approximation theorem summarized in Wikipedia: universal approximation theorem or the univariate approximation theorem for the CNN [6]. However, the expressive capability of ResNet architecture is less studied in theory. In our paper, we analyze the expressive capability for a shallow FNN theoretically. As for ResNet with skip-connections, we empirically illustrate in the experiment section that the incorporation of MPU  also pushes the ResNet toward enhanced performance.

Reviewer comment (Experiments):
The paper only tests the proposed MPU with ResNet18. It is not a very deep network. Is the proposed MPU unable to train deep networks? It is best to give an experiment to train deep networks, such as ResNet101. This is very critical, which determines whether the proposed MPU can be applied in practice.

Author response:
We agree that the performance of the proposed method on a deeper network is also critical. To show this, we have performed benchmarking of the proposed method on ResNet101 architecture, and the result is shown in Section F. We can observe that our MPU still introduces higher accuracy when compared to baseline methods.

Reviewer comment (Presentation):
The organization of section 2 needs to be improved. It is hard to follow.

Author response:
Thank you for your comment, and we apologize for any confusion. To enhance the clarity of Section 2, we have added Section 2.2 after the motivation section to **formally describe** our MPU and the definition of the $n$-dimensional cone. Other than that, we have revised the structure of Section E to clearly show the three aspects of the MPU's implementation and added Section G to discuss the selection of cones. We hope these modifications enhance the readability of the overall paper.

Reviewer comment (Presentation):
When referencing equations, it is better to use “\eqref”.

Author response:
Thanks for your kind suggestion. However, all of the references of equations are already "\eqref". We assume that you are referring to using the format "(1)", where 1 is the equation number, to refer to a certain equation. This is indeed the original reference format provided by the amsmath package of LaTeX. However, in the ICLR template, this command is redefined as "equation 1" or "Equation 1", which is the kind of format in the paper. 

Thanks again for your valuable comments! Please let us know if you have any more questions.

Reviewer comment (Theory):
*[Removed due to answers by authors:]* Theory is basic and simple. But more importantly: it does not really, at the essence, establish why they function should give better results. Sure, you can set the weights just correctly so that a layer is like an iteration. But, so what? The weights are actually learned, and our goal is not to really learn an iteration...

Author response:
**A1**: Thank you for your insightful comment, and we are sorry for any confusion caused by the presentation of the previous manuscript. First, we would like to clarify that we **DO NOT intend to** learn an iteration of the Projected Gradient Descent (PGD) algorithm. The connection between a shallow FNN and one iteration of the projected gradient descent algorithm only serves to **motivate the usage of MPU to replace ReLU**, as in projected gradient descent, the projection can be a general projection to a convex set instead of projection to non-negative numbers (ReLU). The actual neural networks with MPU are by no means relevant to iterations of PGD.

On the other hand, the potential benefit of using an MPU instead of a simple ReLU is validated both theoretically and empirically. To be specific, Theorem 2 theoretically demonstrates that any FNN with ReLU activation can be faithfully reparameterized by an FNN with MPU activation, while the vice versa is impossible regardless of the width of the ReLU network. We believe that this result theoretically guarantees that our proposed structure has the ability to give a better performance than the ReLU function. Moreover, empirical results in Section 4 also show that in the training procedure, the proposed method also gives a better result than most of the existing activation functions tested.

Reviewer comment (Theory):
*[Removed due to answers by authors:]* I am a bit skeptical about the motivation. The way I see it, the whole idea in the activation is to introduce *some* nonlinearity. It does not have to be a lot, to really get a low of expressive power! Just by doing many layers, we can take a really small amount of non-linearity and make it a lot. So, not sure that MIMO really will help...

Author response:
**A2**: Thank you so much for the valuable comments. We agree that we can increase the number of layers to enhance the expressive power of the neural network. However, we would like to first clarify that our approach, which potentially makes one layer more expressive, is **orthogonal** to the "go deeper" approach. The direction of increasing the expressive power of activation units alone is also explored in various activation function research, such as PReLU [1], CReLU [2], and so on.

In numerical experiments, we find that the best test accuracy (Table 1) achieved by a ResNet18 with MPU is similar to that of a ResNet34 using ReLU, which could be an indicator to show in certain settings, it is much more economical to use a "shallower" network with more "non-linear" activation functions, rather than using a "deeper" network with small "non-linear" activations, such as ReLU.  

Moreover, we also added another experiment on the deeper architecture ResNet101, whose result is reported in Appendix Section G. The result also shows that by **combining the benefit** of a deep network and an activation function with more nonlinearity (the proposed Leaky MPU), the test accuracy achieves the highest among all the architectures tested.

Reviewer comment (Evaluation):
*[Added due to answers by authors:]* Limited potential improvements: the authors observed that only using very small input-output size (namely, m=2, which replaces one input one output with two input two output) is the best you can do. Going to m=3 and above does not help. This suggest that there is very little to gain from MIMO non-linearity.

Author response:
**A1**: Thank you for the valuable question! We hope to further explain the superiority of MIMO functions over SISO functions in the following two aspects:
1. **The core difference in the expressive capability of neural networks lies in the shift from SISO AFs to MIMO AFs, rather than that between MIMO AFs with different dimensions.** Intuitively, we can compare the expressive capability of the "true" MIMO AFs (those not reducible to element-wise operations), e.g., the projection to the 2-dimensional second-order cone from $\mathbb{R}^2\to\mathbb{R}^2$, with that of the "fake" MIMO AFs, e.g., the 2-dimensional ReLU function. According to the illustration of our paper, we can also view the 2-dimensional ReLU as the projection from $\mathbb{R}^2$to the polygon $\{(x, y)\mid x\geq 0, y\geq 0\}
$. However, this type of element-wise projection fails to capture the intrinsic variable correlations present in "true" MIMO AFs (This can also be observed by the calculation equation of the MPU in Theorem 4). Generally, the superiority of the MIMO AFs over SISO AFs in the sense of expressive capability is theoretically proved in Theorem 2. Moreover, empirical results in Section 4.1 further illustrate the advantages of the "true" MIMO functions.
2. Indeed, the lower-dimensional MPUs, like the 2D version, tend to outperform their higher-dimensional peers. However, **this is only the case for our MPU, and doesn’t necessarily mean all MIMO AFs will show this pattern.** We still believe that there are a lot more "true" MIMO functions worth exploring, such as the proximal operators in Section 3. We are also not sure whether this phenomenon is also present in other MIMO activation functions. We will continue exploring this direction and try to find the answer behind it!

Reviewer comment (Evaluation):
Empirical improvement is very small, and even then not always achieved.

Author response:
**A3**: Thank you for your question. We agree that, based on the rather limited experiments conducted, the improvements are by no means significant. However, we would like to point out that, for the sake of fairness, the same hyperparameters were maintained  across all experiments, with the sole variation being the Activation  Function (AF). Under these conditions, the enhancement in test accuracy is **typically marginal**. To substantiate our viewpoint, Table 8 of Dubey et al. [3] shows a comprehensive benchmark of the existing Single-Input Single-Output (SISO) activation functions under the same training configuration, such as ReLU, PReLU, ELU, GELU, e.t.c. From these empirical results, we can see that usually, the modification of the activation function alone cannot drastically increase the accuracy of the network, compared to the celebrated ReLU (The best performance on ResNet50 is 94.09% by CELU, ReLU achieves 93.74% already). 

Moreover, to further illustrate the empirical improvement of the proposed MPU, we have added more experiments on both the existing AFs, such as the Winner-Takes-All (WTA)[4], the CReLU[2] and MaxOut[5], and the proposed method on a deeper ResNet101. Both results further illustrate the superiority of the proposed MPU in most settings.

Reviewer comment (Reproducibility):
How are MIMO activations implemented? *[Following rebuttal:]* Still unclear on how exactly the projection itself is implemented.

Author response:
**A2**: Sorry for the confusion caused. Indeed, in the previous version of the paper, the implementation of the cones may be less straightforward for the readers to understand, since the corresponding details are mainly introduced in Appendix E and G and we deeply thank you for pointing this out.

In the revised manuscript, the implementation of the projection is described in the following three aspects, introduced in Section 2.2: Method:
1. Integration of the MPU into neural networks: We've provided a thorough walkthrough in Section 2.2.
2. Formulation of m-dimensional MPUs: With the analytical equations at hand, we've made sure that readers can easily translate the MPU into executable code. This part is detailed in Appendix B and E, and we've included references to it in Section 2.2 to streamline the paper's flow.
3. Parameter selection in MPUs: We address this in Appendix G and have included references in Section 2.2 as well, so readers can navigate the content with ease.

Reviewer comment (Experiments):
Why use small m in the experiments? *[Following rebuttal:]* OK, understood. However, this raises the question on why MPU is a good idea if only very small m is used. See new weakness above.

Author response:
**A5**: Thanks for the question. The dimension $m$ of the MPU serves as a tradeoff between the expressive power of a single MPU (a stronger expressive capability for a larger $m$) and the total number of MPUs. This is because the inputs are flattened and divided into several length-$m$ blocks, so the number of MPU decreases as $m$ increases. After comparing the empirical results in our experiments, we found that the modification from ReLU to the 2-dimensional cone benefits the performance the most, and increasing m to 3 or larger may not result in a better performance.

Indeed, it is beneficial to discuss the choice of cone (including its dimension m), and thank you for pointing this out. Therefore, we have added Section G to discuss the choice of cone and conducted a series of experiments in Section 4, which shows the superiority of 2-dimensional MPUs to 3-dimensional ones with ResNet18. Moreover, we have added the experimental results for $m=4, 5, 6$, which are reported in Table 7. The result also shows the superiority of the MPUs with small dimensions $m$.

Reviewer comment (Experiments):
Table 1: improvements in training error should be compared with cost (MACS here). With MPU you get slighly better errors with slightly more MACS. What happens with ReLU if you increase the MACS a bit, e.g. by doing a few more epochs? Sure, you test ResNet 34 and 50 in which MACS increase. But there is more than one way to increase the MACS... *[Following rebuttal:]* OK, doing more epochs might make sense less, but there are other ways. e.g. slightly wider network?

Author response:
**A6**: We agree that there is more than one way to increase the MACS of neural networks. However, we observe that in ResNet18, the training process already converges so that the performance gain would be minimal or even negative with more epochs. To support our opinion, we have performed an additional experiment on matching the MACS of different activations on ResNet18 with a larger number of training epochs. The results are reported in Table 5 of the revised manuscript. Conversely, the proposed activation function increases the MACS by only 0.5% (71.17G to 71.56G) but is **theoretically guaranteed to have a stronger expressive capability and better empirical results in most cases.**

Moreover, compared to changing the model to ResNet34 or 50, which almost doubles or even triples the MACS, the modification of the activation function seems a more economical choice.

Thanks again for your questions and comments. Please let us know if you have any more questions!

Reviewer comment (Experiments):
*[New following rebuttal:]* The authors use the same hyperparameters across all experiments, with the sole variation being the Activation Function (AF)." Why? And, more importantly, how the results will look if all algorithms use their best hyperparameters?

Author response:
**A4**: Thank you for your insightful question! The reason why we originally adopted the same hyperparameters across all experiments is for the fairness of the experiments. We controlled all other factors as the same, and let the AFs be the only variation, to clearly observe the impact of the AFs. Similar methods are also adopted in many other benchmark papers, such as Dubey et al. [1].

Indeed, it is necessary to dig into the potential of each AF and adopt their best parameters respectively. However, searching for the best hyperparameters for each AF is extremely time-consuming, and we may not be able to provide the final result before the discussion stage ends. 

Thanks again for your time!