Paper: Dynamic Neural Response Tuning (ICLR-2024)

Reviewer comment (Novelty):
Limited comparison with existing methods: While the paper demonstrates the performance of DNRT compared to existing activation functions, it would be helpful to see a more in-depth analysis and discussion of the differences between DNRT and other approaches, including spiking neural networks (SNNs). SNNs are based on neural dynamics and employ dynamic activation functions, which share similarities with the ideas presented in this paper.

Author response:
> **Q1:** (a) Can the authors provide more in-depth comparisons between the proposed DNRT mechanism and existing activation functions, discussing the advantages and disadvantages of each approach? (b) Additionally, please include a comparison with spiking neural networks, as they also employ dynamic activation functions.

We appreciate your constructive suggestion.

**(a)** The major disadvantage of existing activation functions in ANNs is their static nature, making them unable to identify and filter out all irrelevant signals. For instance, ReLU has a fixed activation threshold of 0, which allows it to quickly achieve sparse activation response with its simple and efficient design. However, due to its static nature, there will inevitably be some irrelevant stimuli that exceed the threshold and trigger activation responses in ReLU, hindering the key feature extraction. The same drawbacks exist in activations such as sigmoid, ELU, GELU, and those mentioned in Related Work §2.3. In contrast, the proposed RAA mimics the human brain's dynamic redundant feature filtering mechanism. It adjusts the threshold for each input signal based on the signal's relevance degree, thus avoiding the aforementioned flaw.

**(b) Comparison with SNNs.** Leaky Integrate-and-Fire (LIF) is used as the activation mechanism in Spiking Neural Networks (SNNs), which simulates the process of potential integration and output in biological neurons. When the potential accumulates to a certain threshold, it triggers a pulse output, after which the potential gradually returns to the baseline level through the leakage mechanism.

The dynamic nature of LIF reflects in two aspects:

1. *Time dependence of potential.* At each moment, there will be potential accumulation or release. The potential varies over time. However, it is currently still in the exploration stage, without demonstrating performance advantages over existing approaches.

2. *Learnable thresholds.* The original LIF in SNN used a fixed threshold, and recently it has started supporting learnable thresholds (LIF-LT) (Wang et al., NeurIPS 2022), which shares similarities with our DNRT approach. **However, LIF-LT simply determines the threshold by a scalar learnable parameter, which still lacks flexibility. In LIF-LT, neurons in the same layer share the same threshold and cannot adapt it based on the contextual input. As a result, it is unable to avoid irrelevant features incorrectly triggering activation responses. In contrast, our DNRT judges the relevance of each input signal and adjusts the threshold accordingly for each signal, offering greater flexibility compared to LIF-LT's learnable threshold.**

Additionally, here is the numerical comparison on CIFAR-10. In terms of performance, the mechanism of DNRT remains advantageous.

|Top-1 Acc / %|LIF|LIF-LT|DNRT|
|-|-|-|-|
|ResNet-19|92.9|93.5|**95.6**|

Thank you once again for your valuable feedback. We will incorporate these discussions into the revised version.

Reviewer comment (Experiments):
Lack of generalization analysis: The paper claims that DNRT is applicable to various mainstream network architectures, but it would be beneficial to provide a more detailed analysis of how the proposed method generalizes to other tasks and domains beyond those presented in the experiments.

Author response:
> **Q2:** (a) How does the DNRT mechanism generalize to other tasks and domains beyond those presented in the experiments? (b) Are there any limitations or challenges in applying DNRT to different network architectures?

Thanks for your comment. Here is our explanation:

**(a) Generalization Analysis.** Besides the image classification task, the proposed DNRT can perform various other tasks or domains, including but not limited to semantic segmentation, non-computer vision tasks like GNN-based node classification, and some special tasks like tackling highly imbalanced data. For numerical experimental details, please refer to **the \*initial\* submitted manuscript §5.5 (pp. 8) "Generalization to Other Tasks".**

*(Note: We are not sure what "those presented in the experiments" in your comment refers to. If it refers to §5.5 of the initial paper, then we are willing to provide analysis on more other tasks and domains.)*

**(b)** No, the proposed DNRT is adaptable to various ANN architectures including MLPs, CNNs, ViTs and GNNs. These ANN architectures are employed in a wide range of mainstream deep learning tasks. Table 1,2,3,5,6,7 in the paper can demonstrate the DNRT's broad applicability across different ANN architectures.

Reviewer comment (Reproducibility):
Potential implementation challenges: The paper does not discuss the possible challenges in implementing the proposed DNRT mechanism in real-world applications or the computational cost associated with its use.

Author response:
> **Q3:** What are the potential implementation challenges and computational costs associated with using the DNRT mechanism in real-world applications? Are there any strategies to mitigate these challenges?

Thank you for your valuable feedback. We would like to address each of the points you've raised.

**1. Implementation Challenges.** We have designed our DNRT to be adaptable to various ANN architectures including MLPs, CNNs, ViTs and GNNs. The code is modularized and can be easily integrated into existing systems without requiring extra special hardware support. Therefore, the implementation challenges of applying DNRT in real-world applications are relatively minimal.

**2. Computational Costs**. Firstly, the proposed DNRT is designed with negligible additional **GPU overhead** during both training and inference. *Please refer to the \*initial\* submitted manuscript §A.4 (pp. 14~15) "Resource Demands" for numerical details*. 

Moreover, the impact of DNRT on **computational complexity** and **inference speed** is also minimal. Here we first present a numerical comparison on **computaional complexity**, measured in "FLOPs" (floating-point operations) as follows: 

|FLOPs|ReLU|DNRT|
|-|-|-|
|ViT-T|1.079 G|1.080 G|
|TNT-S|4.849 G|4.856 G|

We have also tested the network's **inference speed**, measured in "Latency" (the time required for model inference on a single image). The "Latency" is obtained, on average, from the model inferring images with 224×224 on a single GeForce RTX 3090 as follows: 

|Latency|ReLU|DNRT|
|-|-|-|
|ViT-T|4.5 ms|4.6 ms|
|TNT-S|10.1 ms|10.5 ms|

The result indicates that DNRT has relatively minimal impact on inference speed. 

Additionally, the gap in GPU overhead and inference speed can be further mitigated through low-level software and hardware optimization.
&nbsp;
Overall, DNRT is suitable for real-world applications while incurring minimal additional computational costs in terms of GPU overhead, computational complexity and inference speed, and we believe there is still a lot of space for low-level software and hardware optimization. We hope that these clarifications address your concerns.

Reviewer comment (Evaluation):
Can the authors provide more in-depth comparisons between the proposed DNRT mechanism and existing activation functions, discussing the advantages and disadvantages of each approach? Additionally, please include a comparison with spiking neural networks, as they also employ dynamic activation functions.

Author response:
> **Q1:** (a) Can the authors provide more in-depth comparisons between the proposed DNRT mechanism and existing activation functions, discussing the advantages and disadvantages of each approach? (b) Additionally, please include a comparison with spiking neural networks, as they also employ dynamic activation functions.

We appreciate your constructive suggestion.

**(a)** The major disadvantage of existing activation functions in ANNs is their static nature, making them unable to identify and filter out all irrelevant signals. For instance, ReLU has a fixed activation threshold of 0, which allows it to quickly achieve sparse activation response with its simple and efficient design. However, due to its static nature, there will inevitably be some irrelevant stimuli that exceed the threshold and trigger activation responses in ReLU, hindering the key feature extraction. The same drawbacks exist in activations such as sigmoid, ELU, GELU, and those mentioned in Related Work §2.3. In contrast, the proposed RAA mimics the human brain's dynamic redundant feature filtering mechanism. It adjusts the threshold for each input signal based on the signal's relevance degree, thus avoiding the aforementioned flaw.

**(b) Comparison with SNNs.** Leaky Integrate-and-Fire (LIF) is used as the activation mechanism in Spiking Neural Networks (SNNs), which simulates the process of potential integration and output in biological neurons. When the potential accumulates to a certain threshold, it triggers a pulse output, after which the potential gradually returns to the baseline level through the leakage mechanism.

The dynamic nature of LIF reflects in two aspects:

1. *Time dependence of potential.* At each moment, there will be potential accumulation or release. The potential varies over time. However, it is currently still in the exploration stage, without demonstrating performance advantages over existing approaches.

2. *Learnable thresholds.* The original LIF in SNN used a fixed threshold, and recently it has started supporting learnable thresholds (LIF-LT) (Wang et al., NeurIPS 2022), which shares similarities with our DNRT approach. **However, LIF-LT simply determines the threshold by a scalar learnable parameter, which still lacks flexibility. In LIF-LT, neurons in the same layer share the same threshold and cannot adapt it based on the contextual input. As a result, it is unable to avoid irrelevant features incorrectly triggering activation responses. In contrast, our DNRT judges the relevance of each input signal and adjusts the threshold accordingly for each signal, offering greater flexibility compared to LIF-LT's learnable threshold.**

Additionally, here is the numerical comparison on CIFAR-10. In terms of performance, the mechanism of DNRT remains advantageous.

|Top-1 Acc / %|LIF|LIF-LT|DNRT|
|-|-|-|-|
|ResNet-19|92.9|93.5|**95.6**|

Thank you once again for your valuable feedback. We will incorporate these discussions into the revised version.

Reviewer comment (Experiments):
How does the DNRT mechanism generalize to other tasks and domains beyond those presented in the experiments? Are there any limitations or challenges in applying DNRT to different network architectures?

Author response:
> **Q2:** (a) How does the DNRT mechanism generalize to other tasks and domains beyond those presented in the experiments? (b) Are there any limitations or challenges in applying DNRT to different network architectures?

Thanks for your comment. Here is our explanation:

**(a) Generalization Analysis.** Besides the image classification task, the proposed DNRT can perform various other tasks or domains, including but not limited to semantic segmentation, non-computer vision tasks like GNN-based node classification, and some special tasks like tackling highly imbalanced data. For numerical experimental details, please refer to **the \*initial\* submitted manuscript §5.5 (pp. 8) "Generalization to Other Tasks".**

*(Note: We are not sure what "those presented in the experiments" in your comment refers to. If it refers to §5.5 of the initial paper, then we are willing to provide analysis on more other tasks and domains.)*

**(b)** No, the proposed DNRT is adaptable to various ANN architectures including MLPs, CNNs, ViTs and GNNs. These ANN architectures are employed in a wide range of mainstream deep learning tasks. Table 1,2,3,5,6,7 in the paper can demonstrate the DNRT's broad applicability across different ANN architectures.

Reviewer comment (Reproducibility):
What are the potential implementation challenges and computational costs associated with using the DNRT mechanism in real-world applications? Are there any strategies to mitigate these challenges?

Author response:
> **Q3:** What are the potential implementation challenges and computational costs associated with using the DNRT mechanism in real-world applications? Are there any strategies to mitigate these challenges?

Thank you for your valuable feedback. We would like to address each of the points you've raised.

**1. Implementation Challenges.** We have designed our DNRT to be adaptable to various ANN architectures including MLPs, CNNs, ViTs and GNNs. The code is modularized and can be easily integrated into existing systems without requiring extra special hardware support. Therefore, the implementation challenges of applying DNRT in real-world applications are relatively minimal.

**2. Computational Costs**. Firstly, the proposed DNRT is designed with negligible additional **GPU overhead** during both training and inference. *Please refer to the \*initial\* submitted manuscript §A.4 (pp. 14~15) "Resource Demands" for numerical details*. 

Moreover, the impact of DNRT on **computational complexity** and **inference speed** is also minimal. Here we first present a numerical comparison on **computaional complexity**, measured in "FLOPs" (floating-point operations) as follows: 

|FLOPs|ReLU|DNRT|
|-|-|-|
|ViT-T|1.079 G|1.080 G|
|TNT-S|4.849 G|4.856 G|

We have also tested the network's **inference speed**, measured in "Latency" (the time required for model inference on a single image). The "Latency" is obtained, on average, from the model inferring images with 224×224 on a single GeForce RTX 3090 as follows: 

|Latency|ReLU|DNRT|
|-|-|-|
|ViT-T|4.5 ms|4.6 ms|
|TNT-S|10.1 ms|10.5 ms|

The result indicates that DNRT has relatively minimal impact on inference speed. 

Additionally, the gap in GPU overhead and inference speed can be further mitigated through low-level software and hardware optimization.
&nbsp;
Overall, DNRT is suitable for real-world applications while incurring minimal additional computational costs in terms of GPU overhead, computational complexity and inference speed, and we believe there is still a lot of space for low-level software and hardware optimization. We hope that these clarifications address your concerns.

Reviewer comment (Evaluation):
Considering the mainstream view that "computation is all you need," it is important to evaluate the additional computational costs and bandwidth bottlenecks introduced by DNRT. This is a concern as activation functions like Softplus and ELU are not widely used due to their lower computational efficiency compared to ReLU. Additionally, there are ongoing discussions about replacing the softmax layer in transformers with ReLU for improved computational efficiency. Could the authors provide an assessment of the impact of DNRT on model inference speed when replacing ReLU?

Author response:
> **Q4:** Considering the mainstream view that "computation is all you need," it is important to evaluate the additional computational costs and bandwidth bottlenecks introduced by DNRT. This is a concern as activation functions like Softplus and ELU are not widely used due to their lower computational efficiency compared to ReLU. Additionally, there are ongoing discussions about replacing the softmax layer in transformers with ReLU for improved computational efficiency. Could the authors provide an assessment of the impact of DNRT on model inference speed when replacing ReLU?

We appreciate your constructive suggestion. The impact of DNRT on model inference speed  when replacing ReLU is negligible. For numerical comparisons on inference speed, **please refer to our response to the previous question (*Q3*) above**. We will include these discussions in the revised version.

&nbsp;

We hope that our response can adequately address your concerns. If necessary, we are more than willing to provide further elaboration. Looking forward to your reply!

Reviewer comment (Presentation):
The default bibliography style should be reformatted to make searching the bibliography easier, e.g., numbered, last name first, initials only except for last name.

Author response:
> Perhaps reformat the bibliography for easier searching, eg numbered, last name first, initials only except for last name.  
>  
Thank you for your constructive comment. If ICLR allows, we will reformat it for easier searching.

Reviewer comment (Presentation):
The bibliography could be reformatted for easier searching, e.g., numbered, last name first, initials only except for last name.

Author response:
> Perhaps reformat the bibliography for easier searching, eg numbered, last name first, initials only except for last name.  
>  
Thank you for your constructive comment. If ICLR allows, we will reformat it for easier searching.

Reviewer comment (Writing):
The reviewer questions whether the first sentence of the abstract is an accurate summary of why ANNs have done so well, as there seems to be much more to the matter.

Author response:
> *"Artificial Neural Networks (ANNs) have gained broad applications across various fields due to their brilliant architecture designs."*  
**Q1:** Is this first sentence an accurate summary of why ANNs have done so well? It seems there is much more to the matter. 

Sorry for the confusion. We will modify the sentence to: "Brilliant architecture design is an important factor for the widespread application of Artificial Neural Networks (ANNs)." The original sentence was intended to highlight the importance of ANN architecture design as the paper proposes a novel activation function, which is part of the ANN architecture.

Reviewer comment (Writing):
The phrase "is achieved by triggering action potentials" is often true, but neurotransmitters and non-spiking neurons can also have prominent roles.

Author response:
> *"The transmission of information in neurons is achieved by triggering action potentials that propagate through axons."*  
**Q2:** "is achieved by triggering action potentials": Often. Neurotransmitters and non-spiking neurons can also have prominent roles.

Thank you for pointing it out. It should be "The transmission of information in neurons is *often* achieved by triggering action potentials that propagate through axons."

Reviewer comment (Writing):
The use of "dynamically adjusts" may be incorrect; from the paper, it seems the activation function is modified to weight inputs differently, but once trained the response properties are fixed, so "dynamically" may not be accurate.

Author response:
> *"RAA dynamically adjusts the response condition based on the strength and characteristics of the input signal."*  
**Q4:** "dynamically adjusts": my sense from the paper is that the activation function is modified to weight inputs differently, but once trained the response does not change dynamically, ie its response properties are fixed. If this is correct, "dynamically" is not a correct word.

Thank you for your comment. In the view of contemporary science, the behavior of biological neurons is driven by complex biological mechanism/rules. Therefore, the dynamic behavior of certain neurons on activation threshold adjustment is also driven by specific rules. In other words, based on specific rules, neurons can possess the ability to behave dynamically. Similarly, in our proposed DNRT in ANNs, the behavior of DNRT is driven by learned complex rules. This behavior, which involves adjusting the activation threshold in response to real-time context, reflecting a high level of flexibility, can thus be considered dynamic.

Additionally, for instance, the Capsule Network ("Dynamic Routing Between Capsules", Hinton et al. NIPS 2017) dynamically selects the feature vectors' propagation route, but this dynamic behavior is in fact driven by a pre-defined routing-by-agreement algorithm. DynamicViT (Rao et al. NeurIPS 2021) trained a lightweight prediction module "to determine which tokens to be pruned in a dynamic way". These methods driven by learned rules also use the word "dynamic", to some extent reflecting their general perspective on dynamic behavior.

Reviewer comment (Writing):
The phrase "ARR is devide" could be clarified by giving brief detail, to parallel the detail already given about RAA in the preceding sentence.

Author response:
> *"ARR is devised to enhance the network’s ability to learn category specificity."*  
**Q5:** "ARR is devised": Maybe give some brief detail about this, to parallel the detail given about RAA is the sentence above.

Thanks for your suggestion. We extend the sentence to: "ARR is devised to enhance the network’s ability to learn category specificity by imposing constraints to the network's response distribution."

Reviewer comment (Novelty):
In paragraph 1, the reviewer suggests finding more comprehensive reviews for ANN progress (e.g., Goodfellow, etc.).

Author response:
> **Q6:** Paragraph 1: Perhaps find more comprehensive reviews for ANN progress (eg Goodfellow, etc).

Thanks for your constructive suggestion. In addition to the references already cited in the paper, we will include more reviews for ANN progress in the revised version including but not limited to:

[1] Goodfellow et al., Deep learning. *MIT press*, 2016.  
[2] Pouyanfar et al., A survey on deep learning: algorithms, techniques, and applications. *ACM computing surveys*, 2018.

Reviewer comment (Theory):
In paragraph 3, the statement "otherwise it suppresses" is true for ReLU but not for other activation functions such as tanh; this would also be a good place to cite literature reviews of activation function varieties.

Author response:
> **Q7:** Paragraph 3 "otherwise it suppresses": This is true for ReLU, but not for other activation functions eg tanh. Also, this would be a good place to cite some good literature reviews of flavors of activation functions.

Thanks for your insightful comment. The sentence will be modified to "otherwise, *in most cases*, it suppresses ...". We will cite more reviews of flavors of activation functions in the revised version including but not limited to:

[1] Sharma et al., Activation functions in neural networks. *Towards data sci*, 2017.  
[2] Nwankpa et al., Activation functions: comparison of trends in practice and research for deep learning. *ArXiv preprint arXiv:1811.03378*, 2018.  
[3] Dubey et al., Activation functions in deep learning: a comprehensive survey and benchmark. *Neurocomputing*, 2022.

Reviewer comment (Writing):
The phrase "An input value ranked later among all inputs" is unclear. It may refer to time-dependent neural response, but this was not addressed in the paper.

Author response:
> **Q8:** "An input value ranked later among all inputs": Does this refer to time-dependent neural response? I did not see this addressed as a technique or effect in the paper.

Sorry for the confusion. This sentence refers to the classical GELU activation function, which is not time-dependent like that in spiking neural networks. In short, GELU assumes that smaller inputs (ranks lower) is more likely to be suppressed, whose curve is similar to ReLU but better simulates reality. We compared GELU with the proposed DNRT in the experiment.

Reviewer comment (Writing):
The claim "always adheres to a gaussian curve" is framed too simply; in addition, within this paragraph, the distinction between ANNs and BNNs is unclear.

Author response:
> *"Due to the presence of slight natural noise, the distribution always adheres to a Gaussian curve."*  
**Q9:** (a) "always adheres to a gaussian curve": Always? This framing is too simple. (b) Also in this paragraph: the distinction between when referring to ANNs or BNNs is unclear. 

(a) We apologize for our wording. A more rigorous expression would be: "Due to the presence of slight natural noise, the distribution *often approximately* adheres to a Gaussian curve." The Gaussian distribution is often employed as an approximate distribution for errors in the natural world, which can be explained by the [central limit theorem](https://en.wikipedia.org/wiki/Central_limit_theorem).

(b) Sorry for the confusion. In this paragraph, the text before "In experiments with ANNs" is discussing BNNs, and the rest is about ANNs. We will modify the first sentence of this paragraph to: "... after the signals aggregate in the *biological* neural system ...".

Reviewer comment (Writing):
The sentence "It is a a common ... final decision." is vague and could be sharpened.

Author response:
> *"It is a common assumption to consider a Gaussian distribution of errors between the model’s predicted value and the true value (the mean value). So, the high Gaussian variances indicate the specificity of the response is not strong enough. It makes the
response less distinct between different categories, affecting the network’s final decision."*  
**Q10:** "It is a common ... final decision.": This strikes me as vague. Can the sentence be sharpened?

Sorry for the confusion. **Based on the experimental observation in Fig. 1(b), 2(b) and 3(b), the ANN's aggregated response in each channel approximately adheres to Gaussian distribution**, which can also be explained by the central limit theorem, i.e., in the case of adding/averaging/aggregating independent and identically distributed (which is also a commonly used assumption in nature and machine learning) random variables, as the sample size increases, the distribution of this sum/mean/aggregation tends to be a Gaussian distribution even if the original variables themselves are not normally distributed. **The Gaussian distribution reflects the presence of error in ANN's responses, and the distribution variance reflects the magnitude of this error.** As shown in Fig. 3(b, left), the response is pretty noisy and severely overlaps between categories. **The high Gaussian variances indicate the specificity of the response is not strong enough, making it difficult to distinguish between different categories through their responses** compared with those in Fig. 3(b, right)**, posing a significant challenge for the ANN to make correct judgments on categories.**

(The sharpened expression would be the connection of all sentences in **bold**.)

Reviewer comment (Writing):
In section 2.1, the phrase "surpasses the threshold" is accurate for spiking neurons but does not account for non-spiking types and local spread of neurotransmitters.

Author response:
> *"When the stimulus surpasses the threshold, an action potential that travels to the terminal is produced, ..."*  
**Q11:** "surpasses the threshold": For spiking neurons (see above comment about non-spiking types and local spread of neurotransmitters).

Thank you for pointing it out. It should be "*For spiking neurons,* when the stimulus surpasses the threshold, an action potential that travels to the terminal is produced, ..."

Reviewer comment (Writing):
In section 2.2, the phrase "pivotal role in neural networks" is ambiguous—does it refer to ANN, BNN, or both?

Author response:
> *"The activation function plays a pivotal role in neural networks ..."*  
**Q12:** "pivotal role in neural networks": Does this refer to ANN, BNN, or both?

Sorry for the confusion. It refers to ANN since this section discusses the previous research on ANN's activation functions. We will modify the sentence to: "The activation function plays a pivotal role in *artificial* neural networks ..."*

Reviewer comment (Theory):
At the bottom of page 3, the statement "Biological NNs..." is important, but it is unclear whether the proposed method truly emulates this or simply modifies the activation function to a new, static form.

Author response:
> *"Biological neural networks comprise neurons responsible for distinct locations with different response thresholds that dynamically adjust to the environmental conditions."*  
**Q13:** Bottom of page 3, "Biological NNs...": This is an important observation. However, does the proposed method really emulate this, or does it modify the activation function to a new, static, form?

We sincerely apologize for any potential confusion caused. **The proposed method is flexibly dynamic _(please refer to our explanation in Q4)_, and have the ability to change its activation threshold based on real-time context, which emulates the BNN's corresponding dynamic working pattern.** It's able to identify and filter out irrelevant signals by activation threshold adjustment to avoid irrelevant features incorrectly triggering activation responses. **Fig. 2(a) & 3(a) of the paper** intuitively demonstrate the effectiveness of the proposed method: With the static activation function, the presented responses are active on almost every channel. Some irrelevant inputs that should not be activated might be mistakenly activated. After using the proposed approach, the activation responses become significantly sparser, implying that most irrelevant features are suppressed. We hope our explanation can address any doubts or concerns you may have.

Reviewer comment (Writing):
In section 3.1, last sentence, while BNNs feature dynamically changing activation functions, it is unclear if the proposed method does this—this may relate to the reviewer's earlier lack of understanding.

Author response:
> *"Biological neurons implement dynamic response conditions, which can help us improve existing activation functions."*  
**Q14:** last sentence: While BNNs feature dynamically changing activation functions, does the proposed method do this **(same as previous comment)**? Perhaps this is really a statement about my lack of understanding of the proposed method.

We apologize once again for any potential confusion caused. **Please refer to our explanation in the previous question (*Q13*) above**, which we believe will adequately address your concern.

Reviewer comment (Writing):
In section 3.2, first sentence, the reviewer believes this is too simple a description.

Author response:
> *"The information aggregated by biological neurons exhibits Gaussian distributions with a high level of category specificity/distinctiveness."*  
**Q15:** first sentence: I believe this is too simple a description.

Thanks for your comment. We have made the description clearer: "For different categories of stimuli, the information aggregated by biological neurons exhibits significant differences, indicating the response specificity of neurons. Due to the presence of slight natural noise, the response often approximately adheres to a Gaussian distribution."

Reviewer comment (Theory):
While the paper draws inspiration from the dynamic nature of biological neurons, a deeper exploration into the biological underpinnings could have been beneficial. More direct parallels between the DNRT mechanism and real-world neural behaviors would strengthen the paper's foundational claims. Consider integrating more biological studies or references that showcase the parallels between DNRT and actual biological neural behaviors, reinforcing the biological accuracy of the proposed mechanism.

Author response:
We appreciate your constructive feedback. Here, we provide more details on DNRT's biological plausibility and its ability to mimic the dynamic behaviors observed in biological neurons:

**(a)** Regarding information transmission, dynamic threshold is a regulatory mechanism of biological neurons, which maintains neuronal homeostasis by adjusting the firing rate of action potentials and can be considered as an adaptation to membrane potential. This dynamic threshold mechanism has been widely observed in various nervous systems [1,2,3]. Moreover, the neuronal response threshold can also be influenced by both intrinsic and extrinsic factors such as medications [4] and nervous system ailments [5]. These biological evidences influenced the design of DNRT's RAA part.

Regarding information aggregation, for different categories of stimuli, the information aggregated by biological neurons exhibits significant differences, indicating response specificity of neurons [6]. Due to the presence of slight natural noise, the response often approximately adheres to a Gaussian distribution [7,8]. These biological evidences influenced the design of DNRT's ARR part.

**References**  
[1] Fontaine et al., Spike-threshold adaptation predicted by membrane potential dynamics in vivo. *PLOS computational biology*, 2014.  
[2] Azouz et al., Dynamic spike threshold reveals a mechanism for synaptic coincidence detection in cortical neurons in vivo. *In Proceedings of the national academy of sciences of the united states of america*, 2000.  
[3] Sun, Experience-dependent intrinsic plasticity in interneurons of barrel cortex layer IV. *Journal of neurophysiology*, 2009.  
[4] Arancio et al., Activity-dependent long-term enhancement of transmitter release by presynaptic 3’, 5’-cyclic gmp in cultured hippocampal neurons. *Nature*, 1995.  
[5] Knowland et al., istinct ventral pallidal neural populations mediate separate symptoms of depression. *Cell*, 2017.  
[6] Kreiman et al., Category-specific visual responses of single neurons in the human medial temporal lobe. *Nature*, 2000.  
[7] Fischer, A history of the central limit theorem: from classical to modern probability theory. *Springer science & business media*, 2010.  
[8] Averbeck et al., Neural correlations, population coding and computation. *Nature reviews neuroscience*, 2006.

**(b)** Theoretically, the proposed RAA incorporates a dynamic activation threshold that adjusts in response to the real-time context. It has the ability to identify and filter out irrelevant signals by dynamic threshold adjustment to avoid irrelevant features incorrectly triggering activation responses, which highly emulates the BNN's corresponding dynamic threshold. The proposed ARR imposes constraints on the neural response distribution to achieve more focused response in each channel, enabling the distinguishability of responses across different categories, thus facilitating the ANN’s final decisions and attempting to approach the high discriminative ability of biological neural networks for categories. In practice, Fig. 2 & 3 of the paper intuitively showcase that DNRT behaves according to the biological plausibility we anticipated.

Reviewer comment (Evaluation):
The paper provides comprehensive experimental evaluations, but a detailed discussion on potential limitations or scenarios where DNRT might not be as effective is missing. Understanding these limitations can provide a more balanced view and guide future work. Dedicate a section to discuss potential limitations, challenges in broader applicability, or scenarios where traditional activation functions might still be preferable.

Author response:
Thank you for your valuable comment.

1\. *Architectures.* DNRT is adaptable to various ANN architectures including MLPs, CNNs, ViTs and GNNs. These ANN architectures are employed in a wide range of mainstream deep learning tasks. Table 1,2,3,5,6,7 in the paper can demonstrate the DNRT's broad applicability across different ANN architectures.

2\. *Tasks.* In certain scenarios e.g. the box regression part of object detection, the ARR mechanism of DNRT may not be applicable. This is because ARR is connected to categories, whereas box regression is not directly associated with categories. Despite this, the generalization ability of DNRT is already commendable, as it can perform various other tasks or domains, including image classification, semantic segmentation, non-computer vision tasks like GNN-based node classification, and some special tasks like tackling highly imbalanced data, whose generalization has been verified in experiments of the paper.

3\. *Datasets.* DNRT was not designed for a specific scenario from the beginning. Its internal mechanism is geared towards various types of data. The experiments in the paper were evaluated on general datasets such as CIFAR, ImageNet, ADE20K, etc., and achieved remarkable results. Therefore, we believe that in specific application scenarios, DNRT can be equally effective.

Reviewer comment (Reproducibility):
The paper mentions the adaptability of RAA and hints at additional parameters, but a clear breakdown of how DNRT introduces and manages these parameters, especially in comparison to traditional activation functions, is lacking. Provide a subsection detailing the parameterization of RAA, including how these parameters are learned, their impact on network complexity, and potential challenges in optimization.

Author response:
We appreciate your valuable comment.

**(a)** RAA comprises *d+1* parameters: a weight vector $\mathbf{w}$ with length *d* and a scalar bias $b$. The code of RAA is modularized to manage these parameters.

**(b)** The impact of these extra parameters on **computaional complexity** and **inference speed** is minimal. Here we first present a numerical comparison on computaional complexity, measured in "FLOPs" (floating-point operations) as follows: 

|FLOPs|ReLU|DNRT|
|-|-|-|
|ViT-T|1.079 G|1.080 G|
|TNT-S|4.849 G|4.856 G|

We have also tested the network's inference speed, measured in "Latency" (the time required for model inference on a single image). The "Latency" is obtained, on average, from the model inferring images with 224×224 on a single GeForce RTX 3090 as follows: 

|Latency|ReLU|DNRT|
|-|-|-|
|ViT-T|4.5 ms|4.6 ms|
|TNT-S|10.1 ms|10.5 ms|

The results indicates these extra parameters have minimal impact on computaional complexity and inference speed. Additionally, the gap in the inference speed can be further reduced through low-level software and hardware optimization.

**\(c)** No, the parameters in RAA can be trained through backpropagation in an end-to-end manner.

We hope that these clarifications could address your concerns.

Reviewer comment (Experiments):
RAA method introduces an additional parameter, W, which may compromise the fairness of comparisons with other activation functions.

Author response:
Thank you for your valuable feedback. Firstly, here is the quantitative comparison of the number of parameters employed for each model **(given in the \*original\* paper submission  §5.7 (pp. 9) & §A.4 (pp. 14~15) "Resource Demands")**.

|Network|#Params of Baseline|#Params of Baseline with RAA only|#Params of Baseline with DNRT (RAA+ARR)|
|-|-|-|-|
|MLP|0.62 M|0.62 M|0.62 M|
|ViT-T|5.75 M|5.76 M|5.76 M|
|DeiT-T|5.91 M|5.92 M|5.92 M|
|CaiT-XXS|11.96 M|11.98 M|11.98 M|
|PVT-T|13.23 M|13.24 M|13.24 M|
|TNT-S|23.76 M|23.77 M|23.77 M|
|AlexNet|61.10 M|61.11 M|61.11 M|
|VGG-11|132.87 M|132.88 M|132.88 M|
|ResNet-18|11.69 M|11.69 M|11.69 M|
|MobileNet|3.50 M|3.51 M|3.51 M|
|ShuffleNetV2|2.28 M|2.28 M|2.28 M|
|Segmenter|6.94 M|6.95 M|6.95 M|
|GCN|7.84e-3 M|7.91e-3 M|7.91e-3 M|
|GraphSAGE|1.55e-2 M|1.56e-2 M|1.56e-2 M|
|ResNet-32|20.62 M|20.63 M|20.63 M|

In the proposed DNRT, ARR does not introduce any parameters. A single RAA comprises only *d+1* parameters: a weight vector $\mathbf{w}$ with length *d* and a scalar bias $b$. The parameter growth rate of the models discussed in the paper ranges from 0.008% to 0.16%, which is negligible.

**Regarding the concern on comparison fairness,** i.e., whether the RAA's introduced parameters are the primary contributor to the performance improvement, we give another experiment where an extra layer of linearity (ex-L) is added into ViT-T's MLPs, with the same parameter count as RAA. The comparison on CIFAR-100 is shown as follows **(also given in the \*original\* paper submission §5.7 (pp. 9) & §A.4 (pp. 14~15) "Resource Demands")**.

|Method|ReLU|GELU|ex-L + ReLU|ex-L + GELU|RAA|
|-|-|-|-|-|-|
|#Params (with ViT-T)|5754472|5754472|5763700|5763700|5763700|
|Top-1 Acc / %|65.7|64.0|66.0|64.2|**66.7**|

The performance on RAA outperforms the pure ex-L, which indicates the primary improvement stems from the RAA intrinsic mechanism (eliminating redundant/irrelevant signals) rather than simply the extra parameters.

Finally, we are open to making necessary revisions to include the parameter counts in the main tables.

Reviewer comment (Novelty):
The authors did not state clearly of the significance of the study. What is the inspiration of this study for neuroscience, or for the development of deep learning?

Author response:
We appreciate your suggestion which can help enhance the depth of our paper. Below, we provide a comprehensive explanation:

ANNs fundamentally aim to replicate the operational principles of the human brain. Nevertheless, **imperfections within such artificial creations remain a constant, inviting us to identify and address these issues**. Biological plausibility (mimicking biological information transmission and aggregation) is one consideration in the design of DNRT(RAA+ARR). Another crucial aspect is that **deficiencies in ANNs concerning information transmission and aggregation do exist**, so the primary objective of DNRT is to **address the corresponding observed deficiencies of ANNs**, and no such targeted alternatives currently exist, which makes the design necessary, and it holds great promise in advancing the field and tackling future challenges for deep learning community.

**Insights of RAA.** Existing activations in ANNs lack in-depth insights and often encounter challenges in achieving desired activation sparsity due to their static nature, as shown in the visual results of §5.6 & §A.3. The proposed RAA addresses this limitation by dynamically adjusting the activation threshold based on the relevance of each input signal. Sparser activation response helps the data fitting since only crucial features get activated while redundant/irrelevant features being suppressed.

**Insights of ARR.** The aggregated response in ANNs tend to exhibit Gaussian distributions with high variances on each channel when stimulated with specific categories, as shown in the visual results of §5.6 & §A.3. The high variances suggests that the network's response to each category has large errors and is not discriminative enough. The proposed ARR addresses this flaw by imposing constraints to the ANN's response distribution, leading to improved category specificity and decision-making.

In conclusion, apart from the aspect of biological plausibility, the motivation behind the development of DNRT(RAA+ARR) stems from the desire to overcome limitations in ANNs concerning information transmission and aggregation by replicating the complex operational principles of the human brain, helping enchance performance across various deep learning mdoels and tasks. As the deep learning community continues to evolve, the integration of DNRT offers a promising avenue for advancing the field and addressing the challenges ahead.

Reviewer comment (Experiments):
In the presented experiments, it is crucial to disclose the number of parameters employed for each model. To ensure a fair comparison, the parameters for the baseline models must be comparable. Please provide this information in a clear and concise manner.

Author response:
Thank you for your valuable feedback. Firstly, here is the quantitative comparison of the number of parameters employed for each model **(given in the \*original\* paper submission  §5.7 (pp. 9) & §A.4 (pp. 14~15) "Resource Demands")**.

|Network|#Params of Baseline|#Params of Baseline with RAA only|#Params of Baseline with DNRT (RAA+ARR)|
|-|-|-|-|
|MLP|0.62 M|0.62 M|0.62 M|
|ViT-T|5.75 M|5.76 M|5.76 M|
|DeiT-T|5.91 M|5.92 M|5.92 M|
|CaiT-XXS|11.96 M|11.98 M|11.98 M|
|PVT-T|13.23 M|13.24 M|13.24 M|
|TNT-S|23.76 M|23.77 M|23.77 M|
|AlexNet|61.10 M|61.11 M|61.11 M|
|VGG-11|132.87 M|132.88 M|132.88 M|
|ResNet-18|11.69 M|11.69 M|11.69 M|
|MobileNet|3.50 M|3.51 M|3.51 M|
|ShuffleNetV2|2.28 M|2.28 M|2.28 M|
|Segmenter|6.94 M|6.95 M|6.95 M|
|GCN|7.84e-3 M|7.91e-3 M|7.91e-3 M|
|GraphSAGE|1.55e-2 M|1.56e-2 M|1.56e-2 M|
|ResNet-32|20.62 M|20.63 M|20.63 M|

In the proposed DNRT, ARR does not introduce any parameters. A single RAA comprises only *d+1* parameters: a weight vector $\mathbf{w}$ with length *d* and a scalar bias $b$. The parameter growth rate of the models discussed in the paper ranges from 0.008% to 0.16%, which is negligible.

**Regarding the concern on comparison fairness,** i.e., whether the RAA's introduced parameters are the primary contributor to the performance improvement, we give another experiment where an extra layer of linearity (ex-L) is added into ViT-T's MLPs, with the same parameter count as RAA. The comparison on CIFAR-100 is shown as follows **(also given in the \*original\* paper submission §5.7 (pp. 9) & §A.4 (pp. 14~15) "Resource Demands")**.

|Method|ReLU|GELU|ex-L + ReLU|ex-L + GELU|RAA|
|-|-|-|-|-|-|
|#Params (with ViT-T)|5754472|5754472|5763700|5763700|5763700|
|Top-1 Acc / %|65.7|64.0|66.0|64.2|**66.7**|

The performance on RAA outperforms the pure ex-L, which indicates the primary improvement stems from the RAA intrinsic mechanism (eliminating redundant/irrelevant signals) rather than simply the extra parameters.

Finally, we are open to making necessary revisions to include the parameter counts in the main tables.

Reviewer comment (Experiments):
It is imperative that the chosen baseline models are well-established and representative of the current state of the art in deep learning. Notably, state-of-the-art deep learning methods have demonstrated the ability to achieve accuracy levels exceeding 85% on the ImageNet dataset. Please confirm that the selected baselines adhere to this standard.

Author response:
We appreciate your constructive feedback. The baseline models in our paper are taken from [`timm`](https://github.com/huggingface/pytorch-image-models) and [`torchvision`](https://github.com/pytorch/vision/tree/main/torchvision) libraries, and the training settings on ImageNet-1K are provided by the [`timm`](https://github.com/huggingface/pytorch-image-models). We apologize for using a smaller batch size and less GPUs than previous literature during training due to the limited computing resources we had, which may result in the difference between the results we obtained and those reported in previous literature. However, in principle, we have ensured that each experiment in a set of comparative studies used the same batch size and other public training settings, allowing the results to fairly demonstrate that our approach can bring improvements across different model architectures.

Moreover, we agree that the chosen baseline models need to represent the current state of the art in deep learning. Therefore, we employ a state-of-the-art model architecture `MaxViT-B` with the input image resolution of 384×384 and apply the proposed DNRT on it, the experimental results on ImageNet-1K are as follows:

|Top-1 Acc / %|ReLU|GELU|DNRT|
|-|-|-|-|
|MaxViT-B|86.0|85.9|**86.5**|

which demonstrate the effectiveness of the proposed DNRT on larger and more advanced models. We will incorporate this information to enhance the overall quality of the manuscript.

Reviewer comment (Novelty):
Apart from the aspect of biological plausibility, it is essential to elucidate the motivations behind the development of RAA and AAR for the deep learning community. Please provide a comprehensive explanation of the underlying inspirations.

Author response:
We appreciate your suggestion which can help enhance the depth of our paper. Below, we provide a comprehensive explanation:

ANNs fundamentally aim to replicate the operational principles of the human brain. Nevertheless, **imperfections within such artificial creations remain a constant, inviting us to identify and address these issues**. Biological plausibility (mimicking biological information transmission and aggregation) is one consideration in the design of DNRT(RAA+ARR). Another crucial aspect is that **deficiencies in ANNs concerning information transmission and aggregation do exist**, so the primary objective of DNRT is to **address the corresponding observed deficiencies of ANNs**, and no such targeted alternatives currently exist, which makes the design necessary, and it holds great promise in advancing the field and tackling future challenges for deep learning community.

**Insights of RAA.** Existing activations in ANNs lack in-depth insights and often encounter challenges in achieving desired activation sparsity due to their static nature, as shown in the visual results of §5.6 & §A.3. The proposed RAA addresses this limitation by dynamically adjusting the activation threshold based on the relevance of each input signal. Sparser activation response helps the data fitting since only crucial features get activated while redundant/irrelevant features being suppressed.

**Insights of ARR.** The aggregated response in ANNs tend to exhibit Gaussian distributions with high variances on each channel when stimulated with specific categories, as shown in the visual results of §5.6 & §A.3. The high variances suggests that the network's response to each category has large errors and is not discriminative enough. The proposed ARR addresses this flaw by imposing constraints to the ANN's response distribution, leading to improved category specificity and decision-making.

In conclusion, apart from the aspect of biological plausibility, the motivation behind the development of DNRT(RAA+ARR) stems from the desire to overcome limitations in ANNs concerning information transmission and aggregation by replicating the complex operational principles of the human brain, helping enchance performance across various deep learning mdoels and tasks. As the deep learning community continues to evolve, the integration of DNRT offers a promising avenue for advancing the field and addressing the challenges ahead.