Paper: DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimization (ICLR-2024)

Reviewer comment (Novelty):
My biggest concerns with the paper are its limited novelty and a lack of proper ablations.

Author response:
> 1. Limited novelty as three mechanisms introduced in DrM closely follow the prior works.

Our work does not claim to reinvent the three mechanisms from scratch. Instead, the central contribution of our study lies in the innovative application of the dormant ratio as a pivotal indicator of the agent's activity level. We uniquely employ the dormant ratio as a guiding principle, integrating it with the three pre-existing mechanisms. This integration allows for a dynamic balancing of exploration and exploitation strategies: the agent is directed to prioritize exploration when the dormant ratio is high, shifting its focus to exploitation as the dormant ratio decreases. We firmly believe that the novel application of the dormant ratio in orchestrating these mechanisms, supported by strong and extensive empirical results across a diverse range of challenging locomotion and manipulation tasks, adds significant value to the community of visual reinforcement learning.

Reviewer comment (Novelty):
None of the three mechanisms introduced by DrM are novel in-and-of themselves. As noted in the paper, "dormant-ratio-guided perturbation" follows the previous work of (D'Oro et al., 2023; Ash & Adams, 2020). Similarly, "dormant-ratio-guided exploitation" is based on the work of Ji et al. (2023). The idea of scheduling the exploration rate goes back a very long way, pre-dating deep RL, and some recent approaches such as Agent57 (Badia et al.) also use a form of adaptive exploration.

Author response:
> 1. Limited novelty as three mechanisms introduced in DrM closely follow the prior works.

Our work does not claim to reinvent the three mechanisms from scratch. Instead, the central contribution of our study lies in the innovative application of the dormant ratio as a pivotal indicator of the agent's activity level. We uniquely employ the dormant ratio as a guiding principle, integrating it with the three pre-existing mechanisms. This integration allows for a dynamic balancing of exploration and exploitation strategies: the agent is directed to prioritize exploration when the dormant ratio is high, shifting its focus to exploitation as the dormant ratio decreases. We firmly believe that the novel application of the dormant ratio in orchestrating these mechanisms, supported by strong and extensive empirical results across a diverse range of challenging locomotion and manipulation tasks, adds significant value to the community of visual reinforcement learning.

Reviewer comment (Novelty):
What is new is the idea of using the dormant ratio to schedule the hyperparameters of these methods. While this is novel, it represents rather limited novelty in the reviewer's opinion.

Author response:
> 1. Limited novelty as three mechanisms introduced in DrM closely follow the prior works.

Our work does not claim to reinvent the three mechanisms from scratch. Instead, the central contribution of our study lies in the innovative application of the dormant ratio as a pivotal indicator of the agent's activity level. We uniquely employ the dormant ratio as a guiding principle, integrating it with the three pre-existing mechanisms. This integration allows for a dynamic balancing of exploration and exploitation strategies: the agent is directed to prioritize exploration when the dormant ratio is high, shifting its focus to exploitation as the dormant ratio decreases. We firmly believe that the novel application of the dormant ratio in orchestrating these mechanisms, supported by strong and extensive empirical results across a diverse range of challenging locomotion and manipulation tasks, adds significant value to the community of visual reinforcement learning.

Reviewer comment (Experiments):
While Figure 10 contains some ablations, the three mechanisms are each ablated in their entirety. The reviewer would prefer that only the adaptive scheduling component be ablated for each method, since this is the novel part. For a fair comparison, DrM ought to be compared against DrQ + perturbation resets (with fixed $\alpha$) + "Blended Exploration and Exploitation operator" (with fixed $\lambda$).

Author response:
> 2. Ablations for adaptive scheduling components: compare DrM against DrQ with fixed factors for perturbation resets, blended exploration and exploitation operator.


Thank you for your valuable suggestion regarding the ablation study. This is indeed a good baseline to compare with in order to show that the three dormant ratio-based components are crucial for the success of DrM. In response, we have updated Figure 10 to include the proposed comparison as an additional baseline. This revised figure now illustrates that DrM substantially outperforms DrQ-v2, with three added mechanisms being fixed and independent of the dormant ratio. This finding further strengthens the contribution of our algorithm, demonstrating its effectiveness and the significance of our approach to use the dormant ratio as a guiding principle.

Reviewer comment (Experiments):
The ablations only consider one domain (Adroit) and the results are aggregated across all Adroit tasks. The reviewer would prefer to see more ablations in greater granularity, as they are the most important part of the experiments. Without them, it is very hard to understand why the improvements work. The argument that DrM lowers the dormant ratio and hence improves performance seems "chicken-egg" to the reviewer; it is unclear which is the cause and which is the effect, and only the dormant-ratio-guided perturbations seem to directly target the dormant ratio.

Author response:
> 3. Ablations only consider one domain (Adroit) and the results are aggregated across all Adroit tasks.

Although we believe that the results of three tasks from one domain are more than sufficient to demonstrate the effectiveness of each component, as opposed to many existing works that only conduct ablation study on one task, we include more ablation studies with 2 tasks on Meta-World to further substantiate our claim. We plot the results on 2 additional Metaworld tasks as well as 3 individual Adroit tasks (as opposed to the aggregated results of Figure 10) in Appendix G.

> 4. The argument that DrM lowers the dormant ratio and hence improves performance seems very "chicken-egg". Only the dormant-ratio-guided perturbations seem to directly target the dormant ratio.

We would like to emphasize, as illustrated in Figure 2 of the main text and Figure 11 in the Appendix, that the dormant ratio effectively indicates the agent's activity level. Notably, the reduction in dormant ratio consistently precedes any increase in reward. This observation suggests that it is not the increase in reward that causes the dormant ratio to drop. Instead, the decrease in dormant ratio leads the agent to actively explore its environment. Subsequently, as the agent begins to acquire reward signals, it starts to learn skills, resulting in an increase in rewards. This underscores the importance of the dormant ratio in the learning process of the agent.

Reviewer comment (Experiments):
Section 3.1 is too unscientific. For example, the claim "This suggests that the dormant ratio acts as an intrinsic metric, influenced more by the diversity and relevance of the agent's behaviors than by its received rewards" is based on one example, and the claim that RL agents learn relevant behaviours before their returns improve sounds dubious. Another too-bold claim is: "As the dormant ratio captures the intrinsic characteristics of an agent's policy network, all the hyperparameters in this approach are robust to tasks and domains." This is never established, and Table 1 in the Appendix suggests that some parameters of DrM require domain-specific tuning.

Author response:
> 5a. Section 3.1 is too unscientific, e.g., the claim "This suggests that the dormant ratio acts as an intrinsic metric, influenced more by the diversity and relevance of the agent's behaviors than by its received rewards". This is all based on one example.

We appreciate your concern regarding the basis of our claim. We would like to clarify that the claim is not solely based on a single example. In fact, the observation consistently emerges across our experiments spanning various tasks, and we have demonstrated this in Figure 11 of the original manuscript, which includes data from three additional DeepMind Control (DMC) tasks.

To further address your concern and provide a clearer understanding, we provide additional qualitative visualizations in addition to the task Hopper Hop (Figure 1). These visualizations showcase the agent's behavior in relation to its dormant ratio throughout the training progress. We have made these visualizations available as videos on our anonymous project webpage, which can be accessed at https://drm-rl.github.io/asset/dormant.mp4.

> 5b. The claim that RL agents learn relevant behaviours before their returns improve sounds dubious. 

In addition to the supplementary videos that qualitatively visualize the agent's behavior in relation to the dormant ratio, here we wish to highlight that when the dormant ratio is low, RL agents begin to execute meaningful actions, actively exploring their environment. This contrasts with scenarios where the dormant ratio is high, and the agent remains largely immobile. It's important to note that these "meaningful actions" are not always directly linked to the task specified by the reward function. Therefore, what we typically observe is a decrease in the agent's dormant ratio prior to any increase in reward. During this phase, although the task reward remains low, the agent begins to actively explore. Subsequently, this effective exploration leads to the agent learning task-relevant behaviors, culminating in an increase in task reward.

> 6. Some parameters of DrM require domain-specific tuning, which is conflicted with the claim in the paper.


Thank you for pointing this out. This indeed deserves further clarification. In Appendix D.2, we now present the performance of DrM across various tasks using a single, unified set of hyperparameters, as opposed to domain-specific ones. We observed that DrM off-the-shelf generally performs well across all tasks with this single set of hyperparameters. However, for certain tasks, finetuning specific DrM hyperparameters does yield additional performance improvements. It's also important to note that the hyperparameter tuning required for DrM is minimal. The adjustments we made were limited to the maximum perturbation rate, selecting from $0.6$ and $0.9$, and modifying the exploitation expectile value by selecting from $0.7$ and $0.9$.

Reviewer comment (Writing):
The claim in the abstract that DrM has "no broken seeds (76 seeds in total)" is vague, as it is unclear what constitutes "broken" and the matter is never returned to in the main body of the paper.

Author response:
> 7. The claim in the abstract that DrM has "no broken seeds (76 seeds in total)" is a little vague (what constitutes "broken"?) and it's never really returned to in the main body of the paper.

“Broken seeds” means seeds where the agents fail to learn a successful policy, and the episode reward of the agent remains low (close to zero) throughout the training process. We appreciate your suggestion on this point, and we have added the additional clarifications on broken seeds in the experiment section.

Reviewer comment (Theory):
In the "dormant-ratio-guided exploitation" section, it is mentioned that "value underestimation occurs as the agent starts to acquire skills, given that $\pi$ is sub-optimal". The reviewer does not follow this reasoning, as they would expect $\pi$ to become less sub-optimal as the agent starts to improve.

Author response:
> 8. In the "dormant-ratio-guided exploitation" section, it's mentioned that "value underestimation occurs as the agent starts to acquire skills, given that $\pi$ is sub-optimal". I don't quite follow this. As the agent starts to improve, wouldn't we expect $\pi$ to become less sub-optimal?


Thanks for your insightful question. We have adjusted the writing of our manuscript to clarify this exploitation mechanism further. Here, the intuition is that, especially in the early stages of training, the replay buffer could contain a very small number of high-quality episodes (i.e., episodes with high reward) that the agent has encountered through exploration. However, in this training stage, $\pi$ is suboptimal, and so $Q$-value is often underestimated due to insufficient exploitation of high-quality samples in the replay buffer. 

Furthermore, this under-exploitation is for more than just the early stages of training. In hard visual continuous control tasks, what we have observed is that effective exploitation from high-quality examples is necessary for the agent to learn an optimal policy. This is evident from the ablation study of three Adroit tasks, where removing this mechanism results in significant performance degradation (Figure 9).

Thus, to mitigate this underestimation, we introduce a value estimator $V$, which estimates the high expectile of $Q$, designed to converge more rapidly than $Q$-values. Therefore, this mechanism allows the RL agent to quickly exploit its historically successful trajectories without introducing additional overestimation.

Reviewer comment (Novelty):
Why is the method using shrink and perturb rather than directly resetting inactive units? Is it possible that S&P is introducing additional exploration due to the noisy updates to the parameters, similarly to the "noisy nets" strategy of Fortunato et al.?

Author response:
1. **Dormant Ratio Based Perturbation**
Dormant ratio based perturbation is a simple mechanism that periodically perturbs the network by scaling down the original network weight (with magnitude of the scaling/perturbing controlled by dormant ratio) and injecting some noise into it.

> Q1: Why is the method using shrink and perturb rather than directly resetting inactive units?

Thanks for your thoughtful question. We conducted an experiment comparing our approach with directly resetting the inactive (dormant) neurons in the updated Appendix F. The results clearly demonstrate that our method significantly outperforms the approach of directly resetting the weights of dormant neurons. Upon analyzing the results, simply resetting weights of dormant neurons does not consider the holistic effect on the entire layer of the network. For example, a part of neurons in the middle layer cannot directly determine the output in a specific action dimension. In a fully connected network, we need to contemplate resetting all neurons rather than selectively discarding weights of dormant neurons. Additionally, neurons that are dormant but have large gradients during a certain period might become active in the next period, making it challenging to determine which neurons' weights should be retained or reset based on a single batch of data. In contrast, our perturbation-based approach encourages exploration, effectively reducing the dormant ratio, enabling the agent to learn better behavior, and ultimately achieving better performance.


> Q2: Is it possible that S&P is introducing additional exploration due to the noisy updates to the parameters, similarly to the "noisy nets" strategy of Fortunato et al.? 
    
Thank you for your insightful question. We would like to clarify that the perturbation mechanism in DrM is fundamentally different from the NoisyNet technique mentioned. In NoisyNet, exploration is encouraged by perturbing the output of the policy/value network, achieved by injecting noise into the weights of the neural network at every timestep. In contrast, our approach in DrM involves updating or refreshing the weights only after a relatively long fixed time interval (2e+5 frames). This key difference underlines the distinct nature of our perturbation mechanism compared to the NoisyNet strategy. Nevertheless, we acknowledge the relevance of the NoisyNet strategy to our work and will include it in our reference list. We have also updated our writing in Section 3.2 accordingly to emphasize the distinction from NoisyNet.

Reviewer comment (Theory):
The exploration strategy appears to actually introduce *more* noise after the network is "activated" than it used before activation due to the max operator. However, the motivation for this noise adaptation is to introduce more noise early in training when the network's behaviours are less interesting. This should have the effect of increasing exploratory behaviour throughout training compared to using a schedule, but not in the way claimed by the paper.

Author response:
> Q2: The exploration strategy appears to actually introduce more noise after the network is "activated" than it used before activation due to the max operator. 

Thanks for your insightful question. This is actually not the case empirically. The incorporation of the max operator in our method serves primarily to prevent a substantial rise in the dormant ratio after initial activation. In practice, however, this operator is almost never invoked. Once the dormant ratio is awakened, it usually decreases sharply, making the second term, $\sigma_\text{linear}(t − t_0)$, the predominant factor. Consequently, in Equation 3, keeping only $\sigma_\text{linear}(t − t_0)$ post-awakening would not significantly change the exploration schedule.

Reviewer comment (Theory):
The use of expectiles introduces a large benefit in training based on the ablation study, but the optimal expectile seems to depend on the environment. Is there additional justification for using expectiles beyond the empirical benefit and the additional knob to tune in the training algorithm?

Author response:
3. **Dormant-ratio-guided exploitation:** 
    > Q1. The use of the term "exploitation" seems inaccurate to me. 

    > Q2. Could the authors provide additional justification for the expectile prediction component of their method, and illustrate what types of environments this might prove detrimental for?


   
    Thank you for raising this important point. We acknowledge that our use of the term "exploitation" might require further clarification. In our context, "exploitation" refers to a strategy that could allow the RL agent to quickly exploit its serendipitous, unexpectedly successful experiences from its exploration process. The key insight that we observe is that, especially in the early stages of training, the replay buffer could contain scarce high-quality episodes (i.e., episodes with high reward) that the agent has encountered through exploration. In this training stage, $\pi$ is suboptimal, and the $Q$-value is often underestimated due to insufficient exploitation of high-quality samples in the replay buffer. Thus, to mitigate this underestimation, we introduce a value estimator $V$, which estimates the high expectile of $Q$, designed to converge more rapidly than $Q$-values. Therefore, this mechanism allows the RL agent to quickly exploit its historically successful trajectories **without introducing additional overestimation**. (This is systematically shown in Ji et al.)

Reviewer comment (Experiments):
There are concerns about the fairness of the comparisons employed in this work, as each of the components of the method seem to increase exploratory behaviour, but the baselines compared against do not incorporate any particularly sophisticated exploration strategies (DrQv2 uses a linear scheduler and it does not appear that A-LIX or TACO improve upon this).

Author response:
6. **Compare with baselines with carefully designed exploration strategies**:

> Q: I have some concerns about the fairness of the comparisons employed in this work: Each of the components of the method seem to increase exploratory behaviour. However, the baselines compared against do not incorporate any particularly sophisticated exploration strategies (DrQv2 uses a linear scheduler and it doesn't look like A-LIX or TACO improve upon this). 

Thank you for your valuable feedback. In response, we have added a new section in the appendix (Appendix E), where we conduct additional experiments on three Adroit tasks to compare DrM against an intrinsic exploration baseline.

Specifically, we compare DrM with Random Network Distillation (RND), a well-known and extensively used method in intrinsic reward-based exploration. We integrated RND into the existing DrQ-v2 implementation to evaluate its performance. The results, detailed in Appendix E and Figure 8, reveal that while RND enhances DrQ-v2 through intrinsic exploration, DrM still markedly outperforms the RND baseline in three Adroit tasks.

Reviewer comment (Experiments):
Given the different optimal values of the expectile and perturbation scale for different environments, the robustness of the method to its hyperparameters is a major concern. It is unclear how much performance degrades if the optimal hyperparameters for one domain are used in another.

Author response:
5. **Hyperparameters**

> c. Given the different optimal values of the expectile and perturbation scale for different environments, the robustness of the method to its hyperparameters is a major concern to me. How much worse is performance if the optimal hyperparameters for one domain are used in another?

Thank you for pointing this out. This indeed deserves further clarification. In Appendix D.2, we now present the performance of DrM across various tasks using a single, unified set of hyperparameters, as opposed to domain-specific ones. We observed that DrM off-the-shelf generally performs well across all tasks with this single set of hyperparameters. However, for certain tasks, finetuning specific DrM hyperparameters does yield additional performance improvements. It's also important to note that the hyperparameter tuning required for DrM is minimal. The adjustments we made were limited to the maximum perturbation rate, selecting from $0.6$ and $0.9$, and selecting the exploitation expectile value from either $0.7$ or $0.9$.

Reviewer comment (Reproducibility):
While the pseudocode in the appendix is appreciated, there are some issues with clarity. In Algorithm 2, it states that one creates a deep copy of the network and stores this in the variable `new_net`, but the next line randomly initializes the weights of `new_net`. This raises the question of what the copy step is for. Also, Algorithm 1 lines 6–7 are extremely vague and should be clarified with an equation.

Author response:
7. **Explanation of details in pseudocode**

> Q: While I appreciate the pseudocode provided in the appendix, there are some issues with clarity. For example, algorithm 2 says that one creates a deep copy of the network and stores this in the variable new_net, but then the next line randomly initializes the weights of new_net. What did the copy step do? Algorithm 1 lines 6-7 are also extremely vague and should be clarified with an equation.

Regarding Algorithm 2, we acknowledge that the step of creating a deep copy of the network into new_net followed by the random initialization of its weights was redundant. The intention was to replicate the network's architecture, not its weights. We have since revised this section in the paper to clarify that only the network's shape is copied.
For Algorithm 1, lines 6-7, we appreciate your input on the vagueness of the description. We have updated the paper to include a more detailed explanation to clearly articulate the calculation process.

Reviewer comment (Experiments):
Many aspects of the algorithm seem unnecessarily complicated. For example, one could simply use an exploration noise value equal to the sigmoid of the dormant unit fraction, or use maximal exploration noise until the awakened threshold is met and then start the noise decay. Additional ablations would help illustrate the importance of these design choices.

Author response:
2. **Awaken Exploration Scheduler**
    > Q1: One could simply use an exploration noise value equal to the sigmoid of the dormant unit fraction discussed, or use a maximal exploration noise until the awakened threshold is met and then start the noise decay
    
    Our objective here was just to introduce a straightforward modification to the existing linear exploration schedule in DrQ-v2, aiming to provide the agent with higher exploration noise when the dormant ratio is high. Although the equation $\displaystyle \frac{1}{1+exp(-(\beta-\hat{\beta})/T)}$ may seem complex at first glance, it essentially aligns with your suggestion of "using maximal exploration noise until awakening." The key difference is only the adaptive adjustment of exploration noise based on the dormant ratio rather than employing a fixed uniform value, as you propose. This nuanced adjustment allows for a more dynamic and responsive exploration strategy tailored to the evolving state of the dormant units. While the key insight here is that dormant ratio can be leveraged as a key tool for efficient exploration in visual RL, our method does not exclude some modifications of our design choice, and we welcome future works to build on top of DrM to perhaps come up with better exploration techniques.

Reviewer comment (Writing):
The use of the term "exploitation" seems inaccurate. Exploitation is usually used to refer to greedy behaviour with respect to the network's predictions. In this method, exploitation refers more to optimistic Q-target updates that bias towards higher expectiles, encouraging visits to states that have occasionally yielded high rewards in the past. This could lead to overestimation of Q-values in noisy environments and potentially result in suboptimal policies where the agent is attracted to high-variance, low-expectation states. The paper would benefit from a clearer discussion of what this component is doing, and potentially a rebrand to characterise it more as a "risk-seeking" parameter rather than an "exploitation" parameter.

Author response:
3. **Dormant-ratio-guided exploitation:** 
    > Q1. The use of the term "exploitation" seems inaccurate to me. 

   
    Thank you for raising this important point. We acknowledge that our use of the term "exploitation" might require further clarification. In our context, "exploitation" refers to a strategy that could allow the RL agent to quickly exploit its serendipitous, unexpectedly successful experiences from its exploration process. The key insight that we observe is that, especially in the early stages of training, the replay buffer could contain scarce high-quality episodes (i.e., episodes with high reward) that the agent has encountered through exploration. In this training stage, $\pi$ is suboptimal, and the $Q$-value is often underestimated due to insufficient exploitation of high-quality samples in the replay buffer. Thus, to mitigate this underestimation, we introduce a value estimator $V$, which estimates the high expectile of $Q$, designed to converge more rapidly than $Q$-values. Therefore, this mechanism allows the RL agent to quickly exploit its historically successful trajectories **without introducing additional overestimation**. (This is systematically shown in Ji et al.)

Reviewer comment (Experiments):
**Weak empirical evaluation** All of the experiments are performed with just four random seeds, which raises questions about the statistical significance of the results. I refer the authors to Patterson et al. (2023) on how to perform good empirical experiments. I realize that it might not be feasible to do 30 runs for all environment-algorithm pairs, but there should be at least one experiment in the paper that will stand the test of time. I suggest that the authors perform at least ten runs for all algorithms in some environments, maybe on the four dense tasks in MetaWorld (from Figure 7).

Author response:
Thank you for raising the question of the limited number of random seeds in our experiments and its possible effect on statistical significance, as referenced in Patterson et al. (2023). It is indeed crucial to increase the number of random seeds for some algorithms that show only marginal improvements. However, in our case, we wish to emphasize that DrM exhibits a substantial performance boost compared to other existing algorithms. The variation of performance across random seeds cannot be the reason to support the consistent and huge performance gain of DrM against all baseline algorithms and across all tasks. 
Nevertheless, understanding the value of robust empirical evidence, we conducted 10 runs of experiments for both DrQ-v2 and DrM across three MetaWorld environments in Appendix H, and cited Patterson et al. (2023) in our reference list. It is evident that DrM is not sensitive to the randomness of seeds, and it consistently maintains a significant performance lead over baseline algorithms.

Reviewer comment (Evaluation):
**Too strong claims** For example, the last line of Section 3.1 says, " ... all the hyperparameters in this approach are robust to tasks and domains.". However, no evidence is provided for this claim. Their method introduces 4 or 5 new hyper-parameters, and different values are used for different environments. This means that the same hyperparameter is not optimal across domains.

Author response:
Thank you for your feedback. We have updated our Appendix D accordingly. We would like to point out that the hyperparameter tuning required for DrM is minimal. The adjustments we made were limited to two hyperparameters: the maximum perturbation rate, selecting from $0.6$ and $0.9$, and modifying the exploitation expectile value by selecting from $0.7$ and $0.9$.
More importantly, in Appendix D.2, we now present the performance of DrM across various tasks using **a single, unified set of hyperparameters**, as opposed to domain-specific ones. We observed that DrM off-the-shelf generally performs well across all tasks with this single set of hyperparameters. However, for certain tasks, finetuning specific DrM hyperparameters does yield additional performance improvements.

Reviewer comment (Presentation):
What value of $\hat{\beta}$ is used in Figures 3 and 4? This should be specified in the figure itself.

Author response:
Thank you for your feedback. In those figures, $\beta=0.2$. We have updated the results in the paper.

Reviewer comment (Reproducibility):
What exactly is the linear schedule, $\sigma_{linear}$, of exploration? The paper says that it's the same as defined in DrQ-v2. However, I believe the paper should be as self-contained as possible, so details like this should be provided in the appendix.

Author response:
Thanks for the valuable suggestion. We have included this in the Table of Appendix D.1. The linear schedule stands for linearly annealing the std of exploration noise until a given number of exploration frames.

Reviewer comment (Theory):
The foundational motivation for dormancy ratio minimization is not entirely convincing. It is not clear why high neuron dormancy should necessarily lead to a decrease in exploration, even if this was empirically observed. It is also not that clear why a lower neuron dormancy may lead to more exploration. I think the paper would benefit from a more careful treatment of this finding, either with further experiment, a toy model/study, theory or merely some text describing why this should be expected.

Author response:
In addressing the concerns about the foundational motivation for dormancy ratio minimization, it is important to clarify the relationship between neuron dormancy and exploration. Our approach is not based on the premise that a lower neuron dormancy directly leads to more exploration. Rather, we focus on using increased exploration as a means to reduce neuron dormancy. The correlation we observe is between neuron dormancy and activity level.
When the policy network exhibits a high dormant ratio, this is typically associated with less active behaviors of the agent. In such scenarios, the strategy is to enhance exploration to decrease the dormant ratio. This process aims to activate a broader range of neurons within the network, potentially leading to the discovery of more effective behaviors. Therefore, the paper emphasizes the use of exploration as a tool to modulate neuron dormancy, rather than suggesting a direct causal relationship between low dormancy and increased exploration.

Reviewer comment (Experiments):
The algorithm proposes three mechansism, each with a hyperparmeter that governs a schedule. This provides several degrees of freedom to improve upon the base algorithm, and no hyperparameter study is conducted. This is not good empirical science, but the empirical demonstration of success on hard problems does at least hold promise for further work.

Author response:
Thank you for your feedback regarding the hyperparameters in our algorithm. We acknowledge the need for a clearer explanation of the motivation behind our scheduling approach. Our method employs a relaxed version of the “0-1” combination, where the hyperparameters dictating the extent of this relaxation are not overly sensitive. We have conducted all our experiments using this consistent set of newly added hyperparameters. The results in the paper have been updated to reflect these experiments, demonstrating the robustness of our approach across different settings.

Reviewer comment (Experiments):
Section 3 (Hyperparameters): The method involves three components, each with at least one hyperparameter. This gives your proposed algorithm a number of additional degrees of freedom over the base algorithm and the other baselines. Looking at the experiments, there is an ablation study for each individual component but no results on the sensitivity to the various hyperparameters. It would have been good to explore this, at least in one of the simpler environments.

Author response:
Thank you for pointing this out. This indeed deserves further clarification. In Appendix D.2, we now present the performance of DrM across various tasks using a single, unified set of hyperparameters, as opposed to domain-specific ones. We observed that DrM off-the-shelf generally performs well across all tasks with this single set of hyperparameters. However, for certain tasks, finetuning specific DrM hyperparameters does yield additional performance improvements. It's also important to note that the hyperparameter tuning required for DrM is minimal. The adjustments we made were limited to the maximum perturbation rate, selecting from $0.6$ and $0.9$, and modifying the exploitation expectile value by selecting from $0.7$ and $0.9$.

Reviewer comment (Experiments):
Section 4 (Ablation study): I appreciate the effort to ablate your algorithm in Adroit, a relatively complex environment. One thing that would help is to show whether each component of DrM is faithful to its motivation. For example, the exploration mechanism could be shown to have a large influence on performance in environments that require exploration and less influence on environments that do not require as much exploration. This would also serve as an opportunity to further elaborate on what the exploitation mechanism accomplishes.

Author response:
Thank you for your question regarding the ablation study. We acknowledge the value of your suggestion to demonstrate how each component of DrM aligns with its intended purpose. However, in practice, our empirical observations indicate that all three mechanisms are integral to DrM’s success. We find that exploration and exploitation are intrinsically interconnected; it is rare to encounter tasks that exclusively demand exploration or exploitation. In reality, a balanced and efficient blend of both exploration and exploitation is necessary for optimal performance.

Reviewer comment (Theory):
Section 2 (visual RL): Is it necessarily the case that your setting is POMDP and that the observations provided to the agent lack some information? While this can be true in some settings, the environment is usually constructed so that the observatiosn do include all necessary information (e.g. concatenation of frames in Atari).

Author response:
Indeed, our experiments have been conducted with frame concatenation to deal with partial observability. Here, adopt the convention of visual RL for both continuous control and Atari games, where the observation is a stack of 3 RGB images of the consecutive state following DrQ-v2. The observation after frame stacking should include all necessary information to make a decision without using a recurrent/transformer-based policy network to incorporate the history.

Reviewer comment (Writing):
Section 2 (Dormant Ratio): You use the term "Linear layer" in both definitions, but I am not sure whether this is required in the definition or what it is that you mean. Does it mean the last layer that usually maps the penultimate activations to the output?

Author response:
We clarify that the term "Linear layer" used in Section 2 refers to all linear/fully connected layers ('nn.Linear'), not just the last output-mapping layer. To avoid confusion, we've updated our paper to replace "Linear layers" with "Fully Connected layers." This change accurately represents our model, which includes both convolutional neural networks and fully connected networks, with the dormant ratio calculations considering all fully connected layers.