Paper: Plasticity-Driven Sparsity Training for Deep Reinforcement Learning (ICLR-2024)

Reviewer comment (Theory):
The lack of theoretical analysis of the relationship between plasticity and feasible pruning ratio seriously weakens the motivation of this paper.

Author response:
Thank the reviewer for pointing out this question, we clarify this weakness in the following three points:
1. **Theoretical Analysis Challenge**: We recognize the difficulty in providing theoretical proof for the loss of plasticity in neural networks. To the best of our knowledge, there is almost no existing theoretical proof in prior work, especially in the domain of DRL. 
2. **Empirical Contribution**: Our study pioneers in empirically establishing the relationship between neural network plasticity and implicit sparse in dense training, offering new insights in this field.
3. **Acknowledging Limitations**: We are aware that our work lacks a comprehensive theoretical framework, a limitation we openly acknowledge, and hope that our empirical findings will inspire further theoretical and empirical exploration.

We clarify that the motivation of the feasible pruning ratio is that WSR only indicates that the weights are approaching 0 but those weights may still contribute to the final representation. The increasing feasible pruning ratio increases aligns significantly with the WSR trend and WSR is a good proxy for the implicit plasticity of dense training. 

On the other hand, the motivation of our work is that implicit sparsity may contribute the decreased performance. We clarify that the performance loss has happened no matter if explicitly pruning the weights or not. The performance without plasticity loss is unknown and infeasible in traditional dense training. Simple reinitializing weights (under the ReLU activate function) that are close to zero over the course of training, under a certain threshold, can lead to significant performance improvement ([Sokar et al., 2023](https://arxiv.org/abs/2302.12902)). The increasing feasibility of pruning ratios does not detract from the validity of our hypothesis.

Reviewer comment (Theory):
The relationship between Weight Shrinkage Ratio proposed in this paper and the plasticity of the neural network is not clear. The increased trend of WSR cannot be totally explained by the loss of plasticity. There are other common reasons, e.g. the high absolute value of initial parameters, weight decay in the optimizer, and implicit regularization in SGD[1]. Causation cannot be inferred simply from correlation.

Author response:
Thank the reviewer for raising this question. We apologize for any lack of clarity in the latter part of Section 4. This section has now been revised for improved precision and coherence. In response, we would like to clarify that our work does not assert that the rising trend of WSR can be totally explained by a loss in plasticity. Rather, our study establishes a link between these two phenomena. Specifically, our results demonstrate that the increasing sparsity in WSR plays a crucial role in the reduction of neural plasticity. Simple reinitializing weights (under the ReLU activate function) that are close to zero, under a certain threshold, can lead to significant performance improvement ([Sokar et al., 2023](https://arxiv.org/abs/2302.12902)),. This improvement room occurs by mitigating inactive or dormant neurons, indicating a loss of plasticity. 

You are absolutely correct that causation cannot be simply inferred from correlation and we also want to clarify that we only claim that the implicit sparsity is a potential key reason for the loss of plasticity. This is precisely why we dedicated a substantial portion of Section 4 to rigorously examine the growth of implicit sparsity in dense training. Our experiments spanned various environments and activation functions, and we meticulously calculated the gradient shrinkage ratio and determined the feasible pruning ratio. While we cannot assert causation with 100% confidence, we have made our best effort to eliminate potential confounders, thereby strengthening our findings: the implicit sparse is a key factor for plasticity loss.

Reviewer comment (Writing):
Minor: I think there is a typo, $L$ in $M_s$ is the number of layers of neural networks, $L$ in calculating mean and variance is the number of neurons of $l$-th layer, and they should not be the same.

Author response:
Thank the reviewer for pointing it out and we have revised it.

Reviewer comment (Reproducibility):
What criteria were chosen in the experiment to determine the distance of the policy when collecting the necessary data into the empty buffer?

Author response:
With the spirit of preserving the simplicity of our proposed algorithm, we opted not to employ any specific criteria but directly reset it in every 0.2M step. This approach has proven efficient in preserving the disparity between the replay buffer and the current policy. The effectiveness of this strategy is explicitly demonstrated in the revised version of our paper in Appendix B.2.

Reviewer comment (Theory):
I am curious about the necessity to maintain plasticity because if plasticity is low, it means that more neurons are useless and can be removed[2], which is a good thing for creating a sparse network. Increased plasticity may also make pruning more difficult.

Author response:
We thank the reviewer for pointing out this important question. First, we hope our clarification in "W1: The feasible pruning ratio seriously weakens the motivation of this paper." is clear to the reviewer. The core insight is that the dormant neurons can be safely removed over the course of training, but the plasticity loss still happens with the result of performance degradation (you can not know!). This is the reason we may want to improve the performance of sparse training in DRL in terms of plasticity and why not adopt the sparse-to-sparse training (loss of plasticity at the beginning).  

We express our gratitude to the reviewer for highlighting this critical inquiry. You are correct the traditional methods for plasticity improvement may make pruning more different by the operation on the neural networks, serving as a trade-off between performance improvement and the target sparse networks. However, our paper aims to mitigate one of the main sources of plasticity loss, i..e, non-stationary, without any operations on the networks but simply resetting the memory in the sparse training of DRL.

Reviewer comment (Writing):
*Definition of Dynamic Weight Rescaling (DWR)*. The paper uses a definition $a^l=h^{l-1} \odot \gamma^l$, but $\gamma^l$ is not defined anywhere. My guess is that it is $\Gamma^l$. But if that is the case and $h^{l-1}$ is the output from the previous layer, how can we have an element-wise product of $h^{l-1}$ and $\Gamma^l$? One is a vector, and the other is a matrix. $\Gamma^l$ is the mask over weights, not neurons?

Author response:
We thank the reviewer for pointing out this. We have revised our paper in Section 5 in our new submission. Specifically, the definition of $\gamma^l$ in our paper aligns with our initial description of the sparse neural network $\mathbf{M}_s$, where $\{\mathbf{M}_s=\Gamma^l: l=1, \ldots, L\}$. This notation was intended to imply that $\gamma^l$ refers to the mask of the weight for the $l^{th}$ layer. We have revised the paper to clarify this in our new submission.

Reviewer comment (Theory):
**Wrong conclusions** The first claim in section 4.2 is that "shrinkage speed increases." But, there is no shrinkage. The weights will only shrink, on average, if the shrinkage ratio is more than 0.5, but the shrinkage ratio is never more than 0.5. So there is no shrinkage; the weights are always getting bigger. The plots show that the weights get larger, not smaller, as training progresses, which is the exact opposite of the main motivation of introducing sparsity.

Author response:
There must be a misunderstanding regarding this aspect of our paper. In response, we first acknowledge that some weights always get bigger in the training process. However, the claim that "shrinkage happens only when WSR is greater than 0.5" is wrong. We clarify that shrinkage happens whenever it is greater than 0. Here is a very simple case with a 1x3 vector to approximate the weight matrix, say [1, 1, 1]. After k time steps, the weight matrix becomes [0, 2, 2]. The WSR is 1/3 < 0.5 but the shrinkage still happens in the first weight (1->0).

Reviewer comment (Evaluation):
**Improper statistical reporting** There are many instances where wrong statistical conclusions are drawn. For instance, Figure 6 says that "Reset buffer distinctly outperforms the Small Buffer strategy in 3 out of 4 tasks, with 2 of these improvements being statistically significant." This is wrong. There is no statistically significant difference between the two algorithms in any of the four cases. We can not conclude that one algorithm is better than the other. I refer the authors to the paper by Patterson et al. (2023) on how to do proper empirical studies in deep RL.

Author response:
We thank the reviewer for pointing it out, and we have revised our paper regarding this in our new submission.

Reviewer comment (Evaluation):
Abstract and introduction: Several claims are made but little evidence or context is provided. For example, what exactly about sparse-to-sparse training "may escalate overall computational cost"? The introduction gives some details, giving references to RLx2 that sparse training is 3x more expensive than dense training. But this is not sufficient evidence of an increase in computational cost, because the sparse models per iteration cost may be more than 3x lower than the dense model.

Author response:
> Abstract and introduction: the statement of sparse-to-sparse training "may escalate overall computation cost"? and the example of RLx2 with 3x training steps. 

Answer: We quote "may escalate the overall computational cost throughout the training process" from [Liu and Tan, 2023](https://arxiv.org/abs/2302.02596) in Section 3.7, page 7, where they note that "some sparse-to-sparse algorithms might take more iterations to converge, hence not always cheaper in terms of the total computation amount throughout training." However, it is important to clarify that the term 'computational cost' encompasses various aspects, including memory cost, floating point operations (FLOPs), and training time. In the context of our paper, particularly in the introduction where we discuss the RLx2 example, we primarily focus on "training steps".

From a hardware perspective, unstructured pruning methods like PlaD and RLx2 cannot be expedited on common GPUs. In other words, the training time for RLx2 (3M steps) is significantly higher than that for standard RL algorithms (1M steps), given the lack of acceleration for unstructured pruning on common GPUs. Our experiments validate this perspective, showing a consistent trend with the aforementioned training cost considerations. In conclusion, we believe this claim is nothing to blame.

Reviewer comment (Theory):
Abstract and introduction: Why should sparsity necessarily contribute to less plasticity? There has been no concrete evidence of this connection in the literature beyond very specific forms of (double and triple) sparsity [cite:@zilly21]. Neither the neuron dormancy nor primacy bias papers provide evidence for this fact. Neuron dormancy, while seemingly related, is about relative magnitude of activations and provides no evidence that lower magnitude activations can or should be removed.

Author response:
> Q1: Why should sparsity necessarily contribute to less plasticity?

Answer:  The connection between sparsity and plasticity is one of the main contributions of our work, which is highlighted at the end of the introduction and expanded in Section 4. At the end of Section 4, we have revised the paper to make it more precious to show that: implicit sparsity suggests the abundance of inactive or dormant neurons. Simply reinitializing the shrinkage weight approaching 0 can lead to improved performance.

Reviewer comment (Theory):
Section 1 (Comments on plasticity): It is stated that the replay buffer is the source of non-stationarity. In a certain sense, the replay buffer may contain information from previous distributions and be a contributing factor to non-stationarity, but to state that it is the primary source of non-stationarity is over-claiming. I do not think periodically removing such a large source of experience is worth it, when you can instead just use an on-policy learning algorithm.

Author response:
> Section 1 (Comments on plasticity): the statement "the replay buffer is the primary source of non-stationarity"

Answer: It appears there has been a misinterpretation of our paper. In the introduction, we specifically claim that "emptying the replay buffer addresses the primary source of plasticity loss in Deep Reinforcement Learning (DRL) training, i.e., non-stationarity." This statement does not suggest that the replay buffer itself is the source of non-stationarity. Rather, it implies that the act of emptying the replay buffer can mitigate the primary source of plasticity loss in DRL, which is non-stationarity. Furthermore, we concur that non-stationarity has multiple sources, as outlined in Section 2.2 of our paper. These sources include dynamic data streams, the introduction of new tasks, and evolving environments. Our intention was to highlight the effectiveness of emptying the replay buffer as a strategy to address plasticity loss due to non-stationarity, not to pinpoint the replay buffer as the sole source of non-stationarity.

> Section 1 (Comments on plasticity): why just use an on-policy learning algorithm to mitigate the non-stationary in the replay buffer?  

Answer: On-policy algorithms are adept at managing non-stationarity by utilizing current data, yet they typically fall short in terms of sample efficiency and variance reduction, areas where off-policy methods excel. Our method, which involves periodically emptying the replay buffer in an off-policy context, is designed to strike a balance between leveraging the benefits of experience replay and addressing the challenges posed by non-stationarity. Therefore, our primary focus is on the off-policy setting.

Reviewer comment (Theory):
Section 2.2 (NN Plasticity): Plasticity and generalization are two separate problems. The ash and adams paper, for example, has no non-stationarity in the traditional sense, and their results do not demonstrate loss of plasticity. Training error can be minimized, but generalization suffers. Whereas plasticity is about an inability to minimize the error. There is a connection between the two, but the nature of this connection has not yet been made clear.

Author response:
> Section 2.2 (NN Plasticity): Plasticity and generalization are two separate problems.

Answer: Thanks for pointing out. We have revised our paper without "generalization".

Reviewer comment (Theory):
Section 2.2 (Non-stationarity in DRL): There are transient sources of non-stationarity in deep RL that are present even in stationary MDPs. But explicitly addressing these is not necessary to design successful DRL algorithms. It remains unclear why this is a particular concern in the training of sparse networks in DRL.

Author response:
> Section 2.2 (Non-stationarity in DRL): It remains unclear why non-stationary particular concern in the training of sparse networks in DRL.

Answer: It seems there has been a misunderstanding regarding our paper. In Section 2.2 of our paper, we discuss plasticity loss in neural networks, citing non-stationarity as a significant contributor. However, we do not connect non-stationarity with sparse DRL algorithms. 

However, Addressing non-stationarity is indeed a critical aspect of sparse training in sparse RL, compared with supervised learning. This is highlighted in recent literature: [Tan et al., 2023] state that "sparse training in DRL is challenging due to the non-stationarity in training data distribution" in their introduction (page 1). They propose two mechanisms to alleviate this issue within the sparse-to-sparse training framework. Similarly, [Graesser et al., 2023](http://arxiv.org/abs/2206.10369) mention in their background section that "previous approaches may not generalize well in the RL setting due to the non-stationarity of the data." These studies underscore the unique challenges posed by non-stationarity in sparse training for RL.

Reviewer comment (Theory):
Section 4: Equating increased sparsity with increased plasticity is not supported by any previous work, and is not clearly demonstrated in this section.

Author response:
> Section 4: Equating increased sparsity with increased plasticity is not supported by any previous work, and is not clearly demonstrated in this section; and Section 4: (Conclusion and connection to sparsity)

Answer: We thank the reviewer for pointing this out, we have revised our paper to clearly demonstrate the connection between the implicit sparsity and plasticity in the last part of Section 4. Concretely, [Sokar et al., 2023](https://arxiv.org/abs/2302.12902) have shown that simply reinitializing weights approaching zero can lead to improved performance. This improvement occurs specifically within inactive or dormant neurons, indicating a potential loss of plasticity.

Reviewer comment (Theory):
Section 4 (Weight Shrinkage Ratio): I do not understand how WSR can "serve as an approximation of the first order gradient of shrinkage / shrinkage speed". First of all, what is the "first-order gradient of shrinkage"? What is shrinkage speed, and what is this speed with respect to? I also do not see how this exposition has anything to do with the study of neural network weights because they are not normally distributed besides at initialization.

Author response:
> Section 4: (Weight Shrinkage Ratio): Why WSR can "serve as an approximation of the first-order gradient of shrinkage/shrinkage speed". 

Answer: The WSR is a statistical metric that is dependent on time. The intuition to understand the shrinkage speed is the connection with distance, velocity, and time. The velocity (WSR) between two points is: $(y_{t} - y_{t-k})/(t - (t-k))$, where WSR is normalized by $N$ and scaled by $k$.  As long as the velocity (WSR)  is greater than 0, it means that the distance is getting farther and farther (weight shrinkage happens). It is important to note that this approximation may fail as the weight may "expand" in the future. However, we show that the rising overlap coefficient of the shrinkage weights, compared with the last checkpoint, suggests that this shrinkage trend of most weights persists throughout the remainder of the training process, as shown in Fig 9 and 10. The shrinkage speed refers to "the speed of weights approach 0".

Reviewer comment (Theory):
Section 4 (Activation function concerns): Why is this referred to as the weight shrinkage ratio when it is defined over activations? If this were about weights, I do not see why the relu activation would drive the weights to zero. If all the relu activations saturate at zero, then the weights remain unchanged and the weight shrinkage ratio would be 0. This would only be a concern for an "Activation shrinkage ratio".

Author response:
> Section 4: (Activation function concerns): Why is this referred to as the weight shrinkage ratio when it is defined over activations? 

Answer: It appears that a misunderstanding has arisen in relation to our paper. In our study, we focused on examining the impact of weight shrinkage ratio across various activation functions, rather than defining a specific weight shrinkage ratio for activations. Note that in a single-layer context, or considering only the forward process, the activation function doesn't affect the weight. However, this dynamic changes when multiple layers are involved in gradient back-propagation, where the weight update is considerably influenced by the activation functions. Our experimental results indicate that scenarios where "all the ReLU activations saturate at zero" did not occur. However, we observed weight shrinkage across different activation functions.

Reviewer comment (Theory):
Section 4 (Gradient shrinkage ratio): While the details on this are sparse, I also do not see how this has anything to do with plasticity. Gradient shrinkage is a desirable property for achieving a local minimum.

Author response:
> Section 4: (Gradient shrinkage ratio): While the details on this are sparse, I also do not see how this has anything to do with plasticity. Gradient shrinkage is a desirable property for achieving a local minimum.

Answer: The same activate functions, we conduct an ablation study on the gradient shrinkage ratio to examine if the gradient information trends align with weight trends. We make no connection between gradient and plasticity, except noting the similar shrinkage trend between gradient and weight during training.

Reviewer comment (Evaluation):
Section 4 (Conclusion and connection to sparsity): This section demonstrated that neural network weights shrink over the course of training, but this does not indicate loss of plasticity and you have provided no evidence for a reduced ability for learning.

Author response:
> Section 4: Equating increased sparsity with increased plasticity is not supported by any previous work, and is not clearly demonstrated in this section; and Section 4: (Conclusion and connection to sparsity)

Answer: We thank the reviewer for pointing this out, we have revised our paper to clearly demonstrate the connection between the implicit sparsity and plasticity in the last part of Section 4. Concretely, [Sokar et al., 2023](https://arxiv.org/abs/2302.12902) have shown that simply reinitializing weights approaching zero can lead to improved performance. This improvement occurs specifically within inactive or dormant neurons, indicating a potential loss of plasticity.

Reviewer comment (Experiments):
Section 5 (Periodic Memory Reset): While resetting the neural network may be computationally wasteful, the primacy bias paper demonstrates that there is far more value in the experience stored in the replay buffer than in the neural network being learned. Furthermore, if plasticity is being lost then it is a property of the neural network and it is not clear at all whether that issue can be alleviated by resetting the replay buffer and using more on-policy experience. Furthermore, these methods use a higher replay ratio because their goal is sample efficiency rather than computational efficiency.

Author response:
> Section 5 (Periodic Memory Reset): While resetting the neural network may be computationally wasteful, the primacy bias paper demonstrates that there is far more value in the experience stored in the replay buffer than in the neural network being learned. 

Answer: It seems there is been a misinterpretation related to our paper in comparison to the primacy bias paper ([Nikishin et al., 2022](https://proceedings.mlr.press/v162/nikishin22a/nikishin22a.pdf)). We clarify that the latter demonstrates the significant value of storing experiments when resetting the memory in DrQ, as shown in their ablation study, which means their comparison is (network resetting w/ memory reset vs. network resetting w/o memory reset) rather than (network resetting vs. memory reset). Nikishin et al., (2022), derive knowledge from the replay buffer, while our approach, PlaD, extracts knowledge directly from the network itself.

> Section 5 (Periodic Memory Reset): It is not clear at all whether that issue can be alleviated by resetting the replay buffer and using more on-policy experience. 

Answer: Thanks for pointing out this question. We would like to clarify that prior studies have identified non-stationarity in learning targets and data flow as a primary factor contributing to the loss of neural network plasticity (Nikishin et al., 2022; Igl et al., 2020; Sokar et al., 2023), as discussed in Section 2.2 of our paper. Prior work seems straightforward by operating on the network to maintain the plasticity, but introducing further computation cost and impacting the policy gradient. In contrast, our work demonstrates that by effectively addressing the issue of non-stationarity, we then can maintain the plasticity of neural networks.

Reviewer comment (Evaluation):
Section 6 (Results, fig 5): I am not able to discern any significant differences from this plot, as many of the error bars are overlapping.

Author response:
> Section 6 (Results, fig 5): I am not able to discern any significant differences from this plot, as many of the error bars are overlapping.
	
Answer: We express our gratitude to the reviewer for their valuable insights in refining this work. To enhance the visual clarity and aesthetic appeal of Fig. 5 in our revised paper, we have omitted the error bars. Detailed error metrics are provided in Appendix B.3. This adjustment allows us to more effectively showcase the superiority of our proposed method regarding the averaged final performance. Notably, our method demonstrates a remarkable performance enhancement over the best baselines, with a significant 30% improvement in the Ant task under 90% sparsity, and a 17% increase in the HalfCheetah task at the same level of sparsity.

Reviewer comment (Experiments):
Section 6 (Results, fig 6): There is only statistically significant evidence of the reset buffer surpassing the small buffer in one task (hopper-v4). While you study off-policy algorithms (SAC), it would be interesting to show how this compares to an on-policy algorithm.

Author response:
> Section 6 (Results, fig 6): There is only statistically significant evidence of the reset buffer surpassing the small buffer in one task (hopper-v4). While you study off-policy algorithms (SAC), it would be interesting to show how this compares to an on-policy algorithm.

Answer: We extend our thanks to the reviewer for their constructive feedback regarding Fig. 6 in our manuscript. We have revised the " statistically significant" to avoid misinterpretation in our new submission. We would like to clarify regarding the on-policy comparison for the following reasons:
1. The primary focus of our research is on off-policy methods, specifically targeting improvements in sample efficiency and variance reduction. These are areas where on-policy methods typically fall short. A major concern in our study is variance and training stability, for which we have developed a dynamic weight-rescaling strategy.
2. In Section 6.3, our comparison with a smaller replay buffer aims to show its potential to reduce the gap between the buffer content and the current policy. This is further corroborated by the policy distance analysis presented in Appendix Section B.2 in our revised paper, which aligns with our hypothesis. Figure 6 highlights the advantage of periodically resetting the memory over a smaller memory with a stepper learning speed over new experiences.

Reviewer comment (Evaluation):
Section 7 (Conclusion): No link between loss of plasticity or sparse training was established.

Author response:
> Section 7 (Conclusion): No link between loss of plasticity and sparse training was established.

Answer: Thank the reviewer for highlighting this aspect. In response, we have updated Section 4 of our paper to explicitly connect implicit sparsity with the loss of plasticity. Specifically,  The research by [Sokar et al., 2023(https://arxiv.org/abs/2302.12902)] reveals a significant insight: reinitializing near-zero weights under the ReLU activation function, when below a certain threshold, markedly improves training performance. This enhancement is attributed to the activation of previously inactive or dormant neurons, counteracting reduced neural plasticity.

Reviewer comment (Novelty):
The motivations behind adopting sparse-to-sparse training in reinforcement learning are not entirely clear. While pruning is theoretically assumed to reduce computational demands, in practice it may not significantly lessen computational burdens on modern hardware due to batching complexities. This raises questions about why sparse-to-sparse training is advantageous for the typically smaller models used in reinforcement learning, as opposed to large models like GPT or LLama where memory conservation benefits are evident. Simply advocating for sparsity because it is a well-studied area does not constitute a strong motivation; a specific and tangible benefit within the RL domain needs to be identified. The authors should explain if sparsity in RL could yield significant improvements in efficiency or performance beyond what dense models can achieve. A more robust justification is needed to establish the importance and potential impact of the proposed Plasticity-Driven Sparsity Training (PlaD) approach.

Author response:
> W1: sparse training might not effectively reduce computational burdens on contemporary hardware due to the intricacies of batching processes.

Answer:  Thank you for this question regarding the reason why training on unstructured pruning methods, we provide the summary following [Liu and Tan, 2023](http://arxiv.org/abs/2302.02596) in 3.5: *Why study unstructured sparsity if it can not be accelerated on common GPUs?* We strongly suggest the reviewer take a deep look at 3.5 if any further question remains. 
1. Unstructured sparsity, recognized for its fine-grained and flexible nature, demonstrates superior performance when compared to other, more structured forms of sparsity.
2. Unstructured sparsity has widely proven its practical acceleration on non-GPU hardware, such as CPUs or customized accelerators.
3. Although the hardware support for unstructured sparsity on "off-the-shelf" commodity GPUs and TPUs may be relatively limited, it has been improving rapidly over the years.

> W1: The motivation of the sparse-to-sparse training on smaller models used in RL. 

Answer: Thank you for this important question. We clarify three reasons why we perform sparse training on smaller models in RL:
1. **Scalability and Cost**: Sparse training on smaller models can be seen as a step towards future-proofing for large RL models in computation and/or latency-constrained scenarios such as Go-AI ([Sliver et al., 2017](https://www.nature.com/articles/nature24270))and controlling plasma ([Degrave et al., 2022](https://www.nature.com/articles/s41586-021-04301-9)). Implementing performance evaluations on larger models is expensive and time-consuming; hence, initial exploration of sparse training on smaller models offers a cost-effective strategy with timely feedback. 
2. **Unique challenges in RL**: In contrast to supervised learning, (online) RL introduces new challenges in the context of sparse training, particularly due to the non-stationarity inherent in bootstrap training from the learning target and training data([Tan et al., 2023](10.48550/arXiv.2205.15043)). Distinct approaches are required to manage these non-stationary aspects in sparse training scenarios, and these challenges arise irrespective of the model size.
3. **Addressing problems unique in traditional RL**: In addition to the well-known benefits of sparse training, it can offer different perspectives to solve the traditional problems in RL, e.g., noisy filtering and state representation ([Vischer et al., 2022](https://openreview.net/forum?id=Fl3Mg_MZR-), [Grooten et al., 2023](http://arxiv.org/abs/2302.06548)). 

> W1: Could sparsity-based training contribute to advancements in efficiency or performance that are not possible with dense models?

Answer: Thanks for the insightful comment on this motivation to introduce sparse training in RL. In addition to the conventional motivation of sparse training for computation cost and inference latency, we clarify that the primary motivation of sparse training is on improving the performance of DRL but also the real-world application. However, there are some potential research that may lead to the performance improvement:
1. Noisy filtering and state representation ([Vischer et al., 2022](https://openreview.net/forum?id=Fl3Mg_MZR-), [Grooten et al., 2023](http://arxiv.org/abs/2302.06548)). 
2. Another interesting direction is highly related to our work: PlaD notably outperforms dense models at high sparsity ratios, evidenced by a 17% performance gain in the Hopper environment with 90% sparsity. Besides, a simple baseline, Static Sparse, can also achieve higher performance in the Hopper environment with 50% sparsity. This unexpected result hints at unexplored mechanisms in sparse training contributing to such efficiency.

Reviewer comment (Experiments):
The strategy of periodically resetting the replay buffer to preserve network plasticity appears counterintuitive given the common belief that larger memory sizes improve performance by avoiding local optima, preventing catastrophic forgetting, and mitigating distribution shifts during training. The efficacy of this approach remains uncertain, and its alignment with established views on the benefits of larger memory needs clarification. The paper should provide a detailed discussion of the trade-offs involved and explain how periodic memory reset impacts performance in reinforcement learning models.

Author response:
> W2: Concerns to periodic memory reset: larger memory sizes correlate with enhanced performance.

Answer:  Thank you for your question regarding the efficacy of addressing the non-stationarity between the replay buffer and the current policy. While traditionally, a larger memory (or a large batch size) is viewed as beneficial for performance enhancement from a traditional perspective, our work indicates that it might actually contribute to a loss of plasticity in terms of the non-stationary between the reply buffer and the current policy, as evidenced by the performance metrics with PlaD.

Our approach involves periodically resetting the memory, which significantly increases the likelihood of the agent learning new objectives while preserving previous knowledge within the network. This is achieved without reinitializing the neural network, thereby reducing computational costs. To substantiate this hypothesis, we have charted the explicit distance between the behavior policy in the replay buffer and the current policy under various scenarios, detailed in Appendix B.2. These findings corroborate our hypothesis.