Paper: MAST: A Sparse Training Framework for Multi-agent Reinforcement Learning (ICLR-2024)

Reviewer comment (Novelty):
The paper utilizes multiple technologies, such as RigL, hybrid TD targets, the Soft Mellowmax operator, and dual buffers, which may make it difficult to discern the specific kernel contribution and novelty.

Author response:
We would like to highlight our contribution in three aspects.
1. **Originality:**
   - Addressing the costliness of training and inference in MARL algorithms, especially on resource-limited devices, our MAST framework stands as the *first* algorithm framework achieving 90% sparsity across multiple MARL algorithms. This marks pioneering research in sparse training for MARL.
2. **Technical Contributions (Value Learning Enhancements):** Recognizing the inadequacies of sparse MARL models in value learning (indicated in Figure 1), we focused on improving the reliability of training data tagets and training data distributions.
    - Notably, we introduced $TD(\lambda)$ targets (Section 4.1) to mitigate sparsification errors and addressed overestimation issues in sparse models using the computationally efficient Soft Mellowmax operator (Section 4.2).
    - In addition, we proposed a dual buffer mechanism to enhance training stability under sparse models by introducing an additional on-policy buffer.
3. **Soundness:** Extensive experiments across popular MARL algorithms validate MAST's pioneering role in sparse training:
    - MAST achieves model compression ratios ranging from 5× to 20× with minimal performance trade-offs (typically under 3%).
    - Remarkably reduces FLOPs required for training and inference by up to 20×.

Reviewer comment (Novelty):
If the motivation lies in the algorithm, the contribution may seem incremental. Additionally, if the paper aims to design an effective framework, it would be necessary to conduct experiments on other benchmarks, such as Google Research Football, to demonstrate its superiority.

Author response:
- **Innovative Framework Design:** Our MAST framework stands as a pioneering effort, marking the first algorithm framework to achieve 90% sparsity across multiple MARL algorithms. This represents groundbreaking research in sparse training for MARL.
- **Experimental Scope:** Our current experiments predominantly focus on SMAC, given the suitability of our base algorithms (QMIX, WQIX, and RES) within this environment, as previously validated in their original papers (Rashid et al. 2020b; Rashid et al. 2020a; Pan et al 2021). Acknowledging the significance of expanding our experimental scope, we aim to include a broader range of algorithms and diverse environments, such as Google Research Football, in our future work.

Reviewer comment (Presentation):
Some curves in the results do not appear to converge at the end. Could this be due to the figures being drawn with smooth weight?

Author response:
In our comparative analysis of different algorithms, we ensure a fair assessment by evaluating their performances over a consistent period of interaction with the environment—specifically, 2 million steps, as prior works such as QMIX (Rashid et al. 2020b), WQMIX (Rashid et al. 2020a), and RES (Pan et al. 2021). It's important to note that some curves in the results may not converge towards the end due to the figures being plotted with smoothed results. In some instances, divergent curves might be attributed to particular seeds leading to poorer outcomes. Additionally, certain experiments might not converge at the designated 2 million steps; these cases involve significantly longer training times, resulting in larger FLOPs, making them incomparable with our MAST framework.

Reviewer comment (Evaluation):
The results were obtained with 4 random seeds. Could you provide information about the variance? Is the method stable?

Author response:
In our revised experiments, we have expanded our seed set by incorporating an additional four seeds. Consequently, the reported results in Table 1 now reflect the average outcomes derived from a comprehensive set of eight seeds. Moreover, we've included detailed standard deviation information in Table 7 within Appendix B.6, focusing specifically on the win rate presented in Table 1. Besides, the standard deviation information is also depicted through the width of shaded areas within the training curves in Appendix B.5. These inclusive representations serve to underscore the stability and consistency of our MAST framework across various scenarios, showcasing its robust performance.

Reviewer comment (Writing):
The writing logic is bad, making readers hard to follow. For example, what is the relationship of the sparse model in SL, single-agent RL and multi-agent RL? Why does MASK apply to QMix series approaches and when MAST is applied to QMIX series algorithms and leverage the RigL method for topology evolution? The authors use too many words on the related work and basic knowledge of MARL, but not clarify the logic clearly.

Author response:
We've integrated the following bullet points in our revised manuscript and would greatly appreciate any specific areas that still require further clarification:

1. **Relationship of Different Sparse Models**
    - Sparse networks, initially proposed in **deep supervised learning**, alongside state-of-the-art sparse training frameworks such as SET (Mocanu et al., 2018) and RigL (Evci et al., 2020), can train a 90%-sparse network without performance degradation from scratch.
    - In **DRL**, the learning target evolves in a bootstrap way (Tesauro et al., 1995), and the training data distribution can be non-stationary (Desai et al., 2019). Existing works (Sokar et al., 2021; Graesser et al., 2022) reveal that directly adopting a DST method in **single-agent RL** fails to uniformly achieve good compression across different environments. RLx2 (Tan et al. 2022) marks the first successful sparse training throughout, surpassing 90% sparsity.
    - As depicted in Figure 1, neither DST nor RLx2 works in **multi-agent RL**. The complexities in **multi-agent RL**, such as inaccurate value estimation and training instability, are addressed by our MAST framework. MAST utilizes a novel hybrid TD(λ) target mechanism, coupled with the Soft Mellowmax operator, which facilitates precise value estimation even in the face of extreme sparsity. Additionally, MAST unveils a dual buffer mechanism designed to bolster training stability in sparse environments.

2. **Why QMIX?**
    - Our focus lies on algorithms adhering to the Centralized Training with Decentralized Execution (CTDE) paradigm (Oliehoek et al., 2008; Kraemer & Banerjee, 2016). Within this paradigm, QMIX (Rashid et al., 2020b) stands as a representative algorithm.

3. **Related Work and Basic Knowledge of MARL**
    - We've condensed and restructured the related work and preliminary sections in our revision.

*References:*
(Tesauro et al., 1995) Gerald Tesauro et al. Temporal difference learning and td-gammon. Communications of the ACM, 38(3):58–68, 1995.
(Desai et al., 2019) Shrey Desai, Hongyuan Zhan, and Ahmed Aly. Evaluating lottery tickets under distributional shifts. EMNLP-IJCNLP 2019, pp. 153, 2019.

Reviewer comment (Theory):
Cannot the discount rate in RL be 1?

Author response:
We consider the case that discount rate is less than 1.
* On the one hand, most Reinforcement Learning algorithms (such as SARSA or Q-learning) and Dynamic Programming algorithms converge solely towards the optimal policy under the discounted reward infinite horizon criteria.
* On the other hand, when the discount rate is below 1, our $TD(\lambda)$ targets play a significant role in mitigating the network fitting error resulting from network sparsification, as discussed in Section 4.1.

Reviewer comment (Experiments):
The experiment is conducted only on 4 seeds, which is not enough and strong in RL scenarios.

Author response:
In our revised experiments, we have expanded our seed set to include an additional four seeds. Consequently, the results reported in Table 1 now represent the average outcomes derived from a total of eight seeds.

Reviewer comment (Experiments):
“These topology adjustments occur infrequently throughout the training process, happening every 200 episodes (about 10,000 steps) in our specific configuration.” How about under other configurations?

Author response:
In our revision, we've conducted an ablation study focusing on the mask update interval ($\Delta_m$), detailed in Table 8 within Appendix B.7. Analysis from Table 8 reveals several key observations:

- Findings indicate that a small $\Delta_m$ negatively impacts performance, as frequent mask adjustments may prematurely drop critical connections before their weights are adequately updated by the optimizer.
- Overall, A moderate $\Delta_m=200$ episodes performs well in different algorithms.

Reviewer comment (Presentation):
Algorithm 1 of the overall procedure is in supplementary, I suggest to contain it in the main text.

Author response:
In our revised version, Algorithm 1 has been relocated to Section 4, accompanied by additional explanations detailing our topology evolution scheme.

Reviewer comment (Writing):
Can the authors give a clear logic of the paper and the innovative solutions in an easy-understood way?

Author response:
**Summary of Paper Logic:**

1. **Issues of Dynamic Sparse Training (DST) in MARL:** DST, stemming from supervised learning, holds potential for accelerating training and compressing models in MARL. However, direct adoption of DST methods in MARL fails to maintain performance at high sparsity levels, as indicated in Figure 1. This points to the challenge of sparse MARL models struggling with value learning, a crucial aspect for policy learning.

2. **Improving Value Learning in Sparse MARL Models:** MAST introduces innovative solutions to address the accuracy of value learning in ultra-sparse models by concurrently refining training data targets and distributions.
   - **Enhancing Learning Targets:** Recognizing larger network fitting errors in sparse models, we introduce $TD(\lambda)$ targets (Section 4.1) to mitigate sparsification errors. Additionally, to address overestimation issues in sparse models, we employ the computationally efficient Soft Mellowmax operator (Section 4.2).
   - **Stabilizing Training Data:** Observing training instability in MARL algorithms under sparse models with a single off-policy buffer, our proposed dual buffer mechanism (Section 4.3) improves stability by introducing an extra on-policy buffer.

3. **An Innovative Solution (MAST) with Empirical Validation:**
   - Integrating these enhancements, our innovative solution, MAST, achieves superior sparsity levels exceeding 90% without performance degradation in MARL, validated through our experiments. This work represents pioneering research in sparse training for MARL.

We have also updated our paper accordingly. Please see our revision.

---
**In conclusion, we extend our gratitude to the reviewer for their invaluable feedback. Should our response effectively address your concerns, we kindly hope that the reviewer could consider raising the score rating for our work. Furthermore, we remain available to address any additional queries or points for discussion.**

Reviewer comment (Writing):
The clarity of this paper needs to be improved. For example, the proposed method uses RigL to sparse the network. However, the details of RigL are missing, which makes it confusing for readers who are not familiar with the sparse training area.

Author response:
We've incorporated additional details about RigL in Section 4 in our revision. We would greatly appreciate any specific areas that still require further clarification.

Reviewer comment (Experiments):
The limitation of this paper is not discussed. For example, there are too many key parameters that need to be fine-tuned, making it infeasible to apply to other complex domains.

Author response:
In our revision, we've discussed the limitations concerning implementation and hyperparameter tuning in Appendix A.4. Additionally, we also provide Table 4 in Appendix B.3 for practical recommendations for both sensitive and insensitive hyperparameters across different algorithms and environments:

1. **For sensitive hyperparameters** (such as $T_0$ and $\lambda$ for hybrid TD($\lambda$) targets and sample partition), we recommend values of $T_0=3/8T$, $\lambda=0.6/0.8$, and a ratio of $5:3$ for the sample partition, where $T$ represents the maximum training step.

2. **For insensitive hyperparameters** (such as $\alpha$ and $\omega$ for the Soft Mellowmax operator), flexibility exists in their settings. However, we recommend following our suggestion of $\alpha=1$ and $\omega=10$. Our algorithm framework demonstrates insensitivity to these parameters, with performance consistently exceeding 85% compared to the original dense performance, as illustrated in Table 3.

*Notably*, as depicted in Table 1, employing the recommended hyperparameter settings (as detailed above) across different algorithms and environments consistently positions our MAST as the top performer.

Reviewer comment (Presentation):
The visualization does not look very informative to the reviewer, as there are no specific patterns for the latent space distribution. Perhaps projecting what connections are removed and what connections are remaining and analyzing why it is like that will be interesting.

Author response:
In our revision, we've provided connection counts for both input and output dimensions in Figure 10 of the main paper and Appendix B.8 for further insight into topology evolution. These new illustrations specifically highlight the non-zero connection counts in both input and output dimensions during the training, shedding light on pruned dimensions. Surprisingly, our findings reveal that a small portion of neurons in both input and output layers prove necessary, while others exhibit minimal or no connections after MAST training. This observation underscores the substantial redundancy present in the original dense network and the efficacy of our MAST framework in removing such redundancy. Notably, this observation aligns with the sparse model obtained in single-agent RL, as demonstrated in Appendix C.11 of (Tan et al., 2022).