Paper: Efficient Multi-agent Reinforcement Learning by Planning (ICLR-2024)

Reviewer comment (Experiments):
The empirical validation of the method is very weak. Despite the authors proposing the method to tackle large action spaces and mentioning the 27m_vs_30m settings as a motivating example in Section 3.2, there is no experiment in these large action space settings.

Author response:
**Q3: There is no experiment in large action space settings like *27_vs_30m*.**

We are sorry for mentioning *27m_vs_30m* in our motivation section while remaining it unevaluated in our experiment section due to the relatively large time consumption for our algorithm in this map (MAPPO uses 10M data for this map). Because of the stress of computing power, we cannot provide another experiment for it. However, we would like to point out that there are still similar tasks like *10m_vs_11m* whose action space is $17^{10}\approx 2\times 10^{12}$, which is also considerably larger than the most complex environment evaluated by Sampled Muzero (Go, $361$ actions). Therefore, while we couldn't include *27m_vs_30m*, our chosen experiments still showcase the algorithm's robustness in complex environments.

We hope this clarification assures you of our algorithm's applicability to a range of scenarios, including those with large action spaces, even though not all could be directly tested due to resource constraints.

Reviewer comment (Experiments):
There’s a lack of experimental validation beyond the SMAC benchmark, and the same benchmark settings are used across all results and ablations. For example, Google Research Football and Multi-agent MuJoCo would be candidates for other tasks.

Author response:
**Q4: There’s a lack of experimental validation beyond the SMAC benchmark, and the same benchmarks settings are used across all results and ablations. For example, Google Research Football and Multi-agent MuJoCo would be candidates for other tasks.**

We agree on the importance of validating the algorithm's performance in other tasks beyond the SMAC benchmark. It would be beneficial to validate the performance of MAZero in other tasks beyond the SMAC benchmark. We further benchmark MAZero on Google Research Football *academy_pass_and_shoot_with_keeper*, the result is shown below(see Appendix E, Table 11), where our method outperforms baselines in terms of sample efficiency.

- Comparisons on *academy_pass_and_shoot_with_keeper,* Google Research Football.
    
    
    | Env steps | MAZero | CDS | RODE | QMIX |
    | --- | --- | --- | --- | --- |
    | 500k | **0.123 ± 0.089** | 0.069 ± 0.041 | 0 | 0 |
    | 1M | **0.214 ± 0.072** | 0.148 ± 0.117 | 0 | 0 |
    | 2M | **0.619 ± 0.114** | 0.426 ± 0.083 | 0.290 ± 0.104 | 0 |

Reviewer comment (Experiments):
The SMAC benchmark is outdated and should be replaced by the SMACv2 benchmark, as many tasks have been shown to be trivially solvable due to the lack of stochasticity in the SMAC benchmark.

Author response:
**Q5: The SMAC benchmark is outdated and should be replaced by the SMACv2 benchmark, as many tasks have been shown to be trivially solvable due to the lack of stochasticity in the SMAC benchmark.**

We appreciate the reviewer for the suggestion. Nevertheless, we did not evaluate MAZero in SMACv2 in our paper for three reasons:

- **Focus on Deterministic Environments**: The primary focus of our research is deterministic environments, as opposed to stochastic ones. MAZero, building upon the legacy of MuZero, is designed for deterministic models. Our Optimistic Search Lambda technique, in particular, is tailored to leverage deterministic models. While testing on SMACv2 is possible, we believe it might detract from the clarity of our research objectives, which are centered on deterministic settings.
- **Baseline Comparability**: The SMAC benchmark has been widely used for evaluating numerous baseline algorithms. Some baselines like MAMBA are benchmarked on SMAC, not SMACv2. To facilitate straightforward reproduction of results and ensure fair and relevant comparisons, we opted for the SMAC benchmark. This choice ensures that our findings are directly comparable to a broad range of existing research.
- **Relevance of SMAC Despite Low Stochasticity**: Even though SMAC is recognized for its lower stochasticity compared to SMACv2, it still presents a meaningful and non-trivial challenge for multi-agent model learning and searching. Evaluating MAZero on SMAC effectively demonstrates our model's capabilities and the efficacy of our search techniques within this context.

We acknowledge the evolving landscape of benchmarks in this field and the potential value of including SMACv2 in future work. However, for the purposes of this paper, we believe that SMAC remains a relevant and suitable choice for demonstrating the strengths and applications of MAZero.

Reviewer comment (Experiments):
The baselines used are relatively few compared to other published MARL works. There should be comparison with other recent methods like MBVD, RODE, CDS, etc.

Author response:
**Q6: The baselines used are relatively few compared to other published MARL works. There should be comparison with other recent methods like MBVD, RODE, CDS, etc.**

We appreciate the reviewer’s suggestion. We have added RODE and CDS as our baseline (see Figure 3) and results show that MAZero significantly outperform them, especially in terms of sample efficiency. We do not add MBVD as the baseline since we fail to find any open-source implementation of it.

Reviewer comment (Experiments):
It’s not clear how many seeds the work used for the main Table 3. The work mentions 3 seeds for follow-up ablations, but this is too few to ensure that the performance is not due to luck. The results should be reported across at least 10 seeds.

Author response:
**Q7: It’s not clear how many seeds the work used for the main Figure 3. The work mentions 3 seeds for followup ablations, but this is too few to ensure that the performance is not due to luck. The results should be reported across at least 10 seeds.**

We have considered the importance of using more seeds for the main Figure 3 to ensure robust evaluation results. We have conducted extra experiments and update Figure 3 with 10 random seeds.

Reviewer comment (Experiments):
To my knowledge, the search-based component of methods like MuZero are very reduced and not significantly useful for many non-board game tasks. For example, while MuZero utilizes a large search depth for Go, it uses a very small search depth for Atari, which raises the questions of how much performance gain the search component contributes. This work mentions that it uses 5 unroll steps, which is still very small. For MCTS-based works like this one, there should be empirical investigations with respect to the number of unroll steps at both training and test time to see whether search is offering any benefits.

Author response:
**Q1: The search-based component of methods like MuZero are reduced and not significantly useful for many non-board game tasks.**

Thank you for raising concerns about the usefulness of search-based methods like MuZero in non-board game tasks. While it is true that model-free RL has achieved remarkable success in various tasks, it is important to recognize the growing importance of search-based methods, such as MCTS, in recent years.

Although search-based methods may seem less significant compared to model-free RL, they  have actually demonstrated valuable contributions across different domains. EfficientZero [1], for example, has achieved impressive results on standard RL benchmarks. According to the paper, it achieves 194.3% mean human performance on the Atari 100k benchmark with only two hours of real-time game experience. Additionally, there are extensions of MuZero in continuous control [2,3] that outperform model-free baselines in terms of data efficiency. These results strongly indicate the effectiveness of search-based methods in these benchmarks.

Moreover, search-based methods have made significant breakthroughs in application domains. For instance, AlphaTensor [4] discovered faster matrix multiplication algorithms, marking the first improvement in this area in 50 years. AlphaDev [5], its sister work, has also found faster sorting algorithms, some of which have been integrated into the standard sort function in the LLVM standard C++ library. Furthermore, these methods have been extended to NP-hard math problems, further demonstrating their potential and applicability.

In the field of AGI research, search-based methods show promise in providing reasonability for LLM-agents. Many recent works [6-8] are exploring the use of search-based methods to enhance the capabilities of large language models, indicating their potential for advancing natural language understanding and reasoning.

Overall, it is crucial to acknowledge that search-based methods have the potential to be utilized in numerous important domains. With this in mind, the objective of this paper is to contribute to the application of search-based methods in the multi-agent domain, serving as a foundation for further exploration and progress in this field.

[1] Ye, Weirui, et al. "Mastering atari games with limited data." *Advances in Neural Information Processing Systems* 34 (2021): 25476-25488.

[2] Hansen, Nicklas, Xiaolong Wang and Hao Su. “Temporal Difference Learning for Model Predictive Control.” *International Conference on Machine Learning* (2022).

[3] Hubert, Thomas, et al. "Learning and planning in complex action spaces." *International Conference on Machine Learning*. PMLR, 2021.

[4] Fawzi, Alhussein, et al. "Discovering faster matrix multiplication algorithms with reinforcement learning." *Nature* 610 (2022): 47-53.

[5] Mankowitz, Daniel J., et al. "Faster sorting algorithms discovered using deep reinforcement learning." *Nature* 618.7964 (2023): 257-263.

[6] Yao, Shunyu, et al. "Tree of Thoughts: Deliberate Problem Solving with Large Language Models." arXiv preprint arXiv:2305.10601 (2023).

[7] Feng, Xidong, et al. "Alphazero-like Tree-Search can Guide Large Language Model Decoding and Training." arXiv preprint arXiv:2309.17179 (2023).

[8] Zhao, Zirui, Wee Sun Lee, and David Hsu. "Large Language Models as Commonsense Knowledge for Large-Scale Task Planning." arXiv preprint arXiv:2305.14078 (2023).

**Q2: For example, while MuZero utilizes a large search depth for Go, it uses a very small search depth for Atari, which raises the questions of how much performance gain the search component contributes. This work mentions that it uses 5 unroll steps, which is still very small.**

It appears there may be a misconception about its role in our model. The ”*unroll step*” refers to the expansion steps during the model-learning phase and is distinct from the search depth used in the search phase. In our work, we set the *unroll step* to 5, mirroring the configuration used by MuZero in board games like Go. This setting, while seemingly modest, allows us to maintain a balance between learning a robust model and computational efficiency. It's important to note that despite an *unroll step* of 5, our model can achieve a search depth exceeding 20, with a total number of simulations ($N$) being 50. This demonstrates that a smaller *unroll step* is effective and sufficient for learning an accurate model, as also evidenced by MuZero's performance in complex games. We chose this approach primarily for efficiency while ensuring minimal model error.

Reviewer comment (Experiments):
As this work mostly treats the multi-agent decision problem as a single-agent RL with a very large action space, the authors should demonstrate $OS(\lambda)$ and AWPO more extensively on (single-agent) decision problems with large action space; indeed, the author could even solely consider these single-agent settings rather than bringing in the multi-agent SMAC benchmark at all. The single-agent bandit experiment in Figure 1 is too simplistic and insufficient to demonstrate the authors’ point. Also, in very large action spaces, MCTS with a limited budget (e.g., $N = 100$ MCTS simulations as considered in this work) seems very sparse and unlikely to simulate the same action multiple times from a given state. This would significantly weaken the motivation for $OS(\lambda)$, which relies on simulating each action many times from a given state. In such large action spaces, a very relevant heuristics baseline is to sample $N$ simulations (each starting with a different action) with the learned policy and simply follow the action given by the best simulation. Overall, there's insufficient experiments demonstrating that $OS(\lambda)$ and AWPO solve the stated large action space problem without suffering from unintended side-effects of the additional optimism, and there should be more trivial heuristics baselines which may also work well for searching large action spaces with a fixed budget.

Author response:
**Q8: The authors should demonstrate OS($\lambda$) and AWPO more extensively on (single-agent) decision problems with large action space**

We have considered demonstrating the effectiveness of OS($\lambda$) and AWPO in single-agent decision problems with large action spaces. Specifically, we choose *LunarLander* environment but discretize action space into $400$ actions. Additionally, we select the *Walker2D* scenario in MoJoCo environment and discretize each dimension of continuous action space into 7 actions, i.e., $6^7 \approx 280,000$ legal actions. Tables 9 and 10 in Appendix D present the results, demonstrating the enhancement of both techniques in terms of learning efficiency and final performance.

- Ablation study on *LunarLander*.
    
    
    | Env steps | MAZero | w/o OS($\lambda$) and AWPO |
    | --- | --- | --- |
    | 250k | **184.6 ± 22.8** | 104.0 ± 87.5 |
    | 500k | **259.8 ± 12.9** | 227.7 ± 56.3 |
    | 1M | **276.9 ± 2.9** | 274.1 ± 3.5 |
- Ablation study on *Walker2D*, MuJoCo.
    
    
    | Env steps | MAZero | w/o OS($\lambda$) and AWPO | TD3 | SAC |
    | --- | --- | --- | --- | --- |
    | 300k | **3424 ± 246** | 2302 ± 472 | 1101 ± 386 | 1989 ± 500 |
    | 500k | **4507 ± 411** | 3859 ± 424 | 2878 ± 343 | 3381 ± 329 |
    | 1M | **5189 ± 382** | 4266 ± 509 | 3946 ± 292 | 4314 ± 256 |

**Q9: In very large action spaces, MCTS with a limited budget (e.g. $N=100$ MCTS simulations as considered in this work) seems very sparse and unlikely to simulate the same action multiple times from a given state. This would significantly weaken the motivation for OS($\lambda$), which relies on simulating each action many times from a given state.** 

Thank you for highlighting concerns regarding the application of OS($\lambda$) in large action spaces with limited MCTS simulation budgets. We recognize the importance of clarity in explaining our methods and appreciate the opportunity to elucidate further.

OS($\lambda$), as employed in our work, is built upon the framework of Sampled MCTS, wherein each node expansion is conducted exactly once. This is a crucial aspect that seems to have been misunderstood by the reviewer. When a node $u$ is expanded, its set of child nodes, typically limited to a size of $K$ (about $5$ or $10$ in our settings), is determined through an on-policy distribution $\beta$. This 'sampled' aspect ensures that each expanded node receives a unique value estimate through a forward pass of the value network, followed by backpropagation to the root. Later, when an expanded node is revisited, we recursively choose a son of it and do the same thing. Therefore, each expanded node will get its value network forwarded exactly once.

Contrary to the classical sampled MCTS, the final value of a node is the average of all estimations in its subtree (each node $x$ in its subtree constitutes an estimation in the form of $r_1+r_2+…+v(x)$), which is too pessimistic when the model is deterministic and action space is large. This motivates us to design a more optimistic value of expanded nodes.

When action space is large and the simulation budget $N$ is limited, the sampling procedure in classical sampled MCTS restricts the degree of the tree to at most $K$ ($K \ll N$). This restriction enables deeper search depth, mitigating the sparsity issues. While OS($\lambda$) performs optimistic value estimation, enhancing the search efficiency by effectively navigating the limited tree structure.

We hope this clarification underscores the suitability and efficacy of OS($\lambda$) in handling the challenges posed by large action spaces and limited simulation budgets.

Reviewer comment (Experiments):
Is MCTS being performed at evaluation time, or is only the learned policy used? If search is being performed at evaluation time, there should be a quantification of the search overhead compared with baselines.

Author response:
**Q10: Is MCTS being performed at evaluation time, or is only the learned policy used? If search is being performed at evaluation time, there should be a quantification of the search overhead compared with baselines.** 

As requested by the reviewer, we have conducted additional ablation study about whether or not perform MCTS in Appendix C, Table 8. Results on SMAC environments show that agents maintain comparable performance without MCTS planning during evaluation.

| Map | Env steps | w MCTS | w/o MCTS | performance ratio |
| --- | --- | --- | --- | --- |
| 3m | 50k | 0.985 ± 0.015 | 0.936 ± 0.107 | 95.0 ± 10.8% |
| 2m_vs_1z | 50k | 1.0 ± 0.0 | 1.0 ± 0.0 | 100 ± 0.0% |
| so_many_baneling | 50k | 0.959 ± 0.023 | 0.938 ± 0.045 | 97.8 ± 4.7% |
| 2s_vs_1sc | 100k | 0.948 ± 0.072 | 0.623 ± 0.185 | 65.7 ± 19.5% |
| 2c_vs_64zg | 400k | 0.893 ± 0.114 | 0.768 ± 0.182 | 86.0 ± 20.4% |
| 5m_vs_6m | 1M | 0.875 ± 0.031 | 0.821 ± 0.165 | 93.8 ± 18.9% |
| 8m_vs_9m | 1M | 0.906 ± 0.092 | 0.855 ± 0.127 | 94.4 ± 14.0% |
| 10m_vs_11m | 1M | 0.922 ± 0.064 | 0.863 ± 0.023 | 93.6 ± 2.5% |
| average performance |  | 100% | 90.1 ± 11.3 % |  |

Reviewer comment (Experiments):
How does MAZero perform in stochastic environments compared to deterministic ones?

Author response:
**Q1: How does MAZero perform in stochastic environments compared to deterministic ones?**

Our algorithm primarily focuses on deterministic environments, as stated in this paper. We have developed a deterministic model based on the previous work of Muzero. Our OS($\lambda$) technique is specifically designed to optimize this deterministic model. However, we have also conducted additional experiments beyond the SMAC environments, including LunarLander, MuJoCo, and Google Research Football, which exhibit more randomness compared to SMAC. The results in Tables 9, 10, and 11 demonstrate that MAZero performs well to some extent in stochastic environments.

- Ablation study on *LunarLander*.
    
    
    | Env steps | MAZero | w/o OS($\lambda$) and AWPO |
    | --- | --- | --- |
    | 250k | **184.6 ± 22.8** | 104.0 ± 87.5 |
    | 500k | **259.8 ± 12.9** | 227.7 ± 56.3 |
    | 1M | **276.9 ± 2.9** | 274.1 ± 3.5 |
- Ablation study on *Walker2D*, MuJoCo.
    
    
    | Env steps | MAZero | w/o OS($\lambda$) and AWPO | TD3 | SAC |
    | --- | --- | --- | --- | --- |
    | 300k | **3424 ± 246** | 2302 ± 472 | 1101 ± 386 | 1989 ± 500 |
    | 500k | **4507 ± 411** | 3859 ± 424 | 2878 ± 343 | 3381 ± 329 |
    | 1M | **5189 ± 382** | 4266 ± 509 | 3946 ± 292 | 4314 ± 256 |
- Comparisons on *academy_pass_and_shoot_with_keeper,* Google Research Football.
    
    
    | Env steps | MAZero | CDS | RODE | QMIX |
    | --- | --- | --- | --- | --- |
    | 500k | **0.123 ± 0.089** | 0.069 ± 0.041 | 0 | 0 |
    | 1M | **0.214 ± 0.072** | 0.148 ± 0.117 | 0 | 0 |
    | 2M | **0.619 ± 0.114** | 0.426 ± 0.083 | 0.290 ± 0.104 | 0 |

Reviewer comment (Experiments):
Can the proposed OS(λ) and AWPO techniques be applied to other model-based MARL algorithms?

Author response:
**Q2: Can the proposed OS($\lambda$) and AWPO techniques be applied to other model-based MARL algorithms?**

OS($\lambda$) and AWPO are specially designed for MCTS algorithms. We don’t think it can be applied to other model-based MARL algorithms like the dreamer-based MBMARL algorithms (e.g. our baseline method MAMBA).

Reviewer comment (Evaluation):
Are there any potential drawbacks or limitations of the proposed network structure that the authors have not discussed?

Author response:
**Q3: Are there any potential drawbacks or limitations of the proposed network structure that the authors have not discussed?**

We recognize the limitations of the proposed network structure and have outlined potential drawbacks. Some potential drawbacks or limitations of the proposed network structure include:

- The reliance on centralized training, which may not scale well to very large numbers of agents or highly heterogeneous environments.
- Although MAZero is more sample-efficiency than model-free MARL method, it is relatively computationally expensive.