Paper: Maximum Entropy Heterogeneous-Agent Reinforcement Learning (ICLR-2024)

Reviewer comment (Writing):
A minor point: the IGO assumption is not explained when first being introduced. I don't think people outside MARL are familiar with this term.

Author response:
Q1: The IGO assumption is not explained.

> A1: Thank you for pointing it out. We have included the definition of IGO and the explanation of its limitations in Appendix A.2.

Reviewer comment (Theory):
(Unclear effect of the sequential updates) Although there is an example [1] to show why the sequential policy updates for the standard MARL objective are needed, an example or technical explanation for the MaxEnt MARL objective is also needed. It is because the authors define the multi-agent soft Q-function (eq. (7)) and local policy update (eq. (8)). Also, the practical objective for the policy (eq. (10)) can be reduced to the expectation of $\alpha\log\pi_{\phi_{i_m}}^{i_m}-Q_{\pi_{old}}^{i_{1:n}}$, which is consistent with the pseudocode of HASAC, since the MA soft Q-function consists of the centralized Q-function $Q_{\pi_{old}}^{i_{1:n}}$ and the entropy term of its complementary agents, which is not subject to optimize. So, in the proposed algorithm, the sequential policy update may be just additional sampling actions of some agents.

Author response:
A1: Thank you for the valuable suggestion. The advantages of sequential updates compared to simultaneous updates in MaxEnt MARL are similar to those mentioned in standard MARL in [1]. MASAC cannot provide a rigorous guarantee of monotonic improvement during training. To illustrate this, let's consider two agents $(1, 2)$ that take actions $a^1, a^2 \in \\{-2, -1, 1, 2\\}$. They receive the reward $r(a^1,a^2) = a^1 \cdot a^2$. Every agent $i$ takes its action $a^i$ from its policy $\pi^i$. We initialize $\pi^1 = (0, 1, 0, 0)$ and $\pi^2 = (0, 0, 1, 0)$. Under this setting, the initial reward is $r(-1, 1) = -1 \cdot 1 = -1$. The maximizing action for agent $1$ is $a_\text{max}^1 = 2$, and for agent $2$ it is $a_{\max}^{2}=-2$. Consequently, MASAC will simultaneously update $\pi^1$ and $\pi^2$, where  $\pi^1$ will assign a highest probability to action 2, and $\pi^2$ will assign a highest probability to action $-2$ (the specific probabilities are influenced by $\alpha$), resulting in a worse outcome. In contrast, sequential updates could avoid such situation, ensuring monotonic improvement.

Following your suggestion, we have conducted an additional ablation study to investigate the effect of sequential updates in the updated submission version. We compare performance of  the original HASAC and HASAC without sequential updates (or MASAC) on Bi-DexHands tasks. As shown in Appendix I, experimental results show that HASAC consistently achieves better performance than MASAC.

Reviewer comment (Experiments):
(Sensitive to the entropy temperature) The Ablation study shows that the proposed algorithm may not be robust to the entropy temperature and converge to different policies according to the temperature. In this paper, each domain has a different entropy temperature; one domain has a fixed temperature, and another has an automatic temperature with a fixed target entropy. A more effective method to tune entropy terms is needed, like ADER[2].

Author response:
A2: We respectfully disagree with the statement that our automatically tuning temperature method is not effective. The automatically adjusting temperature mechanism we implement for HASAC draws on the method proposed in [2], as stated in Appendix H.1. It is similar to the Equation 13 in ADER [3]. However, regarding the method mentioned in ADER for learning individual target entropies, since we only have a centralized Q function, unlike ADER, which has both a centralized value function and local value functions, we cannot directly apply their approach. Nevertheless, we believe that their idea is indeed enlightening and aids in better temperature adjustment. We will explore how to adjust temperature more effectively in future work. Besides this, our method is already fairly effective in practice. As shown in Appendix H.3, HASAC performs well with automatically adjusting $\alpha$ in most tasks; only a few tasks require additional manual tuning of $\alpha$. As shown in the Appendix C2 of ADER, ADER needs to set different values of $k_{\text{ratio}}, \xi, \alpha^i_\text{init}$ for almost every task on SMAC. In contrast, HASAC achieves good performance across various tasks with auto-tuned  $\alpha$, except for setting $\alpha$ to 0.002 on the 'corridor' task.

Reviewer comment (Experiments):
For the weaknesses, could you provide results of MASAC, which is the HASAC without the sequential policy updates, on the benchmarks and the idea of an adaptive entropy temperature tuning method for better exploration?

Author response:
A1: Thank you for the valuable suggestion. The advantages of sequential updates compared to simultaneous updates in MaxEnt MARL are similar to those mentioned in standard MARL in [1]. MASAC cannot provide a rigorous guarantee of monotonic improvement during training. To illustrate this, let's consider two agents $(1, 2)$ that take actions $a^1, a^2 \in \\{-2, -1, 1, 2\\}$. They receive the reward $r(a^1,a^2) = a^1 \cdot a^2$. Every agent $i$ takes its action $a^i$ from its policy $\pi^i$. We initialize $\pi^1 = (0, 1, 0, 0)$ and $\pi^2 = (0, 0, 1, 0)$. Under this setting, the initial reward is $r(-1, 1) = -1 \cdot 1 = -1$. The maximizing action for agent $1$ is $a_\text{max}^1 = 2$, and for agent $2$ it is $a_{\max}^{2}=-2$. Consequently, MASAC will simultaneously update $\pi^1$ and $\pi^2$, where  $\pi^1$ will assign a highest probability to action 2, and $\pi^2$ will assign a highest probability to action $-2$ (the specific probabilities are influenced by $\alpha$), resulting in a worse outcome. In contrast, sequential updates could avoid such situation, ensuring monotonic improvement.

Following your suggestion, we have conducted an additional ablation study to investigate the effect of sequential updates in the updated submission version. We compare performance of  the original HASAC and HASAC without sequential updates (or MASAC) on Bi-DexHands tasks. As shown in Appendix I, experimental results show that HASAC consistently achieves better performance than MASAC. 

A2: We respectfully disagree with the statement that our automatically tuning temperature method is not effective. The automatically adjusting temperature mechanism we implement for HASAC draws on the method proposed in [2], as stated in Appendix H.1. It is similar to the Equation 13 in ADER [3]. However, regarding the method mentioned in ADER for learning individual target entropies, since we only have a centralized Q function, unlike ADER, which has both a centralized value function and local value functions, we cannot directly apply their approach. Nevertheless, we believe that their idea is indeed enlightening and aids in better temperature adjustment. We will explore how to adjust temperature more effectively in future work. Besides this, our method is already fairly effective in practice. As shown in Appendix H.3, HASAC performs well with automatically adjusting $\alpha$ in most tasks; only a few tasks require additional manual tuning of $\alpha$. As shown in the Appendix C2 of ADER, ADER needs to set different values of $k_{\text{ratio}}, \xi, \alpha^i_\text{init}$ for almost every task on SMAC. In contrast, HASAC achieves good performance across various tasks with auto-tuned  $\alpha$, except for setting $\alpha$ to 0.002 on the 'corridor' task.

Reviewer comment (Novelty):
The novelty of the paper is limited, as the main contribution is the application of the Soft Actor-Critic (SAC) algorithm to the multi-agent setting.

Author response:
We respectfully disagree with the statement that the novelty and contribution of our work are limited. Firstly, our novelties are three-fold: We first use PGM to derive MaxEnt objective of MARL and provide a representation of QRE in the context of MaxEnt MARL. Then we develop the theoretically-justified HASAC algorithm to maximize this objective. Finally, we introduced a versatile template, MEHAML, for rigorous algorithmic design.

Secondly, our application of SAC algorithm to MARL is non-trivial. We not only prove rigorous theoretical properties but also demonstrate strong experimental results.

Lastly, ****Reviewer 9JkX**** also agrees with us on the novelty and theoretical soundness, and believes that our work is not a simple combination of MaxEnt RL and MARL.

Reviewer comment (Experiments):
The authors should have tested their method in scenarios where sample efficiency is crucial (such as real robots, stock exchange, etc), given that their proposed method is off-policy.

Author response:
We regret that we may not currently have the capability to test our algorithm on real robots. However, Bi-DexHands [2] and MAMuJoCo [3] are both simulated environments for robotic control where sample efficiency is crucial. As shown in our experimental results, on-policy algorithms exhibit low sample efficiency in these environments, especially in Bi-DexHands, where on-policy algorithms fail to learn any meaningful joint policy within 1e7 steps. In contrast, HASAC shows high sample efficiency in these tasks, demonstrating sample-efficient even compared to other off-policy algorithms. Therefore, we believe that our current experiments could showcase the sampling efficiency of HASAC.

Reviewer comment (Experiments):
The authors should provide more comprehensive experimental results, including results from a larger number of seeds, to fully demonstrate the effectiveness of their method.

Author response:
Thank you for the precious suggestion. We initially ran each algorithm with at least four seeds. Due to the large number of experiments and algorithms, and the fact that each method's performance difference across the four random seeds were not substantial, we did not run additional random seeds.

Following your suggestion, we rerun each algorithm with eight random seeds on three Bi-DexHands tasks, namely ShadowHandCatchAbreast, ShadowHandTwoCatchUnderarm, and ShadowHandLiftUnderarm. We have updated the experimental results in the updated submission version. From the results of these three tasks, the difference between the results from eight seeds and four seeds is minimal, indicating that our original experiments are sufficiently comprehensive to statistically demonstrate the advantages of our algorithm.

Reviewer comment (Experiments):
Could the authors provide more details on how sensitive is the performance of HASAC to the choice of α?

Author response:
Thank you for raising the question. Actually, HASAC is indeed sensitive to $\alpha$, as shown in Section 5.2. Since $\alpha$ affects the stochasticity of each policy, it leads to the convergence to different QREs. Figure 5 and 4(c) intuitively illustrate the impact of different $\alpha$ on the final QREs and performance. Therefore, setting different $\alpha$ for various tasks is necessary to enable policies to discover and converge to higher reward equilibria. To mitigate such sensitivity to $\alpha$, we draw on the method of auto-tuned $\alpha$ mentioned in [1] and have implemented it for HASAC. As shown in our Appendix H, auto-tuned $\alpha$ does achieve this goal. In most tasks, HASAC performs well with automatically adjusting $\alpha$; only a few tasks require additional manual tuning of $\alpha$. This significantly alleviates sensitivity of HASAC to $\alpha$.

Reviewer comment (Experiments):
How does the proposed method perform in scenarios where sample efficiency is crucial? Could the authors provide experimental results in such scenarios?

Author response:
We regret that we may not currently have the capability to test our algorithm on real robots. However, Bi-DexHands [2] and MAMuJoCo [3] are both simulated environments for robotic control where sample efficiency is crucial. As shown in our experimental results, on-policy algorithms exhibit low sample efficiency in these environments, especially in Bi-DexHands, where on-policy algorithms fail to learn any meaningful joint policy within 1e7 steps. In contrast, HASAC shows high sample efficiency in these tasks, demonstrating sample-efficient even compared to other off-policy algorithms. Therefore, we believe that our current experiments could showcase the sampling efficiency of HASAC.

Reviewer comment (Experiments):
Could the authors provide more comprehensive experimental results, including results from a larger number of seeds?

Author response:
Thank you for the precious suggestion. We initially ran each algorithm with at least four seeds. Due to the large number of experiments and algorithms, and the fact that each method's performance difference across the four random seeds were not substantial, we did not run additional random seeds.

Following your suggestion, we rerun each algorithm with eight random seeds on three Bi-DexHands tasks, namely ShadowHandCatchAbreast, ShadowHandTwoCatchUnderarm, and ShadowHandLiftUnderarm. We have updated the experimental results in the updated submission version. From the results of these three tasks, the difference between the results from eight seeds and four seeds is minimal, indicating that our original experiments are sufficiently comprehensive to statistically demonstrate the advantages of our algorithm.

Reviewer comment (Writing):
The contribution of the method in part hinges on the limitations induced by the 'IGO' assumption from prior work (Sec 2, p2), but this is never elaborated on in the paper. Can you define IGO more clearly and explain exactly what limitations this assumption introduces? This will help the reader not intimately familiar with MARL.

Author response:
A1: Thanks for pointing this out. Due to the 9-page limit, we have included the definition of IGO and the explanation of its limitations in Appendix A.2.

Reviewer comment (Experiments):
It seems the design of the drift functional and neighborhood operators will be key to the success of the proposed HASAC, or MEHAML based methods (as you note in Sec. 6). Can you provide any comment on what factors should be taken into consideration in the construction of these terms? E.g. in what ways will this depend on the nature or definition of the MARL task? Some discussion of the design/selection of these terms for your empirical experiments might be helpful here.

Author response:
A3&A4: Thank you for raising the question. We would like to answer it as follows. First, HASAC can be seen as the most natural instance of MEHAML template with a drift functional of $0$ and neighborhood of $\Pi^{i_m}$.  Nevertheless, HASAC has demonstrated effective performance across a majority of tasks.

> Furthermore, when designing more sophisticated drift functionals and neighborhoods, the most crucial factor is undoubtedly to design operators that satisfy the definition. As discussed in [3], drift functional can be designed as KL-divergence, squared L2 distance, etc., while neighborhood can be designed as KL-ball, like in TRPO. What's more, drift functional can be obtained by meta-learning, as shown in [4]. The objective of our theory is primarily theoretical, aiming to provide theoretical guarantees for all these potential methods in MaxEnt MARL. The specifics of which drift functional and neighborhood lead to better performance in a certain task, as well as how to design practical algorithms with different operators, remain for future research.

Reviewer comment (Writing):
The notion of Quantal Response Equilibrium is key to this optimization objective, but I'm not familiar with this term. You provide a citation (#20, also #6), but the paper would be strengthened with a little bit more explanation of this notion in Sec 4.1. E.g. can you give some intuition for what this objective means in practice compared to regular Nash Equilibrium? In what situations is QRE to be preferred over NE?

Author response:
A5: We apologize if the explanation of QRE is not clear enough. In fact, QRE is a generalization of NE. In NE, each player is perfectly rational, deterministically selecting the strategy with highest payoff. In QRE, however, each player is bounded rational, resulting in their strategies being probabilistic distributions related to payoffs. Put simply, QRE represents the equilibria of stochastic behavior, while NE represents the equilibria of deterministic behavior.

> In our case, due to the additional entropy regularization term in the MaxEnt objective, each agent's policy at equilibrium becomes stochastic (Boltzmann distribution) rather than a deterministic one, as demonstrated by Theorem 1. Therefore, it corresponds to QRE. As $\alpha \rightarrow 0$, the MaxEnt objective is reduced to the standard objective, the Boltzmann distribution is reduced to a one-hot distribution, and QRE is reduced to NE. Hence, it is not accurate to claim any inherent advantages of QRE over NE; the observed advantage in experimental results arises from the effectiveness of stochastic policies, which inherently facilitate more efficient exploration, leading to the discovery and convergence of equilibria with higher rewards.

Reviewer comment (Writing):
What is the $\omega$-limit (Point 4 in Theorem 3) - I could not find a definition and am not familiar with this terminology.

Author response:
A6: The $\omega$-limit set is a concept used in the study of dynamic systems, which describes the set of states that a system tends to reach as time progresses indefinitely. In our case, the sequence of joint policies would reach fixed points of soft policy iteration, and these fixed points are QRE policies. Therefore, we refer to these fixed points as the $\omega$-limit set of joint policies, and their $\omega$-limits set consists of QRE.

Reviewer comment (Writing):
Under heading 5.1 - '2 hundred' - could write as '200'.

Author response:
Thank you for the valuable suggestion. We have rewritten '2 hundred' as '200' and added a table of acronyms in Appendix A.1.

Reviewer comment (Writing):
There are a lot of acronyms in this paper - please consider adding a table of acronym definitions in the appendix to aid readers.

Author response:
Thank you for the valuable suggestion. We have rewritten '2 hundred' as '200' and added a table of acronyms in Appendix A.1.