Paper: Attention-Guided Contrastive Role Representations for Multi-agent Reinforcement Learning (ICLR-2024)

Reviewer comment (Theory):
There is no reasonable explanation or formula for the promotion of credit assignment by attention, and the paper only demonstrates the effectiveness of the method through experiments.

Author response:
Thank you for your insightful comments.
Throughout the paper, we have provided explanations for the promotion of more expressive credit assignment in many places.
For example, the first paragraph of Sec. 2.2 (top, page 5) presents the line our research motivation for introducing the attention mechanism as "Popular CTDE algorithms, such as QMIX, realize behavior coordination across agents via a mixing network that estimates joint action-values as a monotonic combination of per-agent values, and the mixing network weights are conditioned on the system’s global state. Naturally, it is interesting to incorporate the learned role information into the mixing network to facilitate skillful coordination across roles. The simplest approach is to concatenate the global state and role representations for generating mixing network weights, while it fails to exploit the internal structure to effectively extract correlations in the role space. Fortunately, the attention mechanism aligns perfectly with our intention by prompting the global state to attend to learned role patterns, thus providing more expressive credit assignment in value decomposition."
On the other hand, the visualization in Sec. 3.3 clearly illustrates that the global state has successfully attended to the learned role patterns, which in turn provides evidence for our attention-guided promotion of credit assignment.

Reviewer comment (Experiments):
How do you know the state encoding after this attention, input to the mix network can have an impact on credit assignment?

Author response:
As stated in A1, the idea of our attention-guided role coordination comes from QMIX's value decomposition.
QMIX realizes behavior coordination across agents (i.e., credit assignment) via the mixing network whose weights are conditioned on the system’s global state.
Intuitively, if we could condition the mixing network weights on extra role representations, we might realize more efficient role coordination for better credit assignment.
Fortunately, the attention mechanism aligns perfectly with our intention by prompting the global state to attend to learned role patterns, thus providing more expressive credit assignment in value decomposition.
Moreover, ablation studies (Sec. 3.1 and Appendix F) and visualization of role coordination (Sec. 3.3) have also verified that our attention module indeed has a positive impact on credit assignment.

Reviewer comment (Novelty):
In cooperative multi-agent reinforcement learning, inferring the role of an agent based on its trajectory is not a novel method and has been proposed in many previous works [1, 2]. Moreover, none of the above-mentioned important papers are cited in the paper.

Author response:
We agree with you that inferring the role of an agent based on its trajectory has been investigated in many previous works, such as those role-based methods in our original paper.
We have indeed included [2] as a Role Emergence method in the Related Work section (top, page 9) of the original paper, and we will also include [1] in our related work in the updated paper to make our literature review more complete and comprehensive.
Compared to existing role-based methods, our method introduces two innovative components, i.e., the contrastive learning formulation for learning discriminative role representations and the attention mechanism to coordinate learned role representations for more expressive credit assignment, and our method obtains SOTA performance on most experiments compared to them.

[1] Cao, Jiahang et al. LINDA: multi-agent local information decomposition for awareness of teammates, Science China Information Science 2021.

[2] Yang, Jiachen et al. Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discovery, AAMAS 2020.

Reviewer comment (Experiments):
The number of baselines used for comparison with ACORM is relatively tiny. Why not use CDS [3] as a baseline, since you mentioned it in the paper?

Author response:
As mentioned in our original paper, there are a large number of related methods that involve agent heterogeneity and role emergence.
For a moderate demonstration, we chose the basic QMIX and the four most relevant methods as our *five* baselines. 
We skipped some methods that generally perform worse than the selected baselines.
For example, ROMA and RODE are both classical role-based methods, and RODE generally outperforms ROMA in the literature.
Hence, we compare ACORM to RODE only.
Further, we have included CDS as an additional baseline in the updated paper to make our evaluation more persuasive.
Extended experimental results are consistent with our previous conclusion, and our method outperforms CDS in most scenarios.

Reviewer comment (Experiments):
SMAC is a relatively old multi-agent testbed. Recently, it has been pointed out that it has a series of problems [4]. I am not against the author evaluating the performance of the algorithm on SMAC, but I feel that the performance of the algorithm should be tested in multiple different domains. Many environments, such as the Google Research Football [5] mentioned in the paper, can be used to enhance the credibility of experimental results.

Author response:
In the original version, we evaluated our method on six *representative* SMAC maps.
Following your suggestion, we have conducted additional experiments on more SMAC maps and on Google research football environments, and provided the extended results in the updated Appendices B and C.
The results on massively extended experiments are generally consistent with our previous conclusion that ACORM obtains the best performance in most scenarios and outperforms baselines by a larger margin on harder tasks.

Reviewer comment (Experiments):
Ablation experiments are insufficient. Compared with the vanilla QMIX, ACORM_w/o_MHA still has an additional MLP and GRU for the global state. One wonders whether what really works is just the representation learning of the state trajectory before input to the Mixing Network.

Author response:
Thank you for your helpful advice.
We have included an additional ablation baseline "ACORM\_w/o\_MHA (Vanilla)" that removes both the attention module and the additional GRU-MLP structure.
In this way, the input to the mixing network is only the current global state $\bf{s}$$^t$, which is kept the same as the original QMIX algorithm.
In Sec. 3.1 of the updated paper and the updated Appendix F, it can be observed that ACORM\_w/o\_MHA (Vanilla) obtains very similar performance compared to ACORM\_w/o\_MHA.
It can be concluded that the effectiveness of our attention module comes from prompting the global state to attend to role representations other than the representation learning of the state trajectory via GRU.

Reviewer comment (Reproducibility):
It is not possible to reproduce the results from the description given in the paper. Some key details (such like $T_{cl}$) are unclear, and some key resources (code) are not furnished.

Author response:
Thank you for your careful reading. 
We have provided the source code of ACORM in the updated supplementary material to ensure reproducibility.
In the original appendices, $T_{cl}$ is the time interval for updating contrastive loss as shown in Algorithm 1 and Tables 2 \& 4.
To make these details clearer, we have modified these notations to be consistent across all appendices.

Reviewer comment (Experiments):
**Experimental Evaluation**: The experiments in the article are conducted solely on SMAC. To my knowledge, various versions of SMAC exist, and different algorithm implementations often involve custom modifications to this environment. Relying solely on SMAC for experiments may not be sufficiently persuasive. It might be beneficial to include experiments from other environments such as GRF and Ma-MuJoCo.

Author response:
**Q1: Reproducibility and more experimental evaluation.**

A1: We have provided our source code in the updated supplementary material to ensure reproducibility.
In the original version, we evaluated our method on six *representative* SMAC maps.
Following your advice, we have conducted additional experiments on more SMAC maps and on Google research football environments, and provided the extended results in the updated Appendices B and C.
The results of massively extended experiments are generally consistent with our previous conclusion that ACORM obtains the best performance in most scenarios and outperforms baselines by a larger margin on harder tasks.

Reviewer comment (Reproducibility):
**Reproducibility**: The supplementary materials do not include the source code, making reproducibility uncertain.

Author response:
**Q1: Reproducibility and more experimental evaluation.**

A1: We have provided our source code in the updated supplementary material to ensure reproducibility.
In the original version, we evaluated our method on six *representative* SMAC maps.
Following your advice, we have conducted additional experiments on more SMAC maps and on Google research football environments, and provided the extended results in the updated Appendices B and C.
The results of massively extended experiments are generally consistent with our previous conclusion that ACORM obtains the best performance in most scenarios and outperforms baselines by a larger margin on harder tasks.

Reviewer comment (Theory):
As previously mentioned, ACORM does not impose constraints on the attention mechanism, and it even utilizes the learned latent variable $z$. How can we ensure its correct execution under the CTDE paradigm?

Author response:
**Q4: ACORM's correct execution under the CTDE paradigm.**

A4: As stated in A2 and A3, we use the same mixing network architecture as QMIX and just modify the input to the mixing network.
During centralized training, we can access the global state $\bf{s}$ and role representations $z_i$ of each agent $i$.
After prompting the global state to attend to role representations, we concatenate the attention output and the original global state as the input to the mixing network for value decomposition (where the hypernetworks produce the weights and biases for mixing network layers).
During execution, each agent can make decisions according to the individual Q-network only, without access to the global state or others' role representations, i.e., decentralized execution.
As we have emphasized in our original paper, ACORM strictly follows the CTDE paradigm and is fully compatible with CTDE methods.

Reviewer comment (Presentation):
Regarding the analysis of Contrastive role representation, Figure 4 is not particularly convincing: In subfigure (b), *(0) is even further from other points in its cluster compared to *(5). While I understand that clustering is done in higher dimensions, this example can be confusing.

Author response:
**Q5: In Fig. 4(b),  agent 0 is even further from other points in its cluster compared to agent 5.**

A5: We cluster agent embeddings in an *unsupervised* way, and there is no ground truth to indicate the oracle cluster assignment.
The effectiveness of our method relies on the mechanism of whether agents with similar behavior patterns tend to be grouped into the same cluster, and whether the contrastive learning objective can push points within the same cluster closer to each other and promote different clusters to be notably separated. 
Fortunately, our method meets this condition in most cases, as shown in Figs. 4-5.
Several outliers, e.g., Agent 0 is even further from other points in its cluster compared to Agent 5 in Fig. 4(b), will not have a fundamental impact on the effectiveness of our method.

Reviewer comment (Experiments):
Additional experiments are needed to bolster the paper's persuasiveness.

Author response:
**Q1: Reproducibility and more experimental evaluation.**

A1: We have provided our source code in the updated supplementary material to ensure reproducibility.
In the original version, we evaluated our method on six *representative* SMAC maps.
Following your advice, we have conducted additional experiments on more SMAC maps and on Google research football environments, and provided the extended results in the updated Appendices B and C.
The results of massively extended experiments are generally consistent with our previous conclusion that ACORM obtains the best performance in most scenarios and outperforms baselines by a larger margin on harder tasks.

Reviewer comment (Experiments):
Extremely Lack of Experiments. There is a lack of experiments as the author only conducted experiments on 6 maps in SMAC. The same applies to the ablation experiments. Hope there will be additional experiments in a wider range of environments and on more maps within SMAC.

Author response:
**Q1: Lack of source code and experiments.**

A1: We have provided the source code of our paper in the updated supplementary material to ensure reproducibility.
In the original version, we evaluated our method on six *representative* SMAC maps.
Following your suggestion, we have conducted additional experiments on more maps within SMAC and on Google research football environments, and provided the extended results in the updated Appendices B and C.
The results of massively extended experiments are generally consistent with our previous conclusion that ACORM obtains the best performance in most scenarios and outperforms baselines by a larger margin on harder tasks.

Reviewer comment (Reproducibility):
The author did not provide the source code to verify.

Author response:
**Q1: Lack of source code and experiments.**

A1: We have provided the source code of our paper in the updated supplementary material to ensure reproducibility.
In the original version, we evaluated our method on six *representative* SMAC maps.
Following your suggestion, we have conducted additional experiments on more maps within SMAC and on Google research football environments, and provided the extended results in the updated Appendices B and C.
The results of massively extended experiments are generally consistent with our previous conclusion that ACORM obtains the best performance in most scenarios and outperforms baselines by a larger margin on harder tasks.

Reviewer comment (Evaluation):
There might be some errors in the analysis. Such as the analysis in Appendix D, there is room for debate regarding the phenomenon of sub-groups. It is incorrect to measure the distance between 2 points within 1 cluster in the original space based on their proximity in the t-SNE space. Evaluating the emergence of sub-groups should start from the original space rather than the two-dimensional space after t-SNE reduction. Similarly, after t-SNE reduction, the distance between clusters still does not reflect the real distance. Therefore, the conclusion of 'their role representations are still closer to each other' in the later part still requires the author's reconsideration.

Author response:
**Q4: Analysis on the visualization of agent embeddings and role representations via t-SNE.**

A4: We agree with you that after t-SNE reduction, the distance between clusters does not match the real *absolute* distance. 
However, what we care about is the *relative* distance between points or clusters.
The t-SNE is a widely used dimensionality reduction technique for embedding high-dimensional data for visualization in a low-dimensional space, in such a way that similar objects are modeled by nearby points and dissimilar objects are modeled by distant points with high probability.
Hence, the distance between points or clusters via t-SNE reduction can reflect their relative distances in the original space.

Reviewer comment (Novelty):
The relationship between the two parts, MHA and CL, in the article is not particularly close; they seem more like two relatively independent components.

Author response:
**Q7: Relationship between MHA and CL.**

A7: We use CL to promote discriminative role representation learning, and utilize an MHA mechanism to coordinate learned role representations for more expressive credit assignment. 
The two innovative components work as a whole to efficiently incorporate the role information into MARL to promote behavior heterogeneity, knowledge transfer, and skillful coordination across agents.

Reviewer comment (Writing):
There are some typos and the writing of the paper needs some improvement.

Author response:
**Q9: There are some typos and the writing of the paper needs some improvement.**

A9: Thank you for your careful advice. We have fixed several typos and improved the writing as much as we can.

Reviewer comment (Evaluation):
In Fig 4. (c), both Agent 5 and 8 are 'Dead Marines'. Why are they clustered into different classes?

Author response:
**Q5: Why dead Marines 5 and 8 are clustered into different classes in Fig. 4(c).**

A5: We cluster agent embeddings in an *unsupervised* way, and there is no ground truth to indicate the Oracle cluster assignment.
The effectiveness of our method relies on the mechanism that agents with similar behavior patterns tend to be grouped into the same cluster. 
Fortunately, our method meets this condition in most cases, as shown in Figs. 4-5.
Several outliers, e.g., dead Marines 5 and 8 are clustered into different groups, will not have a fundamental impact on the effectiveness of our method.
As we can see in Fig. 4(c), Marines \{2,3,4,6,7\} are in the offense team, Marauders \{0,1\} and Marine 5 form one dead group, Marine 8 alone represents another dead group, and Medivac 9 alone forms the rescue group.

Reviewer comment (Experiments):
Why was a new map, 2s3z, introduced for the experiments in MAPPO? Why not directly use the previously employed 3s5z_vs_3s6z map? I would like to see an additional MAPPO experiment on 3s5z_vs_3s6z.

Author response:
**Q10: Additional MAPPO experiment on 3s5z\_vs\_3s6z.**

A10: In the original version, we evaluated MAPPO-based ACORM on three *representative* SMAC maps.
Due to the very limited time for rebuttal revision, the performance of MAPPO-based ACORM on more SMAC maps is not updated.
Debugging MAPPO-style algorithms is generally more time-consuming, since it involves many hyperparameters and empirical tricks.
We hope that the extended experimental evaluation could demonstrate adequate persuasiveness of our method, and we are rushing to complete the evaluation results and trying to update them before the rebuttal deadline.

Reviewer comment (Experiments):
How is the setting cluster_num = 3 applied on the map 2c_vs_64zg when there are only 2 agents available for control on this map? Besides, why didn't the performance decline since it forces the strategies of each agent to diverge as same as the experiment about cluster_num = 5 applied on the map 5m_vs_6m?

Author response:
**Q6: How is the setting cluster\_num=3 applied on the map 2c\_vs\_64zg? Besides, why didn't the performance decline since it forces the strategies of each agent to diverge?**

A6: The number of clusters is less or equal to the number of agents. 
We have specifically pointed out that the cluster number is 2 on the 2c\_vs\_64zg map in the updated Appendices B and D.
When each agent represents a distinct role cluster that forces the strategies of each agent to diverge, the performance may decline compared to those with effective team composition (e.g., K=5 compared to K=2,3,4 on the map 5m\_vs\_6m).
However, it could still obtain improved performance compared to most baselines in Fig. 2, since it can explore cooperatively heterogeneous behaviors via our discriminative role representation learning.

Reviewer comment (Experiments):
In the derivation, both the approximation and logK indicate that a larger value of K yields better results. The experiments conducted in the selected maps have a limited number of agents. It is suggested to have more agents and experiment with larger values of K. For example, experiments with larger K values, such as K=2, 4, 8, 16, can be run in scenarios like 30m and bane_vs_bane.

Author response:
**Q2: It is suggested to have more agents and experiments with larger values of K.**

A2: Following your suggestion, we have conducted additional experiments with larger numbers of agents, including the maps of 30m and bane\_vs\_bane.
Results can be found in the updated Appendix B.

Reviewer comment (Writing):
The paper doesn't explicitly clarify the difference between clustering directly on agent embeddings and on role representations. As it considers role representations to be more discriminative, it's important to further elucidate the necessity of obtaining discriminative representations through contrastive learning.

Author response:
**Q8: Clarify the difference between clustering directly on agent embeddings and on role representations, and further elucidate the necessity of obtaining discriminative representations through contrastive learning.**

A8: As we clarified in the 'Negative Pairs Generation' part of the original paper (bottom, page 4), we perform clustering according to agent embeddings, not role representations.
First, we obtain agent embedding by encoding the agent's trajectory as $e_i^t=f_{\phi}(o_i^t, a_i^{t-1}, e_i^{t-1})$, and the agent embedding is transformed into role representation by a role encoder as $z_i^t=f_{\theta}(z_i^t|e_i^t)$ where the role encoder $\theta$ is optimized by a contrastive learning objective.
As shown in the original paper, we have elucidated the necessity of obtaining discriminative representations through contrastive learning in Sec. 2.1.
Further, experimental performance and ablation studies in Sec. 3.1 have also verified the effectiveness of our constrastive learning module, and visualization in Sec. 3.2 has clearly shown that the role encoder is capable of transforming agent embeddings into more discriminative representations.

Reviewer comment (Experiments):
The model has been added with a global state GRU and a MHA structure. Therefore it increases the number of parameters of the networks. It is recommended to conduct ablation studies with the same network size.

Author response:
**Q3: It is recommended to conduct ablation studies with the same network size, as our model is added with a global state GRU and an MHA structure.**

A3: Following your advice, we have kept the network size of ACORM the same as all ablations for a fair comparison.
Further, we have added another ablation baseline to identify the respective effects of the attention module and the state trajectory encoding via GRU.
The updated experimental results are shown in Sec. 3.1 of the revised paper and in the revised Appendix F.
It can be observed that the ablation performance is generally consistent with that of our original paper.