Paper: DECENTRALIZED MULTI-AGENT REINFORCEMENT LEARNING VIA ANTICIPATION SHARING (ICLR-2024)

Reviewer comment (Theory):
The biggest weakness of this paper is that it fails to establish why AS has better assumptions or performs better than related methods. It also fails to provide intuition for the method.

Author response:
**Motivation for AS and comparison with existing methods**. Thanks for pointing out that the proposed methodology has not been motivated sufficiently. We have taken this important observation into account in our revised manuscript.

In summary, the primary motivation for developing AS was to address the limitations of existing decentralized MARL methods, which often rely on sharing rewards, values, or model parameters. These conventional methods can lead to challenges such as privacy concerns and computational inefficiencies. AS offers a novel approach by focusing on sharing action distributions, which we argue is potentially less invasive and more privacy-preserving for agents compared to exposing detailed model parameters or reward functions directly. This shift in information sharing paradigm is central to AS's proposed contribution. We hypothesize that by sharing anticipated actions instead of full internal model details, agents may be able to achieve more effective coordination while maintaining a greater degree of autonomy and privacy.

We also wish to emphasise that anticipation sharing was undertaken in a distributed manner during training only. The resulting learned policy networks of other agents are not activated at test time. In the revised manuscript (Appendix.E, page 17), we have included a system diagram to help explain the distributed, private nature of the AS approach visually.

Reviewer comment (Novelty):
While the authors have provided many details on AS and have good results, fundamentally, the authors failed to communicate the answers to the following important questions: What is the motivation for AS? What are the existing methods and why would we expect AS to do better than them? What is the problem with the existing methods that AS solves?

Author response:
**Motivation for AS and comparison with existing methods**. Thanks for pointing out that the proposed methodology has not been motivated sufficiently. We have taken this important observation into account in our revised manuscript.

In summary, the primary motivation for developing AS was to address the limitations of existing decentralized MARL methods, which often rely on sharing rewards, values, or model parameters. These conventional methods can lead to challenges such as privacy concerns and computational inefficiencies. AS offers a novel approach by focusing on sharing action distributions, which we argue is potentially less invasive and more privacy-preserving for agents compared to exposing detailed model parameters or reward functions directly. This shift in information sharing paradigm is central to AS's proposed contribution. We hypothesize that by sharing anticipated actions instead of full internal model details, agents may be able to achieve more effective coordination while maintaining a greater degree of autonomy and privacy.

We also wish to emphasise that anticipation sharing was undertaken in a distributed manner during training only. The resulting learned policy networks of other agents are not activated at test time. In the revised manuscript (Appendix.E, page 17), we have included a system diagram to help explain the distributed, private nature of the AS approach visually.

Reviewer comment (Theory):
The abstract states, "In this research, we obviate the need for sharing rewards, values, or model parameters." Sharing action distributions is not a weaker assumption, since it involves sharing a similar amount of information. The paper does not explain why it is a better assumption.

Author response:
**Why sharing action distributions is a better assumption**. Sharing action distributions, as opposed to rewards, values, or model parameters, strikes a balance between the need for effective coordination and maintaining individual agents' privacy. While it does involve sharing a significant amount of information, it circumvents the direct exposure of an agent's internal decision-making process, which can be sensitive in many applications. We believe that this approach can be particularly advantageous in scenarios where privacy and security are paramount, and where the sharing of detailed internal states or policies is not feasible. We have stressed this point more in the revised version.

Reviewer comment (Theory):
The intuition behind the proposed method and the motivation is not totally clear. In Definition 2, the authors define the expectation over individual anticipated advantages. If the understanding is correct, this is the ith agent's advantage of the modelled joint action over the joint actions sampled from π^′, which may be the value of the true joint policy (but this is uncertain). As each agent has a different reward function, the advantages may not even be on the same scale. Thus, it seems dubious to average over all agents' advantages. The paper should provide further intuition for what this quantity means and why we might wish to bound the difference.

Author response:
**Definition 2.**  We appreciate your attention to Definition 2 and the opportunity to clarify it further given its importance. It appears there may have been a misunderstanding regarding the expectation over individual anticipated advantages. This expectation is indeed calculated over the anticipated joint actions, denoted as $\tilde{\pi}^i$, not over $\pi'$. The key distinction here is that each agent in our framework optimizes based on its anticipated joint policy ($\tilde{\pi}^i$), rather than $\pi'$. This is a crucial aspect of our approach, as it enables decentralized optimization - each agent individually anticipates and adapts to the actions of others without the need for central coordination or access to the true joint policy. This decentralized nature of policy anticipation and adaptation is what allows our method to be applicable in settings where sharing detailed internal models or policies is not feasible. In the revised manuscript, we have elucidated this point more clearly to avoid any confusion.

Reviewer comment (Evaluation):
The second biggest weakness is the experimental analysis. The current analysis is too speculative and makes vague statements. For example, in the experimental section the authors state:  
"In comparison, the performance of VS and VPS is not stable across all the tasks. This may imply that the mere act of sharing value or value functions and achieving a consensus on values may not be sufficient to establish cooperative policies. This is because, even though value updates can approximate a system-wide value, **the policy updates of agents lack a coordination mechanism, thereby leading to inferior cooperation performance.** Our algorithm provides a policy coordination mechanism, which directly rectifies the deviation of individual policy update from the collective cooperation goal."  
The bolded claim here is causal and unsubstantiated by evidence. It is simply speculation. The authors should either provide an experiment to support this claim or mark it as a hypothesis. Overall, this section provides only weak insight into why AS is better than the baselines.

Author response:
**Experimental analysis.**  Thank you for the insightful feedback on strengthening our experimental analysis section. We appreciate you highlighting areas needing additional empirical substantiation and clarity.

To address these concerns,  in the revised manuscript (Section 5.2) , we have included a more comprehensive analysis of the experemental results. We have also clearly denoted certain statements as hypotheses where they remain partially speculative.

Importantly, we want to clarify that our method does not focus on explicitly modeling other agents' policies. Rather, each agent calculates anticipations about others' policies to optimize its own objectives. In particular, the anticipation $\pi^{ij}$ is solved by agent i to optimize its own objective (Eq. 11). The constraints in Eq. 11 incorporate the discrepancy between $\pi^{ij}$ and agent j's true action distribution $\pi^{jj}$. While related to teammate modeling, the key distinction is that this discrepancy is not the primary optimization target - it is a constraint when maximizing agent i's objective.

$\pi^{ij}$ is then shared with agent j so it can incorporate agent i's objective when maximizing its own. Additionally, agent i's policy $\pi^{ii}$ is adjusted based on the discrepancy with $\pi^{ji}$, agent j's anticipation of agent i. This enforces agreement between local learning and others' anticipations, going beyond typical teammate modeling objectives.

Regarding the scalability of the method, in the revised Appendix of the manuscript (page 16), we include experiments regarding scalability study.  Please let us know if the revised experiments help assuage the concerns over scalability.

Reviewer comment (Experiments):
Missing experiment: It seems impractical for each agent to have an accurate anticipated policy for all other agents, as it would limit the scalability of the method. The authors should add an experiment on, and discussion of, how inaccuracies in teammate modelling affect the performance of the method.

Author response:
Regarding the scalability of the method, in the revised Appendix of the manuscript (page 16), we include experiments regarding scalability study.  Please let us know if the revised experiments help assuage the concerns over scalability.

Reviewer comment (Presentation):
There should be a figure summarizing the method somewhere in the paper.

Author response:
**Miscellaneous clarity issues.** To improve the overall clarity, as per your suggestion,  in the revised manuscript (Appendix.E, page17), we have incorporated a comprehensive figure summarizing our method, which should aid in better understanding the overall approach- thanks for this.

Reviewer comment (Writing):
The language of "anticipated policies" is unclear. It would be clearer and more reflective of the approach if the paper replaced the terminology "anticipation" with "teammate modelling".

Author response:
**Relation to teamate modelling.** We appreciate your suggestion regarding the clarity of the term 'anticipated policies' in our paper. However, after careful consideration, we have chosen to retain the term "anticipation" rather than adopting "teammate modeling." This decision is based on the fundamental differences between these concepts. Unlike traditional teammate modeling, which involves agents modeling or predicting the actual policies of their peers, our concept of "anticipation" refers to each agent calculating actions of others that would be most beneficial for its own objective. Therefore, the anticipations include the information about individual objectives implicitly. By exchanging the anticipations, individual agents can balance others' objectives and thus the collective performance when optimizing its own objective.

This approach emphasizes solving anticipations serving for the agent's own return optimization, and strategic adaptation based on the anticipations from others that implicitly include the information of their returns, rather than attempting to accurately model their behaviors. This distinction is central to our approach and sets it apart from conventional teammate modeling. In the revised manuscript, we have clarified this terminology and its importance to our research, highlighting how our method's unique perspective contributes to the field of decentralized multi-agent environments. Please see the last paragraph in Section.2 and the last paragraph in Section 4.2 in the revised manuscript.

Reviewer comment (Presentation):
In Algorithm 1, lines 13 and 14, the notation suggests that the policies are shared, not the action distribution. The authors should add a clarifying note to the caption.

Author response:
**Clarification in algorithm 1.** Thanks for pointing out this clarity issue. We have revised lines 13 and 14 of Algorithm 1 to clearly indicate that it is the action distribution, rather than the policies themselves, that are shared.

Reviewer comment (Novelty):
Missing related work: There is a large body of related work on teammate modelling, both in the CTDE setting and the ad-hoc teamwork setting. The authors should see Albrecht et al.'s survey (https://arxiv.org/abs/1709.08071) and add an appropriate subsection of related work.

Author response:
**Missing related work.** We appreciate the reference to Albrecht et al.'s survey on teammate modeling. We have added teammate modeling in Related Work (Section. 2) in our revised manuscript.

Reviewer comment (Experiments):
The simulated environment tests presented seem overly simplistic. Without testing in more complex scenarios, the method's scalability and robustness remain unproven.

Author response:
**Scalability and robustness.** Our simulated tests in relatively simple environments are intended as preliminary validations of the fundamental approach. We recognize that demonstrating scalability and robustness requires evaluation across more heterogeneous, dynamic scenarios.

Per your insightful guidance, in the revised Appendix of the manuscript (page 16), we have expanded the manuscript to include experiments regarding scalability study.  Please let us know if the revised experiments help assuage the concerns over scalability and robustness testing.

Reviewer comment (Writing):
The paper's narrative occasionally lapses into redundancy, indicating a need for more concise and focused writing.

Author response:
**Redundancy in writing**. We have taken your feedback about the occasional redundancy in our writing into consideration. The revised manuscript has been edited accordingly.

Reviewer comment (Theory):
How does the proposed "Anticipation Sharing" method handle non-stationarity introduced by agents continuously updating their anticipations? Continuous updates can lead to a moving target problem in MARL.

Author response:
**Handling non-stationarity**. Our method addresses this challenge by incorporating a mechanism designed to predict and adapt to changes in agents' policies over time. This approach is proactive in nature, aiming to stabilize the learning process and enhance coordination among agents by reducing the unpredictability that arises from these continuous updates. By anticipating potential policy shifts of other agents, each agent can adjust its strategy more effectively, leading to a more stable learning environment. However, fully addressing the complexities of non-stationarity in MARL remains an open challenge requiring further research. This will be a focus as we continue developing the anticipation sharing approach.

Reviewer comment (Theory):
While the paper addresses the challenges of policy discoordination and privacy concerns, how does the proposed method handle issues related to credit assignment, especially when agents have conflicting goals or when their contributions to the global reward are imbalanced?

Author response:
**Credit assignment in conflicting goals**. Credit assignment in multi-agent reinforcement learning typically arises in settings where a singular team reward is shared among agents. Our research, however, focuses on decentralized learning environments where such a team reward is not available, and each agent only has knowledge of its individual reward. In these scenarios, rewards may often be in conflict, meaning an increase in one agent's reward could potentially lead to a decrease in another's.

In our setting, the challenge is not so much about assigning credit from a collective reward, but about maximizing individual returns in a way that also contributes to the overall system's effectiveness. This is where our anticipation sharing framework plays a crucial role. It aims to improve coordination among agents by enabling them to balance their own objective and other agents’ objectives via constraining the policy with the anticipations from others, since the anticipations from others contribute to maximizing other agents’ objectives. 

While our approach focuses on enhancing coordination to potentially reach a maximal total return, it does not directly solve the credit assignment problem typical in environments with shared team rewards. Instead, it addresses the unique challenges of decentralized learning where independent reward maximization might lead to a Nash Equilibrium, rather than a collective optimum. This distinction is critical to understanding the context and objectives of our research in decentralized multi-agent systems. 

We have edited the manuscript to make these points clearer, such as the second paragraph of the Introduction, and the two paragraphs after Eq.11 in Section 4.2.

Reviewer comment (Theory):
It is unfortunately quite difficult to understand the actual mathematical underpinnings of the paper. The first part presents a series of "theorems". However, their proofs do not appear convincing.

Author response:
**Mathematical underpinnings and proofs.** Thank you for your careful reading of our paper and for raising these important points. We appreciate you taking the time to provide such thoughtful feedback.

Regarding Section 4.1, we want to clarify that the key inequalities were already defined and proved in the original Appendix. We have aimed to improve the presentation and clarity of the corresponding theorems in the main text, to better highlight their role in building our proposed solution and strengthen the logical flow. The detailed proofs remain in the Appendix. Please let us know if the revised framing of the theorems makes their implications and connectivity clearer, given that the foundational inequalities were established in the first submission.

We also appreciate you noting the need for more background on the surrogate optimization objective introduced in Section 4.2. In our revised manuscript, we have worked to motivate and explain this component more completely, in order to elucidate its rationale and benefits.

Reviewer comment (Theory):
The authors do not prove the inequalities reported in Section 4.1 (more specifically, Theorem 1, Theorem 2 and Theorem 3).

Author response:
**Mathematical underpinnings and proofs.** Thank you for your careful reading of our paper and for raising these important points. We appreciate you taking the time to provide such thoughtful feedback.

Regarding Section 4.1, we want to clarify that the key inequalities were already defined and proved in the original Appendix. We have aimed to improve the presentation and clarity of the corresponding theorems in the main text, to better highlight their role in building our proposed solution and strengthen the logical flow. The detailed proofs remain in the Appendix. Please let us know if the revised framing of the theorems makes their implications and connectivity clearer, given that the foundational inequalities were established in the first submission.

Reviewer comment (Theory):
The authors introduce a surrogate optimization objective (in Section 4.3) without justifying its introduction. It is very difficult for the reader to understand why it is a good idea to use such a function, which is quite complex computationally.

Author response:
**Surrogate Objectives and Constraints in Section 4.2 and 4.3.** Thank you for your suggestions regarding the presentation of the surrogate objective and the constraints in Section 4.3. Upon reflection, we agree the original drafting could be improved for clarity and accessibility. 

In Section 4.2, we leverage the lower bound given by Theorem 3 to maximize the orginal collective return, since the collective return is intractable without a global view. By omitting irrelevant terms, we have a surrogate objective Eq. 10. Further, to make it feasible in decentralized MARL, we reformulate it from each agent's limited perspective. Remarkably, we can distill the relevant components into a local objective and constraints for each individual agent, which leads to Eq.11. To practically solve Eq. 11, in Section 4.3, we propose an algorithm, involving specific steps, each targeting different aspects of the optimization challenge. In the revised manuscript, we have rewritten Sections 4.2 and 4.3 to explain the surrogate objective and the constraints in a more intuitive, step-by-step manner. Please let us know if the revised explanation provides adequate insight into how the surrogate objective and the constraints emerge from the optimization problem. We welcome any additional feedback on making this section clear and complete.

Reviewer comment (Theory):
The reviewer struggles to understand the introduction of the constraints in Section 4.3. This might be due to a presentation issue, but the derivations of the constraints from the optimization problem listed in Section 4.2 are not straightforward.

Author response:
**Surrogate Objectives and Constraints in Section 4.2 and 4.3.** Thank you for your suggestions regarding the presentation of the surrogate objective and the constraints in Section 4.3. Upon reflection, we agree the original drafting could be improved for clarity and accessibility. 

In Section 4.2, we leverage the lower bound given by Theorem 3 to maximize the orginal collective return, since the collective return is intractable without a global view. By omitting irrelevant terms, we have a surrogate objective Eq. 10. Further, to make it feasible in decentralized MARL, we reformulate it from each agent's limited perspective. Remarkably, we can distill the relevant components into a local objective and constraints for each individual agent, which leads to Eq.11. To practically solve Eq. 11, in Section 4.3, we propose an algorithm, involving specific steps, each targeting different aspects of the optimization challenge. In the revised manuscript, we have rewritten Sections 4.2 and 4.3 to explain the surrogate objective and the constraints in a more intuitive, step-by-step manner. Please let us know if the revised explanation provides adequate insight into how the surrogate objective and the constraints emerge from the optimization problem. We welcome any additional feedback on making this section clear and complete.

Reviewer comment (Evaluation):
The results reported in the evaluation are difficult to explain. Why do some of the methods perform so badly?

Author response:
**Evaluation results**: In our revised manuscript (Section 5.2), we delve into a more detailed analysis to explain the varying performance levels of different methods, including our own. 

However, it is important to note that the primary aim of our study is not to outperform the baseline algorithms in every scenario but to provide a viable alternative in more complex and heterogeneous environments. Our approach is especially relevant in settings where agents cannot exchange values or rewards due to privacy constraints. In such scenarios, our method offers a means of coordination without compromising individual agent privacy.

Additionally, our method is underpinned by a theoretical framework designed for stability across a variety of tasks. This theoretical basis may contribute to more consistent performance, as opposed to methods like VS, which might excel in certain tasks but struggle in others.

Reviewer comment (Evaluation):
For the Pred. task the performance appears very similar to value sharing.

Author response:
**Performance in the Predation task**:  Thank you for raising this insightful point regarding the similar performance observed in the Predation (Pred.) task compared to value sharing (VS). We acknowledge that this outcome warrants further investigation in our revisions.

Our initial hypothesis is that the characteristics of the Pred. task, where agents are homogeneous with a common target, might inherently lend themselves to approaches like VS, thereby explaining the similarity in performance.

In the revised manuscript (Section 5.2) , we have included a more comprehensive analysis of the performance in the Pred. task, exploring why our approach shows similar effectiveness to VS in this particular scenario and highlighting the distinct advantages our method offers in diverse and privacy-sensitive environments.

Reviewer comment (Writing):
What is the difference between value estimation of the other agents vs anticipation?

Author response:
**Difference between value estimation and anticipation**: Thank you for giving us the opportunity to clarify the important difference between value estimation and anticipation. 

Calculating the value function of other agents involves estimating the expected returns or outcomes of their actions, typically based on observed behaviors and strategies. The focus is on understanding what outcomes other agents aim to achieve.

In contrast, as defined in our work, anticipation goes beyond mere outcome estimation. The anticipation $\pi^{ij}$ is solved to optimize agent i's objective (Eq. 11). $\pi^{ij}$ is then sent to agent j, so agent j can balance agent i's objective when maximizing its own. Additionally, agent i's policy $\pi^{ii}$ is adjusted based on the discrepancy with $\pi^{ji}$, agent j's anticipation of agent i. This constraint further enforces agreement between local learning and others' anticipations. 

Therefore, the anticipations include the information about individual objectives implicitly. By exchanging the anticipations, individual agents can balance others' objectives and thus the collective performance when optimizing its own objective. We highlighted this point in our revisded manuscript, see the two paragraphs after Eq.11 in Section 4.2.

We hope that the revised manuscript does a better job at highlighting this conceptual nuance.

Reviewer comment (Theory):
What is the advantage in using the surrogate optimization objective in Section 4.3?

Author response:
**Advantage of using the surrogate optimization objective**: In the original objective (Eq.1), the optimization target is the true joint policy, which is infeasible in decentralized settings where each agent does not have access to other agents' true policies and cannot dictate to others how to update their policies.

The surrogate objective stems from the lower bound (Eq.9) of the original objective, which can be decomposed into individual objectives that optimize the anticipated joint policy consisting of the agent’s own policy and policy anticipations to other agents. The anticipated joint policy is solved by each agent, which is not the true joint policy composed of the true individual policy of each agent, as defined in Definition 1.

The surrogate optimization objective introduced in Section 4.2 offers a more tractable solution to the complex problem of multi-agent coordination under uncertainty. It simplifies the optimization landscape, making it feasible to find solutions that, while not globally optimal, are practically effective. This objective also helps in balancing the computational complexity with the benefits of improved coordination, which is particularly valuable in environments where exact solutions are computationally prohibitive.

We believe that the revised manuscript (Section 4.2) is significantly clearer about these key aspects.

Reviewer comment (Theory):
Can you please motivate/derive the constraints in Section 4.4?

Author response:
**Motivation and derivation of constraints in Section 4.2**:  Thank you for allowing us to clarify the origins of the constraints presented in Section 4.2 (Eq.10 and Eq.11). We appreciate you highlighting this component for further discussion.

You raise an excellent point - the constraints are directly motivated by the lower bound in Section 4.1, Eq. 9. Specifically, the constraints in Eq. 10 derive directly from Eq. 9, with the focus on optimizing $\pi_{new}$ while fixing $\pi_{old}$. They ensure the surrogate optimization aligns with multi-agent practicalities.

Further, Eq. 11 stems from separating out the agents in Eq. 10's constraints and anticipations. This separation is key for decentralized optimization where each agent acts based on its policy and anticipations of others.

We have expanded the manuscript to provide a detailed, step-by-step explanation of how these constraints emerge from the underlying optimization formulation. We aim to elucidate their motivations and relevance to the MARL setting.

Reviewer comment (Evaluation):
Can you please explain the results in Figure 1?

Author response:
**Explanation of Results in Figure 1**: Figure 1 illustrates the performance of our method in various simulated environments. We have enhanced this Figure's explanation by detailing the specific characteristics of each environment and how they interact with our method. This includes a discussion on the factors contributing to the observed performance trends, providing a clearer understanding of the effectiveness of our approach in different settings. Please see Section 5.2 in the revised manuscript.

Reviewer comment (Evaluation):
Can you please explain why we observe differences in terms of performance for the various tasks? Why does your method perform better for certain environments? If the difference is low, it seems that value sharing would be a preferable solution given its lower complexity. What is the trade-off a practitioner should consider in this situation?

Author response:
**Performance for certain environments:** Our approach is especially relevant in settings where agents cannot exchange values or rewards due to privacy constraints. In such scenarios, our method offers a means of coordination without compromising individual agent privacy. Therefore, the aim of our study is not to outperform the baseline algorithms in every scenario but to provide a viable alternative in such challenging settings. Additionally, our method is underpinned by a theoretical framework designed for stability across a variety of tasks. This theoretical basis may contribute to more consistent performance, as opposed to methods like VS, which might excel in certain tasks but struggle in others.

Reviewer comment (Theory):
Can you please explain this sentence: "Theoretically, we established that the difference between agents’ actual action distributions and the anticipations from others bounds the difference between individual and collective objectives."? I am not sure that is the actual theoretical result that you showed in the first part of the paper. Alternatively, this might be considered as an expected fact, which does not need a proof?

Author response:
**Clarification of theoretical statement**: You are correct that this statement (”theoretically, we established that the difference between agents”) stems from our theoretical framework, particularly Theorem 3 and Eq. 9 as described in the manuscript. The LHS of Eq.9 represents the original multi-agent collective objective. The first term on the RHS is the summation of individual objectives from Definition 2 and Eq.6.

Critically, the last term in Eq.9 encapsulates the difference between actual action distributions and anticipations from others. Therefore, as you noted, Eq.9 establishes that this difference bounds the discrepancy between individual and collective goals. This highlights the significance of anticipation sharing for aligning behaviors with group objectives.

Per your helpful feedback, we have significantly revised the presentation of the methodology in the new version of the manuscript to address this point more clearly. Please let us know if the updated explanation provides adequate clarification on how this key theoretical result relates agents' anticipations to collective outcomes. We appreciate you highlighting this opportunity to strengthen the clarity of our framework.