Paper: Detecting Influence Structures in Multi-Agent Reinforcement Learning (ICLR-2024)

Reviewer comment (Evaluation):
The evaluation of the proposed method is not enough. Figure 1 and Figure 2 only show the TIM error and mean TIM approximations in two environments respectively. What about the overall performance improvement of the proposed method?

Author response:
Our evaluation, as depicted in Figures 1 and 2, focuses on demonstrating the practical effectiveness of the algorithms for measuring TIM and SIM. While we acknowledge that a direct comparison to baselines was not provided, this is primarily because there are no existing baselines directly comparable to our approach.
The empirical results validate the practical functionality of our proposed algorithms. The small setting showcases the accuracy of our measurements compared to true values, and the coin game setting demonstrates that TIM and SIM can be reliably estimated in complex scenarios with meaningful semantic implications.

Reviewer comment (Experiments):
This paper did not show any empirical comparison between the proposed method and baselines.

Author response:
Our evaluation, as depicted in Figures 1 and 2, focuses on demonstrating the practical effectiveness of the algorithms for measuring TIM and SIM. While we acknowledge that a direct comparison to baselines was not provided, this is primarily because there are no existing baselines directly comparable to our approach.
The empirical results validate the practical functionality of our proposed algorithms. The small setting showcases the accuracy of our measurements compared to true values, and the coin game setting demonstrates that TIM and SIM can be reliably estimated in complex scenarios with meaningful semantic implications.

Reviewer comment (Reproducibility):
How do TIM and SIM combine with MARL methods?

Author response:
TIM and SIM can be effectively combined with MARL methods in two significant ways. First, TIM and SIM serve as descriptive tools to assess interdependencies among agents in a given system. These measurements are based on observed actions, states, and rewards, enabling their combination with any MARL algorithm. In the coin game experiment, for instance, agents were trained using independent Proximal Policy Optimization (PPO) policies while we simultaneously measured their interdependencies using TIM and SIM. However, we could also have used a cooperative multi-agent learning algorithm such as QMIX [1] to provide insights into which agents exert the most influence under this learning algorithm. TIM and SIM enable this without prior knowledge of the environment.
Secondly, beyond descriptive analysis, TIM and SIM can be employed to enhance the learning of joint policies. For example, SIM can be used as an intrinsic reward signal to prioritize states with higher impact, which are strategically more relevant [2]. Similarly, TIM can facilitate the construction of an approximate coordination graph [3]. However, our paper focuses on the first application, leaving the exploration of these applications for future work.

Reviewer comment (Writing):
What is the main difference between TIM and SIM?

Author response:
TIM and SIM differ in their granularity of capturing interdependencies. SIM provides a state-specific impact analysis. It quantifies the expected impact one agent has on another in a particular state. For example, in different scenarios within the coin game, SIM would reflect varying impacts based on the proximity of agents to coins, capturing nuances in influence that change with the state context.
TIM, on the other hand, represents the expected value of SIM over the stationary distribution of states under the current joint policy. It provides a more holistic view of influence across all states, rather than the state-specific analysis offered by SIM.

Reviewer comment (Theory):
I view the proposed measurement as an alternative way of representing the reward, which changes the problem structure of MMDP. In order to argue for correctness, one would have to prove that the solution to the new problem is exactly the solution to the original problem. How would you relate the two problems?

Author response:
Thank you for your review and the points you have raised. We appreciate the opportunity to clarify aspects of our work and to address your concerns regarding the utility and theoretical foundation of our proposed measurements.

It is important to clarify that influence measurement functions in our work are descriptive inherent properties of a multi-agent Markov Decision Process. They are not intended to serve as objectives for the agents within the system. Therefore, our approach does not alter the fundamental problem. Rather, it provides an additional lens through which the inter-agent dynamics can be understood and quantified. The solutions derived from our measurements are congruent with the solutions to the original problem, offering insights into the influence dynamics without changing the core objective, such as finding a Nash equilibrium.
While our focus is not on altering the objective function itself, we acknowledge that other works have successfully used such measurements to augment the objective function, for instance, in guiding exploration or enhancing cooperation among agents [1, 2]. This demonstrates the potential utility of influence measurements beyond their descriptive role.
Our primary contribution lies in defining a general theoretical framework for describing influence measurement functions. We introduce TIM and SIM as instances within the framework, along with algorithms that guarantee convergence and provide error bounds for these measurements. Furthermore, we demonstrate the application of these theoretical results in complex practical settings, illustrating the utility and applicability of our approach.
We hope these clarifications address your concerns.

### Bibliography:

[1]: Tonghan Wang, Jianhao Wang, Yi Wu, Chongjie Zhang, Influence-based multi-agent exploration, ICLR2020

[2]: Tonghan Wang, Liang Zeng, Weijun Dong, Qianlan Yang, Yang Yu, Chongjie Zhang, Context-Aware Sparse Deep Coordination Graphs, ICLR2022

Reviewer comment (Writing):
Some technical details (e.g., stochastic iteration approximation) are introduced, but not clearly explained.

Author response:
Although there is no proof in the main text, we estimate the background information provided in Section 3.2 to be essential. It helps the reader understand the rationale and design principles behind our proposed measurement algorithms.

Reviewer comment (Reproducibility):
In section 6.1, “using the format of Zhang et al,…” — what exactly is the format? It might be better to provide more details of the environment.

Author response:
We acknowledge that our reference to Zhang et al. was misleading, implying a need to consult their paper for details. That is not the case, as we provide a detailed description of the environment in the appendix. We clarified the reference to this addition in the main text, resolving ambiguity.

Reviewer comment (Writing):
Some notations are a little confusing, e.g., a^{-i} and {a^{bar}}^{i}. They seem to refer to different things, but look quite similar at first glance.

Author response:
Agreed, some of our notations were confusing. We have revised these notations for clarity and consistency.

Reviewer comment (Experiments):
Why emphasize averaged reward settings given that the proposed influence measurement can be applied to discounted reward settings as well?

Author response:
Our decision to emphasize average reward settings stems from a noticeable gap in existing research. To our knowledge, there are currently no interdependency measures that align with our definition of an influence measurement function specifically for average reward settings. This contrasts with the discounted reward setting, where measures like the Value of Interaction [4] already exist. Our work aims to fill this gap by introducing new metrics tailored to the average reward setting, along with corresponding theoretical guarantees.
Additionally, while the underlying definition of influence measurement functions is independent of the setting, TIM and SIM are not. They are specifically designed for the average reward setting. Although there are natural extensions of these measures to the discounted reward setting, the theoretical foundations and proofs of TIM and SIM are intrinsically linked to properties unique to the average reward setting. This setting-specific focus allows for a more rigorous and targeted analysis, ensuring that the theoretical guarantees we provide are robust and applicable within this context.

Reviewer comment (Experiments):
In section 6.2, “deviating from the original game, we employ one-sided penalty system…” — why need to deviate from the original game?

Author response:
In our study, we chose to deviate from the original symmetric game format by implementing a one-sided penalty system in the coin game. The primary reason for this modification was to expose and highlight the differences in interdependencies among the agents. In a symmetric game, one might expect the influence patterns to also be symmetric. However, our aim was to explore and demonstrate more complex influence dynamics. By introducing asymmetry through one-sided penalties, we were able to provide a clearer semantic explanation of these complex dynamics. This decision and its rationale have been more explicitly articulated in the revised paper for greater clarity.

Reviewer comment (Writing):
What is L_add?

Author response:
In our study, L_add is used to control the number of additional influences in our randomly generated environment. This parameter allows us to precisely adjust how much influence each agent has over others. By default, we assume that each agent can influence its own expected return. The value of L_add quantifies the extent of additional influences that go beyond this basic assumption. For example, L_add=0 indicates a scenario where agents do not exert any influence on each other. Conversely, L_add=20 in a system of five agents signifies a situation where every agent has the capacity to influence every other agent.

Reviewer comment (Theory):
It seems like definition 4.1 is related to the potential function in a Markov potential game, and I would ask the authors if they can discuss any connections between the matrix entries for one state-action and a particular potential function.

Author response:
Thank you for your valuable feedback and insightful observations regarding our paper. We appreciate the opportunity to address your questions and clarify certain aspects of our work.
### Connections to Potential Functions
Your observation regarding the connection between our proposed measurement functions and potentials is fitting. We agree that there is a conceptual relationship, but also key differences that distinguish our approach.
Our state- or total-influence measurement functions identify the existence of a potential in agent i’s state-action function due to agent j’s behavior. Specifically, if agent j can influence agent i, this influence manifests as a non-zero potential in our framework.
The Markov potential function, in contrast, is an agent-independent mapping that captures state-value potentials for unilateral deviations in policy by any agent. This means that if agent i changes its behavior, the change in its state-value is indicated by the potential function. However, this function does not account for the impact of agent j’s policy changes on agent i’s state-value. Therefore, it does not adequately represent the interdependencies between agents.
The Markov potential function encapsulates the agents’ incentives, and analyzing the local optima of this function can lead to finding Nash Equilibria. This is a strong property and not universally applicable. Our influence measurement functions describe potentials in strategic deviations among all agents, offering a broader perspective without limiting the strategic complexity inherent in the Markov game. Therefore, the Total Impact Measurement (TIM) and State Impact Measurement (SIM) proposed in Section 5 exist in every Markov game within the average reward setting, irrespective of the existence of a Markov potential function. Furthermore, even in cases where a Markov potential function exists, it may not sufficiently express the complex interdependencies among agents.

Reviewer comment (Theory):
The formulation of the problem itself as a multi-agent MDP is indeed a Markov game.

Author response:
We agree that the common term is Markov or Stochastic game. We have added a comment in the revised manuscript to highlight this connection. We decided to use the terminology provided by Zhang et al. [1] for two reasons. First, we wanted to emphasize the underlying MDP properties as our results and concepts rely heavily on those. Second, it decreases ambiguity regarding the specifics of the reward setting so that readers do not assume that the theoretical results hold for other settings that usually fall under the Markov game term.
We hope that these clarifications and modifications address your concerns. Thank you once again for your constructive feedback.

Bibliography:
[1]: Zhang et al., Fully decentralized Multi-Agent Reinforcement Learning with Networked Agents, ICML2018