Original: I NFLUENCE -B ASED R EINFORCEMENT L EARNING FOR I NTRINSICALLY -M OTIVATED A GENTS
Revised: I NFLUENCE -B ASED R EINFORCEMENT L EARNING FOR I NTRINSICALLY -M OTIVATED A GENTS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Discovering successful coordinated behaviors is a central challenge in Multi-Agent
Revised: Discovering successful coordinated behaviors is a central challenge in Multi-Agent

Original: Reinforcement Learning (MARL) since it requires exploring a joint action space that grows exponentially with the number of agents.
Revised: Reinforcement Learning (MARL) since it requires exploring a joint action space that grows exponentially with the number of agents.

Original: In this paper, we propose a mechanism for achieving sufﬁcient exploration and coordination in a team of agents.
Revised: In this paper, we propose a mechanism for achieving sufﬁcient exploration and coordination in a team of agents.

Original: Speciﬁcally, agents are rewarded for contributing to a more diversiﬁed team behavior by employing proper intrinsic motivation functions.
Revised: Speciﬁcally, agents are rewarded for contributing to a more diversiﬁed team behavior by employing proper intrinsic motivation functions.

Original: To learn meaningful coordination protocols, we structure agents’ interactions by introducing a novel framework, where at each timestep, an agent simulates counterfactual rollouts of its policy and, through a sequence of computations, assesses the gap between other agents’ current behaviors and their targets.
Revised: To learn meaningful coordination protocols, we structure agents’ interactions by introducing a novel framework, where at each timestep, an agent simulates counterfactual rollouts of its policy and, through a sequence of computations, assesses the gap between other agents’ current behaviors and their targets.

Original: Actions that minimize the gap are considered highly inﬂuential and are rewarded.
Revised: Actions that minimize the gap are considered highly inﬂuential and are rewarded.

Original: We evaluate our approach on a set of challenging tasks with sparse rewards and partial observability that require learning complex cooperative strategies under a proper exploration scheme, such as the StarCraft Multi-Agent Challenge.
Revised: We evaluate our approach on a set of challenging tasks with sparse rewards and partial observability that require learning complex cooperative strategies under a proper exploration scheme, such as the StarCraft Multi-Agent Challenge.

Original: Our methods show signiﬁcantly improved performances over different baselines across all tasks.
Revised: Our methods show signiﬁcantly improved performances over different baselines across all tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep Reinforcement Learning (DRL) has been applied to solve various challenging problems, where an agent typically learns to maximize the expected sum of extrinsic rewards gathered as a result of its actions performed in the environment (Sutton et al., 1998).
Revised: Deep Reinforcement Learning (DRL) has been applied to solve various challenging problems, where an agent typically learns to maximize the expected sum of extrinsic rewards gathered as a result of its actions performed in the environment (Sutton et al., 1998).

Original: Multi-Agent Reinforcement Learning (MARL) refers to the task of training a set of agents to maximize collective and/or individual rewards, while existing in the same environment and interacting with each other.
Revised: Multi-Agent Reinforcement Learning (MARL) refers to the task of training a set of agents to maximize collective and/or individual rewards, while existing in the same environment and interacting with each other.

Original: Recent works have shown that agents with coordinated behaviors learn remarkably faster (Roy et al., 2019) since coordination helps the discovery of effective policies in cooperative tasks.
Revised: Recent works have shown that agents with coordinated behaviors learn remarkably faster (Roy et al., 2019) since coordination helps the discovery of effective policies in cooperative tasks.

Original: Nevertheless, achieving coordination among agents still remains a central challenge in MARL (Jaques et al., 2019).
Revised: Nevertheless, achieving coordination among agents still remains a central challenge in MARL (Jaques et al., 2019).

Original: Prominent works often resort to a popular learning paradigm called Centralized Training with Decentralized Execution (CTDE) (Lowe et al., 2017; Foerster et al., 2018), where each agent is evaluated using a centralized critic and has access to extra information about the policies of other learning agents during training.
Revised: Prominent works often resort to a popular learning paradigm called Centralized Training with Decentralized Execution (CTDE) (Lowe et al., 2017; Foerster et al., 2018), where each agent is evaluated using a centralized critic and has access to extra information about the policies of other learning agents during training.

Original: At the time of execution, policies’ actions are restricted to local information only (i.e. their own observations).
Revised: At the time of execution, policies’ actions are restricted to local information only (i.e. their own observations).

Revised: Our work is primarily motivated by the following natural questions:

Revised: In multi-agent settings, how can we quantify the effect of an agent (or player) on other teammates’ behaviors (particularly on their performance)?

Revised: And to what degree exploiting this quantity can lead to coordinated behaviors and consequently better overall performances?

Original: To that end, we propose a novel approach that aims at promoting coordination for cooperative tasks by augmenting CTDE MARL main returnmaximization objective with an additional multi-agent objective that acts as a policy regularizer; we refer to the latter objective as the influence function .
Revised: To that end, we propose a novel approach that aims at promoting coordination for cooperative tasks by augmenting CTDE MARL main return-maximization objective with an additional multi-agent objective that acts as a policy regularizer; we refer to the latter objective as the influence function .

Original: To build intuition, a chosen agent, which we call the “inﬂuencer”, assesses the progress that other agents are making given its current policy and consequently learns behaviors that will result in an improved performance of its teammates.
Revised: To build intuition, a chosen agent, which we call the “inﬂuencer”, assesses the progress that other agents are making given its current policy and consequently learns behaviors that will result in an improved performance of its teammates.

Original: Concretely, we formulate the inﬂuence of an inﬂuencer π as an estimation of the dissimilarity between other agents’ behaviors and their targets given the current behavior of π .
Revised: Concretely, we formulate the inﬂuence of an inﬂuencer π as an estimation of the dissimilarity between other agents’ behaviors and their targets given the current behavior of π .

Original: The inﬂuencer is encouraged to learn behaviors that are expected to minimize that dissimilarity.
Revised: The inﬂuencer is encouraged to learn behaviors that are expected to minimize that dissimilarity.

Original: We also propose two approaches to estimate the inﬂuence and empirically show that they yield unbiased estimates of the true value.
Revised: We also propose two approaches to estimate the inﬂuence and empirically show that they yield unbiased estimates of the true value.

Original: To that end, agents acting upon the proposed coordination paradigm learn to efﬁciently exploit the observed joint action space using available information.
Revised: Agents acting upon the proposed coordination paradigm learn to efﬁciently exploit the observed joint action space using available information.

Original: However, and since the joint space grows exponentially with the number of agents, it is highly unlikely that agents will have access to sufﬁcient information to learn optimal behaviors to solve the task at hand; this problem arises in many scenarios such as sparse-reward environments, thus a proper exploration scheme is often required.
Revised: However, and since the joint space grows exponentially with the number of agents, it is highly unlikely that agents will have access to sufﬁcient information to learn optimal behaviors to solve the task at hand; this problem arises in many scenarios such as sparse-reward environments, thus a proper exploration scheme is often required.

Original: However, many existing multi-agent deep reinforcement learning algorithms still use mostly noise-based techniques (Liu et al., 2021; Rashid et al., 2018;
Revised: However, many existing multi-agent deep reinforcement learning algorithms still use mostly noise-based techniques (Liu et al., 2021; Rashid et al., 2018;

Original: Yang et al., 2018).
Revised: Yang et al., 2018).

Original: Moreover, independent exploration proved to be inefﬁcient in cooperative settings (Roy et al., 2019).
Revised: Moreover, independent exploration proved to be inefﬁcient in cooperative settings (Roy et al., 2019).

Original: Recently, this challenge was addressed through Intrinsic Motivation (IM) (Jaques et al., 2019; Du et al., 2019;
Revised: Recently, this challenge was addressed through Intrinsic Motivation (IM) (Jaques et al., 2019; Du et al., 2019;

Original: Zhou et al., 2020b).
Revised: Zhou et al., 2020b).

Original: Many approaches employ IM to encourage exploration of state-space (Han et al., 2020; Burda et al., 2019) or state-action space (Fayad & Ibrahim, 2021) by identifying novel conﬁgurations and rewarding an agent for visiting them.
Revised: Many approaches employ IM to encourage exploration of state-space (Han et al., 2020; Burda et al., 2019) or state-action space (Fayad & Ibrahim, 2021) by identifying novel conﬁgurations and rewarding an agent for visiting them.

Original: We provide an extension of these ideas into multi-agent settings and further build connection between reward shaping and coordinated behavior learning, where we choose an agent to act as an inﬂuencer (i.e. regularize its standard objective using the inﬂuence function) while other agents learn to maximize the expected sum of both extrinsic and intrinsic rewards.
Revised: We provide an extension of these ideas into multi-agent settings and further build connection between reward shaping and coordinated behavior learning, where we choose an agent to act as an inﬂuencer (i.e. regularize its standard objective using the inﬂuence function) while other agents learn to maximize the expected sum of both extrinsic and intrinsic rewards.

Original: To sum up, our main contributions are threefold: 1) developing an inﬂuence function to promote learning coordinated behaviors and improve team performance; 2) extending exploration via random network distillation to multi-agent settings by crafting a "novelty" function that rewards underexplored behaviors; 3) formulating a novel intrinsic incentive to promote learning diverse team behaviors to help uncover complex behaviors in a collaborative way.
Revised: To sum up, our main contributions are threefold: 1) developing an inﬂuence function to promote learning coordinated behaviors and improve team performance; 2) extending exploration via random network distillation to multi-agent settings by crafting a "novelty" function that rewards underexplored behaviors; 3) formulating a novel intrinsic incentive to promote learning diverse team behaviors to help uncover complex behaviors in a collaborative way.

Original: We demonstrate the effectiveness of our methods on a comprehensive set of challenging tasks which include, but not limited to, the StarCraft Multi-Agent Challenges (SMAC) (Samvelyan et al., 2019) and the Multi-Agent Particle Environments (MAPE) (Mordatch & Abbeel, 2018; Lowe et al., 2017).
Revised: We demonstrate the effectiveness of our methods on a comprehensive set of challenging tasks which include, but not limited to, the StarCraft Multi-Agent Challenges (SMAC) (Samvelyan et al., 2019) and the Multi-Agent Particle Environments (MAPE) (Mordatch & Abbeel, 2018; Lowe et al., 2017).

Original: Empirical results show a signiﬁcant improvement over a wide variety of state-of-the-art MARL approaches.
Revised: Empirical results show a signiﬁcant improvement over a wide variety of state-of-the-art MARL approaches.

Original: We also conduct insightful ablation studies to understand the relative importance of each component of the approach individually.
Revised: We also conduct insightful ablation studies to understand the relative importance of each component of the approach individually.

Original:  B ACKGROUND 2. M ARKOV GAMES
Revised:  B ACKGROUND 2. M ARKOV GAMES

Original: Also called Stochastic games (Littman, 1994), are the foundation for much of the research in multiagent reinforcement learning. Markov games  are a superset of Markov decision process (MDPs) and matrix games, including both multiple agents and multiple states. 
Revised: Markov games (Littman, 1994; Agarwal et al., 2020; Daskalakis et al., 2021) are a superset of Markov decision process (MDPs) and matrix games, including both multiple agents and multiple states. Formally, we consider a setting with N > 2 agents ([ N ] := { 1 , 2 . . . , N } ) who repeatedly select actions in a shared MDP.

Original: Formally, a Markov game consists of  a tuple (cid:104) N, S, A, T, R (cid:105) where: N is a ﬁnite set of agents | N | = n ≥ 2 ; S is a set of states, where the initial states are determined by a distribution ρ : S → [0 , 1] ; A = (cid:81) nk =
Revised: The goal of each agent is to maximize their respective action value function. Formally, an MDP is deﬁned as a tuple G = ( S, N, { A i , r i } i ∈ [ N ] , T, γ, ρ ) , where:

Original: A k is the set of joint actions; and T : S ×
Revised: • S is the ﬁnite states space, where the initial states are determined by a distribution ρ : S → [0 , 1] . Since we consider partially observable testing environments, we denote by s i the information observed by agent i about the global state s ∈ S . A i is a ﬁnite action space for agent i with generic element a i ∈

Original: A × S → [0 , 1] is the transition probability function.
Revised: A i . Using common conventions, we will write A = Π j ∈ [ N ] A j and A − i =

Original: In a Markov game, each agent is independently choosing actions and receiving rewards.
Revised: Π j (cid:54) = i A j to denote the joint action spaces of all agents and of all agents other than i with generic elements a = ( a i ) i ∈

Revised: [ N ] and a − i =

Revised: ( a j ) i (cid:54) =

Revised: j ∈ [ N ] , respectively.

Revised: According to this notation, we have that a = ( a i , a − i ) ∀ i .

Revised: • r

Revised: i : S ×

Revised: A → R is the individual reward function of agent i , i.e., r i ( s, a i , a − i ) is the instantaneous reward of agent i when agent i takes action a i and all other agents take actions a − i at state s ∈ S .

Revised: • T is the transition probability function, for which T ( s (cid:48) | s, a ) is the probability of transitioning from s to s (cid:48) when a ∈ A is the action proﬁle chosen by the agents.

Revised: • γ is a discount factor for future rewards of the MDP, shared by all agents.

Revised: In a Markov game, each agent is independently choosing actions and receiving rewards.

Original: Conventionally, an agent k aims to maximize its own total expected return R k
Revised: Conventionally, an agent k aims to maximize its own total expected return R k

Original: =
Revised: =

Original: (cid:80) Tt =

Revised: (cid:80) Tt =

Original: γ t r ( k ) t whereγ is a discount factor and T is the time horizon.
Revised: γ t r k,t where T is the time horizon.

Original: 2. M ULTI -A GENT D EEP D ETERMINISTIC P OLICY G RADIENT
Revised: 2. M ULTI -A GENT D EEP D ETERMINISTIC P OLICY G RADIENT

Original: MADDPG (Lowe et al., 2017) is a multi-agent extension of the DDPG algorithm (Lillicrap et al., 2015).
Revised: MADDPG (Lowe et al., 2017) is a multi-agent extension of the DDPG algorithm (Lillicrap et al., 2015).

Original: It adapts the CTDE paradigm, where each agent i possesses its own deterministic policy µ ( i ) for action selection and critic Q ( i ) for state-action value estimation, respectively parameterized by θ ( i ) and φ ( i ) .
Revised: It adapts the CTDE paradigm, where each agent i possesses its own deterministic policy µ ( i ) for action selection and critic Q ( i ) for state-action value estimation, respectively parameterized by θ ( i ) and φ ( i ) .

Original: All parametric models are trained off-policy from previous transitions ζ t :=
Revised: All parametric models are trained off-policy from previous transitions ζ t :=

Original: ( o t , a t , r t , o t +1 ) uniformly sampled from a replay buffer D .
Revised: ( s t , a t , r t , s t +1 ) uniformly sampled from a replay buffer D .

Original: Note that o t :=

Original: [ o 1 t , ..., o Nt ] is the joint observation vector and a t :=

Original: [ a 1 t , ..., a Nt ] is the joint action vector, obtained by concatenating the individual observation vectors o ( i ) t and action vectors a ( i ) t of all N agents.

Original: Each centralized critic is trained to estimate the expected return for a particular agent i from the Q-learning loss:
Revised: Each centralized critic is trained to estimate the expected return for a particular agent i from the Q-learning loss:

Original: [Equation]
Revised: [Equation]

Original: Each policy is updated to maximize the expected discounted return of the corresponding agent i :
Revised: Each policy is updated to maximize the expected discounted return of the corresponding agent i :

Original: [Equation]
Revised: [Equation]

Original: Notice that while optimizing an agent’s policy, all agents’ observation-action pairs are taken into consideration.
Revised: Notice that while optimizing an agent’s policy, all agents’ observation-action pairs are taken into consideration.

Original: By that, the value functions of all agents are trained in a centralized, stationary environment, despite happening in a multi-agent setting.
Revised: By that, the value functions of all agents are trained in a centralized, stationary environment, despite happening in a multi-agent setting.

Original: Moreover, this procedure allows for the learning of coordinated strategies, yet needs to be augmented with efﬁcient exploration methods that reward novel action conﬁgurations which may lead to the discovery of higher-return behaviors.
Revised: Moreover, this procedure allows for the learning of coordinated strategies, yet needs to be augmented with efﬁcient exploration methods that reward novel action conﬁgurations which may lead to the discovery of higher-return behaviors.

Original:  M ETHODS 3. B ASIC I NFLUENCE
Revised:  M ETHODS 3. B ASIC I NFLUENCE

Original: Intuitively, one can deﬁne coordination in a team of agents as the behavior of each individual agent being informed by other agents.
Revised: Intuitively, one can deﬁne coordination in a team of agents as the behavior of each individual agent being informed by other agents.

Original: Furthermore, agents’ behaviors can be inter-affected either directly through communication for example or indirectly through task-speciﬁc shared goals and/or rewards or the dynamics of the environment.
Revised: Furthermore, agents’ behaviors can be inter-affected either directly through communication for example or indirectly through task-speciﬁc shared goals and/or rewards or the dynamics of the environment.

Original: We hypothesize that when agents learn in a cooperative setting, they tend to affect each other’s exploitation processes, we conﬁrm the hypothesis throughout the paper and build on that to formalize a general method to foster inﬂuential interactions and learn meaningful coordination protocols.
Revised: We hypothesize that when agents learn in a cooperative setting, they tend to affect each other’s exploitation processes, we conﬁrm the hypothesis throughout the paper and build on that to formalize a general method to foster inﬂuential interactions and learn meaningful coordination protocols.

Original: Speciﬁcally, we introduce a novel framework to assess the inﬂuence that agent π , at timestep t , has on a set of agents upon taking an action a ( π) t in a global state s t . More concretely, consider n agents, namely π, µ 1 , µ 2 ,..., µ n − 1 . 
Revised: Speciﬁcally, consider a setting where N agents with policies π 1 , π 2 , . . . , π N . We assume that the transition dynamics are dependent on agents’ actions ( i.e. T ( s (cid:48) | s, a ) (cid:54) =

Original: Deﬁne µµµ =
Revised: T ( s | s (cid:48) )) , and use tasks that satisfy this assumption to validate our methods. We will write πππ,πππ − j to denote the joint policy of all agents and of all agents other than agent j , respectively. For two ﬁxed agents i and j , the inﬂuence of the behavior of j ’s policy on i ’s performance (expected return) can be quantiﬁed by "marginalizing" out all effects induced by other agents on the expected outcomes of i . More precisely, if the desired quantity is q π j i , then for all ( s, a j ) ∈ S

Original: [ π, µ 1 , ..., µ n − 1 ] T as the joint policy.
Revised: × A j : [Equation] One can note that, analogously to the action-value function Q i , q π j i measures tuple values w.r.t i where the learning dynamics of i conditioned only on the behavior of j .

Original: We use this notation throughout the rest of the paper.
Revised: Naturally, in expectation, q π j i represents the sum of rewards that an agent i is expected to receive when inﬂuenced by π j . Based on above, we say that an agent is an optimal inﬂuencer when agents only inﬂuenced by its policy learn optimal behaviors. To that end, if we ﬁx an agent (say 1 ), then its inﬂuence, F ( π 1 ) , on the rest of the team is deﬁned as follows: [Equation]

Original: Essentially, the agent π , which we call the "inﬂuencer", asks a retrospective question: "How much are agents { µ k } n − 1 k
Revised: where ( α i ) 2 ≤ i ≤ N are positive scaling factors that sum up to 1 and Q i is the action value function of agent i .

Original: =1 (i.e. the"inﬂuencees") expected  to get closer to their target returns after π executes an action a ( π ) t in a global state s t ?" 1 Meaning that state-action pairs that lead agents { µ k } n − 1 k
Revised: Intuitively, since F ( π 1 ) is the difference between the expected rewards of other agents conditioned on the current behavior of π 1 and the maximum returns that can be achieved by other agents in the given task, minimizing F will encourage π 1 to take actions that lead other agents’ conditioned returns ( q π 1 i ) 2 ≤ i ≤ N closer to their maximum unrestricted returns (max u Q i ( s, u )) , which matches our deﬁnition of an optimal inﬂuencer. Hence, the objective functions of agents i ∈ [ N ] are: J i ( θ i ) =

Original: ={+E πππ (cid:2) (cid:80) 
 ≤ t ≤ T γ t r i,t (cid:3) − λF ( π 1 ) 
 i =1 , where θ i are the parameters of π i and λ is a hyperparameter called the inﬂuence importance temperature. Theorem 1.+}
Revised: {+E πππ (cid:2) (cid:80) 
 ≤ t ≤ T γ t r i,t (cid:3) − λF ( π 1 ) 
 i =1 , where θ i are the parameters of π i and λ is a hyperparameter called the inﬂuence importance temperature. Theorem 1.+}

Original: closer to their target returns are considered highly inﬂuential and are rewarded.
Revised: (Inﬂuence Gradient) [Equation] where g ( T ) =

Original: The goal of this section is to show how π can learn effective policies that drive teammates’ behaviors towards their targets by estimating its inﬂuence.
Revised: g ( s, a 1 ) = q π 1 i ( s, a 1 ) − max u Q i ( s, u ) . It has been shown empirically that the existence of one inﬂuencer can greatly improve the overall team performance. Thus, we only consider one inﬂuencer throughout our practical sections and upon that, we call our method "Asymmetric Learning for Inﬂuencing a Team of Agents (ALITA)".

Original: 3.1. I NFLUENCE WITH S INGLE E STIMATOR
Revised: 3.1. I NFLUENCE WITH S INGLE E STIMATOR

Original: Formally, we quantify the inﬂuence F  πof agent π on { µ i } n − 1 i =1 by initializing a network Q cen : S ×
Revised: Practically, we quantify the inﬂuence F ( π 1 ) by initializing a network Q cen : S ×

Original: A → R n − 1 with parameters φ cen ;  Q cen ( . ; φ cen )( i ) estimatesthe updated q-value of agent µ i after frequent visits of π to ( s t , a ( π ) t ) , by minimizing the following loss: [Equation]
Revised: A → R N − 1 with parameters φ cen ; the i -th component of Q cen ( . ; φ cen ) estimates q π 1 i by minimizing the following loss at each timestep t :

Original: [Equation]
Revised: [Equation]

Original: Where y ∈ R n − 1 , y( i) =
Revised: ) where y ∈ R N − 1 , y i =

Original: r( i  ) + γQ ( i ) target ( x (cid:48) ,µµµ ( x (cid:48) ); ¯ φ cen ) ; Q ( i ) target is the target critic of agent µ i , and 
Revised: r i ( x , a ) + γQ ( i ) target ( x (cid:48) ,πππ ( x (cid:48) )) ; Q ( i ) target is the target critic of agent π i , and D t = { T | T = ( x , a , r, x (cid:48) )

Original: D is a buffer containing all agents’ experiences with the exception that  the agent π ’s experience is restricted to ( o π ( s t )  , a ( π ) t ) ; in other words, for all ( x , a , . ) ∈ D , x ( π ) = o π ( s t ) and a ( π ) =
Revised: ⊂ B and x 1 = s 1 ,t , a 1 = a 1 ,t } , where B is a buffer storing all agents’ transitions. The reason for choosing such deﬁnition of D is that when ﬁxing the current information of π (i.e. at timestep t ) while updating Q ceni , we practically marginalize out all agents’ experience (other than 1 and i ).

Original:  a ( π ) t . After obtaining an estimate ofwhat the q-values would be after counterfactual rollouts of π starting from ( o π ( s t ) , a ( π ) t ) , wecan now compute the inﬂuence F  π  :
Revised: Later in this section, we show that this trick yields a fairly good approximation of q π 1 i . After training Q cen , we compute the inﬂuence F ( π 1 ) :

Original: [Equation]
Revised: [Equation]

Original: Where B is a buffer storing all agents’ transitions.
Revised: where z = [max u Q 2 ( x (cid:48) , u ) , . . .

Original: Minimizing F π as a regularizer of its return objective J π , π adjusts its actions’ selections so that { µ k } n − 1 k =1 can reach their goals faster and more efﬁciently, thus achieving sufﬁcient coordination.
Revised: , max u Q N ( x (cid:48) , u )] T .

Original: Note that the second term in the expectation (i.e. the target vector y ) is set to be undifferentiable with respect to π  ’s parameters and thus does not propagate through its network.
Revised: Note that the second term in the expectation (i.e. the target vector z ) is set to be undifferentiable with respect to π 1 ’s parameters and thus does not propagate through its network.

Original: 3.1. I NFLUENCE WITH M ULTIPLE I NDIVIDUAL E STIMATORS
Revised: 3.1. I NFLUENCE WITH M ULTIPLE I NDIVIDUAL E STIMATORS

Original: As seen earlier, agent π  estimates the gap between each agent’s value and its target value by employing a single network.
Revised: As seen earlier, agent π 1 estimates the gap between each agent’s value and its target value by employing a single network.

Original: Another desirable approach is to  use multiple estimators where each estimator, namely Q ( i ) clone , individually calculates a fairly good approximation ofwhat the q-value of µ i would be after counterfactual rollouts of π starting from ( o π ( s t ) , a ( π ) t ) .
Revised: Another desirable approach is to additively decompose the centralized estimator Q cen , i.e. to use multiple estimators where each estimator, namely Q ( i ) clone , individually calculates a fairly good approximation of q π 1 i .

Original: To reduce computational costs and arrive at better estimates, each estimator’s network is initialized with the parameters of the corresponding critic network at each episode (i.e. Q ( i ) clone ← Q( i) ).
Revised: To reduce computational costs and arrive at better estimates, each estimator’s network is initialized with the parameters of the corresponding critic network at each episode (i.e. Q ( i ) clone ← Q i ).

Original: The training is carried out similarly to that of the single estimator setting,
Revised: The training is carried out similarly to that of the single estimator setting,

Original: [Equation]
Revised: [Equation]

Original: The inﬂuence function could be expressed as:
Revised: The inﬂuence function could be expressed as:

Original: [Equation]
Revised: [Equation]

Original: Which approach yields better estimates of the true value of the inﬂuence?
Revised: Which approach yields better estimates of the true value of the inﬂuence?

Original: The inﬂuence of an agent π on a team of agents T was deﬁned as a measure of the improvement in the performance of T given the current behavior of π .
Revised: The inﬂuence of an agent π on a team of agents T was deﬁned as a measure of the improvement in the performance of T given the current behavior of π .

Original: However, measuring the inﬂuence using function approximators might result in inaccurate estimates.
Revised: However, measuring the inﬂuence using function approximators might result in inaccurate estimates.

Original: To resolve this concern, we plot the inﬂuence estimates of the two prior approaches over time while they learn on the Cooperative Navigation 
Revised: To resolve this concern, we plot the inﬂuence estimates of the two prior approaches over time while they learn on the Cooperative Navigation MAPE task (Mordatch

Original: MAPE task (Mordatch & Abbeel, 2018), where the number of agents is N = 6 .
Revised: Abbeel, 2018), where the number of agents is N = 6 .

Original: In Figure (2), we graph the average inﬂuence estimates over 40000 episodes and compare it to the true value.
Revised: In Figure (1), we graph the average inﬂuence estimates over 40000 episodes and compare it to the true value.

Original: The latter can be expressed as the average distance between the true q -value and the true q -target of each agent given the current behavior of one pre-labeled agent (i.e. the inﬂuencer).

Original: This distance isthen averaged over 1000 episodes following the current policies of agents and is reported every 5000 episodes.
Revised: The latter is averaged overepisodes following the current policies of agents and is reported every 5000 episodes.

Original: The plots show a relatively small bias of both methods during learning.
Revised: The plots show a relatively small bias of both methods during learning.

Original: However, as Figure (2) suggests, measuring inﬂuence using multiple individual estimators yielded more accurate values after enough training which substantiates its superiority over the shared network approach.
Revised: However, as Figure (1) suggests, measuring inﬂuence using multiple individual estimators yielded more accurate values after enough training which substantiates its superiority over the shared network approach.

Original: Note that, although conﬁrms our prior hypothesis, this experiment does not reﬂect the importance of employing the inﬂuence on the ﬁnal performance of the agents as we will discuss that in Section (4).
Revised: Note that, although conﬁrms our prior hypothesis, this experiment does not reﬂect the importance of employing the inﬂuence on the ﬁnal performance of the agents as we will discuss that in Section (4).

Original: 3. I NTRINSIC M OTIVATION FOR D IVERSIFIED T EAM B EHAVIOR
Revised: 3. I NTRINSIC M OTIVATION FOR D IVERSIFIED T EAM B EHAVIOR

Original: In this section, we introduce a framework for achieving cooperative exploration by ensuring that agents are consistently tilted towards visiting under-explored state-action conﬁgurations; we start by providing a simple demonstration which shows that the number of environment steps required for all agents to randomly traverse all possible action conﬁgurations increases at least exponentially with thenumber of agents.
Revised: In this section, we introduce a framework for achieving cooperative exploration by ensuring that agents are consistently tilted towards visiting under-explored state-action conﬁgurations; we start by providing a simple demonstration which shows that the number of environment steps required for all agents to randomly traverse all possible action conﬁgurations increases at least exponentially with the of agents

Original: Proposition 1.
Revised: Proposition 1.

Original: Consider an L -action setting of n agents.
Revised: Consider an L -action setting of n agents.

Original: In expectation, the number of steps T needed to visit all L n action conﬁgurations at least once without coordinated exploration grows at least exponentially with the number of agents.
Revised: In expectation, the number of steps T needed to visit all L n action conﬁgurations at least once without coordinated exploration grows at least exponentially with the number of agents.

Original: More concretely, E
Revised: More concretely, E

Original: [ T ] = Ω( nL n ) .
Revised: [ T ] = Ω( nL n ) .

Original: Proof. See Appendix A. To mitigate this issue, we assign agent π  a prediction error as an intrinsic reward to facilitate recognizing and learning novel behaviors:
Revised: To mitigate this issue, we assign agent π 1 a prediction error as an intrinsic reward to facilitate recognizing and learning novel behaviors:

Original: [Equation]
Revised: [Equation]

Original: Where φ is an autoencoder network regularly trained on data generated by the policy π  and λ π is a hyper-parameter that balances the extrinsic and intrinsic reward terms.
Revised: Where φ is an autoencoder network regularly trained on data generated by the policy π 1 and λ π is a hyper-parameter that balances the extrinsic and intrinsic reward terms.

Original: ψ ’s expression stems from the observation that when an autoencoder is trained on data from a particular distribution, it will be good at reconstructing data from that distribution, while it will perform poorly if the data is from a different distribution (Fayad & Ibrahim, 2021; Zhang et al., 2019).
Revised: ψ ’s expression stems from the observation that when an autoencoder is trained on data from a particular distribution, it will be good at reconstructing data from that distribution, while it will perform poorly if the data is from a different distribution.

Original: Thus, by employing ψ as an intrinsic bonus, π  rewards states’ observations and actions that do not belong to the data generated by it.
Revised: Thus, by employing ψ as an intrinsic bonus, π 1 rewards states’ observations and actions that do not belong to the data generated by it.

Original: In practice, φ is designed to be a relatively large network since we want it to be slightly overﬁtted to the training data so that it will not accidentally generalize to behaviors that we may deem novel.
Revised: In practice, φ is designed to be a relatively large network since we want it to be slightly overﬁtted to the training data so that it will not accidentally generalize to behaviors that we may deem novel (Fayad & Ibrahim, 2021; Zhang et al., 2019).

Original: Nevertheless, assigning each agent a ψ is not sufﬁcient as it makes the case equivalent to independent exploration approaches.
Revised: Nevertheless, assigning each agent a ψ is not sufﬁcient as it makes the case equivalent to independent exploration approaches.

Original: Thus, we propose a coordinated exploration method that takes into account other agents’ behaviors, encouraging agents to diversify team behavior while maintaining good performance.
Revised: Thus, we propose a coordinated exploration method that takes into account other agents’ behaviors, encouraging agents to diversify team behavior while maintaining good performance.

Original: Speciﬁcally, we assign agents { µ i } n − 1 i =1 intrinsic penalty deﬁned as:[Equation]
Revised: Speciﬁcally, we assign agents { π i } Ni =2 intrinsic penalty deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: This reward term aims at teaching the agents to recognize previous behaviors and synchronously select novel conﬁgurations.
Revised: This reward term aims at teaching the agents to recognize previous behaviors and synchronously select novel conﬁgurations.

Original: To build intuition, consider a case where N = 2 .
Revised: To build intuition, consider a case where N = 2 .

Original: Whenever ( π, µ ) select an action tuple in the neighborhood of a frequently-visited tuple ( a 1 , a 2 ) in a global state s , ψ will be relatively small and the penalty, r int µ , will be large.
Revised: Whenever ( π, µ ) select an action tuple in the neighborhood of a frequently-visited tuple ( a 1 , a 2 ) in a global state s , ψ will be relatively small and the penalty, r int µ , will be large.

Original: Conversely, if ( π, µ ) encounter a novel tuple, say ( a (cid:48) 1 , a (cid:48) 2 ) in s , the small penalty of µ (Eq.
Revised: Conversely, if ( π, µ ) encounter a novel tuple, say ( a (cid:48) 1 , a (cid:48) 2 ) in s , the small penalty of µ (Eq.

Original: (8)) together with a large reward for π (Eq.
Revised: (11)) together with a large reward for π (Eq.

Original: (7)) will drive both agents to further explore this encounter.
Revised: (10)) will drive both agents to further explore this encounter.

Original: In all, Fig.
Revised: In all, Fig.

Original: (2) shows how this framework can be augmented with the basic inﬂuence introduced earlier to reinforce learning and discovering coordinated behaviors.
Revised: (2) shows how this framework can be augmented with the basic inﬂuence introduced earlier to reinforce learning and discovering coordinated behaviors.

Original:  E MPIRICAL E VALUATION & A NALYSIS
Revised:  E MPIRICAL E VALUATION & A NALYSIS

Original: The goals of our experiments are to: a) verify the performance of our method on a comprehensive set of multi-agent challenges (SMAC, MAPE, sparse-reward settings, and continuous control environments); b) perform ablations to examine which particular components of the proposed framework are important for good performance.
Revised: The goals of our experiments are to: a) verify the performance of our method on a comprehensive set of multi-agent challenges (SMAC, MAPE, sparse-reward settings, and continuous control environments); b) perform ablations to examine which particular components of the proposed framework are important for good performance.

Original: 4. C OOPERATIVE & M IXED G AMES 4.1. S TAR C RAFT M ULTI -A GENT C HALLENGE
Revised: 4. C OOPERATIVE & M IXED G AMES 4.1. S TAR C RAFT M ULTI -A GENT C HALLENGE

Original: StarCraft provides a rich set of heterogeneous units each with diverse actions, allowing for extremely complex cooperative behaviors among agents.
Revised: StarCraft provides a rich set of heterogeneous units each with diverse actions, allowing for extremely complex cooperative behaviors among agents.

Original: We thus evaluate our method on several SC micromanagement tasks from the SMAC 2 benchmark (Samvelyan et al., 2019), where a group of mixed-typed units controlled by decentralized agents needs to cooperate to defeat another group of mixed-typed enemy units controlled by built-in heuristic rules with “difﬁcult” setting; the battles can be both symmetric (same units in both groups) or asymmetric.
Revised: We thus evaluate our method on several SC micromanagement tasks from the SMAC 1 benchmark (Samvelyan et al., 2019), where a group of mixed-typed units controlled by decentralized agents needs to cooperate to defeat another group of mixed-typed enemy units controlled by built-in heuristic rules with “difﬁcult” setting; the battles can be both symmetric (same units in both groups) or asymmetric.

Original: Each agent observes its own status and, within its ﬁeld of view, it also observes other units’ statistics such as health, location, and unit type (partial observability); agents can only attack enemies within their shooting range.
Revised: Each agent observes its own status and, within its ﬁeld of view, it also observes other units’ statistics such as health, location, and unit type (partial observability); agents can only attack enemies within their shooting range.

Original: A shared reward is received on battle victory as well as damaging or killing enemy units.
Revised: A shared reward is received on battle victory as well as damaging or killing enemy units.

Original: Each battle has step limits set by SMAC and may end early.
Revised: Each battle has step limits set by SMAC and may end early.

Original: We consider 4 battle maps grouped intoEasy (2s3z),
Revised: We consider 5 battle maps grouped into

Original:  Hard (5m_vs_6m, 3s_vs_5z), and Super Hard (corridor) against 6 baseline methods using their open-source implementations based on PyMARL (Samvelyan et al., 2019):COMA (Foerster et al., 2018) , IQL (Tan, 1993), VDN (Sunehag et al., 2018), QMIX (Rashid et al., 2018), LIIR (Individual
Revised: Easy (2s3z), Hard (5m_vs_6m, 3s_vs_5z), and Super Hard (corridor, 3s5z_vs_3s6z) againstbaseline methods using their open-source implementations based on PyMARL (Samvelyan et al., 2019): IQL (Tan, 1993), VDN (Sunehag et al., 2018), QMIX (Rashid et al., 2018), LIIR (Individual

Original: Intrinsic Rewards) (Du et al., 2019),and LICA (Implicit Credit Assignment)
Revised: Intrinsic Rewards) (Du et al., 2019), LICA (Implicit Credit Assignment)

Original: (Zhou et al., 2020b).
Revised: (Zhou et al., 2020b), and

Revised: MAVEN (Variational Exploration) (Mahajan et al., 2019), EDTI (Decision-Theoretic Exploration) ,

Revised: EITI (Information-Theoretic Exploration) (Wang et al., 2019).

Original: The corridor map, in which 6 Zealots face 24 enemy Zerglings, requires agents to make effective use of the terrain features and block enemy attacks from different directions.
Revised: The corridor map, in which 6 Zealots face 24 enemy Zerglings, requires agents to make effective use of the terrain features and block enemy attacks from different directions.

Original: A properly coordinated exploration scheme applied to this map would help the agents discover a suitable unit positioning quickly and improve performance, while 2s3z requires agents to learn “focus ﬁre" and interception.
Revised: A properly coordinated exploration scheme applied to this map would help the agents discover a suitable unit positioning quickly and improve performance, while 2s3z requires agents to learn “focus ﬁre" and interception.

Original: For the asymmetric 5m_vs_6m, basic agent coordination alone such as “focus ﬁring” no longer sufﬁces (Du et al., 2019) and consistent success requires extended exploration to uncover complex cooperative strategies such as pulling back units with low health during combat.
Revised: For the asymmetric 5m_vs_6m, basic agent coordination alone such as “focus ﬁring” no longer sufﬁces (Du et al., 2019) and consistent success requires extended exploration to uncover complex cooperative strategies such as pulling back units with low health during combat.

Original: The 3s_vs_5z scenario features three allied Stalkers against ﬁve enemy Zealots.
Revised: The 3s_vs_5z scenario features three allied Stalkers against ﬁve enemy Zealots.

Original: Since Zealots counter Stalkers, the only winning strategy for the allied units is to kite the enemy around the map and kill them one after another, causing the failure of independent learning algorithms to learn good policies in this task.
Revised: Since Zealots counter Stalkers, the only winning strategy for the allied units is to kite the enemy around the map and kill them one after another, causing the failure of independent learning algorithms to learn good policies in this task.

Revised: The 3s5z_vs_3s6z is an extended scenario of 3s_vs_5z where 3 Stalkers and 5 Zealots battle against 3 Stalkers and 6 Zealots—The extra enemy makes this scenario far more challenging.

Revised: For this task, we randomly chose 1 S and 1 Z to act as inﬂuencers because of the heterogeneity of the team.

Revised: In the rest, agents are symmetric, hence the inﬂuencer is randomly chosen at the beginning of training.

Original: For all these scenarios, our method consistently shows the best performance with signiﬁcant learning speed.
Revised: For most of these scenarios, ALITA consistently shows the best performance with signiﬁcant learning speed which conﬁrms the effectiveness of our proposed methods.

Original: Detailed results are reported in Figure (3) as they present the median win rateof the methods during the training across 12 random runs.
Revised: Detailed results are reported in Figure (3) as they present the median win rate during the training across 12 random runs.

Original: (a) 2s3z Easy (b) 3s_vs_5z Hard (c) 5m_vs_6m

Original: [Figure]
Revised: [Figure]

Original: 4.1. S PARSE -R EWARD S ETTINGS
Revised: 4.1. S PARSE -R EWARD S ETTINGS

Original: We test on two additional tasks to show the effectiveness our method on sparse-reward settings and compare it to famous inﬂuence-based coordinated exploration algorithms (Table (1)): EDTI, EITI (Wang et al., 2019), and Social Inﬂuence (Jaques et al., 2019).
Revised: We test on two additional tasks to show the effectiveness our method on sparse-reward settings and compare it to famous inﬂuence-based coordinated exploration algorithms (Table (1)): EDTI, EITI (Wang et al., 2019), and Social Inﬂuence (Jaques et al., 2019).

Original: Sparse Push-Box: A 15 × 15 room is populated with 2 agents and 1 box.
Revised: Sparse Push-Box: A 15 × 15 room is populated with 2 agents and 1 box.

Original: Agents need to push the box to the wall in 300 environment steps to get a reward of 1000.
Revised: Agents need to push the box to the wall in 300 environment steps to get a reward of 1000.

Original: Moreover, both can observe the coordinates of their teammate and the location of the box.
Revised: Moreover, both can observe the coordinates of their teammate and the location of the box.

Original: However, the box is so heavy that only when two agents push it in the same direction at the same time can it be moved a grid.
Revised: However, the box is so heavy that only when two agents push it in the same direction at the same time can it be moved a grid.

Original: Agents need to coordinate their positions and actions for multiple steps to earn a reward.
Revised: Agents need to coordinate their positions and actions for multiple steps to earn a reward.

Original: Sparse Secret Room: A 25 × 25 grid is divided into three small rooms on the right and one large room on the left where 2 agents are initially spawned.
Revised: Sparse Secret Room: A 25 × 25 grid is divided into three small rooms on the right and one large room on the left where 2 agents are initially spawned.

Original: There is one door between each small room and the large room.
Revised: There is one door between each small room and the large room.

Original: A switch in the large room controls all three doors.
Revised: A switch in the large room controls all three doors.

Original: A switch also exists in each small room which only controls the room’s door.
Revised: A switch also exists in each small room which only controls the room’s door.

Original: The agents need to navigate to one of the three small rooms, i.e. the target room, to receive positive reward.
Revised: The agents need to navigate to one of the three small rooms, i.e. the target room, to receive positive reward.

Original: The task is considered solved if both agents are in the 
Revised: The task is considered solved if both agents are in the target room.

Original: The state vector  ( x all agents and binary variables to indicate 
Revised: The state vector contains ( x, y ) locations of all agents and binary variables to indicate if doors are open. [Table]

Original: [Table]
Revised: [Table]

Revised: A notable reason for the good performance of ALITA on the two tasks is that through the intrinsic rewards, agents tend to explore the majority of the possible encounters at the beginning of learning which is crucial for estimating the value of the inﬂuence.

Revised: Exploiting the latter, agents tend to reinforce learning jointly rewarding conﬁgurations.

Original: 4.1. M ULTI -A GENT P ARTICLE E NVIRONMENTS
Revised: 4.1. M ULTI -A GENT P ARTICLE E NVIRONMENTS

Original: To understand how the proposed method helps agents achieve cooperative behavior in nonstationary settings, we conduct experiments on the grounded communication environment 3 proposed in(Mordatch & Abbeel, 2018; Lowe et al., 2017). Each task consists of multiple agents ( N ≥ 2 ) and
Revised: To understand how the proposed method helps agents achieve cooperative behavior in nonstationary settings, we conduct experiments on the grounded communication environment 2 proposed in

Original: L landmarks in a two-dimensional world with continuous space and discrete time. Both agent and landmark entities inhabit a physical location in space and posses descriptive physical characteristics, such as color and shape type. For that purpose, we adapt the DDPG algorithm as a learning framework and train with 10 random seeds. Results of the following tasks are reported in Tables (2, 3, 4).
Revised: (Mordatch & Abbeel, 2018; Lowe et al., 2017).

Original: Cooperative Navigation: In this environment, N agents must collaborate to reach a set of N landmarks with known positions. Agents are rewarded based on how far any agent is from each landmark, meaning that the agents learn to spread with each agent covering one landmark. The agents, which occupy a signiﬁcant physical space, are aware of their relative positions to each other and are further penalized when colliding with each other. [Table]
Revised: The chosen tasks are the Cooperative Navigation,

Original: Cooperative Communication: Here, a stationary speaker must guide a listener in an environment consisting of three landmarks of differing colors. At each episode, one landmark of a particular color is set as a goal for the listener to be reached, however, only the speaker can observe which landmark the listener must navigate to. Moreover, The speaker can produce a communication output at each time step which is observed by the listener. The latter must navigate the environment to reach the correct landmark.
Revised: Cooperative Communication, and Physical Deception .

Original: Agents are collectively rewarded at the end of an episode based on the listener’s distance from the correct landmark. [Table] Physical Deception: This environment consists of N agents and N landmarks, with one landmark as the target of all agents.
Revised: 3 We trained with 10 random seeds and reported results in Tables (2, 3, 4).

Original: The agents are rewarded based on the distance of the closest agent to the target landmark, making it sufﬁcient for only one agent to reach it.

Original: An adversary agent also tries to reach the target landmark, while the agents are penalized as it gets closer to the target.

Original: The adversary, however, does not know which landmark is the target and must deduce it from the agents’ behavior.

Original: For that reason, agents must cooperate to trick the adversary by learning to cover all the landmarks.

Original: This task shows that our algorithm is applicable not only to cooperative interactions but to mixed environments as well.

Original: 4. C ONTINUOUS E NVIRONMENTS
Revised: 4. C ONTINUOUS E NVIRONMENTS

Original: To conﬁrm the scalability of our algorithm to large continuous settings, we measure the performance of our algorithm on a suite of PyBullet (Tan et al., 2018) continuous control tasks, interfaced through[Table]
Revised: To conﬁrm the scalability of our algorithm to large continuous settings, we measure the performance of our algorithm on a suite of PyBullet (Tan et al., 2018) continuous control tasks, interfaced through

Original: OpenAI Gym (Brockman et al., 2016).
Revised: OpenAI Gym (Brockman et al., 2016).

Original: Gym environments, however, are mainly single-agent settings, thus to evaluate our approach, we reframe the problem by introducing an additional learning agent that acts as an auxiliary agent.
Revised: Gym environments, however, are mainly single-agent settings, thus to evaluate our approach, we reframe the problem by introducing an additional learning agent that acts as an auxiliary agent.

Original: Crucially, both agents work collaboratively in order to ﬁnd a region of the solution space where an agent accumulates higher rewards.
Revised: Crucially, both agents work collaboratively in order to ﬁnd a region of the solution space where an agent accumulates higher rewards.

Original: We use TD3  as our learning model and test it against state-of-the-art algorithms in 5 gym environments.
Revised: We use TD3 (Fujimoto et al., 2018) as our learning model and test it against state-of-the-art algorithms in 5 gym environments.

Original: Our algorithm outperforms all baselines across all different environments (e.g. our method attains 131% return of SAC ﬁnal performance on Humanoid-v3).
Revised: Our algorithm outperforms all baselines across all different environments (e.g. our method attains 131% return of SAC ﬁnal performance on Humanoid-v3).

Original: For detailed results, see Appendix B.
Revised: For detailed results, see Appendix C.

Original: 4. A BLATIONS
Revised: 4. A BLATIONS

Original: We further investigate the signiﬁcance of each component along with a symmetric extension of the proposed framework.
Revised: We further investigate the signiﬁcance of each component along with a symmetric extension of the proposed framework.

Original: Speciﬁcally, we consider the three cases: 1) No F : where the inﬂuence function does not contribute to the update rule to any of the policies; 2)
Revised: Speciﬁcally, we consider the three cases: 1) No F : where the inﬂuence function does not contribute to the update rule to any of the policies; 2)

Original: No IM : where a randomly selected agent maximizes both the expected sum of extrinsic rewards along with the inﬂuence function, and other agents’ policies are learned using the DDPG; 3)
Revised: No IM : where a randomly selected agent maximizes both the expected sum of extrinsic rewards along with the inﬂuence function, and other agents’ policies are learned using the DDPG; 3)

Original: Symmetric : where all agents simultaneously play the role of an inﬂuencer and inﬂuencee: they learn to maximize an augmented reward function ( extrinsic and intrinsic ) along with the inﬂuence function.
Revised: Symmetric : where all agents simultaneously play the role of an inﬂuencer and inﬂuencee: they learn to maximize an augmented reward function ( extrinsic and intrinsic ) along with the inﬂuence function.

Original: Results of the experiments conducted on the 2s3z SMAC scenario show that, in the absence of the intrinsic rewards ( No IM ), the agents experience a slightly decreased overall performance when compared to the signiﬁcant decline induced by detaching the inﬂuence function ( No F ).
Revised: Results of the experiments conducted on the 2s3z SMAC scenario show that, in the absence of the intrinsic rewards ( No IM ), the agents experience a slightly decreased overall performance when compared to the signiﬁcant decline induced by detaching the inﬂuence function ( No F ).

Original: In Figure (4), we observe that the agents following the Symmetric approach learn faster, and achieve a signiﬁcantly higher median win rate.
Revised: In Figure (4), we observe that the agents following the Symmetric approach learn faster, and achieve a signiﬁcantly higher median win rate.

Original: This approach, however, doubles the computational costs which restricts its applicability in larger settings.
Revised: This approach, however, doubles the computational costs which restricts its applicability in larger settings.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: We discuss recently developed methods for exploration in RL using intrinsic motivation, coordination in multi-agent RL, and inﬂuence-based coordinated exploration methods subsequently.
Revised: We discuss recently developed methods for exploration in RL using intrinsic motivation, coordination in multi-agent RL, and inﬂuence-based coordinated exploration methods subsequently.

Original: Intrinsic motivation (IM) has been increasingly used both in single-agent RL and multi-agent RL.
Revised: Intrinsic motivation (IM) has been increasingly used both in single-agent RL and multi-agent RL.

Original: A core idea of IM is to encourage the agent to take new actions or visit new states, thus exploring the environment and obtaining more diverse behaviors.
Revised: A core idea of IM is to encourage the agent to take new actions or visit new states, thus exploring the environment and obtaining more diverse behaviors.

Original: One common approach is to approximate state or state-action visitation frequency and add a reward bonus to states the agent rarely covers (Tang et al., 2017; Bellemare et al., 2016; Martin et al., 2017).
Revised: One common approach is to approximate state or state-action visitation frequency and add a reward bonus to states the agent rarely covers (Tang et al., 2017; Bellemare et al., 2016; Martin et al., 2017).

Original: A more related IM approach is to evaluate state visitation novelty (Klissarov et al., 2019; Han et al., 2020; Burda et al., 2019) or state-action visitation novelty (Fayad & Ibrahim, 2021).
Revised: A more related IM approach is to evaluate state visitation novelty (Klissarov et al., 2019; Han et al., 2020; Burda et al., 2019) or state-action visitation novelty (Fayad & Ibrahim, 2021).

Original: Inspired by the latter, we provided a natural extension for this approach to the MARL settings by the learning of a "novelty" function.
Revised: Inspired by the latter, we provided a natural extension for this approach to the MARL settings by the learning of a "novelty" function.

Original: Other works make use of single-agent IM to construct their multi-agent intrinsic reward (Du et al., 2019; Iqbal & Sha, 2019).
Revised: Other works make use of single-agent IM to construct their multi-agent intrinsic reward (Du et al., 2019; Iqbal & Sha, 2019).

Original: Each agent in (Du et al., 2019) learns a distinct intrinsic reward so that the agents are stimulated differently, even when the environment only feedbacks a team reward.
Revised: Each agent in (Du et al., 2019) learns a distinct intrinsic reward so that the agents are stimulated differently, even when the environment only feedbacks a team reward.

Original: This reward helps distinguish the contributions of the agents when the environment only returns a collective reward.
Revised: This reward helps distinguish the contributions of the agents when the environment only returns a collective reward.

Original: In (Iqbal &
Revised: In (Iqbal &

Original: Sha, 2019), each agent has a novelty function that assesses how novel an observation is to it, based on its past experience.
Revised: Sha, 2019), each agent has a novelty function that assesses how novel an observation is to it, based on its past experience.

Original: Their multi-agent intrinsic reward is deﬁned based on how novel all agents consider an agent’s observation.
Revised: Their multi-agent intrinsic reward is deﬁned based on how novel all agents consider an agent’s observation.

Original: A recent work (Liu et al., 2021) assigns agents a common goal while exploring.
Revised: A recent work (Liu et al., 2021) assigns agents a common goal while exploring.

Original: The goal is selected from multiple projected state spaces via a normalized entropy-based technique.
Revised: The goal is selected from multiple projected state spaces via a normalized entropy-based technique.

Original: Then, agents are trained to reach this goal in a coordinated manner.
Revised: Then, agents are trained to reach this goal in a coordinated manner.

Original: Many works studied the cooperative settings in MARL; a straightforward approach is to use independent learning agents (fully decentralized learning).
Revised: Many works studied the cooperative settings in MARL; a straightforward approach is to use independent learning agents (fully decentralized learning).

Original: This approach, however, is shown to perform inadequately both with Q -learning
Revised: This approach, however, is shown to perform inadequately both with Q -learning

Original: (Matignon et al., 2012) and with policy gradient (Lowe et al., 2017).
Revised: (Matignon et al., 2012) and with policy gradient (Lowe et al., 2017).

Original: Therefore, we considered the CTDE paradigm, where each agent’s policy takes its individual observation as many real life applications dictate, while the centralized critic permit for sharing of information during training.
Revised: Therefore, we considered the CTDE paradigm, where each agent’s policy takes its individual observation as many real life applications dictate, while the centralized critic permit for sharing of information during training.

Original: Policy gradient methods have been commonly used along with the
Revised: Policy gradient methods have been commonly used along with the

Original: CTDE paradigm in MARL, either by implementing a single centralized critic for all agents (Foerster et al., 2018), or one centralized critic for each agent (Lowe et al., 2017).
Revised: CTDE paradigm in MARL, either by implementing a single centralized critic for all agents (Foerster et al., 2018), or one centralized critic for each agent (Lowe et al., 2017).

Original: Adopting the latter, we enable agents with different reward functions to learn in competitive and mixed scenarios as well.
Revised: Adopting the latter, we enable agents with different reward functions to learn in competitive and mixed scenarios as well.

Original: Some other works encouraged cooperative interactions between agents by sharing useful information (Yang et al., 2020; Hostallero et al., 2020).
Revised: Some other works encouraged cooperative interactions between agents by sharing useful information (Yang et al., 2020; Hostallero et al., 2020).

Original: In Hostallero et al.
Revised: In Hostallero et al.

Original: (2020), each agent broadcasts a signal that represents an assessment of the effect of the joint actions that all agents take on its expected reward.
Revised: (2020), each agent broadcasts a signal that represents an assessment of the effect of the joint actions that all agents take on its expected reward.

Original: Different from our approach, this signal encourages agents to behave as is expected of them and does not beneﬁt exploration.
Revised: Different from our approach, this signal encourages agents to behave as is expected of them and does not beneﬁt exploration.

Original: As for Yang et al.
Revised: As for Yang et al.

Original: , each agent learns an incentive function that rewards other agents based on their actions.
Revised: , each agent learns an incentive function that rewards other agents based on their actions.

Original: Each agent’s function aims to alter other agents’ behavior to maximize its extrinsic rewards.
Revised: Each agent’s function aims to alter other agents’ behavior to maximize its extrinsic rewards.

Original: To accomplish this, each agent requires access to every other agent’s policy, incentive function, and return making this approach difﬁcult to scale and execute.
Revised: To accomplish this, each agent requires access to every other agent’s policy, incentive function, and return making this approach difﬁcult to scale and execute.

Original: Additionally, Roy et al.
Revised: Additionally, Roy et al.

Original: (2019) proposed two policy regularizers approaches to promote coordination in a team of agent, one of which assumes that an agent must be able to predict the behavior of its teammates in order to coordinate with them, while the other supposes that coordinated agents collectively recognize different situations and synchronously switch to different sub-policies to react to them.
Revised: (2019) proposed two policy regularizers approaches to promote coordination in a team of agent, one of which assumes that an agent must be able to predict the behavior of its teammates in order to coordinate with them, while the other supposes that coordinated agents collectively recognize different situations and synchronously switch to different sub-policies to react to them.

Original: Similarly to our work, (Jaques et al., 2019) proposed a similar idea of rewarding an agent for having a casual inﬂuence on other agents’ actions.
Revised: Similarly to our work, (Jaques et al., 2019) proposed a similar idea of rewarding an agent for having a casual inﬂuence on other agents’ actions.

Original: Their method showed interesting results in terms of learning coordinated behavior.
Revised: Their method showed interesting results in terms of learning coordinated behavior.

Original: However, this casual inﬂuence is designed to reward policies for inﬂuencing other policies’ actions without considering the "quality" of this inﬂuence.
Revised: However, this casual inﬂuence is designed to reward policies for inﬂuencing other policies’ actions without considering the "quality" of this inﬂuence.

Original: Barton et al.
Revised: Barton et al.

Original: (2018) propose causal inﬂuence as a way to measure coordination between agents, speciﬁcally using Convergence
Revised: (2018) propose causal inﬂuence as a way to measure coordination between agents, speciﬁcally using Convergence

Original: Cross Mapping (CCM) to analyze the degree of dependence between two agents’ policies.
Revised: Cross Mapping (CCM) to analyze the degree of dependence between two agents’ policies.

Original: Our method also draws inspiration from the work of (Wang et al., 2019), as they deﬁne an inﬂuence-based intrinsic exploration bonus by the expected difference between the action-value function of one agent and its counterfactual action-value function without considering the state and action of the other agent.
Revised: Our method also draws inspiration from the work of (Wang et al., 2019), as they deﬁne an inﬂuence-based intrinsic exploration bonus, called Value of Interaction (VoI), by the expected difference between the action-value function of one agent and its counterfactual action-value function without considering the state and action of the other agent.

Revised: Particularly, the latter measures the expected behavior of an agent in a situation where it is not inﬂuenced by the other agent.

Revised: Consequently, by maximizing the

Revised: VoI, agents tend to explore meaningful interaction points as the distance between their action-value functions conditioned on other agents and their independent action-value (only conditioned on self behavior) functions is maximized.

Revised: The primary difference from ALITA lies in our deﬁnition of the inﬂuence.

Original:  C ONCLUSIONS & F UTURE W ORK
Revised:  C ONCLUSIONS & F UTURE W ORK

Original: We introduced a novel multi-agent RL algorithm for achieving coordination through assessing the inﬂuence an agent has on other agents’ behaviors.
Revised: We introduced a novel multi-agent RL algorithm for achieving coordination through assessing the inﬂuence an agent has on other agents’ behaviors.

Original: Additionally, we proposed to learn an intrinsic reward for each agent to promote coordinated team exploration.
Revised: Additionally, we proposed to learn an intrinsic reward for each agent to promote coordinated team exploration.

Original: We tested our algorithm on a wide variety of tasks with many challenges, such as partial observability, sparse rewards, and large spaces; these tasks include, but not limited to, SMAC, MAPE, as well as OpenAI gym continuous environments.
Revised: We tested our algorithm on a wide variety of tasks with many challenges, such as partial observability, sparse rewards, and large spaces; these tasks include, but not limited to, SMAC, MAPE, as well as OpenAI gym continuous environments.

Original: Our methods achieved noticeable improvement over prominent algorithms on all tasks.
Revised: Our methods achieved noticeable improvement over prominent algorithms on all tasks.

Original: One promising extension of our algorithm is to use Graph Attention Networks (Zhou et al., 2020a) to learn the importance of the inﬂuencer in determining the inﬂuencees’ policies and to establish a message-passing architecture in networked systems.
Revised: One promising extension of our algorithm is to use Graph Attention Networks (Veli ˇ ckovi ´ c et al., 2017; Zhou et al., 2020a) to learn the importance of the inﬂuencer in determining the inﬂuencees’ policies and to establish a message-passing architecture in networked systems.

Original: The investigation of the effectiveness of these methods is left for future works.
Revised: The investigation of the effectiveness of these methods is left for future works.

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY S TATEMENT

Original: We have provided an illustration of the proposed algorithm in Fig.
Revised: We have provided an illustration of the proposed algorithm in Fig.

Original: (2) along with implementation details and hyperparameters selection in Appendix C. Furthermore, code is submitted with Supplementary Material and each algorithm is evaluated at least 10 times using random seeds on all environments.
Revised: (1) along with implementation details and hyperparameters selection in Appendix D. Furthermore, code is submitted with Supplementary Material and each algorithm is evaluated at least 10 times using random seeds on all environments A M ATHEMATICAL D ETAILS Theorem 1. (Inﬂuence Gradient)

Revised: [Equation]

Original: A P ROOFS
Revised: where g ( T ) = g ( s, a 1 ) = q π 1 i ( s, a 1 ) − max u Q i ( s, u ) . Proof.

Revised: We start by ﬁnding ∇ θ 1 q π 1 i :

Revised: [Equation]

Revised: Let T =

Revised: ( s, a 1 ) ∈ S ×

Revised: A 1 , and φ ( T ) =

Revised: γ E s (cid:48) ,a (cid:48) 1 [ ∇ θ 1 log π

Revised: [Equation]

Revised: a

Revised: [Equation]

Revised: 1 | s

Revised: [Equation]

Revised: ) q π 1 i ( s (cid:48) , a (cid:48) 1 )] where the distribution of s (cid:48) is conditional on s and a 1 .

Revised: Thus, write ∇ θ 1 q π 1 i ( T ) as:

Revised: [Equation]

Revised: ( new tuples T (cid:48)(cid:48) are conditioned on previous ones )

Revised: [Equation]

Revised: ( repeatedly unroll ∇ q

Revised: [Equation]

Revised: 1 i

Revised: [Equation]

Revised: ( normalize the series term )

Revised: [Equation]

Revised: d T is the discounted visitation distribution induced by π 1 )

Revised: [Equation]

Revised: use T (cid:48) to simplify notation )

Revised: Since g ( T ) =

Revised: q π 1 i ( s, a 1 ) − max u Q i ( s, u ) , the gradient of F ( π 1 ) can be expressed as:

Revised: [Equation]

Revised: In all, ∇ θ 1 F ( π 1 ) =

Revised: (cid:88)

Revised: [Equation]

Original: Proposition 1.
Revised: Proposition 1.

Original: Consider an L -action setting of n agents.
Revised: Consider an L -action setting of n agents.

Original: In expectation, the number of steps T needed to visit all L n action conﬁgurations at least once without coordinated exploration grows at least exponentially with the number of agents.
Revised: In expectation, the number of steps T needed to visit all L n action conﬁgurations at least once without coordinated exploration grows at least exponentially with the number of agents.

Original: More concretely, E
Revised: More concretely, E

Original: [ T ] = Ω( nL n ) .
Revised: [ T ] = Ω( nL n ) .

Original: Proof.
Revised: Proof.

Original: Let M = L n .
Revised: Let M = L n .

Original: Since agents tend to visit different action conﬁgurations with no coordinated behavior, one can equivalently say that agents uniformly pick a conﬁguration out of all L n possible conﬁgurations at each step.
Revised: Since agents tend to visit different action conﬁgurations with no coordinated behavior, one can equivalently say that agents uniformly pick a conﬁguration out of all L n possible conﬁgurations at each step.

Original: Let T k be the number of steps to visit the k -th distinct conﬁguration after covering k − 1 distinct action tuples.
Revised: Let T k be the number of steps to visit the k -th distinct conﬁguration after covering k − 1 distinct action tuples.

Original: Observe that:
Revised: Observe that:

Original: [Equation]
Revised: [Equation]

Original: Now, Pr [ T k = i ] = (cid:0)
Revised: Now, Pr [ T k = i ] = [Equation] M (cid:1) i

Original: [Equation]
Revised: [Equation]

Original:  − k − 1 M (cid:1) meaning that T k follows a geometric distribution.
Revised: [Equation] − [Equation] M (cid:1) meaning that T k follows a geometric distribution.

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: Getting back to Eq.
Revised: Getting back to Eq.

Original: [Equation]
Revised: [Equation]

Original: And the conclusion follows.
Revised: And the conclusion follows.

Original: B A DDITIONAL E XPERIMENTS ON C ONTINUOUS E NVIRONMENTS
Revised: B D ETAILS OF MAPE T ASKS

Revised: Cooperative Navigation: In this environment, N agents must collaborate to reach a set of N landmarks with known positions. Agents are rewarded based on how far any agent is from each landmark, meaning that the agents learn to spread with each agent covering one landmark. The agents, which occupy a signiﬁcant physical space, are aware of their relative positions to each other and are further penalized when colliding with each other.

Revised: [Table]

Revised: [Table]

Revised: Cooperative Communication: Here, a stationary speaker must guide a listener in an environment consisting of three landmarks of differing colors.

Revised: At each episode, one landmark of a particular color is set as a goal for the listener to be reached, however, only the speaker can observe which landmark the listener must navigate to.

Revised: Moreover, The speaker can produce a communication output at each time step which is observed by the listener.

Revised: The latter must navigate the environment to reach the correct landmark.

Revised: Agents are collectively rewarded at the end of an episode based on the listener’s distance from the correct landmark.

Revised: Physical Deception:

Revised: This environment consists of N agents and N landmarks, with one landmark as the target of all agents.

Revised: The agents are rewarded based on the distance of the closest agent to the target landmark, making it sufﬁcient for only one agent to reach it.

Revised: An adversary agent also tries to reach the target landmark, while the agents are penalized as it gets closer to the target.

Revised: The adversary, however, does not know which landmark is the target and must deduce it from the agents’ behavior.

Revised: For that reason, agents must cooperate to trick the adversary by learning to cover all the landmarks.

Revised: This task shows that our algorithm is applicable not only to cooperative interactions but to mixed environments as well.

Revised: In the cooperative communication and physical deception tasks, ALITA obtained the highest success rate across all baselines.

Revised: [Table]

Revised: [Table]

Original: Since the formulation of F needs a shared buffer, SAC and TD3 stand as the best off-policy candidates to be incorporated with our framework, as they have shown great performances on many benchmarks.
Revised: Since the formulation of F needs a shared buffer [Table] SAC and TD3 stand as the best off-policy candidates to be incorporated with our framework

Revised: [Table]

Revised: as they have shown great performances on many benchmarks.

Original: SAC, however, uses stochastic policies in general which makes it infeasible to combine with the formulation of r int .
Revised: SAC, however, uses stochastic policies in general which makes it infeasible to combine with the formulation of r int .

Original: Therefore, we use TD3 as our learning model to measure its performance on a suite of PyBullet (Tan et al., 2018) continuous control tasks, interfaced through OpenAI Gym (Brockman et al., 2016).
Revised: Therefore [Table] we use TD3 as our learning model to measure its performance on a suite of PyBullet (Tan et al., 2018) continuous control tasks, interfaced through OpenAI Gym (Brockman et al., 2016).

Original: While many previous works utilized the Mujoco (Todorov et al., 2012) physics engine to simulate the system dynamics of these tasks, we found it better to evaluate our method on benchmark problems powered by PyBullet simulator since it is widely reported that
Revised: While many previous works utilized the Mujoco (Todorov et al., 2012) physics engine to simulate the system dynamics of these tasks, we found it better to evaluate our method on benchmark problems powered by PyBullet simulator since it is widely reported that

Original: PyBullet problems are harder to solve (Tan et al., 2018) when compared to Mujoco.
Revised: PyBullet problems are harder to solve (Tan et al., 2018) when compared to Mujoco.

Original: Also, Pybullet is license-free, unlike Mujoco that is only available to its license holders.
Revised: Also, Pybullet is license-free, unlike Mujoco that is only available to its license holders.

Original: [Figure]
Revised: [Figure]

Original: We compare our method to the original twin delayed deep deterministic policy gradients (TD3); soft actor critic (SAC) (Haarnoja et al., 2018); proximal policy optimization (PPO) (Schulman et al., 2017), a stable and efﬁcient on-policy policy gradient algorithm; deep deterministic policy gradient (DDPG); trust region policy optimization (TRPO) (Schulman et al.,2015);
Revised: We compare our method to the original twin delayed deep deterministic policy gradients (TD3) (Fujimoto et al., 2018); soft actor critic (SAC) (Haarnoja et al., 2018); proximal policy optimization (PPO) (Schulman et al., 2017), a stable and efﬁcient on-policy policy gradient algorithm; deep deterministic policy gradient (DDPG); trust region policy optimization (TRPO) (Schulman et al.,

Original:  Tsallis actor-critic (TAC) (Chen & Peng, 2019), a recent off-policy algorithm for learning maximum entropy policies, where we use the implementation of the authors 4 5 ; and Actor-Critic using Kronecker-Factored Trust Region (ACKTR) (Wu et al., 2017), as implemented by OpenAI’s baselines repository 6 .
Revised: ; Tsallis actor-critic (TAC) (Chen & Peng, 2019), a recent off-policy algorithm for learning maximum entropy policies, where we use the implementation of the authors 4 5 ; and Actor-Critic using Kronecker-Factored Trust Region (ACKTR) (Wu et al., 2017), as implemented by OpenAI’s baselines repository 6 .

Original: Each task is run for at least 1 million time steps and the average return of 15 episodes is reported every 5000 time steps.
Revised: Each task is run for at least 1 million time steps and the average return ofepisodes is reported every 5000 time steps.

Original: To enable reproducibility, each experiment is conducted on 10 random seeds of Gym simulator and network initialization.
Revised: To enable reproducibility, each experiment is conducted on 10 random seeds of Gym simulator and network initialization.

Original: Results of the best performing agent of the two across different methods are reported in Figure (5).
Revised: Results of the best performing agent of the two across different methods are reported in Figure (5).

Original: C T RAINING DETAILS C. G ENERAL C ONFIGURATIONS
Revised: D T RAINING DETAILS D. G ENERAL C ONFIGURATIONS

Original: We use a buffer-size of 10 6 entries and a batch-size of 1024 .
Revised: We use a buffer-size of 10 6 entries and a batch-size of 1024 .

Original: We collect 100 transitions by interacting with the environment for each learning update.
Revised: We collect 100 transitions by interacting with the environment for each learning update.

Original: For all tasks in our hyper-parameter searches, we train the agents for 15 , 000 episodes of 100 steps and then re-train the best conﬁguration for each algorithm-environment pair for twice as long ( 30 , 000 episodes) to ensure full convergence for the ﬁnal evaluation.
Revised: For all tasks in our hyper-parameter searches, we train the agents for 15 , 000 episodes of 100 steps and then re-train the best conﬁguration for each algorithm-environment pair for twice as long ( 30 , 000 episodes) to ensure full convergence for the ﬁnal evaluation.

Original: We use a discount factor γ of 0 .
Revised: We use a discount factor γ of 0 .

Original: 95 , an inﬂuence importance temperature β of 0 . 1 , and a gradient clipping threshold of 0 .
Revised: 95 , an inﬂuence importance temperature λ of 0 . 1 , and a gradient clipping threshold of 0 .

Original: 5 in all experiments unless otherwise speciﬁed.
Revised: 5 in all experiments unless otherwise speciﬁed.

Original: Each cloned critic is updated 4 time per step.
Revised: Each cloned critic is updated 4 time per step.

Original: C. S PARSE P USH B OX AND S PARSE S ECRET R OOM , MAPE, & G YM
Revised: D. S PARSE P USH B OX AND S PARSE S ECRET R OOM , MAPE, & G YM

Original: We use the Adam optimizer (Kingma & Ba, 2014) to perform parameter updates.
Revised: We use the Adam optimizer (Kingma & Ba, 2014) to perform parameter updates.

Original: All models (actors, critics and proxy critics) are parametrized by feedforward networks containing two hidden layers of 128 units excpet for the autoencoder network where we use 7 hidden layers with dimensions (128, 64, 12, 3, 12, 64, 128), respectively.
Revised: All models (actors, critics and proxy critics) are parametrized by feedforward networks containing two hidden layers of 128 units excpet for the autoencoder network where we use 7 hidden layers with dimensions (128, 64, 12, 3, 12, 64, 128), respectively.

Original: All models’ parameters are initialized using Glorot Initialization method (Glorot & Bengio, 2010); while the autoencoder’s parameters are initialized using Kaiming method (He et al., 2015).
Revised: All models’ parameters are initialized using Glorot Initialization method (Glorot & Bengio, 2010); while the autoencoder’s parameters are initialized using Kaiming method (He et al., 2015).

Original: We employ the Rectiﬁed Linear Unit (ReLU) as activation function and layer normalization (Ba et al., 2016) on the pre-activations unit to stabilize the learning.
Revised: We employ the Rectiﬁed Linear Unit (ReLU) as activation function and layer normalization (Ba et al., 2016) on the pre-activations unit to stabilize the learning.

Original: [Table]
Revised: [Table]

Original: . SMAC
Revised: D. SMAC

Original: The architecture of all agent networks is a DRQN (Hausknecht & Stone, 2015) with a recurrent layer comprised of a GRU with a 64 -dimensional hidden state, with a fully-connected layer before and after.
Revised: The architecture of all agent networks is a DRQN (Hausknecht & Stone, 2015) with a recurrent layer comprised of a GRU with a 64 -dimensional hidden state, with a fully-connected layer before and after.

Original: All neural networks are trained using RMSprop ( α = 0 .
Revised: All neural networks are trained using RMSprop ( α = 0 .

Original: 99 with no weight decay or momentum) with learning rate 5 × 10 − 4 .
Revised: 99 with no weight decay or momentum) with learning rate 5 × 10 − 4 .

Original: [Table]
Revised: [Table]

Original: https