C OMMUNICATION -E FFICIENT A CTOR -C RITIC M ETHODS FOR H OMOGENEOUS M ARKOV G AMES
A BSTRACT
Recent success in cooperative multi-agent reinforcement learning (MARL) relies on centralized training and policy sharing.
Centralized training eliminates the issue of non-stationarity MARL yet induces large communication costs and policy sharing is empirically crucial to efﬁcient learning in certain tasks yet lacks theoretical justiﬁcation.
In this paper, we formally characterize a subclass of cooperative Markov games where agents exhibit a certain form of homogeneity such that policy sharing provably incurs no suboptimality.
This enables us to develop the ﬁrst consensus-based decentralized actor-critic method where the consensus update is applied to both the actors and the critics while ensuring convergence.
We also develop practical algorithms based on our decentralized actor-critic method to reduce the communication cost during training, while still yielding policies comparable with centralized training.
I NTRODUCTION
Cooperative multi-agent reinforcement learning (MARL) is the problem where multiple agents learn to make sequential decisions in a common environment to optimize a shared reward signal, which ﬁnds a wide range of real-world applications such as trafﬁc control (Chu et al., 2019), power grid management (Callaway & Hiskens, 2010), and coordination of multi-robot systems (Corke et al., 2005).
Efﬁcient learning for large and complex cooperative MARL tasks is challenging.
Naively reducing cooperative MARL to single-agent RL with a joint observation-action space imposes signiﬁcant scalability issues, since the joint space grows exponentially with the number of agents.
Approaches that treat each agent as an independent RL learner, such as Independent Q-Learning (Tan, 1993), overcome the scalability issue yet fail to succeed in complicated tasks due to the nonstationarity caused by other learning agents’ evolving policies.
To address these challenges, the paradigm of Centralized Training and Decentralized Execution (CTDE) is then proposed, where a centralized trainer is assumed to access to information of all agents during training to approximate the global (action-)value function, whereas each agent only needs local information for its action selection during decentralized policy execution (Lowe et al., 2017;
Foerster et al., 2017).
The centralized critic eliminates non-stationarity during training, while the policy decentralization ensures scalability during execution.
Besides, existing CTDE methods almost always enable policy parameter sharing to further improve learning scalability and efﬁciency, where agents also share the parameters of their decentralized policies.
However, in many real-world scenarios, there is not a readily available centralizer that conveniently gathers the global information from all agents, and therefore agents need to rely on all-to-all communication for centralized training, incurring enormous communication overheads for large numbers of agents.
This motivates us to think about whether it is possible to train agents in a decentralized and communication-efﬁcient manner, while still keeping the beneﬁts of the centralized training of
CTDE.
Moreover, despite its wide adoption, little theoretical understanding has been provided to justify policy parameter sharing.
Agents should at least exhibit a certain level of homogeneity before it is feasible to share their policies.
For example, if the observation and/or action spaces vary across agents, then their decentralized policies cannot even have the same architecture.
Even if it is feasible, it is unclear whether restricting the agents to share their policy parameters will introduce any suboptimality.
In this paper, we address these aforementioned issues centered around the CTDE framework.
We begin by formally characterizing a subclass of Markov games where the cooperative agents exhibit a certain form of homogeneity such that it is not only feasible but also incurs no suboptimality to share their decentralized policies, thus providing a ﬁrst theoretical justiﬁcation for policy parameter sharing.
We then develop a decentralized actor-critic algorithm for homogeneous MGs where agents share their critic and actor parameters with consensus-based updates, for which we prove an asymptotic convergence guarantee with linear critics, full observability, and other standard assumptions.
To our knowledge, this is the ﬁrst decentralized actor-critic algorithm that enjoys provable convergence guarantees with policy (i.e., actor) consensus.
To account for communication efﬁciency, we develop a simple yet effective bandit-based process that wisely selects when and with whom to perform the parameter census update based on the feedback of policy improvement during training.
To further account for partial observability, we develop an end-to-end learnable gating mechanism for the agents to selectively share their observations and actions for learning the decentralized critics.
This series of innovations are capable of transforming any CTDE algorithm into its decentralized and communication-efﬁcient counterpart, with policy consensus in homogeneous MGs for improved efﬁciency.
Our empirical results demonstrate the effectiveness of these innovations when instantiated with a state-of-the-art CTDE algorithm, achieving competitive policy performance with only a fraction of communication during training.
Our contribution is therefore summarized as three-fold: (1) the characterization of a subclass of cooperative Markov games, i.e. homogeneous Markov games (MGs), where policy sharing provably incurs no loss of optimality; (2) a decentralized MARL algorithm for homogeneous MGs that enjoys asymptotic convergence guarantee with policy consensus; and (3) practical techniques that transform
CTDE algorithms to their decentralized and communication-efﬁcient counterparts.
R ELATED W ORK
Communication in cooperative MARL.
Communication is key to solving the issue of nonstationarity in cooperative MARL.
The CTDE paradigm (Lowe et al., 2017; Foerster et al., 2017) assumes a centralized unit during training to learn a joint value function.
Other methods, such as
CommNet (Sukhbaatar et al., 2016) and BiCNet (Peng et al., 2017), do not assume a centralized unit and instead allow agents to share information by all-to-all broadcasting, effectively relying on centralized communication.
These methods require centralized/all-to-all communication that impedes their application to large numbers of agents.
Although follow-up work such as IC3Net (Singh et al., 2018) and VBC (Zhang et al., 2019) proposes algorithms to learn when to communicate, agents there still perform all-to-all communication before others decide whether to receive.
We instead entirely abandon centralized/all-to-all communication, letting each agent decide whom to communicate to purely based on its local observation.
There is another line of work, Networked MARL (NMARL) (Zhang et al., 2018), that where agents lie on a predeﬁned network such that neighboring agents can freely communicate.
Our approach instead learns sparse communication that is dynamically adjusted during decentralized training, even if the predeﬁned network topology can be dense.
Policy parameter sharing and consensus.
Policy parameter sharing is widely adopted in MARL where agents share the same action space, yet it has not been theoretically justiﬁed except under the mean-ﬁeld approximation where the transition dynamics depends on the collective statistics of all agents and not on the identities and ordering of individual agents (Nguyen et al., 2017a;b; Yang et al., 2018).
A recent result from Kuba et al.
(2021) states that enforcing policy parameter sharing in a general cooperative MG can lead to a suboptimal outcome that is exponentially-worse with the number of agents.
We are the ﬁrst to 1) formally characterize the subclass of homogeneous MGs without the notion of mean-ﬁeld approximation, where enforcing policy parameter incurs no suboptimality and 2) develop an algorithm that performs policy parameter sharing in homogeneous MGs in a soft manner with decentralized consensus-based policy update with convergence guarantees.
Zhang &
Zavlanos (2019) also develop a policy consensus algorithm for decentralized MARL, yet they do not assume homogeneity and thus need each agent to represent the joint policy for consensus.
Communication-efﬁcient MARL.
There have been several recent works that also aim to achieve communication-efﬁciency in decentralized MARL.
Chen et al.
(2021b) use pre-speciﬁed commu- nication topology and reduce communication frequency for actor-critic via mini-batch updates; in contrast, our work adaptively learn sparse communication topology during the decentralized training process.
Chen et al.
(2021a) generalize their method of communication-efﬁcient gradient descent from distributed supervised learning (Chen et al., 2018) to distributed reinforcement learning with policy gradient methods, where they assume the existence of a centralized controller that gather the policy gradients from decentralized agents which only communicate when the change in gradient exceeds a predeﬁned threshold; in contrast, our method does not rely on a centralized controller, and we empirically demonstrate the beneﬁt of our adaptive communication learning over a rulebased baseline inspired by Chen et al.
(2021a).
Gupta et al.
(2020) learn discrete messages among agents with a ﬁxed communication topology, where the communicated messages are used to form the policy for action selection rather than for decentralized training.
H OMOGENEOUS M ARKOV G AME
We consider a cooperative Markov game (MG) (cid:104)N , S , A , P, R (cid:105) with N agents indexed by i ∈ N = { 1 , ..., N } , state space S , action space
A =
A 1 × · · · × A N , transition function P : S ×
A × S → [0 , 1] , and reward functions R
=
{ R i } i ∈N with R
i : S ×
A → R for each i ∈ N .
In Section 3, we assume full observability for simplicity, i.e., each agent observes the state s ∈ S .
Under full observability, we consider joint policies, π : S ×
A → [0 , 1] , that can be factored as the product of local policies
π
i : S ×
A i → [0 , 1] , π ( a | s ) = (cid:81)
i ∈N π i ( a i | s ) .
Let r ( s, a ) :
= 1 N (cid:80) i ∈N R i ( s, a ) denote the joint reward function, and let γ ∈ [0 , 1] denote the discount factor.
Deﬁne the discounted return from time step t as G t =
(cid:80) ∞
l =
γ l r t + l , where r t := r ( s t , a t ) is the reward at time step t .
The agents’ joint policy π = ( π 1 , ..., π N ) induce a value function, which is deﬁned as V π ( s t ) =
E s t +1: ∞ ,a t : ∞ [ G t | s t ] , and action-value function Q π ( s t , a t ) =
E s t +1: ∞ ,a t +1: ∞ [ G t | s t , a t ] .
The agents are cooperative in the sense that they aim to optimize their policies with respect to the joint reward function, i.e., max π J ( π ) =
E s 0: ∞ ,a 0: ∞
[ G 0 ] .
3. H OMOGENEOUS MG: D EFINITION , P ROPERTIES , AND E XAMPLES
As along as the action spaces {A i } i ∈N are homogeneous, policy sharing among { π i } i ∈N is feasible.
However, such policy sharing can incur suboptimal joint policies for general MGs, as we will see in an example introduced by Kuba et al.
(2021) and revisited in this subsection.
Here, we characterize a subclass of Markov games in Deﬁnition 1 requiring conditions stronger than homogeneous action spaces, where policy sharing provably incurs no suboptimality.
Deﬁnition 1 (Homogeneous Markov game) .
Markov game (cid:104)N , S , A , P, R (cid:105) is homogeneous if:
(i)
The local action spaces are homogeneous,
i
e., A i =
A j ∀ i, j ∈ N .
Further, the state is decomposed into local states with homogeneous local state spaces,
i
e., s =
( s 1 , ...,
s N ) ∈ S
= S 1 × · · · × S N with S i =
S j ∀ i, j ∈ N .
(ii)
The transition function and the joint reward function are permutation invariant.
Formally, for any s t = ( s 1 t , ..., s Nt ) , s t +
=
( s 1 t
+1 , ...,
s Nt +1 ) ∈ S and a t = ( a 1 t , ..., a Nt ) ∈ A , we have
P ( Ms t +1 | Ms t , Ma t ) =
P ( M (cid:48) s t +1 | M (cid:48)
s t , M (cid:48) a t ) ,
R ( Ms t , Ma t ) =
R ( M (cid:48) s t , M (cid:48) a t )
for any M, M (cid:48) ∈ M , where R ( s, a ) :
= ( R 1 ( s, a ) , ..., R N ( s, a )) , Mx denotes a permutation M of ordered list x = ( x 1 , ..., x N ) , and M is the set of all possible permutations.
(iii)
Each agent i ∈ N has access to a bijective function
o i : S
→ O (i
e., each agent has full observability) that maps states to a common observation space O .
These observation functions { o i } i ∈N are permutation preserving with respect to the state,
i
e., for any s ∈ S and any M ∈ M ,
[Equation]
By Deﬁnition 1, besides requiring homogeneous action spaces, our characterization of homogeneous
MGs further requires that the global state can be factored into homogeneous local states (condition (i))
such that the transition and reward functions are permutation invariant (condition (ii)).
Moreover, condition (iii) requires each agent to have an observation function to form its local representation of the global state.
The main property of the homogeneous MG is that, after representing the global state with the observation functions, policy sharing incurs no suboptimality.
This is formally stated in Theorem 1.
Proofs of all the theorems are included in the appendix.
Theorem 1.
Let Π be the set of state-based joint policies, i.e., Π = { π = ( π 1 , ..., π N ) : π i :
S ×
A i → [0 , 1] } , and let Π o be the set of observation-based joint policies, i.e., Π o = {
π o = ( π 1 o , ..., π No ) : π io : O × A i → [0 , 1] } .
In homogeneous MGs, we have
[Equation]
To provide more intuition for homogeneous MGs, we here give an example from Multi-Agent Particle Environment (MPE) (Lowe et al., 2017) and a non-example from Kuba et al.
Appendix D provides more examples and non-examples show the generality of our homogeneous MG subclass.
Example: Cooperative Navigation.
In a Cooperative Navigation task in MPE, N agents move as a team to cover N landmarks in a 2D space.
The landmarks are randomly initialized at the beginning of an episode, and ﬁxed throughout the episode.
Under full observably where each agent can observe the information (locations and/or velocities) of all agents and landmarks, we can cast a
Cooperative Navigation task as a homogeneous MG by verifying the three conditions in Deﬁnition 1: (i)
The local state of agent i consists of its absolute location l i = ( l i x , l iy ) ∈ R 2 and its absolute velocity v i =
( v ix , v iy ) ∈ R 2 with respect to the common origin, as well as the absolute locations of all N landmarks, { c k = ( c kx , c ky ) } Nk =1 .
Therefore, the location state spaces are homogeneous, and the concatenation of all the location states preserves the global state of the task.
Since local action is the change in velocity, the local action spaces are also homogeneous.
(ii)
The transition function determines the next global state by the current state and all agents’ actions according to physics, and thus it is permutation invariant.
The reward function R
i determines the reward for agent i according to the distances between all the agents and the landmarks to encourage coverage, as well as penalties to discourage collisions if any, resulting in a permutation invariant joint reward function.
(iii)
In MPE, agents’ observations are based on relative , instead of absolute, locations and/or velocities of other agents and/or landmarks.
For Cooperative Navigation, such observations happen to deﬁne observation functions that are bijective and permutation preserving.
Speciﬁcally, function o i yields the observation for agent i that consists of its absolute location l i and velocity v i , the relative location l ij :
=
l j − l i and velocity v ij := v j − v i of other agents j ∈ N \ { i } , and the relative location c ik :=
c k − l i of all the landmarks k = 1 , .., N .
Non-Example: a stateless MG.
Kuba et al.
(2021) recently shows that enforcing policy parameter sharing is exponentially-worse than the optimality without such a restriction in the following stateless MG: Consider a cooperative MG with an even number of N agents, a state s ﬁxed as the initial state, and the joint action space { 0 , 1 } N , where 1) the MG deterministically transits from state s to a terminal state after the ﬁrst time step, and 2) the reward in state s is given by
R ( s, 0 1: N/ 2 , 1 1+ N/ 2: N ) = 1 and R ( s, a 1: N ) = 0 for all other joint actions.
It is obvious that the optimal value of this MG (in state s ) is 1 , while Kuba et al.
(2021) prove that the optimal value under policy parameter sharing is 1 / 2 N − 1 .
This MG is not a homogeneous MG: the agents are relying on the raw state to represent their policies, and therefore their observation functions are identity mappings o
i ( s ) = s , which is not permutation preserving and violates Deﬁnition 1(iii).
3. P OLICY C ONSENSUS FOR H OMOGENEOUS MG S
Theorem 1 theoretically justiﬁes the parameter sharing among the actors with observation-based representations, which enables us to develop the ﬁrst decentralized actor-critic algorithms with consensus update among local (observation-based) actors.
Formally, the critic class Q ( · , · ; ω ) is parameterized with ω to approximate the global action-value function Q π ( · , · ) .
Upon on-policy transition ( s t , a t , { r i t } i ∈N , s t +1 , a t +1 ) sampled by the current policy, the critic parameter ω
i for each agent i ∈ N is updated using its local temporal difference (TD) learning followed by a consensus update (Zhang et al., 2018):
[Equation]
where r it =
R i ( s t , a t ) is the local reward of agent i , δ it =
r it + γQ ( s t +1 , a t +1 ; ω it ) − Q ( s t , a t
[Equation]
ω it ) is the local TD error of agent i , β ω,t > 0 is the critic stepsize, and C ω,t =
[ c ω,t
( i, j )] i,j ∈N is the critic consensus matrix.
The observation-based actor for each agent i ∈ N is parameterized as π i ( a i |
o i ( s ); θ i ) with parameter θ i , which is updated based on the multi-agent policy gradient derived from the critic followed by a consensus update:
[Equation]
where the observation-based actor class π ( ·|· ; θ ) is assumed to be differentiable, β θ,t > 0 is the actor stepsize, and C θ,t
=
[ c θ,t ( i, j )] i,j ∈N is the actor consensus matrix.
Compared with existing decentralized actor-critic methods for cooperative MARL (e.g., (Zhang et al., 2018)), the subclass of homogeneous MGs in Deﬁnition 1 makes it possible to perform actor consensus (i.e., policy consensus) in Equation (2) that is not possible for general MGs.
Theoremstates the convergence of { ω it } and { θ it } generated by (1)(2) with the linear critic class and under standard assumptions on the stepsizes, consensus matrices, and stability.
Theorem 2.
Under standard assumptions for linear actor-critic methods with consensus update, with { ω it } and { θ it } generated from Equations (1) and (2), we have lim t ω it =
ω ∗ and lim t
θ it =
θ ∗ almost surely for any i ∈ N , where θ ∗ is a stationary point associated with update (2), and θ i ∗ is the minimizer of the mean square projected Bellman error for the joint policy parameterized by θ ∗ .
Theorem 2 and its proof generalize the results in Zhang et al.
(2018) to the case where not only the critics but also the actors perform the consensus update.
Please refer to Appendix B which provides the exact assumptions, the convergence points, and our proof.
P RACTICAL A LGORITHM
The convergence of our decentralized actor-critic algorithm in Section 3.2 relies on the assumptions of linear function approximators, full observability, and well-connected consensus.
In this section, we develop a practical algorithm that relaxes these assumptions and achieves communication efﬁciency.
Speciﬁcally, the decentralized actors and critics are represented by neural networks.
We consider the partial observability setting where the agents cannot directly observe the global state s t such that their observation functions { o i } are not bijective.
Further, similar to Network
MARL (Zhang et al., 2018), we assume the agents can communicate through a time-variant network
G t :=
( N , E t ) with vertex set N and directed edge set E t ⊆ { ( i, j ) : i, j ∈ N , i (cid:54) = j } .
Denote the neighbors of agent i at time step t as N it :=
{ j : ( i, j ) ∈
E t } .
With agents only partially observing the global state, we in Section 4.1 develop an architecture for the agents to learn to share their local observations and actions in a communication efﬁcient manner.
To achieve communication efﬁciency on the actor-critic parameter consensus, in Section 4.2 we develop an effective bi-level multi-armed bandit for the agents to learn to exchange the parameters only when it beneﬁts learning.
Appendix
E provides implementation details of our algorithm.
4. E FFICIENT OBSERVATION - ACTION COMMUNICATION
We primarily focus on the type of partial observability where the state is not fully observably by individual agents but jointly observable, i.e., the mapping from s t to { o i ( s t ) } i ∈N is bijective.
For example, this joint observability is satisﬁed in Cooperative Navigation, where all agents’ observations can determine the state.
Thus, each agent can use the observations and actions of its own as well as from its neighbors for its critic to approximate the global action-value.
To encourage communication efﬁciency, we propose an architecture, communication network , that selects a subset C it ⊆
N it for observation-action communication, such that agent i ’s critic becomes Q i ( { ( o kt , a kt ) } k ∈{ i }∪C it ) .
For the texts below, we abuse notation o it :
=
o i ( s t ) to denote the observation and omit the subscript of time step t when the context is clear.
Communication network.
The communication network C i of agent i outputs c ij ∈ { 0 , 1 } indicating whether to communicate with neighbor j ∈ N i , i.e., C i = { j ∈ N i : c ij = 1 } .
Speciﬁcally, we choose an L -layer graph convolutional networks (GCN) to implement C i , which can deal with arbitrary input size determined by |N
i | and achieve permutation invariance.
Speciﬁcally, the input to this GCN is a fully connected graph with one vertex ( o i , e ij ( o i )) per neighbor j ∈ N i , where e i j
( o i ) embeds information of neighbor j that can be extracted from o i .
For example, when agents’ identities are observable, e ij ( o i ) can be (the embedding of) the ID of neighbor j .
When identities are not observable, e ij ( o i ) preserves information speciﬁc to neighbor j , such as the physical distance from j to i in MPE.
The GCN’s last layer outputs the logits { l ij } from which { c ij } j ∈N i are sampled.
To enable differentiability of the sampling, we use the reparameterization trick Straight-Through
Gumbel-Softmax (Jang et al., 2016).
Actor and critic networks.
Our proposed communication network is compatible with any multiagent actor-critic architecture.
Our experiments mainly explore deterministic actors a i
=
π i ( o i ) .
Similar to the communication network, the critic network Q i ( { ( o k , a k ) } k ∈{ i }∪C i ) is also implemented by an L -layer GCN to deal with arbitrary input size and achieve permutation invariance, where the input to the ﬁrst layer is the fully connected graph with vertices { ( o k , a k ) } k ∈{ i }∪C i .
Training.
Critic Q
i directly guides agent i ’s actor update using the deterministic policy gradient.
Critic Q i itself is updated to minimize the TD loss L i TD =
E o t ,a t ,r t ,o t +1 [( Q it − y it ) 2 ] , where o t :=
( o 1 t , ..., o Nt ) is the joint observation, a t :=
( a 1 t , ..., a Nt ) is the joint action, Q it :=
Q i ( { ( o kt , a kt ) } k ∈{ i }∪C it ) is the abbreviated notation for the critic value of agent i at timestep t , and y it :
= r it + γQ it +1 is the TD target.
Due to the differentiability enabled by Gumbel-Softmax, the gradient can ﬂow from Q i to communication network C i .
Commonly used in the literature (Jang et al., 2016), the update of C i is guided by a regularization term α | 1 |C it | (cid:80) j ∈C it Softmax ( l ij ) − η | , which places a restriction on the amount of communication allowed deﬁned by rate η .
4. A BI - LEVEL BANDIT FOR PARAMETER CONSENSUS
The parameter consensus deﬁned in Equations (1)(2) requires each agent i to communicate with all other agents j where the ( i, j ) entry of the consensus matrix is non-zero.
To achieve communication efﬁciency, existing literature mainly considers gossip algorithms where each agent only communicated with one neighbor j per communication round (Boyd et al., 2006), i.e., the consensus matrix entries satisfy c
( i, j )
=
c ( i, i ) = 1 / 2 .
Here, we develop a novel bi-level multi-armed bandit to further improve communication efﬁciency over gossip algorithms, where at each round each agent chooses whether or not to perform consensus at the high level, and if yes, chooses which neighbor to perform gossip consensus update.
For ease of exposition, we assume that 1) the agents perform a gradient step and decide whether and how to perform parameter consensus every episode indexed by m = 1 , 2 , ... , and 2) every agent can always choose from all other agents for parameter consensus, i.e., N it =
N \ { i } .
Extensions to more general settings are straightforward.
We next formally describe this bi-level bandit for an arbitrary agent i , dropping the superscript i for convenience.
Arms.
The high-level is a 2 -armed bandit determining whether to perform consensus at each round m .
Denote the selected high-level arm at round m as x 1 m , where x 1 m = 0 , 1 corresponds to performing and not-performing consensus, respectively.
The low-level is a ( N − 1) -armed bandit, and we let x 2 m ∈ N \ { i } denote the selected low-level arm at round m .
Rewards.
We design different reward functions for the arms in the two levels, as they have different goals.
The high-level bandit aims to 1) reduce communication while 2) maintaining a reasonable performance of the learned consensus matrices.
Requirement 2) can be captured by the difference of the episodic rewards at different time steps, and requirement 1) can be measured by the frequency of selecting to perform consensus.
Let G m be the total rewards of episode m .
To compute the reward r 1 m for the high-level, we ﬁrst normalize G m using the latest l episodes high-level records to fulﬁll requirement 2), followed by the rewarding or penalizing depending on the sign of normalized G m to fulﬁll requirement 1), and ﬁnally mapped to to [ − 1 , 1] .
The low-level bandit only considers the performance of the learned policy, and the reward r 2 m can be computed by the normalization of G m using latest l episodes low-level records, then be mapped to [ − 1 , 1] .
Equation (3) shows the details.
[Equation]
if r m ≥
[Equation]
otherwise
[Equation]
The reward functions designed above are in nature non-stationary, since the agents are continuously updating their policies, which directly inﬂuence the values of the episodic reward difference.
Here we choose the adversarial bandit algorithm Exponentially
Weighted Average Forecasting (CesaBianchi & Lugosi, 2006) to learn the bi-level bandit.
E XPERIMENTS
Our experiments aim to answer the following questions in Sections 5.1-5.3, respectively: 1) How communication-efﬁcient is our algorithm proposed in Section 4 against baselines and ablations?
2) How empirically effective is policy consensus?
Speciﬁcally, compared with not using policy consensus, can policy consensus converge to better joint policies faster?
What are the qualitative properties of the learned communication rules?
Environments.
We evaluate our algorithm on three tasks in Multi-Agent Particle Environment (MPE) with the efﬁcient implementation by Liu et al.
(2020), each of which has a version with
N = 15 agents and another with N = 30 agents.
As described in Section 3.1, these MPE environments can be cast as homogeneous MGs provided full observability and the permutation preserving observation functions.
We set the communication range to be k = 10 < N nearest agents for all the environments to introduce partial observability.
The details of the observation functions in each environment are as follows.
Cooperative Navigation: There are 15 , 30 landmarks for N = 15 ,respectively.
The observation of an agent contains its own absolute location, the relative locations of the nearest 10 agents, and the relative location of the 11 nearest landmarks.
Cooperative Push: N cooperating agents are tasked to push a large ball to a target position.
There are 2 , 2 landmarks for
N = 15 , 30 respectively.
The observation of an agent contains its own absolute location, the relative locations of the nearest 10 agents, and the relative location of the 2 landmarks.
Predator-and-Prey:
N cooperating predators (agents) are tasked to capture p preys.
The preys are pre-trained and controlled by the environment.
There are 5 , 10 preys and 5 , 10 landmarks (blocks) for N = 15 ,respectively.
Each predator can see p = 3 , 5 preys, l = 3 , 5 landmarks, k = 10 , 10 other predators for N = 15 , 30 respectively.
The observation of an agent contains its own absolute location, the relative locations of the nearest k agents (predators), the nearest p preys, and the relative location of the l nearest landmark.
Baselines.
We use Permutation Invariant Critic (Liu et al., 2020), the state-of-the-art CTDE actorcritic algorithm on MPE, as our centralized training algorithm to derive the following decentralized training baselines.
Full-Communication employs all-to-all communication for both observationaction and parameters, i.e., at each time step, each agent receives the observations and actions from all other neighbors for its critic, as well as critic and policy parameters from all other neighbors for its consensus update.
Independent learning ( IL ) employs no communication, i.e., each agent uses its local observations and actions only for its critic and performs no consensus update.
Random selects at random 1) a subset of neighbors for observation-action sharing and 2) a single neighbor for gossip parameter consensus.
For parameter consensus, we consider a Rule-based baseline, where each agent saves a copy of the parameters from the latest communications with other agents.
When agent i considers doing communication at t , it calculates the l 1 norm of the parameters of Q i and ˆ Q j which is the latest copy of j ’s parameters it saves.
The norm serves as the score to rank the other agents.
Intuitively, the high parameter difference implies dramatic behavioral differences.
The agent j with the highest score is selected to do parameter consensus, and the copy of agent j ’s critic parameter is recorded by agent i .
For fair comparisons, the Random and Rule-based baselines incur the same communication cost (i.e., the fraction of neighbors for observation-action communication, and the frequency for parameter consensus) as learned by our bandit method.
5. C OMMUNICATION EFFICIENCY AGAINST BASELINES AND ABLATIONS
Figure 1 shows the learning curves comparing our communication-efﬁcient decentralized actor-critic algorithm described in Section 4 against the baselines of Full-Communication, IL, and Random, with the observation-action communication rate set as 50% .
The results clearly verify the importance of observation-action and parameter communication, as Full-Communication outperforms IL by signiﬁcant margins uniformly in all the tasks.
As will be conﬁrmed in Section 5.3, our algorithm complies with the given observation-action communication rate, and its bandit learning chooses to communicate with a neighbor for parameter consensus only roughly 95% less frequently than Full-
[Figure]
[Figure] Communication.
Our algorithm signiﬁcantly outperforms the Random baseline that uses the same amount of communication in almost all the tasks, and achieves performances comparable with FullCommunication in several tasks.
Remarkably, Random barely outperforms IL in Cooperative Push, suggesting that efﬁcient communication for decentralized training is challenging.
Ablation: parameter communication.
We perform ablations on the three environments with
N = 15 agents.
Fix the observation-action communication pattern to be learned by the proposed communication network, we can see in Figure 2 that parameter consensus strategy learned by our bi-level bandit outperforms the Random parameter consensus baseline, the Rule-based parameter consensus strategy, and even the parameter consensus using Full-Communication in Predator-andPrey, and behaves comparably to parameter consensus using Full-Communication in Cooperative
Push, using random parameter consensus and Rule-based parameter consensus in Cooperative Navigation.
Noticeably, the Rule-based algorithm behaves similar to the Random parameter consensus baseline in the three scenarios.
A possible explanation is that the parameters of the other agents an agent records are outdated, as the Rule-based algorithm uses the same communication frequency (around 95%) learned by the bi-level bandit.
Another explanation is that parameter consensus harms exploration, and the balance of them cannot be handled by the Rule-based algorithm which only considers marinating homogeneous behaviors between agents.
Ablation: observation-action communication.
Figure 3 shows that increasing communications is not always helpful.
In particular, ﬁxing our communication network as the communication pattern, learning with 20% observation-action communication rate performs the worst among the three environments.
The one learned with 80% observation-action communication rate performs the best in
Cooperative Navigation, performs similarly to the one learned with 50% observation-action communication rate in Cooperative Push, and even performs worse in Predator-and-Prey than the one learned with learned with 50% observation-action communication rate.
5. E FFECTIVENESS OF POLICY CONSENSUS
Theorem 2 assures that policy consensus in distributed actor-critic methods for Homogeneous MGs provably converges to a local optimum, yet it remains open questions whether such a local is good
[Figure]
[Figure]
[Figure]
and whether the convergence is fast. Our experiments in this subsection empirically investigate these questions by comparing the learning curves with and without policy parameter consensus.
We separately consider 1) the setting where the agents employ all-to-all communication for their critics and parameter consensus, such that the assumptions of full observability and well-connected consensus matrices in Theorem 2 are satisﬁed, and 2) the setting where the agents learn to communicate efﬁciently with our communication network and the bandit.
The results in Figure 4 show that policy parameter consensus is always beneﬁcial with our communication-efﬁcient method and, surprisingly, it can negatively impact the training with all-to-all communication (e.g., Predator-and-Prey).
One plausible explanation is that, while it might speed up convergence, policy parameter consensus can harm exploration, leading to worse local optima
[Figure]
5. Q UALITATIVE ANALYSES OF THE LEARNED COMMUNICATION RULE
[Figure]
We ﬁrst qualitatively examine the observation-action communication rule learned by our communication network.
We plot the output of our communication network, i.e., the probabilities of communicating with the k = 10 distance-sorted neighbors, as the learning progresses.
Interestingly, the rule learned by our communication network encourages communicating with the nearest neighbors.
For example, in Cooperative Navigation with the 50% communication rate as shown in Figure 5, the probabilities of agents communicating with nearest 5 neighbors are over 75%, around 25% for the 6th nearest agent, around 0% for the other neighbors.
We provide the counterparts of Figure 5 for all the environments in Appendix C.
Figure 6 shows the probability of choosing the high-level arm of performing consensus as the learning progresses, averaged across the runs of all the environments.
The result shows that, with the designed bandit’s reward function, the average probability of selecting to communicate decreases from around 50% to less than 10%.
C ONCLUSION
In this paper, we characterize a subclass of cooperative Markov games where the agents exhibit a certain form of homogeneity such that policy sharing provably incurs no loss of optimality.
We develop the ﬁrst multi-agent actor-critic algorithm for homogeneous MGs that enjoys asymptotic convergence guarantee with decentralized policy consensus.
For practical usage, we propose techniques that can efﬁciently learn to communicate with other agents in exchange of observations, actions, and parameters.
The empirical results show that our proposed algorithm performs better than several baselines in terms of communication efﬁciency.
A P ROOF OF T HEOREM
The set of bijections { o i } i ∈N induces a one-to-one mapping between Π and Π o , and therefore the ﬁrst equality holds.
For the second equality, consider an arbitrary state s =
( s 1 , ..., s N ) ∈ S and the permutation M that swaps a pair of agents ( i, j ) , such that Ms = M ( ..., s i , ..., s j , ... )
= ( ..., ( Ms ) i =
s j , ..., ( Ms ) j
= s i , ... ) . .
Due to the permutation invariance of the transition and reward functions by condition (ii) of Deﬁnition 1, there exists an optimal state-based joint policy π ∗ ∈ Π such that π i ∗ ( ·| s ) =
π j ∗ ( ·| Ms ) .
Consider the corresponding optimal observation-based joint policy π ∗ o ∈ Π o that is the bijective mapping of π ∗ , such that π i ∗ o ( ·| o i ( s ))
=
π i ∗ ( ·| s ) and π j ∗ o ( ·| o j ( Ms )) =
π j ∗ ( ·| Ms ) .
We therefore have
[Equation] [Equation]
Further, since { o i } i ∈N are permutation preserving by condition (iii) of Deﬁnition 1, we have o i ( s ) =
o j ( Ms ) ∈ O in Equation (4).
Since Equation (4) holds for arbitrary s ∈ S and i, j ∈ N , and thus it follows that
[Equation]
5) i.e., the second equality holds.
This concludes the proof.
A. I LLUSTRATIVE EXAMPLE FOR THE PROOF
[Equation]
) 𝑠 $# = 𝑠 !
=
𝐿,∆ 1,2, 3,1,2 permute 𝑜 !
= ∆∇∆ 𝑜 " = ∇∆∆ 𝑜 # = ∆∆∇ 𝑜 $!
=
∇∆∆ = 𝑜 " 𝑜 $" =
∆∆∇ = 𝑜 # 𝑜 $# = ∆∇∆ = 𝑜 !
𝑜 ! ,𝑜 " ,𝑜 # = 𝑜′ # ,𝑜′ ! ,𝑜′ " Definition 1(iii)      𝜋 ∗&!
𝑜 ! =
↑ 𝜋 ∗&" 𝑜 " =↓ 𝜋 ∗&# 𝑜 # =↑ 𝜋 ∗&!
𝑜 $! =
↓= 𝜋 ∗&" 𝑜 " 𝜋 ∗&" 𝑜 $" =↑= 𝜋 ∗&# 𝑜 # 𝜋 ∗&# 𝑜 $# =↑= 𝜋 ∗&!
𝑜 !
𝜋 ∗&’ 𝑜 = 𝜋 ∗&( 𝑜 ∀(𝑖,𝑗,𝑜)
We here provide an illustrative example in Figure 7 to aid the proof.
The example Markov game consists of three agents i ∈ { 1 , 2 , 3 } placed on the left ( L ), middle ( M ), and right ( R ), one in each position, with a triangle placed in front of each agent that is either pointing up ( (cid:52) ) or down ( (cid:53) ).
The agents have homogeneous action spaces {↑ , ↓} .
The agents also have homogeneous local state spaces { L, M, R } × {(cid:52) , (cid:53)} , repenting its position and the shape in front.
The game ends after the ﬁrst timestep, and the agents share the following reward function: 1) if the number of the (cid:52) is even, reward is +1 when the agents behind (cid:52) choose to go up ( ↑ ) and the agents behind (cid:53) choose to go down ( ↓ ); 2) if the number of the (cid:52) is odd, reward is +1 when the agents behind (cid:52) choose to go down ( ↓ ) and the agents behind (cid:53) choose to go up ( ↑ ); 3) reward is 0 otherwise.
Thus, the Markov game satisﬁes Deﬁnition 1(i)(ii).
The agents’ local observations preserve their absolute positions (i.e., L , M , or R ) and consist the three shapes ordered clockwise starting from the shape right in front.
For example, the local observation of agent i = 1 on the left of Figure 7 is o
=
( L, (cid:52) (cid:53) (cid:52) ) .
As veriﬁed by the second row in Figure 7, the local observations are permutation preserving to satisfy Deﬁnition 1(iii).
Therefore, the Markov game is homogeneous by Deﬁnition 1.
The third row in Figure 7 veriﬁes Equations (4)(5) that are key to the proof.
B P ROOF OF T HEOREMB. A SSUMPTIONS
We make the following assumptions that are necessary to establish the convergence.
Assumption 1.
The Markov game has ﬁnite state and action spaces and bounded rewards.
Further, for any joint policy, the induced Markov chain is irreducible and aperiodic.
Assumption 2.
The critic class is linear, i.e., Q ( s, a ; ω ) =
φ ( s, a ) (cid:62) ω , where φ ( s, a ) ∈ R K is the feature of ( s, a ) .
Further, the feature vectors φ ( s, a ) ∈ R K are uniformly bounded by any ( s, a ) .
The feature matrix Φ ∈ R |S||A|× K has full column rank.
Assumption 3.
The stepsizes β ω,t and β θ,t satisfy
[Equation]
In addition, lim t β ω,t +1 β − 1 ω,t = 1 .
Assumption 4.
We assume the nonnegative matrices C t ∈ { C ω,t , C θ,t } satisfy the following conditions: (i) C t is row stochastic (i.e., C t 1 = 1 ) and E [ C t ] is column stochastic (i.e., 1 (cid:62) E
[ C t ] = 1 (cid:62) ) for all t > 0 ; (ii) The spectral norm of E [( W (cid:62) t ( I − 1 N 11 (cid:62) )
W t ] is strictly smaller than one; (iii) W t and ( s t , { r it } ) are conditionally independent given the σ -algebra generated by the random variables before time t .
Assumption 5.
The critic update is stable, i.e., sup t (cid:13)(cid:13) ω
it (cid:13)(cid:13) < ∞ , for all i .
For the actor update, { θ it } belongs to a compact set for all i and t .
B. C RITIC CONVERGENCE
In this subsection, we establish critic convergence under a ﬁxed joint policy in Lemma 3.
Speciﬁcally, given a ﬁxed joint policy π = ( π 1 , ..., π N ) , we aim to show that the critic update converges to ω π , which is the unique solution to the Mean Square Projected Bellman Error (MSPBE):
[Equation]
which also satisﬁes
[Equation]
where T π is the Bellman operator for π , Π is the projection operator for the column space of Φ , and
D π = diag
[ d π ( s, a ) : s ∈ S , a ∈ A ] for the stationary distribution d π induced by π .
Lemma 3.
Under the assumptions , for any give joint policy π , with distributed critic parameters ω it generated from Equation 1 using on-policy transitions ( s t , a t , r t , s t +1 , a t +1 ) ∼ π , we have lim t ω i t = ω
π almost surely (a.s.) for any i ∈ N , where ω π is the MSPBE minimizer for joint policy π .
Proof.
We use the same proof techniques as Zhang et al.
Let φ t = φ ( s t , a t ) , δ t =
[ δ 1 t , ..., δ Nt ] (cid:62) , and ω t =
[ ω 1 t , ...,
ω
Nt ] (cid:62) .
The update of ω t in Equationcan be rewritten in a compact form of ω t +
=
( C ω,t ⊗ I )( ω t + β ω,t y t ) where ⊗ is the Kronecker product, I is the K × K identity matrix, and y t =
[ δ 1 t φ (cid:62) t , ..., δ Nt φ (cid:62) t ] (cid:62) ∈ R KN .
Deﬁne operator (cid:104)·(cid:105) :
R KN → R K as (cid:104)
[Equation]
=
[Equation] N [Equation] ⊗ I )
[Equation]
= 1 N (cid:88) i ∈N ω i for any ω = [( ω 1 ) (cid:62) , ..., ( ω N ) (cid:62) ] (cid:62) ∈ R KN with ω i ∈ R K for any i ∈ N .
We decompose ω t into its agreement component 1 ⊗ (cid:104) ω t (cid:105) and its disagreement component ω ⊥ ,t := ω t − 1 ⊗ (cid:104) ω t (cid:105) .
To prove ω t = ω ⊥ ,t + 1 ⊗ (cid:
) ω
t (cid:105) a.s. −−→ 1 ⊗ ω π , we next show ω ⊥ ,t a.s. −−→ 0 and
(cid:104) ω
t (cid:105) a.s. −−→ ω
π respectively.
Convergence of ω ⊥ ,t a.s. −−→ 0 .
We ﬁrst establish that, for any M > 0 , we have
[Equation]
To show Equation 6, let {F t } be the ﬁltration of F t = σ ( r τ − 1 s τ , a τ , ω τ , C ω,τ − 1 ; τ ≤ t ) , J =
N ( 11 (cid:62) ⊗ I ) such that Jω t = 1 ⊗ (
cid:104) ω t (cid:105) , ( I − J ) ω t = ω ⊥ ,t .
The following facts about ⊗ will be useful:
[Equation]
This enables us to write ω ⊥ ,t +1 as ω ⊥ ,t +1 =( I − J ) ω t
[Equation]
=(
I
[Equation]
) [( C ω
[Equation]
t ⊗ I )( [Equation] t + β ω,t y t )] =( I
[Equation]
[Equation]
t
[Equation]
[Equation]
(cid:104)
[Equation]
t
[Equation]
+
[Equation] ⊥ ,t + β [Equation] ,t y t )] ( By Equation 7 and Assumption 4, we have ( C ω,t ⊗ I )( 1 ⊗ (cid:104) ω t (cid:105) ) =
( C ω,t 1 ) ⊗ ( I (cid:104)
ω t (cid:105) )
= 1 ⊗ (cid:104) ω t (cid:105) ) =(
I − J )
[ 1 ⊗ (
cid:104) ω t (cid:105) +
( C ω,t ⊗ I )( ω ⊥ ,t + β ω,t y t )]
=( I − J )
[( C ω,t ⊗ I )( ω ⊥ ,t + β ω,t y t )]
(( I − J )( 1 ⊗ (
cid:104) ω t (cid:105) )
= 0)
=[
( I − 11 (cid:62) /N )
⊗ I ]
[( C ω,t ⊗ I )( ω ⊥ ,t + β ω,t y t )]
( I − J
=
( I − 11 (cid:62) /N )
⊗ I )
=[
( I − 11 (cid:62) /N )
C ω,t ⊗ I ]( ω ⊥ ,t + β ω,t y t ) ( By Equation 7 ) .
We then have
[Equation]
x
[Equation]
=
[Equation]
A
[Equation]
x (cid:62) x ) ( C ω,t and ( r t , y t ) are independent conditioning on F t )
[Equation]
( where ρ is the spectral norm of E [ C (cid:62)
[Equation] [Equation]
( I
[Equation] [Equation]
/N
[Equation]
C ω,t ]) =
[Equation]
( By Cauchy-Schwarz |
[Equation]
u,
[Equation]
u
[Equation]
( Quantities are deterministic given F t )
[Equation]
Since E [ (cid:107) y t (cid:107) 2 |F t ] =
E [ (cid:80) i ∈N (cid:13)(cid:13)
δ it φ t (cid:13)(cid:13) 2 |F t ] with δ it =
r t + γφ (cid:62) t ω
it − φ (cid:62) t +1 ω
it , by Assumptionsand 2 the rewards r t and the features φ t are bounded, and thus we have that E [ (cid:107) y t (cid:107) 2 |F t ] is bounded on set { sup τ ≤ t (cid:107)
ω τ (cid:107) ≤ M } for any given M > 0 .
We can then following the proof of Lemma 5.
in Zhang et al.
and its sequel to show Equation 6 and conclude the step of ω ⊥ ,t a.s. −−→ 0 .
Convergence of (cid:104) ω t (cid:105) a.s. −−→ ω
π .
We write the update of (cid:104) ω t (cid:105) as
[Equation]
( By Equation 7 )
[Equation]
We rewrite the above update as
[Equation]
where
[Equation]
We can verify that the following conditions hold (with probability 1) regarding the update of (cid:104) ω t (cid:105) in
Equation 9:
E [ (cid:104) δ t (cid:105) φ t | F t ] is Lipschitz continuous in (cid:104) ω t (cid:105)
[Equation]
ξ t is a martingale difference sequence and satisﬁes E [ (cid:107)
[Equation]
t +1 (cid:107) 2 | F t ] ≤ K (1 + (cid:107) ω t (cid:107) 2 ) for some constant K ,
such that the conditions in Assumption B.1 of Zhang et al.
(2018) are satisﬁed (with probability 1) and the behavior of Equation 9 is related to its corresponding ODE (see Theorem B.2 in Zhang et al.
[Equation] [Equation]
Note that ( γP π − I ) has all eigenvalues with negative real parts, so does (Φ (cid:62)
D π
( γP π − I )Φ) since Φ is assumed to be full column rank.
Hence, the ODE is globally asymptotically stable, with its equilibrium satisfying
[Equation]
which is the MSPBE minimizer, i.e., (cid:104) ω (cid:105)
=
ω
π .
This concludes the step of (cid:104) ω t (cid:105)
a.s. −−→ ω π and the proof of Lemma 3.
B. A CTOR CONVERGENCE
In this subsection, we establish the convergence of actor update with critic parameters ω it in Equation 2 replaced with the critic convergence point established in Lemma 3.
Then, by the two-timescale nature of the algorithm, we establish the convergence of { ω it } and { θ it } generated by Equation 1 and
Equation 2.
Let θ =
[( θ 1 ) (cid:62) , ..., ( θ N ) (cid:62) ] (cid:62) and ω θ be the critic convergence point for joint policy parameterized by θ as established in Lemma 3.
Deﬁne
[Equation]
for an arbitrary θ .
We study the variant of Equation 2 where ω it is replaced by ω θ t :
[Equation]
which can be rewritten as
[Equation]
where y t,θ t =
[( A 1 t,θ t · ψ 1 t,θ t ) (cid:62) , ..., ( A Nt,θ t · ψ Nt,θ t ) (cid:62) ] (cid:62) .
Similar to the critic convergence, we make the decomposition θ t = θ ⊥ ,t + 1 ⊗ (cid:104)
θ t (cid:105) and then show θ ⊥ ,t a.s. −−→ 0 and convergence of (cid:104)
θ t (cid:105) respectively.
Convergence of θ ⊥ ,t a.s. −−→ 0 .
In light of the argument for ω ⊥ ,t a.s. −−→ 0 in the proof of Lemma 3, it sufﬁces to show that the the boundedness of y t,θ t .
Here, y t,θ t =
[( A 1 t,θ t · ψ 1 t,θ t ) (cid:62) , ..., ( A Nt,θ t · ψ Nt,θ t ) (cid:62) ] (cid:62) is bounded because 1)
A it,θ t =
Q ( s t , a t ; ω θ t ) is bounded since ω θ t is the MSPBE minimizer; (2) ψ it,θ t is bounded since by Assumption 5 it is a continuous function over a compact set.
Convergence of (cid:104) θ t (cid:105) .
We write the update of (cid:104) θ t (cid:105) in Equation 10 as
[Equation]
( By Equation 7 ) .
We rewrite the above update as
[Equation]
where
[Equation]
where F t = σ ( θ τ , τ ≤ t ) , π (cid:104) θ t (cid:105) is the joint policy where each individual policy is parameterized by (cid:104)
θ t (cid:105) .
Note that ξ t is a martingale difference sequence.
By Assumption 5 ξ t is bounded and further by Assumption 3 we have (cid:80) t (cid:107) β θ,t ξ t (cid:107) 2 < ∞ .
By arguments in the proof of Theorem 4.7 in Zhang et al.
, we can apply Kushner-Clark lemma and conclude that (cid:104) θ t (cid:105) converges almost sure to a point in the set of asymptotically stable equilibria of
[Equation]
[Table]
C V ISUALIZATION OF THE LEARNED COMMUNICATION RULE
[Figure]
D E XAMPLES AND N ON -E XAMPLES OF H OMOGENEOUS MG D. E XAMPLES
MPE tasks with homogeneous agents.
We have explained in Section 3.1 that the Cooperative
Navigation task in MPE is an example of homogeneous MG.
By repeating the same arguments, we can show that all other MPE tasks with homogeneous agents are example of homogeneous MG, including cooperative push and predator-and-prey as we have used for our experiments in Section 5.
Speciﬁcally, each agent’s observation contains its absolution location and velocity, as well as the relative location and/or velocity of other agents and environment objects (e.g., landmarks, the ball and the target position in cooperative push, preys).
For predator-and-prey, either the predators or the preys form a team of homogeneous agents.
SMAC scenarios with homogeneous ally units.
StarCraft Multi-Agent Challenge (SMAC) is another benchmark environment for cooperative MARL.
In a number of SMAC scenarios, the team of agents consists of ally units of a single unit type (e.g., Marines), and they are tasked to defeat an enemy team controlled by the environment, with examples including 3m, 8m, 25m, 8m vs 9m, 2m vs 1z, 6h vs 8z, etc.
Each ally unit (i.e., agent) observes the following attributes of both ally and enemy units: distance, relative x, relative y, health, shield, and unit type .
Thus, an SMAC scenario with homogeneous ally units is similar to MPE’s predator-and-prey in the sense how it satisﬁes the conditions in Deﬁnition 1.
Thus, SMAC scenarios with homogeneous ally units are homogeneous
MGs.
Team sports.
Sports with homogeneous players forming a team are homogeneous MGs, with examples including basketball, American football, soccer(associate football)/ice hockey excluding the goalkeeper.
In these team sports, players’ local views naturally forms their observations that satisfy Deﬁnition 1(iii).
Trafﬁc with homogeneous vehicles.
Trafﬁc consisting vehicles of the same type (e.g., the same car-following model) is an example of homogeneous MG.
Like team sports, vehicles’ local views naturally forms their observations that satisfy Deﬁnition 1(iii).
Unlike team sports, these vehicles are unnecessarily cooperative, but their reward functions are permutation invariant to satisfy Deﬁnition 1(ii).
Surveillance with drones.
Drone surveillance is an application of cooperative MARL, where a set of (homogeneous) drones is tasked to collectively monitor a ground area.
Since the drones’ objective is to cover the ground area, the task is analogous to MPE’s Cooperative Navigation to satisfy the conditions in Deﬁnition 1.
D. N ON -E XAMPLES
MPE tasks with heterogeneous agents.
As the counterpart of Cooperative Navigation with homogenous agents, Liu et al.
(2020) introduce heterogeneous navigation where half of the agents are small and fast and the other half are big and slow.
In such an MPE task, the transition function is not permutation invariant, and therefore it is not a homogeneous MG.
SMAC scenarios with heterogeneous ally units.
If the ally units in an SMAC scenario are of different types, then the transition function is not permutation invariant, and therefore it is not a homogeneous MG.
These SMAC scenarios include 2s3z, 3s5z, MMM2, etc.
Multi-Agent MuJoCo
In a MuJoCo task, a robot aims to learn an optimal way of motion.
MultiAgent MuJoCo
(Peng et al., 2020) controls each part of the robot with an agent, for example, a leg for a spider.
Since the parts of a robot are heterogeneous, Multi-Agent MuJoCo can violate condition (i) of Deﬁnition 1.
E I MPLEMENTATION D ETAILS E. A RCHITECTURE O VERVIEW
𝑄 𝑖 (𝐺𝐶𝑁) 𝑎𝑔𝑒𝑛𝑡𝑗 𝑎𝑔𝑒𝑛𝑡𝑘 𝑎𝑔𝑒𝑛𝑡𝑙 𝑎𝑔𝑒𝑛𝑡𝑞 𝑎𝑔𝑒𝑛𝑡𝑖 𝑜 𝑡 𝑘 ;𝑎 𝑡 𝑘 𝑜 𝑡 𝑙 ;𝑎 𝑡 𝑙 𝑜 𝑡 𝑖 ;𝑎 𝑡 𝑖 𝑎𝑔𝑒𝑛𝑡𝑗 𝑎𝑔𝑒𝑛𝑡𝑘 𝑎𝑔𝑒𝑛𝑡𝑙 𝒩 𝑡𝑖 = {𝑗,𝑘,𝑙} 𝑜 𝑡𝑖 ;𝑒 𝑘𝑖 (𝑜 𝑡𝑖 ) 𝑜 𝑡𝑖 ;𝑒 𝑗𝑖 (𝑜 𝑡𝑖 ) 𝑜 𝑡𝑖 ;𝑒 𝑙𝑖 (𝑜 𝑡𝑖 ) 𝐶 𝑖 (𝐺𝐶𝑁 + 𝐺𝑢𝑚𝑏𝑒𝑙 − 𝑆𝑜𝑓𝑡𝑚𝑎𝑥)
𝒞 𝑡𝑖 𝑗
=
𝒞 𝑡𝑖 𝑘 = 𝒞 𝑡𝑖 𝑙 = 𝒞 𝑡 𝑖 𝜋 𝑖 𝑜 𝑡 𝑖 𝑎𝑔𝑒𝑛𝑡𝑝
P SEUDOCODE
1: init { C i } Ni =1 , { Q i } Ni =1 , { π i } Ni =1 , memory , { records il } Ni =1 , { records ih } Ni =1 .
2: for episode m = 1 , 2 , . . .
, L do 3: for agent i = 1 , 2 , . . .
, N do 4: sample x 1 m from agent i ’s high-level bandit.
5: if x 1 m =
=
then 6: sample x 2 m from agent i ’s low-level bandit.
Agent i does parameter consensus with agent x 2 m .
8: end if 9: end for 10:
G m =11: for t = 1 , 2 , . .
, T do 12:
Get o t +1 , r t by interacting with the environment; Put o t , a t , o t +1 , r t into the memory.
G m ← r t + γG m 14: end for 15: if len(memory) > batch size then 16: for i = 1 , 2 , . . .
, N do 17: sample a batch from memory.
18: l i critic =(cid:46) Loss for agent i ’s critic and comm net.
19: for o it , a it , o it +1 , r it in batch[i] do 20:
C it ← C i ( o it ) , C it +1 ← C i
( o it +1 ) 21:
Q it ← Q
i ( { ( o kt , a kt ) } k ∈{ i }∪C it ); Q it +1 ← Q i ( { ( o kt +1 , a kt +1 ) } k ∈{ i }∪C
it +1 ) 22: y it :=
r it + γQ it + 23: l i critic ← l i critic
+ ( Q it − y it )
+ α | 1 |C it | (cid:80) j ∈C it Softmax ( l ij ) − η | 24: end for 25: l i critic ← l i critic / batch size 26:
Send l i critic to the Adam optimizer for the update.
27: end for 28: for i = 1 , 2 , . . .
, N do 29: sample a batch from memory.
30: l i actor =(cid:46) Loss for agent
i ’s actor.
31: for o it in batch[i] do 32: a it ← π i ( o it ) 33:
C it ← C i ( o it ) , C it +1 ← C i
( o it +1 ) 34:
Q it ← Q
i ( { ( o kt , a kt ) } k ∈{ i }∪C it ) 35: l i actor ← l i actor − Q it 36: end for 37: l i ← l i actor / batch size 38:
Send l i actor to the Adam optimizer for the update.
39: end for 40: for agent i = 1 , 2 , . .
. , N do 41:
Push G m into records ih .
Keep the latest l elements of records ih ; Compute r 1 m designed in (3) 43:
Update agent i ’s high-level bandit by r 1 m .
44: if x 1 m =
= 0 then 45:
Push G m into records il .
Keep the latest l elements of records il ; Compute r 2 m designed in (3) 47:
Update agent i ’s low-level bandit by the r 2 m .
48: end if 49: end for 50: end if 51: end for
E. H YPERPARAMETERS
[Table]
network
[Table]
searched in [
[Table] [Table]
F E XPERIMENTS ON A T OY E XAMPLE OF H OMOGENEOUS MG
[Figure]
Theorem 2 proves the asymptotic convergence of our decentralized actor-critic updates in Equations (1)(2) with actor consensus for homogeneous MGs, which generalizes the asymptotic convergence result without actor consensus (Zhang et al., 2018).
While the actor-critic updates converge asymptotically both with and without actor consensus, obtaining their convergence rates require non-trivial ﬁnite-time analysis that remains an open problem.
Here, we empirically compare the actor-critic updates in Equations (1)(2) with and without actor consensus on a toy example of homogeneous MG, leaving the ﬁnite-time analysis for future work.
The toy example.
We have provided the stateless MG in Kuba et al.
(2021) in Section 3.1 as a nonexample.
If we augment each agent i with a unique local state s i , then it is easy to verify that these local states satisfy Deﬁnition 1(ii) and it is ease to construct local observations, o i =
( s i , s 1 , .., s N ) , that satisfy Deﬁnition 1(iii), such that the MG becomes an example of homogeneous MG.
Results.
We deﬁne the unique local states by the trigonometric function, s i = cos( i − 1 N − 1 π ) , i = 1 , ..., N .
We use feature function φ ( s, a ) = concat[ { s i , one hot( a i ) } i ] for the linear critic, and parameterize the actor as a linear function of o i followed by softmax over the two actions.
The consensus matrix is 1 /N everywhere for both the critics and the actors.
For effective training, we 1) replace the sparse reward function with a denser one, R ( s, a ) :
= mean i : s i ≥ 0 { 1 [ a i = 1] } − mean i : s i < 0 { 1 [ a i = 1] } , such that the optimal joint action is a i = 1 for i ≤ N/ 2 and a i
=for i > N/ 2 that gets a reward of +1 , and 2) instead of using the decaying stepsizes as suggested in Assumption 3, which we found is not effective for training, we use the optimizer of Adam for adaptive learning rates.
Figure 10 show the results for N = 10 , 50 .
While both converge, actor consensus slightly improves the learning efﬁciency.
G E XPERIMENTS WITH V ARIOUS A MOUNTS O F C OMMUNICATION
[Figure]