A M AXIMUM M UTUAL I NFORMATION F RAMEWORK FOR M ULTI -A GENT R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper, we propose a maximum mutual information (MMI) framework for multi-agent reinforcement learning (MARL) to enable multiple agents to learn coordinated behaviors by regularizing the accumulated return with the mutual information between actions.
By introducing a latent variable to induce nonzero mutual information between actions and applying a variational bound, we derive a tractable lower bound on the considered MMI-regularized objective function.
Applying policy iteration to maximize the derived lower bound, we propose a practical algorithm named variational maximum mutual information multi-agent actor-critic (VM3-AC), which follows centralized learning with decentralized execution (CTDE).
We evaluated VM3-AC for several games requiring coordination, and numerical results show that VM3-AC outperforms other MARL algorithms in multi-agent tasks requiring coordination.
I NTRODUCTION
With the success of RL in the single-agent domain (Mnih et al. (2015); Lillicrap et al.
, MARL is being actively studied and applied to real-world problems such as trafﬁc control systems and connected self-driving cars, which can be modeled as multi-agent systems requiring coordinated control (Li et al.
; Andriotis & Papakonstantinou (2019)).
The simplest approach to MARL is independent learning, which trains each agent independently while treating other agents as a part of the environment.
One such example is independent Q-learning (IQL) (Tan (1993)), which is an extension of Q-learning to multi-agent setting.
However, this approach suffers from the problem of non-stationarity of the environment.
A common solution to this problem is to use fully-centralized critic in the framework of centralized training with decentralized execution (CTDE) (OroojlooyJadid & Hajinezhad (2019);
Rashid et al.
For example, MADDPG (Lowe et al.
(2017)) uses a centralized critic to train a decentralized policy for each agent, and COMA (Foerster et al.
(2018)) uses a common centralized critic to train all decentralized policies.
However, these approaches assume that decentralized policies are independent and hence the joint policy is the product of each agent’s policy.
Such non-correlated factorization of the joint policy limits the agents to learn coordinated behavior due to negligence of the inﬂuence of other agents (Wen et al.
de Witt et al.
Thus, learning coordinated behavior is one of the fundamental problems in MARL (Wen et al.
Liu et al.
In this paper, we introduce a new framework for MARL to learn coordinated behavior under CTDE.
Our framework is based on regularizing the expected cumulative reward with mutual information among agents’ actions induced by injecting a latent variable.
The intuition behind the proposed framework is that agents can coordinate with other agents if they know what other agents will do with high probability, and the dependence between action policies can be captured by the mutual information.
High mutual information among actions means low uncertainty of other agents’ actions.
Hence, by regularizing the objective of the expected cumulative reward with mutual information among agents’ actions, we can coordinate the behaviors of agents implicitly without explicit dependence enforcement.
However, the optimization problem with the proposed objective function has several difﬁculties since we consider decentralized policies without explicit dependence or communication in the execution phase.
In addition, optimizing mutual information is difﬁcult because of the intractable conditional distribution.
We circumvent these difﬁculties by exploiting the property of the latent variable injected to induce mutual information, and applying variational lower bound on the mutual information.
With the proposed framework, we apply policy iteration by redeﬁning value functions to propose the VM3-AC algorithm for MARL with coordinated behavior under CTDE.
R ELATED W ORK
Learning coordinated behavior in multi-agent systems is studied extensively in the MARL community.
To promote coordination, some previous works used communication among agents (Zhang & Lesser (2013);
Foerster et al.
; Pesce & Montana (2019)).
For example, Foerster et al.
(2016) proposed the DIAL algorithm to learn a communication protocol that enables the agents to coordinate their behaviors.
Instead of relying on communication, Jaques et al.
(2018) proposed social inﬂuence intrinsic reward which is related to the mutual information between actions to achieve coordination.
The purpose of the social inﬂuence approach is similar to our approach and the social inﬂuence yields good performance in social dilemma environments.
The difference between our algorithm and the social inﬂuence approach will be explained in detail and the effectiveness of our approach over the social inﬂuence approach will be shown in Section 6. Wang et al.
(2019) proposed an intrinsic reward capturing the inﬂuence based on the mutual information between an agent’s current actions/states and other agents’ next states.
In addition, they proposed an intrinsic reward based on a decision-theoretic measure.
Although they used the mutual information to enhance exploration, our approach focuses on the mutual information between simultaneous actions capturing policy correlation not inﬂuence.
Besides, they considered independent policies, whereas policies are correlated in our approach.
Some previous works considered correlated policies instead of independent policies.
For example,
Liu et al.
(2020) proposed explicit modeling of correlated policies for multi-agent imitation learning, and Wen et al.
(2019) proposed a recursive reasoning framework for MARL to maximize the expected return by decomposing the joint policy into own policy and opponents’ policies.
Going beyond adopting correlated policies, our approach maximizes the mutual information between actions which is a measure of correlation.
Our framework can be interpreted as enhancing correlated exploration by increasing the entropy of own policy while decreasing the uncertainty about other agents’ actions.
Some previous works proposed other techniques to enhance correlated exploration (Zheng & Yue (2018); Mahajan et al.
For example, MAVEN addressed the poor exploration problem of QMIX by maximizing the mutual information between the latent variable and the observed trajectories (Mahajan et al.
However, MAVEN does not consider the correlation among policies.
B ACKGROUND
We consider a Markov Game (Littman (1994)), which is an extention of Markov Decision Process (MDP) to multi-agent setting.
An N -agent Markov game is deﬁned by an environment state space S , action spaces for N agents
A 1 , · · · , A N , a state transition probability T : S ×
A ×S → [0 , 1] , where
A = (cid:81) {+N i =
+}
A i is the joint action space, and a reward function R : S ×
A → R .
At each time step t ,
Agent i executes action a it ∈
A i based on state s t ∈ S .
The actions of all agents a t =
( a 1 t , · · · , a Nt ) yield next state s t +1 according to T and yield shared common reward r t according to R under the assumption of fully-cooperative MARL.
The discounted return is deﬁned as R t = (cid:80) ∞
τ = t γ τ r τ , where γ ∈ [0 , 1) is the discounting factor.
We assume CTDE incorporating resource asymmetry between training and execution phases, widely considered in MARL (Lowe et al.
(2017); Iqbal & Sha (2018)
; Foerster et al.
Under CTDE, each agent can access all information including the environment state, observations and actions of other agents in the training phase, whereas the policy of each agent can be conditioned only on its own action-observation history τ
it or observation o it in the execution phase.
For given joint policy π = ( π 1 , · · · , π N ) , the goal of fully cooperative MARL is to ﬁnd the optimal joint policy π ∗ that maximizes the objective J ( π ) =
E π (cid:2) R 0 (cid:3) .
Maximum Entropy RL The goal of maximum entropy RL is to ﬁnd an optimal policy that maximizes the entropy-regularized objective function, given by
[Equation]
It is known that this objective encourages the policy to enhance exploration in the state and action spaces and helps the policy avoid converging to a local minimum.
Soft actor-critic (SAC), which is based on the maximum entropy RL principle, approximates soft policy iteration to the actor-critic method.
SAC outperforms other deep RL algorithms in many continuous action tasks (Haarnoja et al.
We can simply extend SAC to multi-agent setting in the manner of independent learning.
Each agent trains its decentralized policy using decentralized critic to maximize the weighted sum of the cumulative return and the entropy of its policy.
We refer to this method as Independent SAC (I-SAC).
Adopting the framework of CTDE, we can replace decentralized critic with centralized critic which incorporates observations and actions of all agents.
We refer to this method as multi-agent soft actor-critic (MA-SAC).
Both I-SAC and MA-SAC are considered as baselines in the experiment section.
T HE P ROPOSED M AXIMUM M UTUAL I NFORMATION F RAMEWORK
We assume that the environment is fully observable, i.e., each agent can observe the environment state s t for theoretical development in this section, and will consider partially observable environment for practical algorithm construction under CTDE in the next section.
Under the proposed MMI framework, we aims to ﬁnd the policy that maximizes the mutual information between actions in addition to cumulative return.
Thus, the MMI-regularized objective function for joint policy π is given by
[Equation]
[Equation]
where a it ∼ π i ( ·| s t ) and α is the temperature parameter that controls the relative importance of the mutual information against the reward.
As aforementioned, we assume decentralized policies and want the decentralized policies to exhibit coordinated behavior.
By regularization with mutual information in the proposed objective function (2), the policy of each agent is implicitly encouraged to coordinate with other agents’ policies without explicit dependency by reducing the uncertainty about other agents’ policies.
This can be seen as follows:
Mutual information is expressed in terms of entropy and conditional entropy as
[Equation]
If the knowledge of π i ( ·| s t ) does not provide any information about π j ( ·| s t ) , the conditional entropy reduces to the unconditional entropy, i.e., H ( π j ( ·| s t )
| π i ( ·| s t ))
=
H ( π j ( ·| s t )) , and the mutual information becomes zero.
Maximizing mutual information is equivalent to minimizing the uncertainty about other agents’ policies conditioned on the agent’s own policy, which can lead the agent to learn coordinated behavior based on the reduced uncertainty about other agents’ policies.
𝒔 𝒕+
𝒔 𝒕 𝒂 𝒕+𝟏𝒂 𝒕+ 𝒂 𝒕 𝒂 𝒕𝒔 𝒕+
𝒔 𝒕 𝒂 𝒕+𝟏𝒂 𝒕+ 𝒂 𝒕 𝒂 𝒕𝒛 𝒕+𝟏 𝒛 𝒕 the variable to the standard
However, direct optimization of the objective function (2) is not easy.
Fig.
1(a) shows the causal diagram of the considered system model described in Section 3 in the case of two agents with decentralized policies.
Since we consider the case of no explicit dependency, the two policy distributions can be expressed as π 1 ( a 1 t | s t ) and π 2 ( a 2 t | s t ) .
Then, for given environment state s t observed by both agents, π 1 ( a 1 t | s t ) and π 2 ( a 2 t | s t ) are conditionally independent and the mutual information
I ( π 1 ( ·| s t ); π 2 ( ·| s t ))
= 0 .
Thus, the MMI objective (2) reduces to the standard MARL objective of only the accumulated return.
In the following subsections, we present our approach to circumvent this difﬁculty and implement the MMI framework and its operation under CTDE.
4. I NDUCING M UTUAL I NFORMATION U SING L ATENT V ARIABLE
First, in order to induce mutual information among agents’ policies under the considered system causal diagram shown in Fig. 1(a), we introduce latent variable z t .
For illustration, consider the new diagram with latent variable z t in Fig. 1(b).
Suppose that the latent variable z t has a prior distribution p ( z t ) , and assume that both actions a 1 t and a 2 t are generated from the observed random variable s t and the unobserved random variable z t .
Then, the policy of Agent i is given by the marginal distribution π i ( ·| s t ) =
(cid:82)
z π i ( ·| s t , z ) p ( z ) dz marginalized over z .
With the unobserved latent random variable z , the conditional independence does not hold for a 1 t and a 2 t and the mutual information can be positive, i.e., I ( π 1 ( ·| s t ); π 2 ( ·| s t )) > 0 .
Hence, we can induce the mutual information between actions without explicit dependence by introducing the latent variable.
In the general case of N agents, we have π ( a 1 , · · · , a N | s ) =
E z
[ π 1 ( a 1 | s, z ) · · · π N ( a N | s, z )] .
Note that in this case we inject a common latent variable z into all agents’ policies.
4. V ARIATIONAL B OUND OF M UTUAL I NFORMATION
Even with non-trivial mutual information I ( π i ( ·| s t ); π j ( ·| s t )) , it is difﬁcult to directly compute the mutual information.
Note that we need the conditional distribution of a j t given ( a it , s t ) to compute the mutual information as seen in (4), but it is difﬁcult to know the conditional distribution directly.
To circumvent this difﬁculty, we use a variational distribution q ( a jt | a it , s t ) to approximate p ( a jt | a it , s t ) and derive a lower bound on the mutual information
I ( π i ( ·| s t ); π j ( ·| s t ))
=:
I ij ( s t ) as
[Equation]
where the inequality holds because KL divergence is always non-negative.
The lower bound becomes tight when q ( a jt | a it , s t ) approximates p ( a jt | a it , s t ) well.
Using the symmetry of mutual information, we can rewrite the lower bound as
[Equation]
Then, we can maximize this lower bound of mutual information by using the tractable approximation q ( a it | a jt , s t ) .
4. M ODIFIED P OLICY I TERATION
In this subsection, we develop policy iteration for the MMI framework.
First, we replace the original
MMI objective function (2) with the following tractable objective function based on the variational lower bound (5): [Equation] (cid:88)
[Equation]
log [Equation] s t ) (cid:17)(cid:35) , (6) where q ( a jt | a it , s t ) is the variational distribution to approximate the conditional distribution p ( a jt | a it , s t ) .
Then, we determine the individual objective function ˆ J i ( π i , q ) for Agent i as the sum of the terms in (6) associated with Agent i ’s policy π i or action a it , given by ˆ J i ( π i , q ) =
E π (cid:34) ∞ (cid:88) t = γ
[Equation]
+ [Equation] (cid:88)
[Equation]
(cid:104) [Equation] (cid:124) (cid:123)(cid:122) (cid:125) (
[Equation]
(cid:105)(cid:17)(cid:35) , (7) where β = αN is the temperature parameter.
Note that maximizing the term (a) in (7) implies that each agent maximizes the weighted sum of the policy entropy and the return, which can be interpreted as an extension of maximum entropy RL to multi-agent setting.
On the other hand, maximizing the term (b) with respect to π i means that we update the policy
π i so that
Agent j well predicts Agent i ’s action by the ﬁrst term in (b) and Agent i well predicts Agent j ’s action by the second term in (b).
Thus, the objective function (7) can be interpreted as the maximum entropy
MARL objective combined with predictability enhancement for other agents’ actions.
Note that predictability is reduced when actions are uncorrelated.
Since the policy entropy term H ( π i ( ·| s i )) enhances individual exploration due to maximum entropy principle (Haarnoja et al.
and the term (b) in (7) enhances predictability or correlation among agents’ actions, the proposed objective function (7) can be considered as one implementation of the concept of correlated exploration in
MARL (Mahajan et al.
Now, in order to learn policy π i to maximize the objective function (7), we modify the policy iteration in standard RL.
For this, we redeﬁne the state and state-action value functions for each agent as follows:
[Equation]
where q ( i,j ) ( a it , a jt , s t ) (cid:44) q ( a i t | a jt , s t ) q ( a jt | a i t , s t ) .
Then, the Bellman operator corresponding to V π i and Q π i is given by
[Equation]
where
[Equation]
In the policy evaluation step, we compute the value functions deﬁned in (14) and (15) by applying the modiﬁed Bellman operator T π repeatedly to any initial function Q 0 i .
Lemma 1.
(Variational Policy Evaluation).
For ﬁxed π and the variational distribution q , consider the modiﬁed Bellman operator T π in (16) and an arbitrary initial function Q
i : S ×
A → R , and deﬁne Q k +1 i
=
T [Equation] ki .
Then, Q ki converges to Q π i deﬁned in (15).
Proof .
See Appendix A.
In the policy improvement step, we update the policy and the variational distribution by using the value function evaluated in the policy evaluation step.
Here, each agent updates its policy and variational distribution while keeping other agents’ policies ﬁxed as follows: ( π ik
[Equation]
[Equation]
where a − i (cid:44) { a 1 , · · · , a N }\{ a i } .
Then, we have the following lemma regarding the improvement step.
Lemma 2.
(Variational Policy Improvement).
Let π inew and q new be the updated policy and the variational distribution from (30) in Appendix A.
Then, Q π inew ,π − i old i (
[Equation]
[Equation] old
[Equation]
, a ) for all ( s, a ) ∈ ( S × A ) .
Proof .
See Appendix A.
The modiﬁed policy iteration is deﬁned as applying the variational policy evaluation and variational improvement steps in an alternating manner.
Each agent trains its policy, critic and the variational distribution to maximize its objective function (7).
A LGORITHM C ONSTRUCTION
Summarizing the development above, we now propose the variational maximum mutual information multi-agent actor-critic (VM3-AC) algorithm, which can be applied to continuous and partially observable multi-agent environments under CTDE.
The overall operation of VM3-AC is shown in
Fig.
Under CTDE, each agent’s policy is conditioned only on local observation, and centralized critics are conditioned on either the environment state or the observations of all agents, depending on the situation (Lowe et al.
Let x denote either the environment state s or the observations of all agents ( o 1 , · · · , o N ) , whichever is used.
In order to deal with the large continuous state-action spaces, we adopt deep neural networks to approximate the required functions.
For Agent i , we parameterize the variational distribution with ξ i as q ξ i ( a j | a i , o i , o j ) , the state-value function with ψ i as V iψ i ( x ) , two action-value functions with θ i,
and θ i, 2 as Q
iθ i, 1 ( x , a ) , Q iθ i, 2 ( x , a ) , and the policy with φ i as π iφ i ( a | o i ) =
E z
[ π iφ i ( a | o i , z )] .
We assume normal distribution for the latent variable which plays a key role in inducing coordination among agents’ policies, i.e., z t ∼ N (0 , I ) , and further assume that the variational distribution is Gaussian distribution with constant variance σ 2 , i.e., q ξ i ( a j | a i , o i , o j )
=
N ( µ ξ i ( a i , o i , o j ) , σ 2 ) , where µ ξ i ( a i , o i , o j ) is the mean of the distribution.
Centralized Training As aforementioned, the policy is the marginalized distribution over the latent variable z , where the policies of all agents take the same z t generated from N (0 , I ) as an input variable.
We perform the required marginalization based on Monte Carlo numerical expectation as follows:
[Equation]
and we use L = 1 for simplicity.
The parameterized value functions, policy, and variational distributions are trained similarly to the training in SAC.
Due to space limitation, training detail and pseudo code are provided in Appendices B and C.
Decentralized Execution In the centralized training phase, we pick actions ( a 1 , · · · , a N ) by using
Monte Carlo expectation based on common latent variable z l generated from zero-mean Gaussian distribution, as seen in (13).
We consider two methods to achieve the same operation in the decentralized execution phase.
First, this can be done by making all agents have the same Gaussian random sequence generator and distributing the same seed to this random sequence generator only once in the beginning of the execution phase.
This eliminates the necessity of communication for sharing the latent variable.
In fact, this way of sharing z l can be applied to the centralized training phase too.
Second, we exploit the property of zero-mean Gaussian latent variable z l .
That is, we simply replace z l with zero vector with the matching dimensions in the decentralized execution phase.
This substitution method does not deteriorate the performance much as seen in Section 6 since the latent variable distribution is zero-mean Gaussian and zero has the highest density.
Thus, the proposed algorithm is fully operative under CTDE.
E XPERIMENT
In this section, we provide numerical results to evaluate VM3-AC.
We considered four baselines: 1) MADDPG (Lowe et al.
- an extension of DDPG with a centralized critic to train a [Figure] decentralized policy for each agent.
2) Multi-agent actor-critic (MA-AC) - a variant of VM3-AC ( β = 0) without the latent variable.
3) Multi-agent variational exploration (MAVEN) (Mahajan et al.
Similarly to VM3-AC, MAVEN introduced latent variable and variational approach for optimizing the mutual information.
However, MAVEN does not consider the mutual information between actions but consider the mutual information between the latent variable and trajectories of the agents.
Inﬂuence MOA (IMOA) also known as social inﬂuence (SI) (Jaques et al.
The
IMOA method models p ( a jt +1 | a it , s it ) , where agents j and i are inﬂuencee and inﬂuencer, respectively, and adds intrinsic reward given by the mutual information I ( a it ; a jt +
| s t ) between inﬂuencer’s current action and inﬂuencee’s next timestep action not the mutual information
I ( a it ; a jt | s t ) between two agents’ simultaneous actions, which is considered in our MMI framework.
Both MAVEN and IMOA are implemented on the top of MA-AC since we consider continuous action-space environments.
We evaluated the proposed algorithm and the baselines in three multi-agent environments with the varying number of agents: multi-walker (Gupta et al.
predator-prey (Lowe et al.
(2017)), and cooperative navigation (Lowe et al.
We modiﬁed the original environments to require further coordination among agents.
For example, we increased the size of the agent and collision reward in the cooperative navigation.
Hence, the agent should consider other agents more while achieving its goal.
The detailed setting of each environments is provided in Appendix D.
6. R ESULT
Fig.
3 shows the learning curves for the considered three environments with the different number of agents.
The y-axis denotes the average of all agents’ rewards averaged over 7 random seeds, and the x-axis denotes time step.
The hyperparameters including the temperature parameter β and the dimension of the latent variable are provided in Appendix E. As shown in Fig.
3, VM3-AC outperforms the baselines in the considered environments.
Especially, in the case of the multi-walker environment, VM3-AC has large performance gain.
This is because the agents in the multi-walker environment are required especially to learn coordinated behavior to obtain high rewards.
In addition, the agents in the predator-prey environment, where the number of agents is four, should spread out in groups of two to get more reward.
In this environment, VM3-AC also has large performance gain.
Thus, it is seen that the proposed MMI framework improves performance in complex multi-agent tasks requiring high-quality coordination.
It is observed that both MAVEN and IMOA outperform the basic algorithm MA-AC but not VM3-AC.
Hence, the mutual information between the latent variable and trajectory (used in MAVEN) and the mutual information between the action of the agent and the next time step action of other agents (used in IMOA) are not as effective for coordinated behavior as the mutual information between agents’ simultaneous actions used for VM3-AC.
6. A BLATION S TUDY
In this section, we provide ablation study on the major techniques and hyperparameter of VM3-AC: 1) mutual information versus entropy 2) the latent variable, 3) the temperature parameter β , and 4) injecting zero vector instead of the latent variable z to policies in the execution phase.
Mutual information versus entropy: The proposed MMI framework maximizes the sum of entropy and variational conditional probability, which provides a lower bound of mutual information between actions.
As aforementioned, maximizing the entropy and variational conditional probability enhance exploration and predictability for other agents’ actions, respectively.
Hence, the proposed MMI framework enhances correlated exploration among agents.
We compared VM3-AC with MA-SAC, which is an extension of maximum entropy RL to multi-agent setting.
We performed an experiment in the predator-prey environment with four agents where the number of required agents to catch the prey is two.
In this environment, the agents started at the center of the map.
Hence, the agents should spread out in the group of two to catch preys efﬁciently.
Fig.4 shows the positions of the four agents at ﬁve time-steps after the episode starts.
The ﬁrst row and the second row in Fig.4 show the results of VM3-AC and MA-SAC in the early stage of the training, respectively.
It is seen that the agents of
VM3-AC explore in the group of two while the agents of MA-SAC tend to explore independently.
We provided the performance comparisons of VM3-AC with MA-SAC in Fig.5 (a) and (b).
Latent variable: The role of the latent variable is to induce mutual information among actions and promote coordinated behavior.
We compared VM3-AC and VM3-AC without the latent variable (implemented by setting dim ( z ) = 0 ) in the multi-walker environment.
In both cases, VM3-AC yields better performance that VM3-AC without the latent variable as shown in Fig.5(a) and 5(b). [Table]
Injecting zero vector instead of the latent variable: As mentioned in Section 5, we replace the latent variable with zero vector to execute actions without communication in the execution phase.
We compared the performance of decentralized policies which use zero vector and decentralized policies which use the latent variable assuming communication.
We used deterministic evaluation based on 20 episodes generated by the corresponding deterministic policy, i.e., each agent selects action using the mean network of Gaussian policy π i φ i .
We averaged the return over 7 seeds, and the result
[Figure] is shown in Table 3. It is seen that the zero vector replacement method yields almost the same performance and enables fully decentralized execution without performance loss. Temperature parameter β : The role of temperature parameter β is to control the relative importance between the reward and the mutual information.
We evaluated VM3-AC by varying β
=
15] in the multi-walker environment with N = 3 and N = 4 .
Fig. 5(c) and 5(d) show that VM3-AC with the temperature value around [0 . 05 , 0 . 1] yields good performance.
C ONCLUSION
In this paper, we have proposed the MMI framework for MARL to enhance multi-agent coordinated learning under CTDE by regularizing the cumulative return with mutual information among actions.
The MMI framework is implemented practically by using a latent variable and variational technique and applying approximate policy iteration.
Numerical results show that the derived algorithm named VM3-AC outperforms other baselines, especially in multi-agent tasks requiring high coordination among agents.
Furthermore, the MMI framework can be combined with other techniques for cooperative MARL, such as value decomposition (Rashid et al.
to yield better performance.
A PPENDIX A: V ARIATIONAL POLICY EVALUATION AND POLICY IMPROVEMENT
In the main paper, we deﬁned the state and state-action value functions for each agent as follows:
[Equation]
Lemma 3.
(Variational Policy Evaluation).
For ﬁxed π and the variational distribution q , consider the modiﬁed Bellman operator T π in (16) and an arbitrary initial function Q
i : S ×
A → R , and deﬁne Q k +1 i
=
T π Q ki .
Then, Q ki converges to Q π i deﬁned in (15).
[Equation]
where
[Equation]
Proof.
Deﬁne the mutual information augmented reward as T π Q i ( s t , a t ) =
=
r
[Equation]
t
[Equation]
γE
[Equation]
(18) =
r
[Equation]
t
[Equation]
a t )
+ γE
[Equation]
1 ∼ p
[Equation]
(19) + γE s t +1 ∼ p, a t +1 ∼ π (cid:34)
Q i (
[Equation]
t +
[Equation]
a t +1 ) (cid:35) (20) =
r π ( s t , a t ) + γE s t +1 ∼ p, a t +1 ∼ π (cid:34)
[Equation]
(cid:35) (21)
Then, we can apply the standard convergence results for policy evaluation.
Deﬁne
[Equation]
(22) for v =
[ Q ( s, a )] s ∈S , a ∈A .
Then, the operator T π is a γ -contraction.
[Equation]
[Equation]
Note that the operator T π has an unique ﬁxed point by the contraction mapping theorem, and we deﬁne the ﬁxed point as Q π i ( s, a ) .
Since
[Equation]
27) we have
[Equation]
and this implies
[Equation]
We proved the variational policy evaluation in the ﬁnite state-action sets.
We can expand it to the inﬁnite state-action sets by assuming follows:
• Assume that Q functions for π are in L inﬁnity • From Folland (1999), L inﬁnity is Banach space • From Agarwal et al.
(2018), by Banach ﬁxed point theorem, Q function should be converge to a unique point in L inﬁnity space and that is the
Q
function of given π
Lemma 4.
(Variational Policy Improvement).
Let π inew and q new be the updated policy and the variational distribution from (30).
Then, Q π inew ,π − i old i
[Equation]
) for all ( s, a ) ∈ ( S × A ) .
[Equation]
Proof.
Let π new be determined as
[Equation]
Then, the following inequality is hold
[Equation]
[Equation]
A PPENDIX B: A LGORITHM C ONSTRUCTION 7. C ENTRALIZED T RAINING
The value functions V iψ i ( x ) , Q iθ i ( x , a ) are updated based on the modiﬁed Bellman operator deﬁned in (16) and (17).
The state-value function V iψ i ( x ) is trained to minimize the following loss function:
[Equation]
where
[Equation]
(cid:35) , D is the replay buffer that stores the transitions ( x t , a t , r t , x t +1 ) , and Q imin ( x t , a it ) =
min [ Q iθ i, 1 ( x t , a it ) , Q iθ i, 2 ( x t , a it )] is the minimum of the two action-value functions to prevent the overestimation problem Fujimoto et al.
The two action-value functions are updated by minimizing the loss
[Equation]
where
[Equation]
and V ψ i is the target value network, which is updated by the exponential moving average method.
We implement the reparameterization trick to estimate the stochastic gradient of policy loss.
Then, the action of agent i is given by a i = f φ i ( s ; (cid:15) i , z ) , where (cid:15) i ∼ N (0 , I ) and z ∼ N (0 , I ) .
The policy for agent i and the variational distribution are trained to minimize the following policy improvement loss,
L
[Equation]
i ,
[Equation]
i ,
[Equation]
=
[Equation]
where
[Equation]
Since approximation of the variational distribution is not accurate in the early stage of training and the learning via the term (a) in equation 45 is more susceptible to approximation error, we propagate the gradient only through the term (b) in equation 45 to make learning stable.
Note that minimizing − log q ξ i ( a j | a i , s t ) is equivalent to minimizing the mean-squared error between a j and µ ξ i ( a i , o i , o j ) due to our Gaussian assumption on the variational distribution.
A PPENDIX C: P SEUDO C ODE
Centralized training
phase
Initialize parameter φ i , θ i , ψ i , ψ i , ξ i , ∀ i ∈ { 1 , · · · , N } for episode = 1 , 2 , · · · do
Initialize state s 0 and
each agent observes o ifor t < T and s t (cid:54) =
terminal do
Generate z t ∼ N (0 , I ) and select action a it ∼ π i ( ·| o it , z t ) for each agent
i
Execute a t and each agent i receives r t and o it +
Store transitions in D end for for each gradient step do
Sample a minibatch from D and generate z l ∼ N (0 , I ) for each transition.
Update θ i , ψ i by minimizing the loss (42) and (43)
Update φ i , ξ i by minimizing the loss (44) end for
Update ψ
i using the moving average method end for
Decentralized execution phase
Initialize state s 0 and
each agent observes o ifor each environment step do
Select action a it ∼ π i ( ·| o it , z t ) where z t = −→ 0 (or sample from the Gaussian random sequence generator with the same seed)
Execute a t and each agent i receives o it + end for
A PPENDIX D: E NVIRONMENT D ETAIL
Multi-walker
The multi-walker environment, which was introduced in Gupta et al.
(2017), is a modiﬁed version of the BipedalWalker environment in OpenAI gym to multi-agent setting.
The environment consists of N bipedal walkers and a large package.
The goal of the environment is to move forward together while holding the large package on top of the walkers.
The observation of each agent consists of the joint angular speed, the position of joints and so on.
Each agent has 4-dimensional continuous actions that control the torque of their legs.
Each agent receives shared reward R 1 depending on the distance over which the package has moved and receives negative local compensation R 2 if the agent drops the package or falls to the ground.
An episode ends when one of the agents falls, the package is dropped or T time steps elapse.
To obtain higher rewards, the agents should learn coordinated behavior.
For example, if one agent only tries to learn to move forward, ignoring other agents, then other agents may fall.
In addition, the different coordinated behavior is required as the number of agents changes.
We set T = 500 , R 2 = − 10 and R 1 = 10 d , where d is the distance over which the package has moved.
We simulated this environment in three cases by changing the number of agents ( N = 2 , N = 3 , and N = 4 ).
All algorithms used neural networks to approximate the required functions.
In the algorithms except
I-SAC, we used the neural network architecture proposed in Kim et al.
(2019) to emphasize the agent’s own observation and action for centralized critics.
For agent i , we used the shared neural network for the variational distribution q ξ i ( a jt | a it , o it , o jt ) for j ∈ { 1 , · · · , N }\{ i } , and the network takes the one-hot vector which indicates j as input.
Experimental details are given in Appendix E.
Agent
Landmark
[Figure] [Figure]
Predator-prey
The predator-prey environment, which is a standard task for MARL, consists of
N predators and M preys.
We used a variant of the predator-prey environment into the continuous domain.
The initial positions on the predators are randomly determined, and those of the preys are in the shape of a square lattice as shown in ﬁgure (b).
The goal of the environment is to capture as many preys as possible during a given time T .
A prey is captured when C predators catch the prey simultaneously.
The predators get team reward Rwhen they catch a prey.
After all of the preys are captured and removed, we set the preys to respawn in the same position and increase the value of R 1 .
Thus, the different coordinated behavior is needed as N and C change.
The observation of each agent consists of relative positions between agents and other agents and those between agents and the preys.
Thus, each agent can access to all information of the environment state.
The action of each agent is two-dimensional physical action.
We set
R 1 = 10 and T = 100 .
We simulated the environment with three cases: ( N = 2 , M = 16 , C = 1 ), ( N = 3 , M = 16 , C = 1) and ( N = 4 , M = 16 , C = 2) .
Cooperative navigation Cooperative navigation, which was proposed in Lowe et al.
(2017), consists of N agents and L landmarks.
The goal of this environment is to occupy all landmarks while avoiding collision with other agents.
The agent receives shared reward R 1 which is the sum of the minimum distance of the landmarks from any agents, and the agents who collide each other receive negative reward − R 2 .
In addition, all agents receive R 3 if all landmarks are occupied.
The observation of each agent consists of the locations of all other agents and landmarks, and action is two-dimensional physical action.
We set R 2 = 10 , R 3 = 1 , and T = 50 .
We simulated the environment in the cases of ( N = 3 , L = 3 ).
A PPENDIX E: H YPERPARAMETER AND T RAINING D ETAIL
The hyperparameters for MA-AC, MA-SAC, MADDPG, and VM3-AC are summarized in Table 2.
[Table]
[Table] [Table]