D EALING WITH N ON -S TATIONARITY IN MARL VIA T RUST -R EGION D ECOMPOSITION
Anonymous authors
Paper under double-blind review
A BSTRACT
Non-stationarity is one thorny issue in cooperative multi-agent reinforcement learning (MARL).
One of the reasons is the policy changes of agents during the learning process.
Existing work has studied the various consequences caused by non-stationarity and put forward several indicators to measure these consequences.
This makes the objectives or goals of existing algorithms are inevitably inconsistent and disparate.
In this paper, we introduce a novel notion, the δ - stationarity measurement, to explicitly measure the non-stationarity of a policy sequence, which can be further proved to be bounded by the KL-divergence of consecutive joint policies.
A straightforward but highly non-trivial way is to control the joint policies’ divergence, which is difﬁcult to estimate accurately by imposing the trust-region constraint on the joint policy.
Although it has lower computational complexity to decompose the joint policy and impose trust-region constraints on the factorized policies, simple policy factorization like mean-ﬁeld approximation will lead to more considerable policy divergence, which can be considered as the trust-region decomposition dilemma.
We model the joint policy as a pairwise
Markov random ﬁeld and propose a trust-region decomposition network (TRD-Net) based on message passing to estimate the joint policy divergence more accurately.
The M ulti-
A gent M irror descent policy algorithm with T rust region decomposition, called MAMT , is established by adjusting the trust-region of the local policies adaptively in an end-to-end manner.
MAMT can approximately constrain the consecutive joint policies’ divergence to satisfy δ -stationarity and alleviate the non-stationarity problem.
Our method can bring noticeable and stable performance improvement compared with baselines in cooperative tasks of different complexity.
I NTRODUCTION
Learning how to achieve effective collaboration in multi-agent decision-making tasks, such as multiplayer games (Berner et al., 2019;
Vinyals et al., 2019; Ye et al., 2020), resource allocation (Zimmer et al., 2021; Sheng et al., 2022), and network routing (Mao et al., 2020a;b), is a signiﬁcant problem in cooperative multi-agent reinforcement learning (MARL).
Although deep reinforcement learning (RL) has achieved great success in single-agent environments, its adaptation to the multi-agent system (MAS) still faces many challenges due to the complicated interactions among agents.
This paper focuses on one of these thorny issues, i.e., non-stationarity, caused by changing agents’ policies during the learning process.
Speciﬁcally, the state transition function and the reward function of each agent depend on the joint action of all agents.
The policy change of other agents leads to the change of above two functions for each agent.
Recently, many works have been proposed to deal with this non-stationarity problem.
These works can be divided into two categories (Papoudakis et al., 2019): targeted modiﬁcations on standard RL learning schemes (Lowe et al., 2017;
Foerster et al., 2018b;
Iqbal & Sha, 2019; Baker et al., 2019), and opponent information estimation and sharing (Raileanu et al., 2018; Foerster et al., 2018a; Rabinowitz et al., 2018; Al-Shedivat et al., 2018).
Both categories aim to mitigate the negative impact of policy changes to solve the non-stationarity problem.
These algorithms have studied the various consequences caused by non-stationarity and put forward several indicators to measure these consequences.
The objectives or goals of these algorithms are inevitably inconsistent and disparate, which makes them ineffective in general.
Additionally, those methods also require either an unexpansive training scheme or excessive information exchange, which signiﬁcantly increase the training costs.
Recently, some work has shown that naive parameter sharing, a particular case of opponent information sharing, can effectively alleviate the non-stationarity problem (Gupta et al., 2017; Terry et al., 2020).
However, we can prove that naive parameter sharing can lead to an exponentially worse suboptimal outcome with the increasing number of agents.
From the perspective of each agent, solving the non-stationarity problem in cooperative MARL can be transformed into solving multiple non-stationary Markov decision processes (MDPs) (Jaksch et al., 2010; Ortner et al., 2020; Cheung et al., 2019; Mao et al., 2021).
In these MDPs, the interactive environment is composed of multiple agents , and the reason for the non-stationary environment is the changing of agent policies.
If the policies of all agents change slowly, the environment can remain stable.
In this case, each agent could converge to optimality, i.e., pursue the best response to other agents, which will alleviate the non-stationarity problem.
Moreover, the changing opponents will signiﬁcantly hinder the agent’s learning in MAS (Radanovic et al., 2019; Lee et al., 2020;
Mao et al., 2021).
To address this issue, a popular class of RL algorithms focus on limiting the divergence between consecutive policies of the learned policy sequence by imposing direct constraints or regularization terms.
These methods are referred to as trust-region-based or proximity-based algorithms (Schulman et al., 2015; 2017; Tomar et al., 2020).
However, the precise connection between joint policy divergence and the non-stationarity is still unclear, which motivates us to analyze the relationship theoretically.
Furthermore, directly adding the trust-region constraint to the joint policy will make the problem intractable.
Therefore, effective factorization of the joint policy and trust-region constraint is the key to improve the algorithm’s efﬁciency (Oliehoek et al., 2008).
In this paper, we propose a novel notion called δ -stationarity to measure the stationarity of a given policy sequence.
The core of δ -stationarity is the opponent switching cost , which is inspired by the local switching cost (Bai et al., 2019; Gao et al., 2021) that measure the changing behavior of an single-agent RL, and is used to measures the changing of the agent’s joint behavior.
Furthermore, the relationship where δ -stationarity is bounded by the KL-divergence of consecutive joint policies is theoretically established.
Similar to existing works (Ortner et al., 2020; Cheung et al., 2019; Mao et al., 2021), which use dynamic regret (Jaksch et al., 2010) to measure the optimality of solving non-stationary MDPs algorithms, we can also prove a ˜ O
( D 3 / 2 max |O| 1 / 2 δ 1 /
i
T ) dynamic regret bound for each agent.
This provides theoretical support to impose the trust-region constraint on the joint policy to alleviate the non-stationarity problem.
However, directly dealing with the trust-region constraint on the joint policy is computationally expensive.
The natural idea is to decompose the joint policy while imposing trust-region constraints on the factorized policies.
However, simple policy factorization like mean-ﬁeld approximation will lead to siniﬁcant policy divergence, which is considered as the trust-region decomposition dilemma .
To this end, we model the joint policy as a Markov random ﬁeld and propose a trust-region decomposition network (TRD-Net) to adaptively factorize the trust-region of the joint policy and approximately satisfy the δ -stationarity.
Local trust-region constraints are imposed on factorized policies, which can be efﬁciently solved through mirror descent.
The TRD-Net constructs the relationship among factorized policies, factorized trust-regions, and the estimated joint policy divergence.
To accurately estimate the joint policy divergence, an auxiliary task, which borrowes the idea in ofﬂine RL to measure the degree of out-of-distribution, is constructed to train the TRD-Net.
The proposed algorithm is denoted as M ulti-
A gent M irror descent policy optimization with T rust region decomposition, i.e., MAMT .
MAMT can alleviate the non-stationarity problem and factorize joint trust-region constraint into trust-region constraints on local policies, which could signiﬁcantly improve the learning effectiveness and robustness.
Our contributions mainly consist of the following: 1) We propose a formal non-stationarity deﬁnition of cooperative MARL, δ -stationarity, which is derived from the local switching cost.
δ -stationarity is bounded by the joint policy divergence and bounds the dynamic regret of each agent.
The novel trust-region decomposition scheme based on message passing and mirror descent could approximately satisfy the δ -stationarity through a computationally efﬁcient way; 3)
Our proposed algorithm could bring noticeable and stable performance improvement in multiple cooperative tasks of different complexity than baselines.
T HE S TATIONARITY OF THE L EARNING P ROCEDURE
In our work, we consider a cooperative multi-agent task that can be modelled by a cooperative
POSG (Hansen et al., 2004)
(cid:104)I , S , {A i } ni =1 , {O i } ni
=1 , P , E , {R i } ni =1 (cid:105) , where I represents the nagent space.
s ∈ S represents the true state of the environment.
We consider partially observable settings, where agent i only accessible to a local observation o i ∈
O i according to the emission function E ( o i | s ) .
At each timestep, each agent i selects an action a i ∈
π i ( a | o i ) , forming a joint action a = (cid:104) a 1 , . . . , a n (cid:105) ∈
×A i , results in the next state s (cid:48)
according to the transition function
P ( s (cid:48) | s, a ) and a reward r i
= R i
( s, a ) and
∂ R i (cid:48)
∂ R i (cid:62) 0 where i and i (cid:48) are a pair of agents.
Intuitively, this means that there is no conﬂict of interest for any pair of agents.
To explore and mitigate the non-stationarity in cooperative MARL, we ﬁrst need to model the nonstationarity explicitly.
To emphasize, the non-stationarity is not an intrinsic attribute of a cooperative
POSG but is additionally introduced when using a speciﬁc learning algorithm.
As mentioned above, solving the non-stationarity problem in cooperative MARL can be transformed into solving multiple dynamic non-stationary MDPs, where the non-stationary transition probability
T i and reward function R i is caused by the changing of environment (other agents policies).
Switching cost is a standard notion in the literature to measure the changing behavior of an single-agent RL algorithm (Cesa-Bianchi et al., 2013; Bai et al., 2019; Gao et al., 2021).
We consider the following deﬁnition of the (local) switching cost from Bai et al.
Deﬁnition 1 (Local switching cost) .
Let H be the horizon of the MDP and K be the number of episodes that the agent can play.
The local switching cost (henceforth also ”switching cost”) between any pair of policies ( π, π (cid:48) ) is deﬁned as the number of ( h, o ) pairs on which π and π (cid:48) are different:
[Equation]
where [ H ] := 1 , · · · , H and o, O are the local observation and observation space.
For an singleagent RL algorithm that employs policies ( π 1 , · · · , π K ) , its local switching cost is deﬁned as
[Equation]
In MARL, it needs more attention that the magnitude of the change of other agents’ joint policy at each learning step.
Thus, we propose the following opponent switching cost to measure the changing behavior of opponents by naturally extending the local switching cost:
Deﬁnition 2 (Opponent switching cost) .
Let H be the horizon of the MDP and K be the number of episodes that the agent can play, so that total number of steps T :=
HK .
The opponent switching cost of agent i is deﬁned as the maximum Kullback–Leibler divergence of all ( t, o ) pairs between any pair of opponents’ joint policies ( π − i , π (cid:48)− i ) on which π − i and π (cid:48)− i are different:
[Equation]
where [ T ] := 1 , · · · , T and o , O are the joint observation and observation space.
The larger the changing magnitude of other agents’ joint policy is, the larger the opponent switching cost is, and the more serious non-stationarity problem will be suffered (Radanovic et al., 2019; Lee et al., 2020; Mao et al., 2021).
Therefore, opponent switching cost could play an effective role to measure the non-stationarity.
Based on the deﬁnition of opponent switching cost, the (non-)stationarity of the learning procedure can be further deﬁned as follows:
Deﬁnition 3 ( δ -stationarity of the learning procedure) .
For a MAS containing n agents, if we have d i switch (cid:0) π − i , π (cid:48)− i (cid:1) ≤ δ i , then the learning procedure of agent i is δ i -stationary.
Further, if all agents are δ -stationary with corresponding { δ i } , then the learning procedure of entire multi-agent system is δ -stationary with δ = 1 n (cid:80) i δ i .
There is a strong relationship between the Deﬁnition 3 and the changing of the T i , R i for each agent i , which is stated by Lemma 1 and Lemma 2:
Lemma 1.
If any agent i satisﬁes δ
i -stationarity, then the total variation distance of its two consecutive transition probability distribution p t ( ·| o , a i ) and p t +1 ( ·| o , a i ) is bounded by δ i , i.e.,
[Equation]
Lemma 2.
Assume the absolute value of joint reward is less than 1 .
If any agent i satisﬁes δ istationarity, then the total variation distance of its two consecutive reward function r t ( o , a i ) and r t +1 ( o , a i ) is bounded by δ i , i.e., D TV (cid:0) r t ( o , a i ) , r t +1 ( o , a i ) (cid:1) ≤ 2 ln 2 · δ 1 / 2 i .
Lemma 1 and Lemma 2 indicate that the δ -stationarity is a reasonable mathematical description of the non-stationarity problem (Hernandez-Leal et al., 2017; Papoudakis et al., 2019; Padakandla, 2021), which can measure the non-stationarity of the learning procedure.
Besides, we ﬁnd that controlling the δ -stationarity of the learning procedure can obtain tighter (dynamic) regret bound for each agent i , which indicates smaller distance to the set of coarse correlated equilibria (CCE) in ﬁnite games (Hannan, 2016; Hart & Mas-Colell, 2000; Hsieh et al., 2021).
Based on the Lemma 1 and
Lemma 2, Theorem 1 states that the (dynamic) regret of each agent i can be bounded by corresponding δ i (the deﬁnitions of D max , B r and B p are shown in appendix):
Theorem 1.
Consider the learning procedure of a MAS satisﬁes the δ -stationarity and each agent satisﬁes the δ i -stationarity.
Let H be the horizon and K be the number of episodes, so that total number of steps T :=
HK .
In addition, suppose that T ≥ B r + 2 D max B p > 0 , then a ˜ O ( D 3 / 2 max |O| 1 / 2 δ 1 / 4 i
T ) dynamic regret bound is attained for each agent i .
However, computing the KL-divergence of the consecutive opponents’ joint policies ( π t − i , π t +1 − i ) in Theorem 1 still has high computational complexity.
For a MAS with n agents, n constraints on the joint policy divergence should be modeled simultaneously.
Before proposing the algorithm, we still need to relax the constraints on the joint policy divergence to increase efﬁciency.
Based on the following theorem, the constraints on the joint policy divergence of the other agents can be limited by the joint policy divergence of all agents, independent of the agent number.
Both detailed proofs of
Lemma 1, Theorem 1 and the following Theorem 2 can be found in the appendix.
Theorem 2.
For a multi-agent system, the maximum KL-divergence of all agents’ consecutive joint policies ( π, π (cid:48) ) is the upper bound of the average opponent switching cost
[Equation]
With the above Theorem 2, it only needs to impose only one trust-region constraint on all agents‘ joint policy to control the non-stationarity of the entire learning procedure.
This allows us to design a
MARL algorithm with better effectiveness to approximate the deﬁned δ -stationarity.
T HE P ROPOSED MAMT M ETHOD
According to Theorem 2, we need to properly constrain the maximum divergence of consecutive joint policies to make the learning procedure more stable and efﬁcient.
In other words, the algorithm needs to balance between eliminating the non-stationarity (i.e., δ → 0 ) and fast learning (i.e., δ → ∞ ).
To emphasize, based on Deﬁnition 3, Theorem 1 and Theorem 2, we can modify the cooperative
MARL formulation by adding stationarity constraint.
This problem imposes a constraint that
the
KL divergence is bounded at every point in the state space.
While it is motivated by the theory, this problem is impractical to solve due to the large number of constraints.
Instead, we can use a heuristic approximation, similar as Schulman et al.
(2015), which considers the average KL divergence:
[Equation] [Equation]
where π represents the joint policy of all agents; o :
= ( o 1 , · · · , o i , · · · , o n ) represents the joint observation with n be the agent number.
D is the replay buffer.
Considering the low sample efﬁciency of MARL, we did not adapt the fully on-policy learning, but use off-policy training with a small replay buffer.
In this way, it can be ensured that samples are less different from the current policy thereby alleviating the instability caused by off-policy training.
Directly imposing a constraint on the joint policy as in (1) will make the problem signiﬁcantly more difﬁcult and computationally intractable.
A straightforward way is to evenly distribute the trust-region to all agents based on mean-ﬁeld policy approximation (as shown in Figure 6 in the appendix).
However, this will bring up the problem which we call trust-region decomposition dilemma .
3. TRUST - REGION D ECOMPOSITION D ILEMMA
Formally, inspired by Lowe et al.
(2017), Foerster et al.
(2018b) and Iqbal & Sha (2019)
, we ﬁrst introduce the mean-ﬁeld approximation assumption.
Assumption 1 (Mean-Field Approximation) .
We use the mean-ﬁeld variation family π to estimate the joint policy of all agents, i.e., π ( a | o ) =
(cid:81) ni
=
π i ( a i |
o i ) .
Based on mean-ﬁeld approximation assumption, the joint policy trust-region constraint can be factorized into trust-region constraints of local policies with the following theorem.
Theorem 3.
For any consecutive policies which are belong to mean-ﬁeld variational family π and π k in the policy sequence obtained by any learning algorithm, the KL divergence constraint on joint policy in (1) is equivalent with the following summation constraints of local policies, i.e.,
[Equation]
Further the joint summation trust-region constraint (2) can be equivalently decomposed into the following local trust-region constraints, i.e.,
[Equation]
with (cid:80) ni =
δ i = δ , 0 ≤ δ i ≤ δ ; π i and π ki represent the consecutive local policies of agent i ;
o i and u i represent the local observation and initial local observation distribution of agent i respectively.
Due to the summation constraint term (2), it is still need to jointly train the policies of all agents with limited effectiveness and solvability.
Therefore, we further equivalently transform (2) into (3) * .
Based on Theorem 3, problem (1) can be reformulated as
[Equation]
The trust-region decomposition scheme adopted in the above, we called MAMD, is based on the mean-ﬁeld approximation assumption (i.e., formulation (4)).
The similar optimization problem is also obtained by Li & He (2020) that try to implement TRPO for MARL through distributed consensus optimization.
However, decomposing the trust-region inappropriately will make the algorithm converge to sub-optimal solutions, which can be numerically proved through the following example.
By considering a simple MAS with three agents i, j, k , we assume that the agent k is independent with two agents i and j .
Then for the multi-agent system, we have p
i
o (cid:48) | o , a i , a j , a k ) =
p i ( o (cid:48)
| o , a i , a j ) , p j ( o (cid:48) | o , a j , a i , a k ) = p j ( o (cid:48) | o , a j , a i ) , p k ( o (cid:48) | o , a k , a i , a j ) = p ( o (cid:48) | o , a k ) .
The change of agent k ’s policy cannot affect the state transition probability of i and j , and vice versa.
As a result, the constraints on the agent i and agent j are “insufﬁcient” but the constraint for agent k is “excessive”, when we decompose the trust-region δ in (2) into three parts equally, i.e., δ/ 3 .
This phenomenon is called trust-region decomposition dilemma † in this paper.
Theoretically, the reason for the trust-region decomposition dilemma might be the inaccurate estimation of the joint policy divergence based on the mean-ﬁeld approximation assumption.
The simple summation of local policies’ divergence is not equal to the divergence of the joint policy, while the gap might be enormous.
Once the joint policy divergence cannot be accurately estimated, the non-stationarity cannot be judged accordingly, and imposing constraints on it may run counter to our goals.
3. T HE MAMT A LGORITHM F RAMEWORK
Solving the trust-region decomposition dilemma need to accurately model the relationship between the local policy divergences and the joint policy divergence.
However, calculating the joint policy divergence is intractable.
MAMT solves this dilemma from another perspective, i.e., directly learning the relationship through the following three steps.
First, approximating the joint policy divergence;
Then, using a differentiable function, i.e., the trust-region decomposition network (TRD-Net), to ﬁt the relationship between the approximated joint policy divergence and local trust-regions; Finally, adaptively adjust local trust regions by optimizing the approximated joint policy divergence.
The rest of this section will be organized in three aspects: joint policy divergence approximation, the trust-region decomposition network, and local trust-region optimization.
Joint Policy Divergence Approximation.
From the analysis of the trust-region decomposition dilemma, it can be seen that the dependence between agents is an essential factor affecting the joint policy divergence.
Formally, we use a non-negative real number, coordination coefﬁcient , to represent the dependency between two agents ‡ .
In the learning procedure, the coordination relationship between agents is changing with the learning of policies and completing tasks.
Therefore, we model the cooperative relationship between two agents based on counterfactual (Foerster et al., 2018b; Jaques et al., 2019).
In Jaques et al.
the causal inﬂuence reward of agent i w.r.t.
opponent j is calculated by c ti,j = D KL (cid:2) p (cid:0) a tj | a ti , o tj (cid:1) (cid:107) p (cid:0) a tj | o tj (cid:1)(cid:3) , where p (cid:0) a tj | s tj (cid:1) =
(cid:80) ˜
a ti p (cid:0) a tj | ˜ a ti , o tj (cid:1) p (cid:0) ˜ a ti
| o tj (cid:1) .
It can be seen that the greater the causal inﬂuence reward, the tighter the coordination between the two agents.
However, when calculating the causal inﬂuence reward, the agent’s policy needs to be modiﬁed.
That is, the agent needs to explicitly rely on the actions of other agents when making decisions.
To solve this problem, we modiﬁed the process of calculating the counterfactual baseline in Foerster et al.
(2018b)
based on the idea of Jaques et al.
and obtained a new way of calculating the coordination coefﬁcient
[Equation]
where Q i ( · ) is the centralized critic of agent i .
a is the joint action of all agents, and a \ j is the joint action of all agents expect for j ; σ is the threshold to keep sparsity.
According to Theorem 2, Deﬁnition 2 and Assumption 1, after modeling the coordination coefﬁcient between agents, does it mean that the weighted summation of the local policy divergence, i.e.,
[Equation]
/n
[Equation]
)] can accurately estimate the joint policy divergence?
It can be seen from Deﬁnition 3 that the agents’ policies do not directly cause the non-stationarity.
The agent cannot directly observe the opponent’s policy but only the action sequences.
This explicit information directly affects the learning process of the agent and leads to non-stationarity.
Therefore, we borrowed the idea in ofﬂine RL to measure the degree of out-of-distribution through the model-based
RL (Kidambi et al., 2020; Yu et al., 2020).
Speciﬁcally, each agent i has a prediction model
h φ ji for each other agent j , predicting the other agent’s actions based on its local history observation.
The prediction model can naturally replace the old policy π (cid:48) since it is trained using historical information.
Thus the summation of the divergence between the predicted action distribution and the actual action distribution of all other agents could represent the non-stationarity of the agent i , i.e.,
[Equation]
[Equation]
where “ns” denotes the “non-stationarity”; π ψ j is the policy of agent j which is parameterized by ψ j ; Π ns represents the projection function, which constrains D ti, ns to a speciﬁc range.
Kim et al.
(2020) also introduces inter-agent action prediction, but Kim et al.
(2020) is to promote collaboration, and this paper is to estimate the non-stationarity of the joint policy better.
The joint policy divergence can then be approximated by the summation of all local non-stationarities D t ns
=
(cid:80) i D ti, ns .
Trust-Region Decomposition Network.
To achieve more reasonable joint policy decomposition, recent works (B¨ohmer et al., 2020; Qu et al., 2020; Li et al., 2021) modeled the joint policy as a
Markov random ﬁeld with pairwise interactions (shown in the Figure 6 in the appendix) based on graph neural networks (GNN).
The joint policy divergence is related to the local policies and local trust-regions.
Inspired by these methods, a similar mechanism is utilized in this paper to decompose the joint policy divergence into local policies and local trust regions (which can be input to the GNN) with pairwise interactions.
The employed GNN is denoted as trust-region decomposition network and the network structure is shown in Figure 1.
The approximated joint policy divergence D ns can then be used as an surrogate supervision signal to train the trust-region decomposition network.
Formally, the loss function to learn the trust-region decomposition network can be formulated as
[Equation]
where θ and ˆKL t i are the parameters and the output of the trust-region decomposition network.
Local Trust-Region Optimization.
Based on the approximated joint policy divergence and the trust-region decomposition network, the learning of local trust-region δ i can be formulated by the trade-off between two parts, i.e., the non-stationarity of the learning procedure and the performance of all agents.
Formally, the learning objective of δ i is to maximize
[Equation]
where δ = { δ i } ni =1 and π ( δ ) represent the joint policy is related to the δ i of all agents; ˆKL π ( δ ) i denotes the output of the TRD-Net which is parameterized by θ .
Finally, the learning of { δ i } and θ can be modeled as a bilevel optimization problem (Dempe & Zemkoho, 2020)
[Equation]
We can employ the efﬁcient two-timescale gradient descent method to simultaneously perform
[Figure]
gradient update for both δ and θ .
Speciﬁcally, we have
[Equation]
where α k β k → 0 indicates that { θ k } k ≥ 0 updates faster than { δ k } k ≥ 0 .
In practical, we make α k and β k equal but perform more gradient descent steps on θ , similar as Fujimoto et al.
At the same time, in order to ensure that the updated δ k +1 can meet the constraint, that is, (cid:80) i δ i,k +1 <
δ , we added an additional regular term ReLU( (cid:80) i
δ i,k +1 − δ ) to F ( δ ) during algorithm training
Algorithm Summary.
By directly combing the MAAC (Iqbal & Sha, 2019) and the mirror descent technique in MDPO (Tomar et al., 2020), we can get the algorithm framework for solving problem formulated in Equation (1) after decomposing the trust-region constraint.
Each agent i has its local policy network π ψ i and a local critic network Q ζ i , which is similar to MADDPG § (Lowe et al., 2017).
All critics are centralized updated iteratively by minimizing a joint regression loss function, i.e.,
[Equation]
) with y i
=
r i + γ E a (cid:48) ∼ π ¯ ψ ( o (cid:48) )
[ Q ¯ ζ i ( o (cid:48) , a (cid:48) ) − α log( π ψ i ( a (cid:48)
i | o (cid:48) i ))] , and the parameters of the attention module are shared among agents; π ¯ ψ represents the joint target policy (similar as target Q -network in DQN) of all agents; Q ¯ ζ i represents the target local critic of agent i .
α denotes the temperature balancing parameter between maximal entropy and rewards.
As for the individual policy updating step, the mirror descent (Tomar et al., 2020) is introduced to deal with the decomposed local trust-region constraints:
[Equation]
, where b ( o, a \ i ) denotes the counterfactual baseline (Foerster et al., 2018b); π ψ old i represents the policy of agent i at last step; δ i is the assigned trust-region range; KL[ ψ i (cid:107) ψ old i ] = log( π ψ i ( a i | o i ))
− log( π ψ old i ( a i | o i )) .
However, if we only perform a single-step SGD, the resulting gradient would be equivalent to vanilla policy gradient and misses the purpose of enforcing the trust-region constraint.
As a result, the policy update at each iteration k involves m SGD steps as
[Equation]
For off-policy mirror descent policy optimization, performing multiple SGD steps at each iteration becomes increasingly time-consuming as the value of m grows, Therefore, similar with Tomar et al.
(2020), we resort to staying close to an m step old copy of the current policy while performing a single gradient update at each iteration of the algorithm.
This copy is updated every m iterations with the parameters of the current policy.
The pseudo-code of MAMT is shown in the Algorithm 1.
E XPERIMENTS
This section aims to verify the effectiveness of the trust-region constraints, the existence of the trust-region decomposition dilemma, and the capacity of the TRD-Net with 4 cooperative tasks
Spread , Multi-Walker , Rover-Tower , Pursuit (more details are in the appendix).
Require: Randomly initialize π ψ i , Q ζ i , θ and δ
i 1: Initialize empty replay buffer D 2: Assign Q ¯ ζ i ← Q ζ i , { π ¯ ψ i ,π ψ old i } ← π ψ i 3: for i = 1, 2, ... do 4:
Collect interaction τ i with π ψ i 5:
Update replay buffer D ← { τ i } 6: for j = 1, 2, ... do 7:
Sample transitions B from D 8:
Compute {C ti, \ i , D ti, ns } with Eq.
Update δ ti ,θ with Eq.10:
Update Q ζ i ,π ψ i by Eq.
Update target networks Q ¯ ζ i and π ¯ ψ i 12:
Update old policy π ψ old
i
Baselines .
The ﬁsrt CCDA algorithm MADDPG (Lowe et al., 2017) is chosen.
Our proposed trust-region technique motivates us to set the combination of PPO and MADDPG as another baseline, MA-PPO (Yu et al., 2021).
The
MAAC (Iqbal & Sha, 2019) which is based on the attention mechanism is also considered.
MA-PPO’s policy network also uses an attention mechanism similar to MAAC.
LOLA ¶ (Foerster et al., 2018a) algorithm that models opponent agents to solve the non-stationarity problem is compared as the baseline.
LOLA is only compared in the Spread environment due to the poor scalability.
MAMD , i.e., MAMT without the
TRD-Net, is also considered.
Comparisons .
we compare MAMT with all baselines and get the following three conclusions:
[Figure]
Combine the single-agent trust-region technique with MARL directly cannot bring stable performance improvement.
The averaged episode rewards of all methods in four environments are shown in
Figure 2.
From the comparison of MAAC and MA-PPO, it can be seen that directly combining the single-agent trust-region technique with MARL cannot bring stable performance improvement.
In the Rover-Tower and Pursuit , MA-PPO did not bring noticeable improvement in ﬁnal performance;
MA-PPO only has a subtle improvement in the simple Spread environment and even causes performance degradation in Multi-Walker .
In contrast, MAMT can bring noticeable and stable performance improvements in all environments.
It is worth noting that MAMD also shows promising results.
This shows that even a simple trust-region decomposition can bring noticeable improvement in simple scenarios.
This further veriﬁes the rationality and effectiveness of non-stationarity modeling.
[Figure]
MAMT achieves the trust-region constraint on the local policies and minimizes the non-stationarity of the learning procedure.
Below we conduct an in-depth analysis of the reasons why MAMT can noticeably and stably improve the ﬁnal performance.
MAMT uses an end-to-end approach to adaptively adjust the sizes of the trust-regions of the local policies according to the current
We dependencies between agents to constrain the KL divergence of the joint policy more accurately.
Therefore, we analyze MAMT from the two perspectives: the trust-region constraint satisfaction of the local policies and the non-stationarity satisfaction via surrogate measure.
Figure 3 shows the average value of the KL divergence of all agents’ local policies during learning procedures in different environments.
It can be seen from the ﬁgure that MAMT controls the policy divergence.
The non-stationarity of the learning procedure is challenging to calculate according to the deﬁnition, so we use the surrogate measure D ns to represents the non-stationarity.
It can be seen from Figurethat as the learning progresses, the non-stationarity is gradually decreasing.
[Figure] [Figure]
meanvariance (c) Rover-Tower.
AgentAgent-
AgentAgent- AgentAgent-
Agent0.2 0.4 0.
meanvariance (d) Pursuit.
Trust-region decomposition works better when there are more agents and more complex coordination relationships.
Figure 2 shows that MAMT and MAMD have similar performance in the Spread and Multi-Walker , but MAMT is noticeably better than MAMD in the Rover-Tower and Pursuit .
To analyze the reasons for this phenomenon, we separately record the coordination coefﬁcients between agents in the learning procedure and the sizes of the trust-regions of the local policies, see Figureand Figure 4.
It can be seen from Figure 5 that the Spread and Multi-Walker are relatively simple, and the number of agents is small.
This makes the interdependence between agents is very tight (larger mean) and will not change over time (variance is microscopic).
Moreover, the corresponding trust-region sizes are similar to coordination coefﬁcients.
These ﬁgures show that in simple coordination tasks with a small number of agents, trust-region decomposition is not a severe problem, which leads to the fact that the MAMT cannot bring a noticeable performance improvement.
In the other two environments, the dependence between agents is not static (the variance in Figure 5 is more considerable) due to the more signiﬁcant number of agents and more complex tasks.
This also makes the sizes of the trust-regions of the agent’s local policies ﬂuctuate signiﬁcantly in different stages (the variance in Figure 4 is more considerable).
MAMT alleviates the trust-region decomposition dilemma through the TRD-Net, which brings a noticeable and stable performance improvement.
C ONCLUSION
In this paper, we deﬁne the δ -stationarity to explicitly model the stationarity in the learning procedure of the multi-agent system and provide a theoretical basis for proposing an achievable learning algorithm based on joint policy trust-region constraints.
To solve the trust-region decomposition dilemma caused by the mean-ﬁeld approximation and to estimate the joint policy divergence more accurately, we propose an efﬁcient and robust algorithm MAMT combining message passing and mirror descent with the purpose to satisfy δ -stationarity.
Experiments show that MAMT can bring noticeable and stable performance improvement and more suitable for large-scale scenarios with complex coordination relationships between agents.
Ethics Statement.
Our method is not a generative model, nor does it involve the training of superlarge-scale models.
The training data is sampled in the simulated environments, so it does not involve fairness issues.
Our method also does not involve model or data stealing and adversarial attacks.
Reproducibility Statement.
The source code of this paper is available at https:// anonymous.4open.science/r/MAMT .
We specify all the training details (e g., hyperparameters, how they were chosen), the error bars (e g., with respect to the random seed after running experiments multiple times), and the total amount of compute and the type of resource used (e g., type of GPUs and CPUs) in the Section 4 or the Appendix G. We also cite the creators of partial code of our method and mention the license of the assets in the Appendix G
Supplementary Material A P RELIMINARIES
Cooperative POSG.
POSG (Hansen et al., 2004) is denoted as a seven-tuple via the stochastic game (or Markov game)
[Equation]
where n denotes the number of agents; I represents the agent space; S represents the ﬁnite set of states; A i , O i denote a ﬁnite action set and a ﬁnite observation set of agent i ∈ I respectively;
A =
A 1 × A 2 × · · · × A n is the ﬁnite set of joint actions; P ( s (cid:48) | s, a ) denotes the Markovian state transition probability function, where s, s (cid:48) ∈ S represent states of environment and a = { a i } ni =1 ,
a i ∈ A i represents the action of agent i ; O
=
O 1 ×
O 2 × · · · × O n is the ﬁnite set of joint observations; E ( o | s ) is the Markovian observation emission probability function, where o
=
{ o i } ni
=1 , o i ∈
O i represents the local observation of agent i ; R i : S ×
A × S → R denotes the reward function of agent i and r
i ∈ R i is the reward of agent i .
The game in POSG unfolds over a ﬁnite or inﬁnite sequence of stages (or timesteps), where the number of stages is called horizon .
In this paper, we consider the ﬁnite horizon case.
The objective for each agent is to maximize the expected cumulative reward received during the game.
For a cooperative POSG, we quote the deﬁnition in Song et al.
[Equation]
where i and i (cid:48) are a pair of agents in agent space I ; π i and π i (cid:48) are the corresponding policies in the policy space
Π i and Π i (cid:48) respectively.
Intuitively, this deﬁnition means that there is no conﬂict of interest for any pair of agents.
Mirror descent method in RL.
The mirror descent method (Beck & Teboulle, 2003) is a typical ﬁrst-order optimization method, which can be considered an extension of the classical proximal gradient method.
In order to minimize the objective function f ( x ) under a constraint set x ∈ C ⊆ R n , the basic iterative scheme at iteration k + 1 can be written as
[Equation]
where B ψ ( x , y ) :
= ψ ( x ) − ψ ( x ) − (cid:104)∇ ψ ( y ) , x − y (cid:105) denotes the Bregman divergence associated with a strongly convex function φ and γ k is the step size (or learning rate).
Each reinforcement learning problem can be formulated as optimization problems from two distinct perspectives, i.e.,
[Equation]
Geist et al.
and Shani et al.
(2020) have utilized the mirror descent scheme (10) and update the policy iteratively as follows
[Equation]
where KL( · , · ) denotes the Bregman divergence corresponding to negative entropy.
B R ELATED W ORKS B. T ACKLE N ON -S TATIONARITY IN MARL
Many works have been proposed to tackle non-stationarity in MARL.
These methods range from using a modiﬁcation of standard RL training schemes to computing and sharing additional other agents’ information.
B.1. M ODIFICATION OF S TANDARD RL T RAINING S CHEMES
One modiﬁcation of the standard RL training schemes is the centralized critic and decentralized actor (CCDA) architecture.
Since the centralized critics can access the information of all other agents during training, the dynamics of the environment remain stable for the agent.
Lowe et al.
(2017) combined DDPG (Lillicrap et al., 2016) with CCDA architecture and proposed MADDPG algorithm.
Foerster et al.
(2018b) proposed a counterfactual baseline in the advantage estimation and used the REINFORCE (Williams, 1992) algorithm as the backbone in the CCDA architecture.
Similar with Lowe et al.
(2017), Iqbal & Sha (2019) combined SAC (Haarnoja et al., 2018)
with CCDA architecture and introduced attention mechanism into the centralized critic design.
In addition, Iqbal & Sha (2019) also introduced the counterfactual baseline proposed by Foerster et al.
(2018b).
CCDA alleviated non-stationarity problems indirectly makes it unstable and ineffective, and our experiments have also veriﬁed this.
Speciﬁcally, on the one hand, the centralized critic does not directly affect the agent’s policy but only inﬂuences the update direction of the policy through the gradient.
The policy modeling does not take into account the actions of other agents like centralized critics (considering the decisions of other agents in centralized critics is the crucial improvement of this type of algorithm) but only based on local observations.
On the other hand, and more importantly, even if centralized critics explicitly consider the actions of other agents, they cannot mitigate the adverse effects of non-stationarity.
Centralized critics only consider the sample of other agent’s policy distribution.
Once the other agent’s policy change drastically and frequently (corresponding to more severe non-stationarity), they need more samples to ”implicit” modeling the outside environment, which leads to higher sample complexity.
Another modiﬁcation to handle non-stationarity in MARL is self-play for competitive tasks or population-based training for cooperative tasks (Papoudakis et al., 2019).
Tesauro (1995) used self-play to train the TD-Gammon, which managed to win against the human champion in Backgammon.
Baker et al.
(2019) extended self-play to more complex environments with continuous state and action space.
Liu et al.
and Jaderberg et al.
(2019) combined population-based training with self-play to solve complex team competition tasks, a popular 3 D multiplayer ﬁrst-person video game, Quake III
Arena Capture the
Flag , and MuJoCo Soccer , respectively.
However, such methods require a lot of hardware resources and a well-designed parallelization platform.
B.1. C OMPUTING AND S HARING A DDITIONAL I NFORMATION
In addition to modifying the standard RL training scheme, there are also methods to solve non-stationarity problems by computing and sharing additional information.
One naive approach is parameter sharing and use agents’ aggregated trajectories to conduct policy optimization at every iteration (Gupta et al., 2017; Terry et al., 2020).
Unfortunately, this simple approach has signiﬁcant drawbacks.
An obvious demerit is that parameter sharing requires that all agents have identical action spaces, i.e. , A i =
A j , ∀ i, j ∈ N , which limits the class of MARL problems to solve.
Importantly, enforcing parameter sharing is equivalent to putting a constraint θ i
= θ j , ∀ i, j ∈ N on the joint policy space.
In principle, this can lead to a suboptimal solution.
To elaborate, we have following proposition proposed by Kuba et al.
Proposition 1 (suboptimal) .
Let’s consider a fully-cooperative game with an even number of agents n , one state, and the joint action space { 0 , 1 } n , where the reward is given by r ( 0 n/ 2 , 1 n/ 2 ) =
r ( 1 n/ 2 , 0 n/ 2 ) = 1 , and r ( a 1: n ) =
for all other joint actions.
Let J ∗ be the optimal joint reward, and J ∗ share be the optimal joint reward under the shared policy constraint.
Then
[Equation]
This proposition shows that parameter sharing can lead to a suboptimal outcome that is exponentially-worse with the increasing number of agents.
In addition, there are many other ways to share or compute additional information among agents.
Foerster et al.
proposed importance sampling corrections to adjust the weight of previous experience to the current environment dynamic to stabilize multi-agent experience replay.
Raileanu et al.
and Rabinowitz et al.
(2018) used additional networks to predict the actions or goals of other agents and input them as additional information into the policy network to assist decision-making.
Foerster et al.
(2018a) accessed the optimized trajectory of other agents by explicitly predicting the parameter update of other agents when calculating the policy gradient, thereby alleviating the non-stationarity problem.
These explicitly considering other agents’ information are also called modeling of others .
To solve the convergence problem shown by Foerster et al.
(2018a)
under certain n-player and non-convex games, Letcher et al.
(2019) presents Stable Opponent
Shaping(SOS).
This new method interpolates between Foerster et al.
(2018a) and a stable variant named LookAhead, which is proved that converges locally to equilibria and avoids strict saddles in all differentiable games.
Different from Foerster et al.
(2018a);
Letcher et al.
(2019) directly predicting the opponent’s policy parameters, Xie et al.
(2020) solves the non-stationary problem by predicting and inﬂuencing the latent representation of the opponent’s policy.
Recently, Al-Shedivat et al.
(2018) transformed non-stationarity problems into meta-learning problems, and extended MAML (Finn et al., 2017) to MAS to ﬁnd an initialization policy that can quickly adapt to non-stationarity.
However, Al-Shedivat et al.
(2018) treats other agents as if they are external factors whose learning it cannot affect, and this does not hold in practice for the general multi-agent learning settings.
Kim et al.
(2021) combines Foerster et al.
(2018a) to consider both an agent’s own non-stationary policy dynamics and the non-stationary policy dynamics of other agents in the environment.
Due to the unique training mechanism of the above methods, they are difﬁcult to extend to the tasks of more than 2 agents.
B. TRUST - REGION M ETHODS B.2. TRUST - REGION M ETHODS IN S INGLE -A GENT RL
trust-region or proximity-based methods, resonating the fact they make the new policy lie within a trust-region around the old one.
Such methods include traditional dynamic programming-based conservative policy iteration (CPI) algorithm (Kakade & Langford, 2002), as well as deep RL methods, such as trust-region policy optimization (TRPO)
(Schulman et al., 2015) and proximal policy optimization (PPO) (Schulman et al., 2017).
TRPO used line-search to ensure that the KL divergence between the new policy and the old policy is below a certain threshold.
PPO is to solve a more relaxed unconstrained optimization problem, in which the ratio of the old and new policy is clipped to a speciﬁc bound.
Wu et al.
(2017) (ACKTR) extended the framework of natural policy gradient and proposed to optimize both the actor and the critic using Kronecker-factored approximate curvature (KFAC) with trust-region.
Nachum et al.
(2018) proposed an off-policy trust-region method,
Trust-PCL, which introduced relative entropy regularization to maintain optimization stability while exploiting off-policy data.
Recently, Tomar et al.
(2020) used mirror decent to solve a relaxed unconstrained optimization problem and achieved strong performance.
B.2. TRUST - REGION M ETHODS IN MARL
Extending trust-region methods to MARL is highly non-trivial.
Despite empirical successes, none of them managed to propose a theoretically-justiﬁed trust-region protocol in multi-agent learning.
Instead, they tend to impose certain assumptions to enable direct implementations of TRPO/PPO in MARL problems.
For example, IPPO (de Witt et al., 2020) assume homogeneity of action spaces for all agents and enforce parameter sharing which is discussed above.
Yu et al.
(2021) proposed MAPPO which enhances IPPO by considering a joint critic function and ﬁner implementation techniques for on-policy methods.
Yet, it still suffers similar drawbacks of IPPO.
Wen et al.
(2021) adjusted PPO for MARL by considering a game-theoretical approach at the meta-game level among agents.
Unfortunately, it can only deal with two-agent cases due to the intractability of Nash equilibrium.
Hu & Hu (2021) developed Noisy-MAPPO that targets to address the sub-optimality issue; however, it still lacks theoretical insights for the modiﬁcation made on MAPPO.
Recently,
Li & He (2020) tried to implement TRPO for MARL through distributed consensus optimization; however, they enforced the same trust-region for all agents (see their Equation (7)) which, similar to parameter sharing, largely limits the policy space for optimization, and this also will make the algorithm face the trust-region decomposition dilemma.
The method HATRPO (Kuba et al., 2021) based on sequential update scheme is proposed from the perspective of monotonic improvement guarantee.
But at the same time, sequential update scheme also limit the scalability of the algorithm.
In addition, Jiang & Lu (2021) proposes a MARL algorithm to adjust the learning rates of agents adaptively.
Limiting the size of the trust-region of each agent’s local policy is related to limiting the learning rate of its local policy.
They focus on speeding up the learning speed, instead of solving non-stationarity problems, However, Jiang & Lu (2021) has no theoretical support, and matching the direction of adjusting the learning rates with the directions of maximizing the Q values may cause overestimation problems.
C P SEUDO - CODE OF MAMT
This section gives the pseudo-code of the MAMT algorithm (see Algorithm 2 || ) and
the
MAMT algorithm without trust-region decomposition network (see Algorithm 3).
For convenience, we named the latter MAMD.
In the MAMD algorithm, the trust-region constraint is equally distributed to the local policies of all agents.
D TRUST - REGION D ECOMPOSITION D ILEMMA Algorithm 2 MAMT
Require:
Initialize behavior policy π ψ i and main centralized critic Q ζ i for each agent i , empty replay buffer D , trust-region decomposition network f θ − and g w + , local trust-region
δ
i 1: Set main critic do
Execute environment 6:
Observe next local observation i , local reward r i local done signal d of agent i 7:
Store ( { o i } , { a i } , { r i } , { o (cid:48) i } , { d i } ) in D 8: if it’s time to update then 9: for j in range(however many updates) do 10:
Sample a batch of transitions B from D 11:
Compute {C ti, \ i } with Eq.12:
Compute {D ti, ns } with Eq.13: for slow update do 14:
Update δ ti with Eq.15: for fast update do 16:
Update f θ − , g w + with Eq.17:
Update centralized critic of all agents by Eq.18:
Update individual policy of all agents by Eq.19:
Update target networks Q ¯ ζ i and π ¯ ψ i 20: if it’s time to update then 21:
Update old policy π ψ old
i
In order to verify the existence of the trust-region decomposition dilemma, we deﬁned a simple coordination environment.
We extend the Spread environment of Section 4 toagents and 3 landmarks, labeled Spread-3 .
We deﬁne different Markov random ﬁelds (see
Figure 7) of the joint policy by changing the reward function to inﬂuence the transition function of each agent indirectly .
Speciﬁcally, the reward function of each agent is composed of two parts: the minimum distance between all agents and the landmarks, and the other is the collision.
For the leftmost MRF in Figure 7, all agents are independent.
The reward function of each agent is only related to itself, only related to the minimum distance between itself and a speciﬁc landmark, and will not collide with other agents.
We labeled this situation as Spread-3Sep .
For the middle MRF in Figure 7, the reward function of agent j is the same as that of
Spread-3-Sep , which is only related to itself; but agent i and k are interdependent.
For agent i , its reward function consists of the minimum distance between i and k and the landmark and whether the two collide.
The reward function of agent k is similar.
We labeled this situation as Spread-3-Mix .
Finally, the rightmost MRF is consistent with the standard environment settings, labeled Spread-3-Ful .
To verify the existence of the trust-region decomposition dilemma, we compare the performance of three different algorithms.
First, we select MAAC without any trust-region constraints as the baseline, labeled MAAC .
Secondly, we choose MAMD based on meanﬁeld approximation and naive trust-region decomposition as one of the algorithms to be compared, labeled MAMD .
Finally, we optimally assign trust-region based on prior knowledge.
For Spread-3-Sep , we do not impose any trust-region constraints, same as MAAC ; for Spread-3-Mix , we only impose equal size constraints on agent i and k ; and for Spread3-Ful , we impose equal size constraints on all agents, same as MAMD .
We labeled these optimally decomposition as MAMD-OP .
The performance is shown in Figure 8.
[Figure]
It can be seen from the ﬁgure that inappropriate decomposition of the trust-region will negatively affect the convergence speed and performance of the algorithm.
The optimal decomposition method can make the algorithm performance and convergence speed steadily exceed baselines.
Note that the three algorithms compared here are all based on the MAAC with centralized critics.
After we change the reward function of the agent, the information received by these centralized critics has more redundancy in some scenarios (for example, in Spread-3-Sep and Spread-3-Mix ).
To exclude the algorithm’s performance from being affected by this redundant information, we output the attention weights in MAAC, as shown in Figure 9.
It can be seen from the ﬁgure that different algorithms can ﬁlter redundant information well, thus eliminating the inﬂuence of redundant information on the convergence speed and performance of the algorithm.
j k 0.
i k 0.
i j 0.
(a) MAAC in Spread-3-Sep. j k 0.
i k 0.
i j 0.
(b) MAMD in Spread-3-Sep. j k 0.
i k 0.
i j 0.
(c
MAMD-OP in Spread-3-Sep. j k 0.
i k 0.
i j 0.
j k 0.
i k 0.
i j 0.
(e) MAMD in Spread-3-Mix.
j k 0.
i k 0.
i j 0.
(f) MAMD-OP in Spread-3-Mix.
E P ROOFS E. P ROOF OF L EMMAProof.
D TV ( p t ( ·| o , a i ) , p t +1 ( ·| o , a i ))
=
max o ,a i | p t +1 ( ·| o , a i ) − p t ( ·| o , a i ) | = max o ,a i | (cid:90) p ( ·| o , a ) · (cid:0) π t +1 − i ( a − i | o ) − π t − i ( a − i | o ) (cid:1) d a − i | ≤ max o (cid:90) (cid:12)(cid:12)
π t
+1 − i ( a − i | o ) − π t − i ( a − i | o ) (cid:12)(cid:12)
d a − i = max o (cid:13)(cid:13)
π t +1 − i ( o )
[Equation]
π t − i ( o ) (cid:13)(cid:13)
[Equation]
· max o
[Equation] [Equation]
t
[Equation]
i ( o
[Equation]
(cid:107)
[Equation]
t
[Equation]
1 − i ( o )) ≤ 2 ln 2 · δ 1 / 2 i .
E. P ROOF OF L EMMAProof.
D TV ( r t ( o , a i ) , r t +1 ( o , a i ))
=
max o ,a i | r t +1 ( o , a i ) − r t ( o , a i ) | = max o ,a i | (cid:90) r ( o , a ) · (cid:0) π t +1 −
i ( a − i | o ) − π t − i ( a − i | o ) (cid:1) d a − i | ≤ max o (cid:90) (cid:12)(cid:12)
π t
+1 − i ( a − i | o ) − π t − i ( a − i | o ) (cid:12)(cid:12)
d a − i = max o (cid:13)(cid:13)
π t +
[Equation]
E. P ROOF OF T HEOREM
Proof.
In this paper, we model the learning procedure of each agent in a multi-agent system as a dynamic non-stationary MDP.
From each agent’s perspective, the quantities r t ( o , a i ) ’s and p t ( ·| o , a i ) ’s of each agent i vary across different t ’s in general.
Following Besbes et al.
(2014), Cheung et al.
and Mao et al.
(2021), we quantify the variations on r t ( o , a i ) ’s and p t ( ·| o , a i ) ’s in terms of their respective variation budgets B r , B p
[Equation]
>
[Equation] [Equation]
B r
[Equation]
t , where B r
[Equation]
t =
[Equation]
B p
[Equation]
where B p
[Equation]
To measure the convergence to the best-response from each agent’s perspective, we consider an objective of minimizing the dynamic regret (Jaksch et al., 2010; Besbes et al.,
2014; Cheung et al
[Equation]
, 2019; Mao et al
[Equation] [Equation]
In the oracle (cid:80) Tt =1 ρ ∗ t , the summand ρ ∗ t is the optimal long-term average reward of the stationary MDP, i.e., other agents follow the ﬁxed optimal policies, with state transition distribution p i,t and mean reward r i,t .
Below we give a deﬁnition and an assumption.
Deﬁnition 4 (Communicating MDPs and Diameter) .
Consider a set of states S , a collection A
= {A s } s ∈S of action sets, and a state transition distribution ¯ p
= { ¯ p ( · | s, a ) } s ∈S ,a ∈A s .
For any s, s (cid:48) ∈
S and stationary policy π , the hitting time from s to s (cid:48) under π is the random variable Λ ( s (cid:48) | π, s ) :
= min { t : s t +
= s (cid:48) , s
= s, s
τ +1 ∼ ¯ p
( · | s τ , π ( s τ )) ∀ τ } , which can be inﬁnite.
We say that is a communicating MDP
iff D := max s,s (cid:48) ∈S min stationary π E [Λ ( s (cid:48) | π, s )] is ﬁnite.
The quantity D is the diameter (Jaksch et al., 2010) associated with ( S , A , ¯ p ) .
Assumption 2 (Bounded Diameters) .
For each t ∈ [ T ] , the tuple ( S , A , p t ) constitutes a communicating MDP with diameter at most D t .
We denote the maximum diameter as
D max = max t ∈{ 1 ,...,T } D t .
Then we have following proposition (Cheung et al., 2019) from each agent’s perspective:
Proposition 2.
Consider an instance ( S , A , T, p, r ) from each agent’s perspective that satisﬁes Assumption 2 with maximum diameter D max and has variation budgets B r , B p for rewards and transition distributions respectively.
In addition, suppose that T ≥ B r + 2 D max B p > 0 , then it holds that
[Equation]
The maximum is taken over all non-anticipatory policies Π ’s.
We denote (cid:8)(cid:0) s Π t , a Π t (cid:1)(cid:9) T t
=1 as the trajectory under policy Π , where a Π t ∈
A s Π t is determined based on Π and H t − 1 ∪ (cid:8) s Π t (cid:9) , and s Π t +1 ∼ p t (cid:0) · | s Π t , a Π t (cid:1) for each t .
The proof of Proposition 2 is shown in (Cheung et al., 2019).
Based on Lemma 1 and
Lemaa 2, we can easily obtain B r ≤ 2 ln 2 · δ 1 / 2 i · T and B p ≤ 2 ln 2 · δ 1 / 2 i · T · |O| .
Then we have following corollary:
Corollary 1.
Consider an instance ( S , A , T, p, r ) from each agent’s perspective that satisﬁes Assumption 2 with maximum diameter D max and has variation budgets B r , B p for rewards and transition distributions respectively.
In addition, suppose that T ≥
B r
[Equation]
2 D max B p > 0 , then it holds that
[Equation]
According to the Deﬁnition 4, Theorem 1 is proved.
E. P ROOF OF T HEOREM
Proof.
[Equation]
i | π (cid:48)− i (cid:3) . So we have KL [ π | π (cid:48) ] ≥ 1 n (cid:80) i KL (cid:2) π − i | π
(cid:48)− i (cid:3) .
Take the maximum value on both sides of the inequality, Theorem 2 is proved.
Since a similar conclusion is reached in Li & He (2020), we will make a brief comparison with it here.
Theorem 2 establishes the relationship between the maximum KL-divergence of the consecutive joint policies of all agents and the maximum KL-divergence of the consecutive joint policies of other agents.
It establishes the theoretical connection between the KL-divergence of the consecutive joint policies of all agents and environmental nonstationarity.
The Equation (8) of Li & He (2020) extends the KL divergence constraint of the TRPO algorithm to the multi-agent scenario and establishes the connection between the divergence of the joint policy of all agents and the divergence of local policy of each agent.
E. P ROOF OF T HEOREM
Proof.
E o ∼
[Equation]
o
[Equation]
o
[Equation]
< δ ⇐⇒ E o ∼ µ (cid:20)(cid:90) a π ( ·| o ) log π ( ·| o )
π k ( ·| o ))
d a (cid:21) <
δ ⇐⇒ E
o ∼ µ (cid:34)(cid:90) a π ( ·| o ) (cid:32) n (cid:88)
i = log π i ( ·| o i )
π ki ( ·| o i ))
(cid:33) d a (cid:35) <
δ ⇐⇒ E
o ∼ µ (cid:34) n (cid:88) i =
(cid:90) a π ( ·| o ) log π i ( ·| o i )
π ki ( ·| o i ))
d a (cid:35) <
δ ⇐⇒ n (cid:88) i =
E o ∼ µ
[Equation]
⇐⇒ n (cid:88) i =
(cid:20)(cid:90)
[Equation]
We ﬁrst simplify the integral term of the inner layer
[Equation]
We replace the original formula with the simpliﬁed one
[Equation]
where we denote KL (cid:0) π i ( ·| o i ) , π ki ( ·| o i ) (cid:1) as δ ( o i ) .
For the outer integral term, we have (cid:90) o µ ( o ) δ
( o i )
d o ⇐⇒ (cid:90)
o \ i × o i µ ( o \ i ) µ ( o i ) δ
( o i )
d o \ i
d o i ⇐⇒ (cid:90)
o
i µ ( o i ) δ
( o i ) (cid:34)(cid:90)
o \ i µ ( o \ i )
d o \ i (cid:35) d o i ⇐⇒ (cid:90) o i µ ( o i ) δ
( o i )
d o i ⇐⇒
E
o i ∼ u
i
[ δ ( o i )] .
Overall, we have
E
[Equation]
⇐⇒ n (cid:88)
[Equation]
=
[Equation]
o
[Equation]
⇐⇒ n (cid:88) i =
[Equation]
o
[Equation]
o
[Equation]
< δ ⇐⇒ n (cid:88) i =
E
[Equation]
<
[Equation]
⇐⇒ n
[Equation]
i =
E o
i
[Equation]
u i (cid:104)
[Equation]
< δ.
[Table]
F M ORE D ISCUSSION ON THE L INEAR R EGRET
The bound on Theorem 1 has a linear dependence on the number of time intervals T = HK .
Therefore, the average regret doesn’t vanish to zero as T becomes large, even when small non-stationarity.
But from the experimental results, our algorithm has a faster convergence speed and better performance than baselines.
Therefore, the rest of this section will expand from the following two aspects.
First, although theoretically only linear regret can be achieved, it can be close to sub-linear in actual implementation; second, we can achieve theoretical and implementation sub-linearity through some improvements that will bring a higher computational load. [Equation]
MAMT I S A N EAR -S UBLINEAR R EGRET I MPLEMENTATION
First, we explain why controlling the δ -stationarity can only achieve linear regret.
Tablelists the sublinear regret bounds that current SOTA algorithms to solve the single-agent non-stationarity problem can reach.
It is worth noting that these single-agent algorithms cannot modify ∆ because this is the intrinsic property of the external environment.
But in our method, the environment is made up of other agents’ policies, so ∆ can be adjusted.
As can be seen from the table, the power of ∆ =
( B r + B p ) and T add up to 1 .
And from the proof of Theorem 1, B r and B p are both the summation over T , i.e., B r = (cid:80)
Tt =1 B tr and B p = (cid:80) Tt =1 B tp .
However, opponent switching cost considers the maximum value of the continuous policy divergence in the entire learning procedure, so we have
[Equation]
where 0 < a ≤ 1 / 3 .
In general, constraining δ -stationarity produces a linear regret because the opponent switching cost is a relatively loose constraint.
The key to achieving sublinear regret is to constrain the consecutive joint policy divergence at each timestep, not the maximum or expected value of the entire learning procedure.
Therefore, we propose [Table] a tighter constraint based on opponent switching cost, and prove that by imposing this constraint on the consecutive joint policy divergence, the algorithm can achieve sub-linear regret
˜ O
( D 3 / 2 max |O| 1 / 2 ∆ 1 / 4 δ T 1 / 2 ) .
First, we deﬁne the temporal opponent switching cost.
Deﬁnition 5 (Temporal opponent switching cost) .
Let H be the horizon of the MDP and
K be the number of episodes that the agent can play, so that total number of steps T :=
HK .
The temporal opponent switching cost of agent i is deﬁned as the Kullback–Leibler divergence (over the joint observation space O at speciﬁc timestep t ) between any pair of opponents’ joint policies ( π − i , π (cid:48)− i ) on which π − i and π (cid:48)− i are different: [Equation] [Table] [Equation]
[Table]
[Equation]
where o , O are the joint observation and observation space and t ∈ T .
Correspondingly, we can also deﬁne temporal δ t -stationarity.
Deﬁnition 6 (Temporal δ t -stationarity) .
For a MAS containing n agents, if we have d i,t switch (cid:0) π − i , π (cid:48)− i (cid:1) ≤ δ ti , then the learning procedure of agent i is δ ti -stationary at timestep t.
Further, if all agents are δ -stationary with corresponding { δ ti } , then the learning procedure of entire multi-agent system is δ t -stationary with δ t = 1 n (cid:80) i δ ti .
Similarly, we can also get Lemma 1 and Lemma 2 about temporal δ t -stationarity.
Since the content is similar, we won’t repeat them here.
Then we have B r ≤ 2 ln 2 · (cid:80) Tt =
δ ti 1 /and B p ≤ 2 ln 2 · (cid:80) Tt =
δ ti 1 / 2 · |O| .
In addition, we denote, B iδ =
(cid:80) Tt =
δ ti 1 / 2 .
With these deﬁnitions and lemmas, we can get the following theorem.
Theorem 4.
Let H be the horizon and K be the number of episodes, so that total number of steps T :=
HK .
Consider the learning procedure of a MAS satisﬁes the δ t -stationarity and each agent satisﬁes the δ ti -stationarity at each timestep t ∈ T .
In addition, suppose that T ≥ B r + 2 D max B p > 0 , then a ˜ O ( D 3 / 2 max |O| 1 / 2 B iδ 1 / 2 T 1 / 2 ) sublinear dynamic regret bound is attained for each agent i .
The proof is similar to Theorem 1.
Theorem 4 shows that if we want to achieve a sublinear regret, we need to constrain the consecutive joint policy divergence at each timestep instead of only constraining the maximum value of the policy divergence.
Looking back at the implementation of the MAMT, although the algorithm has a global hyperparameter δ to constrain the adaptively adjusted { δ i } Ni =1 , the constraint only works when (cid:80) i
δ i > δ .
We have observed in the experiment that most of the time (cid:80) i δ i ≤ δ (see Table 2).
In this way, a different δ t = (cid:80) i δ ti at each timestep t constrain the consecutive joint policy divergence, thereby approximately achieving a sub-linear regret.
F. F UTURE W ORK TO A CHIEVE S UBLINEAR R EGRET
Again, Theorem 4 shows that if we want to achieve a sub-linear regret, we need to constrain the consecutive joint policy divergence at each timestep.
This means that we need to set a hyperparameter δ t at every timestep t to constrain the consecutive joint policy divergence, and a manual adjustment will bring an extremely complicated workload.
[Figure]
A more realistic way is to treat the δ t of each timestep as an additional optimization variable and formulate it as a dual variable about delta ti and the policy parameter ψ ti .
Then the dual gradient descent can be used to optimize δ t , similar to the adaptive reward scale learning in the soft-actor-critic (SAC, Haarnoja et al.
algorithm.
In addition, non-parameter optimization methods, such as evolutionary methods, can also be used to adjust the δ t .
But no matter which technique is adopted, it will bring a higher computational load.
How to balance the computational load and algorithm performance, we leave it to the future to explore.
G E XPERIMENTAL D ETAILS G. E NVIRONMENTS
Spread.
This environment (Lowe et al., 2017) has 2 agents, 2 landmarks.
Each agent is globally rewarded based on how far the closest agent is to each landmark (sum of the minimum distances).
Locally, the agents are penalized if they collide with other agents ( −for each collision).
Multi-Walker.
In this environment (Terry et al., 2020a), bipedal robots attempt to carry a package as far right as possible.
A package is placed on top of 3 bipedal robots.
A positive reward is awarded to each walker, which is the change in the package distance.
Rover-Tower.
This environment (Iqbal & Sha, 2019) involves 8 agents, 4 of which are “rovers” and another 4 which are “towers”.
In each episode, rovers and towers are randomly paired.
The pair is negatively rewarded by the distance of the rover to its goal.
The rovers are unable to see in their surroundings and must rely on communication from the towers, sending one of 5 discrete messages.
Pursuit.
30 blue evaders and 8 red pursuer agents are placed in a grid with an obstacle.
The evaders move randomly, and the pursuers are controlled (Terry et al., 2020a).
Every time the pursuers surround an evader, each of the surrounding agents receives a reward of 5 , and the evader is removed from the environment.
Pursuers also receive a reward of 0 .every time they touch an evader.
G. O THER D ETAILS
Random seeds.
All experiments were run for 8 random seeds each.
Graphs show the average (solid line) and std dev (shaded) performance over random seed throughout training.
Performance metric.
Performance for the on-policy (MA-PPO) algorithms is measured as the average reward across the batch collected at each epoch. Performance for the off-policy [Table] num
[Table]
step size from 10,000 to 50, num epochs per step steps per update buffer size
[Table] batch size [Table] batch handling
Shuffle transitions num critic attention heads value loss
MSE modeling policy loss
CrossEntropyLoss discount
[Table]
optimizer
Adam adam lr 1eadam mom
[Table]
adam eps 1elr decay 0.
policy regularization type
L policy regularization coefficient 0.
modeling policy regularization type
L modeling policy regularization coefficient 0.
critic regularization type
L critic regularization coefficient 1.
critic clip grad 10 * num of agents policy clip grad 0.
soft reward scale modeling policy clip grad 0.
trust-region decomposition network clip grad 10 * num of agents trust-region clip from 0.01 tonum of iteration delay in mirror descent tsallis q in mirror descent 0.
δ in coordination coefficient 0.
algorithms (MAMT, MAMD, LOLA, MADDPG, and MAAC) is measured by running the deterministic policy (or, in the case of SAC, the mean policy) without action noise fortrajectories and reporting the average reward over those test trajectories.
Network Architecture.
Figure 11 shows the detailed parameters of the three main networks in the MAMT algorithm.
Hyperparameters.
Table 3 shows the default conﬁguration used for all the experiments of our methods (MAMD and MAMT) and baselines in this paper.
We do not ﬁne-tune the hyperparameters of baselines and use the default setting as same as original papers, see
Table 4,5,6 and 7.
The hyperparameter ﬁne-tune range of our methods are shown in Tableand 9.
Considering the long training time of the MARL algorithm, we did not train all hyperparameter combinations to the pre-deﬁned maximum number of episodes for MAMT.
We ﬁrst train all hyperparameter combinations to one-sixth of the maximum number of episodes and select the top-sixth hyperparameter combinations with the best performance (deﬁned in Performance metric. ).
Then we train the selected one-sixth combination to one-third of the maximum number of episodes and select the best-performing one-sixth combination.
Finally, we train the remaining combinations to the maximum number of episodes and select the best hyperparameter combination.
[Table]
step size from 10,000 to 50, num epochs per step steps per update buffer size
[Table]
batch size [Table] batch handling
Shuffle transitions num critic attention heads value loss
[Table]
discount
[Table]
optimizer
Adam adam lr 1eadam mom
[Table]
adam eps 1elr decay 0.
policy regularization type
L policy regularization coefficient 0.
critic regularization type
L critic regularization coefficient 1. critic clip grad 10 * num of agents policy clip grad 0. soft reward scale [Table]
step size from 10,000 to 50, num epochs per step steps per update buffer size [Table] batch size batch handling Shuffle
transitions
value loss
[Table]
discount [Table] optimizer Adam adam lr 1eadam mom
[Table] adam eps 1elr decay 0. policy regularization type
L policy regularization coefficient 0.
critic regularization type L critic regularization coefficient 1. critic clip grad 10 * num of agents policy clip grad 0.
Hardware.
The hardware used in the experiment is a server with 128 G memory and
NVIDIA 1080Ti graphics cards with 11 G video memory.
The Code of Baselines.
The code and license of baselines are shown in following list:
[Table]
step size from 10,000 to 50, batch size
[Table]
num critic attention heads
[Table]
value loss
[Table]
Huber delta
[Table]
GAE
[Table]
discount [Table]
optimizer
[Table]
policy regularization type
L policy regularization coefficient 0.
critic regularization type
L critic regularization coefficient 0.
critic clip grad policy clip grad use reward normalization
TRUE use feature normalization
TRUE
[Table]
buffer size
[Table]
batch size
[Table]
batch handling
Shuffle transitions value loss
[Table]
discount
[Table]
optimizer
[Table]
adam
[Table]
adam
[Table]
decay
[Table]
regularization type
L regularization coefficient
[Table]
clip grad 10 * num of agents
• MADDPG (Lowe et al., 2017): https://github.com/ shariqiqbal2810/maddpg-pytorch , MIT License; • MAAC (Iqbal & Sha, 2019): https://github.com/shariqiqbal2810/ MAAC , MIT License; • MA-PPO: https://github.com/zoeyuchao/mappo , MIT License; • LOLA
(Foerster et al., 2018a):
https://github.com/alexis-jacq/ LOLA_DiCE and https://github.com/geek-
ai/MAgent , MIT License. [Table] [Table] [Figure]
[Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
Learning curves are smoothed by averaging over a window of 11 epochs.
Source code is available at https://anonymous.4open.science/r/MAMT .
H M ORE R ESULTS AND A BLATION S TUDIES
Figure 13 to Figure 23 show the performance indicators of all agents in 4 environments under different random seeds.
In addition, we have also conducted additional ablation studies to analyze the importance of different modules in MAMT.
These modules mainly include mirror descent, coordination coefﬁcients, and modeling of others.
Regarding the mirror descent.
Considering the low sample efﬁciency of MARL, we did not adopt the fully on-policy learning but used off-policy training with a small replay buffer.
In this way, it can be ensured that samples are less different from the current policy, thereby alleviating the instability caused by off-policy training.
The methods such as TRPO and
PPO are all on-policy algorithms.
Although there are some off-policy trust-region methods, such as Trust-PCL, etc., they are all more complicated.
To this end, we adopted a more concise technique, Mirror Descent, as the optimization method of MAMT.
MAMT can
[Table]
also use the on-policy method as the backbone, such as PPO.
To this end, we have added a comparative experiment, replacing Mirror Descent with the PPO algorithm (MAMT-PPO).
Regarding the coordination coefﬁcients.
In the MAMT algorithm, the counterfactual baseline is used in policy learning and used in the calculation of coordination coefﬁcients.
To verify the relationship between the correctness of the modeling of the dependencies between agents and the algorithm’s performance, we have added two comparative experiments to the revised version.
Speciﬁcally, we used two pre-deﬁned ways to set the coordination coefﬁcients, set to a ﬁxed value (MAMT-Fixed) of 1 / ( n − 1) (where n is the number of agents), and random sampling (MAMT-Random, random sampling from the range of (0 , 1) and then input into the softmax function).
Regarding the modeling of others.
We borrowed the idea in ofﬂine RL to measure the degree of out-of-distribution (see Equation (5)) through the model-based RL to reasonably characterize the non-stationarity or the consecutive joint policy divergence.
However, a more intuitive and simple method is to directly perform a weighted summation of the consecutive local policy divergence (the weights are the coordination coefﬁcients).
The reason why we did not adopt this scheme is based on the following considerations.
It can be seen from Deﬁnition 3 that the agents’ policies do not directly cause the non-stationarity.
The agent cannot directly observe the opponent’s policy but only the action sequences.
This explicit information directly affects the learning process of the agent and leads to non-stationarity.
To verify the correctness of the idea, we also did additional comparative experiments.
Speciﬁcally, we changed the calculation method of coordination coefﬁcient to the weighted summation of consecutive local policy divergence (MAMT-WS), i.e.
[Equation] [Equation]
We compared the above ablation algorithms under 4 cooperative tasks, and the experimental results are shown in Table 10.
The following points can be seen from the Table 10.
First of all, from the comparison results of MAMT-PPO and MAMT, it can be seen that our method is not signiﬁcantly affected by the optimization method of the policy learning, and both achieve the best results on the four cooperative tasks.
However, MAMT-PPO is an on-policy method, which makes its sample effectiveness lower than MAMT.
In Figure 12, we have made statistics on the number of episodes required by the two to achieve the performance in Table 10.
It can be seen from the table that MAMT only needs about half of the data to achieve the same performance as MAMT-PPO.
Secondly, from the results of MAMT-Random and MAMT-Fixed, the coordination coefﬁcient is signiﬁcant to the algorithm’s performance, which is also consistent with the theoretical analysis of the trust-region decomposition dilemma in the main body.
Here we have observed two interesting phenomena.
First, we found that randomizing the coordination coefﬁcient will seriously affect the performance of MAMT.
The possible reason is that the wrong trust-region constraint on the local policy will hinder the agent’s learning.
Second, we found that setting the coordination coefﬁcient to the same ﬁxed value will make the performance of the MAMT algorithm approach that of MAMD.
This shows that the TRD-Net ﬁts the relationship between the local trust regions and the approximated consecutive joint policy divergence well.
Third, it can be seen from the results of MAMT-WS that the effect of approximating the
KL divergence of consecutive joint policies by using the weighted summation of the KL divergence of consecutive local policy policies is very unstable.
In all cooperative tasks,
MAMT-WS has greater variance. In simple tasks, such as Spread ( 2 -agents) and MultiWalker ( 3 -agents), MAMT-WS can have an effect close to MAMT. But once the difﬁculty of the task increases, the effect of MAMT-WS will even be at the same level as MAMTRandom. We believe that the main reason for this instability is that the agent will cause rapid changes in local policies due to exploration in the early stages of learning.
AgentAgent-
AgentAgent-5 AgentAgent-
Agent0.2 0.4 0.
meanvariance (a) Agent 1.
AgentAgent-
AgentAgent-5 AgentAgent-
Agent0.2 0.4 0.
meanvariance (b) Agent 3.
AgentAgent-
AgentAgent-4 AgentAgent-
Agent0.2 0.4 0.
meanvariance (c) Agent 5.
AgentAgent-
AgentAgent-4 AgentAgent-
Agent0.2 0.4 0.
meanvariance (d) Agent 7.
AgentAgent-
AgentAgent-5 AgentAgent-
Agent0.2 0.4 0.
meanvariance (e) Agent 2.
AgentAgent-
AgentAgent-5 AgentAgent-
Agent0.2 0.4 0.
meanvariance (f) Agent 4.
AgentAgent-
AgentAgent-4 AgentAgent-
Agent0.2 0.4 0.
meanvariance (g) Agent 6.
AgentAgent-
AgentAgent-4 AgentAgent-
Agent0.2 0.4 0.
meanvariance (h) Agent 8.
[Figure]
[Figure] [Figure]
[Figure]
(a) Agent 1.
[Figure]
AgentAgent-
Agent-5 AgentAgent-
Agent0.2 0.4 0.
meanvariance (b) Agent 3.
AgentAgent-
AgentAgent-4 AgentAgent-
Agent0.2 0.4 0.
meanvariance (c) Agent 5.
AgentAgent-
AgentAgent-4 AgentAgent-
Agent0.2 0.4 0.
meanvariance (d) Agent 7.
AgentAgent-
AgentAgent-5 AgentAgent-
Agent0.2 0.4 0.
meanvariance (e) Agent 2.
AgentAgent-
AgentAgent-5 AgentAgent-
Agent0.2 0.4 0.
meanvariance (f) Agent 4.
AgentAgent-
AgentAgent-4 AgentAgent-
Agent0.2 0.4 0.
meanvariance (g) Agent 6.
AgentAgent-
AgentAgent-4 AgentAgent-
Agent0.2 0.4 0.
meanvariance (h) Agent 8.
[Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
R EFERENCES FOR S UPPLEMENTARY M ATERIAL