M ODEL -B ASED D ECENTRALIZED P OLICY O PTIMIZA -
TION
Anonymous authors
Paper under double-blind review
A BSTRACT
Decentralized policy optimization has been commonly used in cooperative multiagent tasks.
However, since all agents are updating their policies simultaneously, from the perspective of individual agents, the environment is non-stationary, resulting in it being hard to guarantee monotonic policy improvement.
To help the policy improvement be stable and monotonic, we propose model-based decentralized policy optimization (MDPO), which incorporates a latent variable function to help construct the transition and reward function from an individual perspective.
We theoretically analyze that the policy optimization of MDPO is more stable than model-free decentralized policy optimization.
Moreover, due to non-stationarity, the latent variable function is varying and hard to be modeled.
We further propose a latent variable prediction method to reduce the error of latent variable function, which theoretically contributes to the monotonic policy improvement.
Empirically,
MDPO can indeed obtain superior performance than model-free decentralized policy optimization in a variety of cooperative multi-agent tasks.
I NTRODUCTION
Decentralized multi-agent reinforcement learning (MARL) has been commonly used in practice for cooperative multi-agent tasks, e.g. , traffic signal control (Wei et al., 2018), unmanned aerial vehicles (Qie et al., 2019), and IoT (Cao et al., 2020), where global information is inaccessible.
Independently performing policy optimization using local information, e.g. , independent PPO (Schulman et al., 2017) (IPPO), is one of the most straightforward methods for decentralized MARL.
Recent empirical studies (de Witt et al., 2020; Yu et al., 2021a; Papoudakis et al., 2021) demonstrate that IPPO performs surprisingly well in several cooperative multi-agent benchmarks, which shows great promise for fully decentralized policy optimization.
However, since all agents are updating policies, from the perspective of an individual agent, the environment is non-stationary (Zhang et al., 2019).
Thus, the monotonic policy improvement, which can be achieved by policy optimization in single-agent settings (Schulman et al., 2015; 2017), may not be guaranteed in decentralized MARL.
Concretely, in policy optimization, the state visitation frequency is assumed to be stationary since the agent policy is limited to slight updates, which is necessary to guarantee monotonic policy improvement (Schulman et al., 2015).
However, in decentralized multi-agent settings, as all agents are updating policies simultaneously, the state visitation frequency will change greatly, which contradicts the fundamental assumption of policy optimization, thus the monotonic improvement of policy optimization may not be preserved.
To address this problem, we resort to exploiting the environment model to stabilize the state visitation frequency and help monotonic policy improvement.
However, learning an environment model in decentralized settings is non-trivial, since the information of other agents, e.g. , other agents’ policies, is not observable and changing.
Therefore, we introduce a latent variable to help distinguish different transitions resulting from the unobservable information.
And then we build an environment model for each agent, which contains a transition function, a reward function, and a latent variable function that learns the latent variable given observation.
The agents are trained using independent policy optimization methods, e.g. , TRPO (Schulman et al., 2015) or PPO (de Witt et al., 2020), on both the experiences generated by the environment model and collected in the environment.
Since the environment is non-stationary, the latent variable function is also varying during learning.
We theoretically show that independently performing policy optimization on experiences generated by the environment model with the varying latent variable function can obtain more stationary observation visitation frequency than on the experiences collected in the non-stationary environment.
Thus, independent policy optimization goes more stable on the environment model.
Moreover, to obtain monotonic improvement, the gap between the return of interacting with the environment and the return predicted by the environment model should be small.
We theoretically analyze that the return gap is bounded by the prediction error of the latent variable function.
As the latent variable function is varying due to non-stationarity, to minimize the prediction error, we propose a latent variable prediction method that uses the historical variables to predict the future variable.
Thus, the latent variable prediction can reduce the return gap and help the monotonic policy improvement.
The proposed algorithm, model-based decentralized policy optimization ( MDPO ), is theoretically grounded and empirically effective for fully decentralized learning.
We evaluate MDPO on a variety of cooperative multi-agent tasks, i.e. , a stochastic game, multi-agent particle environment (MPE) (Lowe et al., 2017), and multi-agent MuJoCo
(Peng et al., 2021a).
MDPO outperforms the model-free independent policy optimization baseline, and the proposed latent variable prediction additionally obtains performance gain, verifying that MDPO can help stable and monotonic policy improvement in fully decentralized learning.
P RELIMINARIES
Dec-POMDP.
A cooperative multi-agent task is generally modeled as a decentralized partially observable Markov decision process (Dec-POMDP) (Oliehoek & Amato, 2016).
Specifically, a
Dec-POMDP is defined as a tuple G =
{S , I , A , O , Ω , P, R, γ } .
S is the state space, I is the set of agents, and
A =
A 1 × · · · × A |I| is the joint action space, where A i is the action space for each agent i .
At each state s , each agent i ∈ I merely gets access to the observation o i ∈
O , which is drawn from observation function Ω( s, i ) , and selects an action a
i ∈
A i , and all the actions form a joint action a ∈ A .
The state transitions to next s ′ according to the transition function
P ( s ′ | s, a ) : S ×
A × S → [0 , 1] , and all agents receive a shared reward r
= R ( s, a ) : S ×
A → R .
The objective is to maximize the expected return η ( π ) =
E [ (cid:80) ∞ t =0 γ t r t | ρ 0 , π ] under the joint policy of all agents π and distribution of initial state ρ 0 , where γ ∈ [0 , 1) is the discounted factor.
The joint policy π can be represented as the product of each agent’s policy π i .
Also we denote π − i as the joint policy of all agents except i .
Fully decentralized learning.
We consider the fully decentralized way to solve the Dec-POMDP (Tan, 1993; de Witt et al., 2020), where each agent independently learns a policy and executes actions without communication or parameter sharing in both training and execution phases.
Since all agents are updating policies, from the perspective of individual agents, the environment is nonstationary, which fundamentally challenges decentralized learning (Zhang et al., 2019).
The existing decentralized MARL methods are limited.
Independent Q-learning (IQL) (Tan, 1993) and independent policy optimization, e.g. , IPPO (de Witt et al., 2020), are the most straightforward fully decentralized algorithms.
Despite good empirical performance (Papoudakis et al., 2021), due to non-stationarity, these methods lack theoretical guarantees.
IQL has no convergence guarantee, to the best of our knowledge.
Although there has been some study (Sun et al., 2022), IPPO may not guarantee policy improvement by independent policy optimization, since the assumption of stationary state visitation frequency for policy optimization may not hold in fully decentralized settings, which we will discuss in the following.
Monotonic policy improvement.
In Dec-POMDP, from a centralized perspective, we can obtain a
TRPO objective (Schulman et al., 2015) of the joint policy π for the monotonic improvement,
[Equation]
where ρ π old ( s ) = (cid:80) t =
γ t Pr( s t = s
| π old ) is the discounted state visitation frequency given π old , similarly for ρ π new ( s ) , A π old is the advantage function under π old , D maxKL ( π old ∥
π new )
= max s D KL ( π old ( ·| s )
∥
π new ( ·| s )) , and C is a constant.
From (1) to (2) is an approximation or assumption (Schulman et al., 2015).
As ρ π new is unknown and the policy is limited to slight updates, ρ π new is approximated by ρ π old .
However, in fully decentralized MARL, this assumption may not hold, as all agents are updating their policies simultaneously and their joint policy may change significantly especially when the number of agents is large.
This will severely affect the performance of independent policy optimization.
Although we can constrain the policy update of each agent to be slight like TRPO, this leads to much slower convergence, especially in fully decentralized
MARL, where the joint policy has a much larger search space and is merely optimized by independent learning of individual agents.
M ODEL -B ASED D ECENTRALIZED P OLICY O PTIMIZATION
In this paper, we provide a novel perspective and resort to the environment model to bridge the gap between ρ π new and ρ π old for each agent such that the monotonic joint policy improvement can be potentially achieved by fully decentralized policy optimization.
As illustrated in the following, we turn the learning process into a Dyna-style (Sutton, 1990) decentralized model-based method with the green path 1 .
Each agent i additionally learns a decentralized model using local information from policy rollout and can optionally perform policy optimization on the experiences from model rollout.
When optimizing policy with model rollout, we essentially have 1 , which means the state visitation frequency in model rollout ( ρ model ) is more stable.
Thus, the approximation from (1) to (2) becomes acceptable under looser constraints of policy update.
Further, we can bound the gap between the returns of policy rollout ( η ) and model rollout ( η model ), formally in 2 .
Once the bound ( B ) is controllable throughout the learning process, it can potentially guarantee the monotonic improvement of the joint policy in the real environment.
[Table]
Thus, 1 and 2 together highlight the potential benefits of incorporating an environment model for decentralized policy optimization.
In the following, we discuss how to learn such a decentralized model, theoretically investigate its benefits for decentralized policy optimization, and analyze the return bound for monotonic policy improvement.
3. L ATENT V ARIABLE M ODEL
In decentralized learning, from the perspective of each agent i , the transition function and reward function are respectively,
[Equation]
where a − i denotes the joint action of all agents except i .
As other agents are also updating their policies, P i and R i are varying throughout the learning process, which is the well-known nonstationarity problem.
Moreover, as each agent i usually obtains observation instead of state in decentralized learning, the model can only be learned on ( o i , a i , o ′ i , r ) .
Thus, it is challenging to construct an environment model from the perspective of an individual agent.
To build a decentralized environment model, we introduce a latent variable
z i , which helps distinguish different transitions resulting from varying unobservable information of the full state and other agents’ policies.
Then the transition function and the reward function can be redefined as:
[Equation]
R
[Equation]
, z i
[Equation]
As we discuss fully decentralized learning, we drop the subscript of i for simplicity in the following.
To model the transition function and the reward function with the latent variable, we define the latent variable function from the perspective of an individual agent, ψ ( z | o ) , which indicates the probability of latent variable z given observation o .
As z is related to the policies of other agents, ψ ( z | o ) also varies during policy updates.
A latent variable model consists of three modules: transition function P θ , reward function R ϕ , and latent variable function ψ ω , to predict the next observation and reward.
As the impact of unobservable information is designed to merely reflect on the latent variable, although other agents update their policies, the transition function and reward function stay constant and only the latent variable function varies.
We learn such a model by maximizing the likelihood of experiences of policy rollout D ,
[Equation]
We examine the correlation between latent variable learned end-to-end and inaccessible information in a simple setting, and the learned latent variable is indeed correlated with the inaccessible information.
More details can be found in Appedix B.
Moreover, when using the learned latent variable model to train an agent, we adopt k -step branched model rollout in MBPO (Janner et al., 2019) to avoid compounding model error due to longhorizon rollout.
Concretely, at each policy update of an agent, we sample h -step length experiences { ( o 1 , a 1 , o ′ 1 , r 1 ) , · · · , ( o h , a h , o ′ h , r h ) } from policy rollout D and perform k -step model rollout starting from the last observation o ′ h under current policy π .
The policy π is updated on the merged ( h + k ) -step experiences { ( o 1 , a 1 , o ′ 1 , r 1 ) , · · · , ( o h + k , a h + k , o ′ h
+ k , r h + k ) } by policy optimization, e.g. , PPO (Schulman et al., 2015).
3. S
Now, we turn to analyze the benefits of such a model-based method over model-free independent policy optimization.
We first theoretically analyze that independently performing policy optimization e.g., , TRPO (Schulman et al., 2015) or PPO (Schulman et al., 2017), on the latent variable model can make the learning process more stable.
In decentralized learning, from the perspective of an agent, given the true latent variable function ψ , the discounted observation visitation frequency of D obtained by policy rollout is defined as
[Equation]
where ρ π,ψt ( o ) ≜
Pr ( o t = o ) and o t is the observation at timestep t of experience from D .
Note that ρ π,ψ ( o ) is an unbiased estimate of discounted observation visitation frequency when interacting in the environment.
Similarly, ρ π,ψ ω denotes the discounted observation visitation frequency for experiences obtained by model rollout.
During the learning process, π n and ψ n respectively denote the policy and latent variable function after the n th policy update.
Then, we have the following theorem.
All proofs are available in Appendix A.
Theorem 1.
Define ∆ ρ n ( o ) ≜ ρ π n ,ψ n ( o ) − ρ π n − 1 ,ψ n − 1 ( o ) , and denote ∥ ∆ ρ n ∥ ≜ max o | ρ n ( o ) − ψ n − 1 ( o ) | , similarly for ∥ ∆ π n ∥ and ∥ ∆ ψ n ∥ .
It holds that,
[Equation]
where E π ≜ max n ∥ ∆ π n ∥ , E ψ ≜ max n ∥ ∆ ψ n ∥ and C is a constant.
Assume ψ nω =
(1 − α ) ψ n − 1 ω + αψ n and ψ 0 ω =
ψ 0 2 .
It holds that E ψ > E ψ ω and the bound above is lower when substituting ψ with ψ ω .
According to Theorem 1, the divergence of discounted observation visitation frequency is bounded by the divergence of policy and latent variable function.
Again, the policy divergence can be constrained via policy optimization, like TRPO.
Thus, the main difference lies in the divergence of latent variable function.
As indicated by Theorem 1, the learned latent variable function ψ ω has a smaller divergence between consecutive policy rollouts than the true latent variable function ψ .
Therefore, independent policy optimization on experiences generated by the latent variable model can obtain more stationary observation visitation frequency than on the experiences collected in the varying environment, so the learning process of independent policy optimization becomes stable on the model.
3. R ETURN B OUNDS
We then analyze the bound of return gap between interacting in the environment and interacting with the model.
If the return improvement of interacting with the model is higher than the bound, the agent can obtain the monotonic policy improvement when interacting in the environment.
However, the return of interacting in the environment is hard to analyze in decentralized learning since the policies of other agents are inaccessible, we turn to analyze the return in policy rollout, which is an unbiased estimate of expected return in the environment.
Several bounds have been introduced in MBPO (Janner et al., 2019) for the return bound analysis, which however are not sufficient in decentralized learning.
Thus, we need to introduce two new bounds that indicate the divergence of latent variable function between consecutive policy rollouts and the error of learned latent variable function.
Formally, with ψ n , ψ nω , and π n respectively referring to the true latent variable function, the learned latent variable function, and the policy of the n th policy rollout, we denote the bounds as follows:
• reward bound r max ≜ max o,a,z max { R ( o, a, z ) , R ϕ
( o, a, z ) } ; • policy divergence ϵ π ≜ max o D TV ( π ∥ π n ) , where D TV is total variation distance; • transition function error ϵ θ ≜ max t E a t ,z t ∼ ( π,ψ nω )
D TV ( P ( o t +1 | o t , a t ) ∥
P θ ( o t +1 | o t , a t , z t )) ; • latent variable function divergence ϵ ψ ≜ max o D TV (cid:0) ψ n ∥ ψ n +1 (cid:1) ; • learned latent variable function error ϵ ω ≜ max o D TV ( ψ n ∥ ψ nω ) .
Additionally, we use several notations to represent different returns.
The return in n th policy rollout with the true latent variable function ψ is denoted as η ( π, ψ ) , the return in model rollout with the n th learned latent variable function ψ ω is denoted as η model ( π, ψ nω ) , and the return in k -step branched model rollout with h -step experiences of n th policy rollout is denoted as η branch (( π n , π ) , ( ψ, ψ nω )) .
Now we analyze the return bound of model rollout and branched model rollout with the newly introduced ϵ ψ and ϵ ω in the following two theorems.
Theorem 2.
The return gap between n + 1 th policy rollout and model rollout with n th learned model is bounded as:
[Equation]
Theorem 3.
The return gap between n + 1 th policy rollout and branched model rollout with n th learned model is bounded as:
[Equation]
According to Theorem 2 and 3, we can guarantee the monotonic improvement in the environment via improving the return in model rollout or branched model rollout beyond a bound linear to ( ϵ π , ϵ θ , ϵ ω , ϵ ψ ) .
In these bounds, ϵ θ and ϵ ω are limited via supervised learning and ϵ π is constrained by policy optimization.
However, ϵ ψ is left unrestricted .
In the following, we try to find a better bound in which all elements are controllable.
3. L ATENT V ARIABLE P REDICTION
In order to restrict the impact of divergence of the latent variable function, we introduce one new error bound, which measures the divergence between the learned latent variable function and the true latent variable function in incoming policy rollout.
Formally, such an error bound in n th policy rollout is defined as:
[Equation]
Now we use ˆ ϵ ω in place of ϵ ψ to analyze the return bound of model rollout and branched model rollout again in the following two theorems.
Theorem 4.
The return gap of n + 1 th policy rollout and model rollout with n th learned model is bounded as:
[Equation]
Theorem 5.
The return gap of n + 1 th policy rollout and branched model rollout with n th learned model is bounded as:
[Equation]
Now all elements of the bounds are controllable, once we can constrain ˆ ϵ ω in the learning process.
To achieve this, we introduce a latent variable prediction function, which predicts the latent variable distribution given observation o in incoming policy rollout via latent variable distributions of o in the latest l − 1 policy rollouts.
However, as the true latent variable function cannot be obtained directly for an agent, the latent variable prediction function f ζ can instead minimize:
[Equation]
With such a latent variable prediction function, ˆ ϵ ω is controllable.
3. A LGORITHM
With all the theoretical analysis and discussions above, we are ready to present the learning of model-based decentralized policy optimization (MDPO).
As illustrated in Figure 1, the environment model consists of transition function P θ , reward function R ϕ , latent variable prediction function f ζ , and latent variable functions { ψ ω 1 , · · · , ψ ω l } over recent l consecutive policy rollouts.
The experiences of the l consecutive policy rollouts
D env =
{D 1 , · · · , D l } are also stored.
After the latest policy rollout l , we update the transition function and reward function, and learn the latent variable function ψ ω l of policy rollout l by optimizing the objective:
[Equation]
where ψ ω l is obtained by updating ψ ω l − 1 using D l , while P θ and R ϕ are updated using D env to make sure they are stable across policy rollouts.
Then, the latent variable prediction function f ζ is updated using D l by optimizing the objective: [Equation]
[Equation]
For model rollout, the model predicts the transition in incoming policy rollout given observation o and action a via l − 1 latest learned latent variable functions ( ψ 2 , · · · , ψ l ) as: [Equation]
[Equation]
Finally, the policy is updated using the branched model rollout by policy optimization, such as PPO or TRPO.
We summarize the full learning procedure of MDPO in Algorithm 1.
Algorithm 1.
MDPO
E XPERIMENTS
For evaluation, we compare MDPO, MDPO without latent variable prediction (denoted by MDPO w/o prediction), and independent PPO (IPPO)
(Schulman et al., 2017) on a set of cooperative multiagent tasks including a stochastic game, multi-agent particle environment (MPE) (Lowe et al., 2017), and multi-agent MuJoCo (Peng et al., 2021b).
We do not consider StarCraft multi-agent challenge (SMAC) (Samvelyan et al., 2019), because IPPO has been shown to perform very well in SMAC (de Witt et al., 2020; Papoudakis et al., 2021), close enough to centralized training with decentralized execution methods like QMIX (Rashid et al., 2018) and MAPPO (Yu et al., 2021a).
Thus, the gain of
MDPO may not be clearly evidenced there.
By experiments, we try to answer the following three questions:
Does the latent variable model help to generate experiences with more stationary observation visitation frequency experimentally?
Does latent variable prediction help to control ˆ ϵ ω ?
Does MDPO help to improve performance in decentralized learning
For a fair comparison, the network architecture and hyperparameters are the same for IPPO and
MDPO.
The number of environment steps taken in each round (policy rollout, network update) is consistent and thus we compare the performance of methods under the same number of environment steps and policy updates.
Note that since we consider fully decentralized learning, for all methods, agents do not use parameter-sharing.
Indeed, parameter-sharing should not be allowed in decentralized learning (Terry et al., 2020).
More details on experiment settings, implementation, and hyperparameters are available in Appendix C.
4. S TOCHASTIC G AME
The stochastic game is a cooperative game with 30 observations (states), 3 agents, and 5 actions for each agent, and every episode consists of 40 steps.
The transition function and the shared reward function are randomly generated.
The game is chosen to verify our theoretical results.
Figure 2 (left) shows the learning curves of MDPO, MDPO w/o prediction, and IPPO, among which
MDPO performs better throughout the learning process.
With a finite observation space in this game, we calculate the divergence of observation visitation frequencies ( ∥ ∆ ρ ∥ in Section 3.2) in consecutive rollouts.
Concretely, we calculate the L1 distance of observation visitation frequency over all observations in consecutive rollouts (policy rollouts for IPPO and branched model rollouts for MDPO), and their curves are shown in Figure 2 (mid).
We can see that the latent variable model generates experiences with more stationary observation visitation frequency than IPPO, which is consistent with Theorem 1.
This may account for the superior performance of MDPO w/o prediction over IPPO.
We also examine how well the latent variable prediction helps to control the prediction error ( ˆ ϵ ω ).
As the real latent variable function is inaccessible, we examine ˆ ϵ ω by comparing how well the learned environment model predicts with and without latent variable prediction.
Specifically, we measure the

     URXQG         REVHUYDWLRQYLVLWDWLRQIUHTXHQF\
      URXQG    
 SUHGLFWUHZDUG/GLVWDQFH   
   SUHGLFWREVHUYDWLRQ&URVV(QWURS\

     URXQG          DYHUDJHUHWXUQ ,332 0'32ZRSUHGLFWLRQ 0'3
      URXQG 
       5HJXODU3RO\JRQ&RQWURO       URXQG        D Y H U D J H  U H W X U Q &RRSHUDWLYH1DYLJDWLRQ ,3320'32ZRSUHGLFWLRQ0'3 mean cross-entropy of the next observation distribution predicted by the n th learned model and the ground truth in the n + 1 th round, and the mean L1 distance of predicted reward and ground truth reward.
The curves are shown in Figure 2 (right).
The lower prediction error of MDPO indicates that latent variable prediction error ( ˆ ϵ ω ) is controlled at a lower level than without latent variable prediction.
Moreover, as shown in Figure 2 (mid), the divergence of observation visitation frequency of MDPO and MDPO w/o prediction are similar but much lower than IPPO, which indicates ϵ ω is still under control in MDPO.
This indicates that MDPO can well control both ϵ ω and ˆ ϵ ω .
As MDPO helps to handle non-stationarity in multi-agent settings from the perspective of an individual agent, it will be natural to also apply MDPO to non-stationary single-agent settings.
So, we modify this stochastic game into a non-stationary single-agent game and show that MDPO also outperforms the baselines.
More details are available in Appendix D.
4. MPE
MPE is a multi-agent environment with continuous observation.
In our MPE tasks, agents observe their own positions, velocity, and others’ relative positions.
And agents are expected to fulfill a certain goal via controlling their accelerations in every direction which is continuous in our experiments.
Two tasks of MPE, 4-agent Cooperative Navigation and 5-agent Regular Polygon Control, are chosen for performance comparison.
In 4-agent Cooperative Navigation, 4 agents learn to cooperate to reach 4 landmarks respectively.
In 5-agent Regular Polygon Control, 4 agents learn to cooperate with another agent, which is controlled by a fixed policy, aiming to form a regular pentagon, and the reward is given according to the similarity to a regular pentagon.
Figure 3 shows the learning curves of all methods.
Generally, MDPO w/o prediction performs better than IPPO, which verifies that the latent variable model can help decentralized policy improvement by making the observation visitation frequency more stationary.
And MDPO outperforms MDPO w/o prediction, which verifies latent variable prediction can reduce the gap between the return of interaction and the return predicted by the environment model.
It is worth noting that the unobservable information required to fulfill the goal is at completely different levels in the two tasks.
Concretely, acknowledging the general direction of others is enough to decide which landmark to approach in Cooperative Navigation.
Yet the precise positions of others


    URXQG      D Y H U D J H  U H W X U Q +DOI&KHHWDKî 
    URXQG      +RSSHUî      URXQG         $QWî 
    URXQG      D Y H U D J H  U H W X U Q +DOI&KHHWDKî 
    URXQG         +DOI&KHHWDK>@      URXQG      +DOI&KHHWDKî ,332 0'32ZRSUHGLFWLRQ 0'3 matter to form a regular polygon in Regular Polygon Control and are hard to learn accurately.
Thus,
MDPO performs well in Cooperative Navigation since the very early learning stage, while it does not perform well in Regular Polygon Control before 600 rounds.
Although a more accurate model is required in Regular Polygon Control, MDPO still converges to better performance.
And this indicates a progressive pattern in prediction also works when prediction is hard to be fairly accurate.
4. M ULTI -A GENT M U J O C O
Multi-agent MuJoCo is a continuous multi-agent robotic control environment, based on OpenAI’s
Mujoco Gym environments.
In a multi-agent MuJoCo task, each agent controls several joints of the robotic to move forward, where both the observation space and action space are continuous.
We choose 3-agent Hopper, 4-agent Ant, and 4 versions of HalfCheetah with different agent numbers or joint allocation for performance comparison.
Details of joint allocation are given in Appendix C.
As illustrated in Figure 4, MDPO consistently performs better in these tasks with different difficulties and various agent numbers.
Compared with MPE, agents in multi-agent MuJoCo have deeper impacts on each other due to the interaction between adjacent joints.
Consequently, the transitions of each agent are closely related to the policies of other agents.
Thus, non-stationarity caused by policy updates of other agents is severer in these tasks, resulting in IPPO struggling and converging to low performance.
Moreover, note that MDPO w/o prediction performs almost the same as IPPO or even worse in some tasks.
The poor performance of MDPO w/o prediction is a consequence of a larger ϵ ψ caused by strongly associated agents in these tasks.
Thus, latent variable prediction is necessary in these tasks with closely associated agents.
C ONCLUSION
In this paper, we propose model-based decentralized policy optimization (MDPO).
By introducing a latent variable into the environment model, we theoretically show the model helps to generate experiences with more stationary observation visitation frequency and benefits decentralized policy optimization.
Furthermore, We theoretically analyze that the return bound for monotonic policy improvement is controllable by the prediction error of the latent variable function.
Consequently, we propose a latent variable prediction method to constrain the prediction error.
We examine all the theories and designs via experiments on a set of cooperative multi-agent tasks.
Results verify our theoretical results and show MDPO indeed obtains superior performance over model-free decentralized policy optimization.
A. O BSERVATION V ISITATION F REQUENCY D IVERGENCE
In this section
[Equation]
we provide proofs for the upper bound of observation visitation frequency divergence.
Lemma 1.
Given two pairs of policy and latent variable function, ( π 1 , ψ 1 ) and ( π 2 , ψ 2 ) .
∀ o ∈
O
[Equation]
it holds that
[Equation]
where ∥
[Equation]
Proof.
(cid:88)
[Equation]
Lemma 2 (Timestep observation visitation frequency recursion) .
Given two pairs of policy and latent variable function ( π 1 , ψ 1 ) and ( π 2 , ψ 2 ) , we define :
[Equation]
It holds that
[Equation]
Proof.
For observation visitation frequency at timestep t +
[Equation]
there is a recurrence relation:
[Equation]
Thus
[Equation]
the divergence of observation visitation frequency at timestep t + 1 can be processed corre- spondingly:
[Equation]
( o ) (cid:88) a,z ( P ( o ′| a, o, z ) π 1 ( a | s ) ψ 1 ( z | o ))
(cid:33) − (cid:88) o (cid:32) ρ π 2 ,ψ 2 t ( o ) (cid:88) a,z ( P ( o ′| a, o, z ) π 2 ( a | s ) ψ 2 ( z | o )) (cid:33) =
(cid:88) o
(cid:32) ρ π 1 ,ψ 1 t ( o ) (cid:88)
a,z ( P ( o ′| a, o, z ) ( π 1 ( a | o ) ψ 1 ( z | o ) − π 2 ( a | o ) ψ 2 ( z | o )))
(cid:33) + (cid:88) o
(cid:32) ∆ ρ ( π 1 ,ψ 1 ) , ( π 2 ,ψ 2 ) t ( o ) (cid:88) a,z ( P ( o ′| a, o, z ) π 2 ( a | o ) ψ 2 ( z | o )) (cid:33)
Using Lemma
[Equation]
we can bound the divergence of observation state frequency at timestep t + 1 :
[Equation]
Lemma 3 (discounted observation visitation frequency divergence bound) .
Given two pairs of policy and latent variable function ( π 1 , ψ 1 ) and ( π 2 , ψ 2 ) , with the same distribution of initial observation ρ 0 ( o )
[Equation]
it holds that
[Equation]
where C is a certain constant.
Proof.
We transform it to the cumulative form of the timestep
[Equation]
and scale it using Lemma 2:
[Equation]
γ
[Equation]
Thus
[Equation]
we get bound discounted observation visitation frequency divergence:
[Equation]
A. L ATENT V ARIABLE F UNCTION D IVERGENCE
In this section
[Equation]
we provide proof for divergence comparison between latent variable function of policy rollout and learned latent variable function in the model
[Equation]
Lemma 4 (Latent variable function divergence comparison) .
Assume ψ nω =
(1 − α ) ψ n − 1 ω + αψ n , and initially ψ 1 ω = ψ 1 , where n is the n th policy rollout.
Then,
E ψ > E ψ ω , where E ψ ≜ max n (cid:13)(cid:13) ψ n − ψ n − 1 (cid:13)(cid:13) and E ψ ω ≜ max n (cid:13)(cid:13) ψ nω − ψ n − 1 ω (cid:13)(cid:13) .
Proof.
Firstly, we can construct such a recursive inequality:
[Equation]
Thus, we can expand it recursively:
[Equation]
= (cid:13)(cid:13)
ψ n − ψ n − 1 (cid:13)(cid:13)
+ (1 − α ) (cid:13)(cid:13)
ψ n − 1 − ψ n − 2 (cid:13)(cid:13) + · · · + (1 − α ) n − 1 (cid:13)(cid:13) ψ 1 − ψ 0 (cid:13)(cid:13)
≤E ψ (cid:16) 1 + (1 − α ) + · · · + (1 − α ) n − 1 (cid:17) <
E ψ α .
Using this inequality, we can zoom (cid:13)(cid:13) ψ nω − ψ n − 1 ω (cid:13)(cid:13) :
(cid:13)(cid:13) ψ nω − ψ n − 1 ω (cid:13)(cid:13) =
(cid:13)(cid:13) (1 − α ) ψ n − 1 ω + αψ n − ψ n − 1 ω
(cid:13)(cid:13) =
α (cid:13)(cid:13) ψ n − ψ n − 1 ω
(cid:13)(cid:13)
z < E ψ .
Thus, E ψ ω = max n (cid:13)(cid:13) ψ nω − ψ n − 1 ω (cid:13)(cid:13) <
E ψ .
Now we combine Lemma 3 and 4 to prove Theorem 1.
Theorem 1 (Latent variable model benefits) .
Define ∆ ρ n ( o ) ≜ ρ π n ,ψ n ( o ) − ρ π n − 1 ,ψ n − 1 ( o ) , and denote ∥ ∆ ρ n ∥ ≜ max o | ρ n ( o ) − ψ n − 1 ( o ) | , similarly for ∥ ∆ π n ∥ and ∥ ∆ ψ n ∥ .
It holds that, ∥ ∆ ρ n ∥ ≤ C ( E π + E ψ ) , where E π ≜ max n ∥ ∆ π n ∥ , E ψ ≜ max n ∥ ∆ ψ n ∥ and C is a constant.
Assume ψ nω =
(1 − α ) ψ n − 1 ω + αψ n , and initially ψ 0 ω
= ψ 0 .
It holds that E ψ > E ψ ω and the bound above is lower when substituting ψ with ψ ω .
Proof.
Using Lemma 3, we can scale ∥ ∆ ρ n ∥ ,
[Equation]
Using Lemma 4, E ψ > E ψ ω .
Thus, the bound above is lower for E ψ ω .
A. L EMMAS FOR R ETURN B OUND A NALYSIS
In this section, we prove several lemmas as preparations for return bound analysis.
Lemma 5 (TVD bound of joint distribution) .
Consider two joint distributions of n + 1 variables like this:
[Equation]
We can bound the total variation distance of the joint distributions as:
[Equation]
Proof.
We start the proof from a basis case when n = 1 :
[Equation]
Similarly
[Equation]
we can prove the case of multi-variables:
[Equation]
Before proving following lemmas, we clarify the premise our discuss in this section is based on.
In a
Dec-POMDP, denote the co-occurrence probability of tuple ( o, a, z ) at timestep t as P t ( o, a, z ) ≜
P ( o t
= o, a t = a, z t = z ) .
Consider two Dec-POMDPs different merely in transition function and reward function, G 1 , G 2 .
Prepresents the probability in G 1 while P 2 for G 2 .
Different policies and latent variable functions, ( π 1 , ψ 1 ) and ( π 2 , ψ 2 ) , are used to rollout respectively in G 1 and G 2 , We denote several bound between them:
reward bound
[Equation]
policy bound: ϵ π ≜ max o D TV ( π 1 ∥ π
[Equation]
latent variable function bound: ϵ ψ ≜ max o D TV ( ψ 1 ∥ ψ 2 ) ; transition function bound: ϵ m ≜ max t E o,a,z ∼ P t −
[Equation]
Additionally, consider the branched model rollout mentioned in Section 3.
Policy, latent variable function, transition function, and reward function vary before and after the model rollout branch.
We denote these functions via superscripts ’Pre’ for function before branch and ’Post’ for functions after branch.
Correspondingly, when discussing branched model rollout, we extend the bounds above:
[Equation]
Lemma 6 (Observation distributions TVD bound) .
The total variation distance of observation distributions at timestep t , P 1 ( o t ) and P 2 ( o t ) , can be bounded as below:
[Equation]
Proof.
[Equation]
o,a,z
P t − 1( o, a, z ) P 1 ( o t | o, a, z ) − P t − 1( o, a, z ) P 2 ( o t | o, a, z ) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ≤ (cid:88)
o t (cid:18) (cid:88) o,a,z (cid:12)(cid:12) P t − 1( o, a, z ) − P t − 1( o, a, z ) (cid:12)(cid:12) P 1 ( o t | o, a, z ) + (cid:88)
o,a,z
P t − 1( o, a, z ) | P 1 ( o t | o, a, z ) − P 2 ( o t | o, a, z ) | (cid:19) =  (cid:88) o,a,z (cid:12)(cid:12) P t − 1( o, a, z ) − P t − 1( o, a, z ) (cid:12)(cid:12) (cid:88)
o t
P 1 ( o t | o, a, z ) + E o,a,z ∼ P t −
D TV ( P 1 (
[Equation]
|
[Equation]
[Equation]
( o
[Equation]
| [Equation]
, a, z )) =
[Equation]
(cid:0) P
[Equation]
[Equation]
[Equation]
m
According to Lemma 5,
[Equation] Lemma 7 (Rollout return bound) .
The gap between rollout returns in G 1 with ( π 1 , ψ 1 ) and G 2 with ( π 2 , ψ 2 ) is bounded as:
[Equation]
Proof.
[Equation]
Using Lemma 5 and 6,
[Equation]
Thus,
[Equation]
Lemma 8 (Branched rollout return bound) .
In k -step branched rollout with h -step length before the branch taking into consideration, denote the gap between branched rollout returns in G 1 with (cid:0)(cid:0) π Pre, π Post(cid:1) , (cid:0)
ψ Pre, ψ Post(cid:1)(cid:1) and G 2 with (cid:0)(cid:0)
π Pre, π Post(cid:1) , (cid:0) ψ Pre, ψ Post(cid:1)(cid:1) as: | η 1 − η
[Equation]
| ≜ (cid:12)(cid:12) η branch(cid:0)(cid:0)
[Equation]
Pre
[Equation]
[Equation]
Post
[Equation]
(cid:1) [Equation]
(cid:0)
[Equation]
Pre
[Equation]
[Equation]
Post
[Equation]
(cid:1)(cid:1) [Equation]
branch
[Equation]
(cid:0)(cid:0)
[Equation]
Pre
[Equation]
, π Post(cid:1)
[Equation]
(cid:0) ψ Pre, ψ Post(cid:1)(cid:1)(cid:12)(cid:12) which is bounded as:
[Equation]
Proof.
According to Lemma 6,
[Equation]
which stays in branched rollout case.
We can discuss δ t ≜ D TV ( P t 1 ( o, a, z ) ∥
P t 2 ( o, a, z )) with different t value: when t ≤ h ,
[Equation]
when h < t ≤ h + k ,
[Equation]
Postψ when t > h + k , δ t ≤ h
[Equation]
[Table]
Using the inequalities above
[Equation]
we can write:
[Equation]
∞ (cid:88) t = h
+ k + (cid:20) γ t (cid:16) ( t − k ) (cid:0) ϵ Preπ + ϵ Preψ + ϵ Prem (cid:1)
+ ϵ Preπ + ϵ Preψ + k (cid:0)
ϵ Postπ + ϵ Postψ + ϵ Postm (cid:1)
+ ϵ Postπ + ϵ Postψ (cid:17)(cid:21) ≤ (cid:32) ∞ (cid:88) t
= γ t h
+ γ h + k ∞ (cid:88) t
= γ t t (cid:33) (cid:0)
ϵ Preπ + ϵ Preψ + ϵ Prem (cid:1) + (cid:32) ∞ (cid:88) t
= γ t (cid:33) (cid:0)
ϵ Preπ + ϵ Preψ (cid:1) + (cid:32) ∞ (
cid:88) t
= h
+ γ t (cid:33) (cid:0) k (cid:0)
ϵ Postπ + ϵ Postψ + ϵ Postm (cid:1)
+ ϵ Postπ + ϵ Postψ (cid:1)
= (cid:32)
h 1 − γ + γ h
+ k + (1 − γ )(cid:33) (cid:0) ϵ Preπ + ϵ Preψ + ϵ Prem (cid:1) + 1 − γ (cid:0) ϵ Preπ + ϵ Preψ (cid:1) + γ h + 1 − γ (cid:0) k
(cid:0)
ϵ Postπ + ϵ Postψ + ϵ Postm (cid:1)
+ ϵ Postπ + ϵ Postψ (cid:1)
Thus, | η 1 − η 2 | ≤ 2 r max δ ≤ 2 r max 1 − γ (cid:20) (cid:18) h
+ γ h
+ k + (1 − γ ) (cid:19) (cid:0) ϵ Preπ + ϵ Preψ + ϵ Prem (cid:1) + (cid:0)
ϵ Preπ + ϵ Preψ (cid:1)
+ γ
[Equation]
A. P ROOF OF RETURN BOUND
In this section, we provide proofs of return bound in different cases.
Theorem 2 (Rollout return bound for decentralized model) .
The gap between return in n + 1 th policy rollout and return in model rollout with n th learned model η model ( π, ψ nω ) is bounded as:
[Equation]
Proof.
[Equation]
Apply Lemma 7 to L 1 and L 2 :
[Equation]
Thus,
[Equation]
Theorem 3 (Branched rollout return bound for decentralized model) .
The gap between return in n + 1 th policy rollout η ( π, ψ n +1 ) and return in k -step branched rollout with h -step experience with n th learned model η branch (( π nD , π ) , ( ψ n , ψ nω )) is bounded as:
[Equation]
Proof.
[Equation]
Apply Lemma 8 to L 1 and L 2 :
[Equation]
Thus
[Equation]
≤ 2 r max
[Equation]
Theorem 4 (Rollout return bound for decentralized model with prediction error) .
The gap between return in n + 1 th policy rollout η ( π, ψ n +1 ) and return in model rollout with n th learned model η model ( π, ψ nω ) is bounded as:
[Equation]
Proof.
[Equation]
Apply Lemma 7 to L 1 and L 2 :
[Equation]
Thus,
[Equation]
Theorem 5 (Branched rollout return bound for decentralized model with prediction error) .
The gap between return in n + 1 th policy rollout η ( π, ψ n +1 ) and return in k -step branched rollout with h -step experience with n th learned model η branch (( π nD , π ) , ( ψ n , ψ nω )) is bounded as:
[Equation]
Proof
[Equation]
Apply Lemma 8 to L 1 and L 2 :
[Equation]
Thus
[Equation]
δ ≤ 2 r max
[Equation]
[Figure]
B V ERIFICATION ON L EARNED L ATENT V ARIABLE
To examine how related the learned latent variable and the inaccessible information are, we designed a simple tabular case, where policies, transition matrix, and reward matrix are preset.
There arestates and 3 agents with 2 actions for each and the space of latent variable is set to be 4.
For agent 0 , we collect experiences ⟨ s, a 0 , s ′ , r ⟩ and train a latent variable model end-to-end.
For visualization, we design ψ ω as an explicit network to fetch learned z 0 and preset the forward pass for the state in P θ to avoid the correspondence being conditioned on the state.
Then, we sample z 0 from learned latent variable function network ψ ω for each experience in the buffer, and then calculate the conditional probabilities, P ( z 0 | a − 0 ) and P ( a −
| z 0 ) .
As shown in Figure 5, there is a one-to-one correspondence between the latent variable and other agents’ joint action.
This demonstrates the latent variable model can implicitly capture inaccessible information relevant to transition and reward via end-to-end learning.
C E XPERIMENT D ETAILS C. E NVIRONMENT S ETTING
In this section, we introduce the environment settings we used in the experiments.
Stochastic Game.
In our stochastic game, there are 30 observations, 3 agents and 5 actions for each agent, and episode length is limited to 40 steps.
We generate a transition matrix T and a reward matrix R in advance as transition function and reward function.
Concretely, T is a matrix in shape of [30 , 5 , 5 , 5 , 30] and R is a matrix in shape of [30 , 5 , 5 , 5 , 1] .
At each timestep t , given observation o t and agent joint actions (cid:0) a 0 t , a 1 t , a 2 t (cid:1) , the transition is:
[Equation]
MPE.
In our MPE tasks, agents observe their own positions, velocity, and others’ relative positions.
And actions of agents control their accelerations in every direction which is continuous in our experiments.
In MPE tasks, the episode length is limited to 40 steps.
4-Agent Cooperative Navigation.
In 4-agent Cooperative Navigation, as shown in Figure(left), 4 agents learn to cooperate to reach 4 landmarks respectively.
Concretely, we denote the radius of agent i as d i , position of agent i as ( x ia , y ia ) and position of landmark i as ( x il , y il ) , and the reward is:
[Equation]
where p c is a collision penalty:
[Equation]
Thus, the reward upper bound at each step is − 4 .
[Figure]
5-Agent Regular Polygon Control.
In 5-agent Regular Polygon Control, as shown in Figure(right), 4 agents learn to cooperate with another agent, which is controlled by a fixed policy, aiming to form a regular pentagon.
The fixed policy is that the acceleration of the agent is always in the direction of the relative position between the center of the other 4 agents and itself.
And the reward is given according to the area S of current pentagon scaled by its perimeter C , which formally is:
[Equation]
form a convex pentagon 0 , otherwise, and represents the area of its similar pentagon with a perimeter of 10.
So when the pentagon is a regular pentagon, S scaled comes to its maximum, 5 cot π 5 .
Additionally, two penalty items are given.
Bound penalty, p b , is used to restrict agents to stay in bounds : bound ( x ) =
[Equation]
[Equation]
1 .e
[Equation]
otherwise , [Equation]
(cid:2) bound ( x i )
+ bound ( y i ) (cid:3) , where ( x i , y i ) is the position of agent i .
Collision penalty, p c , is as same as that in 4-agent
Cooperative Navigation.
Finally, we design the reward as:
[Equation]
where max operator helps to distinguish when pentagon is relatively large and min operator handles the situation being divided by zero.
This task is more difficult than Cooperative
Navigation.
Multi-Agent MuJoCo.
In our multi-agent MuJoCo experiments, the state in MuJoCo environment, which describes the position, velocity, angular velocity of each joint, etc, is used as the observation distributed to each agent.
Specifically, in Ant task, we only use dimensions from 0 to 26 of the state.
We limit the episode length of Halfcheetah to 250 steps, and 500 steps for Ant and Hopper.
We provide the joint allocation of each task in Table 1.
C. I MPLEMENTATION & H YPERPARAMETERS
In this section, we provide details for implementation and hyperparameters.
For the experiment environment, we adopt MPE (MIT license) and MuJoCo
Gym (MIT license).
For
PPO, we follow the version in OpenAI’s Spinning Up (MIT license).
[Table]
[Table]
[Table]
All neural networks used in our implementation are in the form of Multi-Layer Perception (MLP).
Particularly, the transition function and reward function are respectively learned using an ensemble formed by 3 individual versions of the last layer. The likelihood of next observation is multiplied by a coefficient c o ′ to balance the scale of the elements in (3) and (4).
The hidden size and activation function used in the networks are provided in Table 2.
And the parameters used in training are provided in Table 3.

     URXQG       REVHUYDWLRQYLVLWDWLRQIUHTXHQF\       URXQG     SUHGLFWUHZDUG/GLVWDQFH      SUHGLFWREVHUYDWLRQ&URVV(QWURS\

     URXQG        DYHUDJHUHWXUQ
,332 0'32ZRSUHGLFWLRQ 0'3
In the implementation of latent variable function, both deterministic and stochastic latent variable satisfy our analysis, and we choose between them according to environment properties and experimental performance.
In MPE and stochastic game except for Appendix B, we use deterministic latent variable with L2 regularization.
In stochastic game in Appendix B, we use Category distribution.
And in multi-agent MuJoCo environment, we use Gaussian distribution.
As for the implementation of transition function and reward function, we use Category distribution for transition function in stochastic game and deterministic output for others.
The experiments are carried out on Intel i9-10900K CPU and NVIDIA GTX 3080Ti GPU.
The training of stochastic game task costs 6 hours, while it takes 14 hours for each MPE task, andhours for each multi-agent MuJoCo task.
D A DDITIONAL R ESULTS
Since MDPO helps to handle non-stationarity in decentralized MARL from the perspective of an individual agent, it will be natural and easy to also apply MDPO to single-agent RL in non-stationary environments.
In this section, we investigate how MDPO performs in such a non-stationary singleagent environment. We adopt the cooperative stochastic game into a single-agent non-stationary version. Concretely, we fix the policies of two agents and leave only one agent to update its policy. And we generate 5 noise matrices ( N 0 , · · · , N 4 ) randomly, which are in the same shape as the transition matrix ( T ) and will influence the transition probability in a rotating manner.
Formally, in n th policy rollout, the transition matrix is T + N n mod 5 , and we guarantee such a transition matrix is reasonable when generating noise matrices.
We compare the performance of MDPO, MDPO w/o prediction, and IPPO on the single-agent nonstationary stochastic game, and the learning curves are shown in Figure 7.
As illustrated in Figure(left), MDPO still performs the best in the single-agent non-stationary environment.
As shown in
Figure 7 (right), latent variable prediction helps to predict the non-stationary transition, and as no noise is applied to the reward matrix, there is merely a slight difference in reward prediction.
Since the latent variable in this environment (noise matrices) is in a regular rotation, the prediction function is easier to learn than in decentralized MARL settings.
However, unlike in decentralized MARL, non-stationarity in this setting will not fade away in pace with policy convergence, thus MDPO w/o prediction may keep oscillating and generate experiences with larger observation visitation frequency divergence than MDPO, which is shown in Figure 7 (mid).
Generally, MDPO also works in single-agent non-stationary environments, especially when there is a regular pattern of non-stationarity.
More thorough studies are left as future work.
E A DDITIONAL R ELATED W ORK
By utilizing an environment model, model-based RL has shown many advantages, such as sample efficiency (Wang et al., 2019) and exploration (Pathak et al., 2017).
Many paradigms have been
[Figure]
proposed on how to exploit the environment model.
Model-based planning methods, such as model predictive control, select actions through model rollouts.