F ULLY D ECENTRALIZED M ODEL - BASED P OLICY O PTIMIZATION WITH N ETWORKED A GENTS
Anonymous authors
Paper under double-blind review
A BSTRACT
Model- based RL is an effective approach for reducing sample complexity
However, when it comes to multi-agent setting where the number of agent is large
, the model estimation can be problematic due to the exponential increased interac tions
In this paper, we propose a decentralized model-based reinforcement learning algorithm for networked multi-agent systems, where agents are cooperative and communicate locally with their neighbors.
We analyze our algorithm theoretically and derive an upper bound of performance discrepancy caused by model usage and provide a sufﬁcient condition of monotonic policy improvement.
In our experiments, we compare our algorithm against other strong multi-agent baselines and demonstrate that our algorithm not only matches the asymptotic performance of model-free methods but also largely increases its sample efﬁciency
I NTRODUCTION
Many real world problems, such as autonomous driving, wireless communications, multi-player games can be modeled as multi-agent RL problems, where multiple autonomous agents coexist in a common environment, aiming to maximize its individual or team reward in the long term by interacting with the environment and other agents.
Unlike single-agent tasks, multi-agent tasks are more challenging, due to partial observations and unstable environments when agents update their policies simultaneously.
Therefore, there are hardly any one-ﬁts-all solutions for MARL problems.
Examples include networked systems control (NSC) (Chu et al., 2020), in which agents are connected via a stationary network.
They perform decentralized control based on its local observations and messages from connected neighbors.
Examples of networked systems include connected vehicle control (Jin & Orosz, 2014), trafﬁc signal control (Chu et al., 2020), etc.
Despite the success of multi-agent reinforcement (RL) algorithms, their performance relies on a massive amount of model usage.
Typically, a multi-agent RL algorithm needs millions of interaction with the environment to converge.
On the other hand, model-based reinforcement learning (MBRL) algorithms, which utilize predictive models of the environment to help data collection, are empirically more data-efﬁcient than model-free approaches.
Although model inaccuracy performs as a bottleneck of policy quality in model-based algorithms, we can still learn a good policy with an imperfect model (Luo et al., 2019), especially combined with the trick of branched rollout (Janner et al., 2019) to limit model usage.
Experimentally, MuZero (Schrittwieser et al., 2020), a modelbased RL algorithm, succeeded in matching the performance of AlphaZero on Go, chess and shogi, and becomes state-of-the-art on Atari games.
Model-based MARL is not fully investigated.
Existing MB-MARL algorithms either limit their ﬁeld of research on speciﬁc scenario, e.g. two-player zero-sum Markov game (Zhang et al., 2020) or pursuit evasion game (Bouzy & M´etivier, 2007), or use tabular RL method (Bargiacchi et al., 2021).
MB-MARL for multi-agent MDPs is still an open problem to be solved (Zhang et al., 2019), with profound challenges such as scalability issues caused by large state-action space and incomplete information of other agents’ state or actions.
In this paper, we develop decentralized model-based algorithms on networked systems, where agents are cooperative, and able to communicate with each other.
We use localized models to predict future states, and use communication to broadcast their predictions.
To address the issue of model error, we adopt branched rollout (Janner et al., 2019) to limit the rollout length of model trajectories.
In the policy optimization part, we use decentralizd PPO (Schulman et al., 2017) with a extended value function.
At last, we analyze these algorithms theoretically to bound the performance discrepancy between our method and its model-free, centralized counterpart.
At last, we run these algorithms in trafﬁc control environments (Chu et al., 2020; Vinitsky et al., 2018) to test the performance of our algorithm.
We show that our algorithm increases sample efﬁciency, and matches the asymptotic performance of model-free methods.
In summary, our contributions are three-fold.
Firstly, we propose an algorithmic framework, which is a fully decentralized model-based reinforcement learning algorithm, which is named as
D ecentralized M odel-based P olicy O ptimization (DMPO).
Secondly, we analyze the theoretical performance of our algorithm.
Lastly, empirical results on trafﬁc control environments demonstrate the effectiveness of DMPO in reducing sample complexities and achieving similar asymptotic performance of model-free methods.
R ELATED W ORK
Model-based methods are known for their data efﬁciency (Kaelbling et al., 1996), especially compared with model-free algorithms.
There is a vast literature on the theoretical analysis of modelbased reinforcement learning.
In a single-agent scenario, monotonic improvement of policy optimization has been achieved (Luo et al., 2019; Sun et al., 2018), and a later work improved the performance of model-based algorithms by limiting model usage (Janner et al., 2019).
But these analysis is restricted to single-agent scenarios, whereas ours addresses multi-agent problems.
On the other hand, Networked System Control (NSC) (Chu et al., 2020) is a challenging setting for
MARL algorithm to take effect.
Some multi-agent algorithms falls into centralized training decentralized execution (CTDE) framework.
For example, QMIX (Rashid et al., 2018) and COMA (Foerster et al., 2018) all use a centralized critic.
In a large network, however, centralized training might not scale.
In many scenarios, only fully decentralized algorithms can be used.
Zhang et al.
(2018) proposed an algorithm of NSC that can be proven to converge under linear approximation.
Qu et al.
(2020a) proposed truncated policy gradient, to optimize local policies with limited communication.
Baking in the idea of truncated Q -learning in (Qu et al., 2020a), we generalize their algorithm to deep RL, rather than tabular RL.
Factoring environmental transition into marginal transitions can be seen as factored MDP.
Guestrin et al.
(2001) used Dynamic Bayesian Network to predict system transition.
Simao & Spaan (2019) proposed a tabular RL algorithm to ensure policy improvement at each step.
However, our algorithm is a deep RL algorithm, enabling better performance in general tasks.
There are some works on applying model-based methods in MARL settings.
A line of research focuses on model-based RL for two-player games.
For example, Brafman & Tennenholtz (2000) solved single-controller-stochastic games, which is a certain type of two-player zero-sum game;
Bouzy & M ´ etivier (2007) performed MB-MARL in the pursuit evasion game; Zhang et al.
(2020) proved that model-based method can be nearly optimally sample efﬁcient in two-player zero-sum
Markov games.
Bargiacchi et al.
(2021) extended the concept of prioritized sweeping into a MARL scenario.
However, this is a tabular reinforcement algorithm, thus unable to deal with cases where state and action spaces are relatively large, or even continuous.
In contrast to existing works, our algorithm is not only applicable to more general multi-agent problems, but is also the ﬁrst fully decentralized model-based reinforcement learning algorithm.
P ROBLEM S ETUP
In this section, we introduce multi-agent networked MDP and model-based networked system control.
Networked MDP
We consider environments with a graph structure.
Speciﬁcally, n agents coexist in an underlying undirected and stationary graph G =
( V , E ) .
Agents are represented as a node in the graph, therefore V = { 1 , ..., n } is the set of agents.
E ⊂ V × V comprises the edges that represent the connectivity of agents.
Agents are able to communicate along the edges with their neighbors.
Let N i denote every neighbor of agent i, and ¯ N i
= N i ∪{ i } .
Furthermore, let N κi denote the κ -hop neighborhood of i , i.e. the nodes whose graph distance to i is less than or equal to κ .
For the simplicity of notation, we also deﬁne N κ − i =
V \ N κ i .
The corresponding networked MDP is deﬁned as ( G , {S i , A i } i ∈V , p, r ) .
Each agent i have their local state s i ∈ S i , and perform action a i ∈
A i .
The global state is the concatenation of all local states: s = ( s 1 , ..., s n ) ∈ S :
= S 1 × ...
×S n .
Similarly, the global action is a = ( a 1 , ..., a n ) ∈
A :=
A 1 × ...
× A n .
For the simplicity of notation, we deﬁne s N i to be the local states of every agent in
N i , that is, given N i = { j 1 , ..., j c } , then s N i =
( s j 1 , ..., s j c ) .
a N i , s N κi , a N κi are deﬁned similarly.
The transition function is deﬁned as: p ( s (cid:48) | s, a ) : S ×
A → S .
Each agent possess a localized policy π
θ i
i ( a i | s ¯ N i ) that is parameterized by θ i ∈ Θ i , meaning the local policy is dependent only on states of its neighbors and itself.
We use θ =
( θ 1 , ..., θ n ) to denote the tuple of localized policy parameters, and π θ ( a | s )
= (cid:81) ni
=
π θ i
i ( a i | s ¯ N i ) denote the joint policy.
We also assume that reward functions is only dependent on local state and action: r i ( s i , a i ) , and the global reward function is deﬁned to be the average reward r ( s, a )
= 1 n (cid:80) ni
=1 r
i
s i , a i ) .
The goal of reinforcement learning is to maximize the expected sum of discounted rewards, denoted by η :
[Equation]
where γ ∈ (0 , 1) is the temporal discount factor.
We deﬁne the stationary distribution under policy π to be d π ( s ) .
Independent Networked System Networked system may have some extent of locality, meaning in some cases, local states and actions do not affect the states of distant agents.
In such systems, environmental transitions can be factorized, and agents are able to maintain local models to predict future local states.
We deﬁne Independent Networked System (INS) as follows:
Deﬁnition 1.
An environment is an Independent Networked System (INS) if:
[Equation]
INS might be an assumption that is too strong to hold.
However, for the dynamics that cannot be factorized, we can still use an INS to approximate it.
Let D TV denote the total variation distance between distributions, we have the following deﬁnition:
Deﬁnition 2.
( ξ -dependent) Assume there exists an Independent Networked System ¯ p such that ¯
[Equation]
=
[Equation]
=1 p i
[Equation]
s
[Equation]
i | s ¯ N i , a i ) .
An environment is ξ -dependent, if:
[Equation]
To explain the intuition behind this deﬁnition, we point out that ξ is actually the lower bound of model error when we use local models
ˆ p ( s ¯ N i , a i ) .
Recall that p ( s (cid:48) | s, a ) is the real environment transition, ¯ p = (cid:81) ni
=1 p i ( s (cid:48)
i | s ¯ N i , a i ) is the product of marginal environment transitions, and ˆ p ( s, a ) = (cid:81) n i =1 ˆ p
i ( s (cid:48)
i | s ¯ N i , a i ) is the product of model transitions.
Then the universal model error D ( p (cid:107) ˆ p ) can be divided into two parts: dependency bias D ( p (cid:107) ¯ p ) and model error D (¯ p (cid:107) ˆ p ) :
D ( p (cid:107) ˆ p ) ≤ D ( p (cid:107) ¯ p ) + D (¯ p (cid:107) ˆ p ) .
Then for a ξ -dependent system, when models become very accurate, meaning D (¯ p (cid:107) ˆ p ) ≈ 0 , sup D ( p (cid:107) ˆ p ) ≈ sup D ( p (cid:107) ¯ p ) = ξ .
While D can be any appropriate distance metric, we use the
TV-distance hereafter for the ease of presentation.
In the following, we develop theory under both
INS and ξ -dependent scenarios.
D ECENTRALIZED M ODEL - BASED P OLICY O PTIMIZATION
In this section, we formally present Decentralized Model-based Policy Optimization (DMPO), which is a fully decentralized model-based reinforcement learning algorithm.
Compared with independent multi-agent PPO, DMPO is augmented in three ways: localized model, policy with one-step communication, and extended value function.
We introduce the detail of localized model in 4.1.
Policy and value functions are introduced in 4.2.
The illustration of our algorithm is given in Figure 1.
All the components mentioned above are analyzed in Section 5.
We argue that under certain conditions, our algorithm ensures monotonic policy improvement.
[Figure]
4. D ECENTRALIZED P REDICTIVE M ODEL
To perform decentralized model-based learning, we let each agent maintain a localized model.
The localized model can observe the state of 1-hop neighbor and the action of itself, and the goal of a localized model is to predict the information of the next timestep, including state, reward and done.
This process is denoted by ˆ p
i ( s (cid:48)
i , r (cid:48) i , d (cid:48)
i | s ¯ N i , a i ) .
We implement a localized model with graph convolutional networks (GCN).
Recall that agents are situated in a graph G =
( V , E ) .
In the ﬁrst step, a node-level encoder encodes local state into node embedding,
[Equation]
Then we perform one step of graph convolution as follows,
[Equation]
In this way, h 1 i is dependent only on s ¯ N i and a i .
Finally, a node-level decoder generates the prediction of state, reward and done from h 1 i as follows:
[Equation]
Note that we predict the next state with a skip connection, because empirically, it’s more efﬁcient to predict the change of the state rather than the state itself.
In practice, the data are all stored locally by each agent.
Data that are collected in the environment by agent i is denoted as D envi , and those generated by predictive model is denoted as D modeli .
Scaling model-based methods into real tasks can result in decreased performance, even if the model is relatively accurate.
One main reason is the compound modeling error when long model rollouts are used, and model error compound along the rollout trajectory, making the trajectory ultimately inaccurate.
To reduce the negative effect of model error, we adopt a branched rollout scheme proposed in (Janner et al., 2019).
In branched rollout, model rollout starts not from an initial state, but from a state that was randomly selected from the most recent environmental trajectory τ .
Additionally, the model rollout length is ﬁxed to be T .
This scheme is shown to be effective in reducing the negative inﬂuence of model error both theoretically and empirically.
To deal with the bias of model trajectories, at each model rollout, we allow the algorithm to fall back to the real trajectory with probability 1 − q 0 , where q 0 is a hyperparameter.
We describe the detailed framework of model usage and experiment storage in Algorithm 1.
4. P ROXIMAL P OLICY O PTIMIZATION WITH E XTENDED V ALUE F UNCTION
To optimize the policies, we need to adopt an algorithm that can exploit network structure, whilst remaining decentralized.
Independent RL algorithms that observes only local state are fully decentralized, but they often fail to learn an optimal policy.
Centralized algorithms that utilize centralized critics often achieve better performance than decentralized algorithms, but they might not scale to large environments where communication costs are expensive.
We propose Proximal Policy Optimization with extended value function , which is deﬁned as
V i ( s N κi ) =
E s Nκ − i ∼ d π [ (cid:80) ∞ t =
r ti | s 0 N κi
=
s N κi ] , i ∈ V .
The intuition behind extended value function comes from (Qu et al., 2020a), where truncated Q-function Q ( s N κi , a N κi ) is initially proposed.
In 5.3, we prove that V i ( s N κi ) is a good approximation of V i ( s ) , with a difference decreasing exponentially with κ .
To generate the objective for extended value function, or return R i , we use reward-to-go technique.
However, because model rollout is short, standard reward-to-go returns would get a biased estimation of V i .
To resolve this issue, we add the value estimation of the last state to the return.
In this way, with a local trajectory τ i
= { ( s ti , a ti , r ti , ( s (cid:48) ) ti , d ti , log π ti ) , t = 0 , 1 , ..., T − 1 } ,
the objective of
V ti ( s N κi ) is
[Equation]
and the loss of value function is deﬁned as L valuei = 1 m (cid:80) m ∈D [Equation] i
[Equation]
training, extended value functions V i are reduced via communication to their κ -hop neighbors to generate an estimation of global value function,
[Equation]
and advantages ˆ A i are computed on ˜ V i with generalized advantage estimation (GAE) (Schulman et al., 2015) for policy gradient update.
The surrogate loss function of a DMPO agent is deﬁned as
[Equation]
similar to PPO-Clip loss.
The communication of κ step might seem costly, yet information of N κi is only used in the training phase.
We argue that in the training phase, algorithms are less sensitive with latency than execution.
Furthermore, since model-based learning can effectively increase sample efﬁciency, we might tolerate more communication.
T HEORETICAL A NALYSIS
In this section, we analyze DMPO theoretically.
In 5.2, we derive a bound between the true returns and the returns under a model ˆ p in a networked system.
In 5.3, we prove that extended value function
V i ( s N κi ) is a good approximation of V i ( s ) , and with extended value function, the true policy gradient can also be approximated.
5. B ACKGROUND : M ONOTONIC M ODEL - BASED P OLICY O PTIMIZATION
Let η [ π ] denote the returns of the policy in the true environment, ˆ η [ π ] denote the returns of the policy under the approximated model.
To analyze the difference between η [ π ] and ˆ η [ π ] , we need to construct a bound
[Equation]
where C is a non-negative function, and π D is the data-collecting policy.
According to equation 8, if every policy update ensures an improvement of ˆ η [ π ] by at least C , η [ π ] will improve monotonically.
This inequality was ﬁrst presented in single agent domain (Janner et al., 2019).
In this work, we extend this to the multi-agent networked system, aiming to achieve monotonic team reward improvement.
In this work, we let π indicate a collective policy π =
[ π 1 , ..., π n ] , and the model ˆ p be an INS ˆ p ( s (cid:48) | s, a )
= (cid:81) ni
=
ˆ p
i ( s (cid:48)
i | s ¯ N i , a i ) that approximating the true MDP.
In DMPO, each agent learns a localized model ˆ π i , policy π i ( |
s N k ) , critic V i ( s N κi ) ,
making it never a trivial extension.
We give the detailed analysis in 5.2.
5. A NALYSIS OF R ETURNS B OUND
In model-based learning, different rollout schemes can be chosen.
The vanilla rollout assumes that models are used in an inﬁnite horizon.
The branched rollout performs a rollout from a state sampled by a state distribution of previous policy π D , and runs T steps in ˆ π according to π .
Based on different rollout schemes, we can construct two lower bounds.
Under vanilla rollout , real return and model return can be bounded by model error and policy divergence.
Formal results are presented in Theorem 1.
The detailed proof is deferred to Appendix C.
Theorem 1.
Consider an independent networked system.
Denote local model errors as (cid:15) m i
= max s ¯ Ni ,a i D TV [ p i ( s (cid:48)
i | s ¯ N i , a i ) (cid:107)
ˆ p
i ( s (cid:48)
i | s ¯ N i , a i )] , and divergences between the data-collecting policy and evaluated policy as (cid:15) π i
= max s ¯
Ni
[Equation]
TV [
[Equation]
| s
[Equation]
) (cid:107) π ( a i | s ¯ N i )] .
Assume the upper bound of rewards of all agents is r max .
Let η p [ π 1 , ..., π n ] denote the real returns in the environment.
Also, let η ˆ p
π 1 , ..., π n ] denote the returns estimated in the model trajectories, and the states and actions are collected with π D .
Then we have:
[Equation]
Intuitively, the term (cid:80) ∞ k =0 γ k +1 | ¯ N ik | n would be in the same magnitude as 11 − γ , which might be huge given the choice of γ , making the bound too loose to be effective.
To make tighter the discrepancy bound in Theorem 1, we adopt the branched rollout scheme.
The branched rollout enables a effective combination of model-based and model-free rollouts.
For each rollout, we begin from a state sample from d π D , and run T steps in each localized ˆ π i .
When branched rollout is applied in an INS,
Theorem 2 gives the returns bound.
Theorem 2.
Consider an independent networked system.
Denote local model errors as (cid:15) m i
= max s ¯ Ni ,a i D TV [ p i ( s (cid:48)
i | s ¯ N i , a i ) (cid:107)
ˆ p
i ( s (cid:48)
i | s ¯ N i , a i )] , and divergences between the data-collecting policy and evaluated policy as (cid:15) π i = max
[Equation]
¯ Ni
[Equation]
TV [
[Equation]
D
[Equation]
i
[Equation]
¯
[Equation]
i ) (cid:107) π
( a i | s ¯ N i )] .
Assume the upper bound of rewards of all agents is r max .
Let η p [ π 1 , ..., π n ] denote the real returns in the environment.
Also, let η branch
π 1 , ..., π n ] denote the returns estimated via T -step branched rollout scheme.
Then we have:
[Equation]
n
[Equation]
Comparing the results in Theorem 1 and 2, we can see that branched rollout scheme reduced the coefﬁcient before (cid:15)
m i from (cid:80) ∞ k =
γ k +1 | ¯ N ik | n ≤
[Equation]
to (cid:80)
[Equation]
This reduction explains that empirically, branched rollout brings better asymptotic performance.
Also, if we set T = 0 , this bound turn into a model-free bound.
This indicates that when (cid:15) m i is lower than (cid:15)
π i allowed by our algorithm, a model might increase the performance.
In reality, not every system satisﬁes the deﬁnition of INS.
Yet we can generalize Theorem 2 into a ξ -dependent system.
Corollary 1.
Consider an ξ -dependent networked system.
Denote local model errors as (cid:15) m i =
[Equation]
i , a i )] , and divergences between the data-collecting policy and evaluated policy as (cid:15) π i = max
[Equation] [Equation]
s ¯ N i ) (cid:107) π
( a i | s ¯ N i )] .
Assume the upper bound of rewards of all agents is r max .
Let η p [ π 1 , ..., π n ] denote the real returns in the environment.
Also, let η branch
π 1 , ..., π n ] denote the returns estimated via T -step branched rollout scheme.
Then we have:
[Equation]
, ..., π n ] − η branch [ π 1 , ..., π n ] |
[Equation]
(cid:15) m i
[Table]
(cid:0)
T −
[Table]
(cid:88) k
= γ k +1 | ¯ N ik | n (cid:1) + (cid:15)
π i · (cid:0) ∞ (cid:88) k
= T γ k +1 | ¯ N ik | n (cid:1) (cid:105)
The proof can also be found in Appendix C. Compared to Theorem 2, Corollary 1 is more general, as it is applicable to the multi-agent systems that are not fully independent.
Intuitively, if a networked system seems nearly independent, local models will be effective enough.
The bound indicates that when the policy in optimized in a trust region where D ( π, π D ) ≤ (cid:15) π i , the bound would also be restricted, making monotonic update more achievable.
5. E XTENDED V ALUE F UNCTION
In this section, we analyze the effect of extended value function.
The idea of extended value function
V i ( s N κi ) comes from truncated Q-function Q i ( s N κi , a N κi ) proposed in (Qu et al., 2020a).
We prove that extended value function is an approximation of the real value function.
The detailed proof of
Theorem 3 is deferred to Appendix C.
Theorem 3.
Deﬁne
[Equation]
and
[Equation] [Equation]
s
[Equation]
= s ] , then:
[Equation]
From Theorem 3, it is straightforward that the global value function can be approximated with the average of all extended value functions: | V ( s ) − 1 n (cid:80) n i =
1 V i
( s N κi )
| ≤ r max 1 − γ γ κ .
In PPO, value functions are used for calculating advantages ˆ A
[Equation]
t +1) )
− V ( s ( t ) ) , and we have proven that V ( s ) can be estimated with the average of extended value functions 1 n (cid:80) ni
=1 V i
s N κi ) .
In practice, an agent might not get the value function of distant agents.
However, we can prove that ˜ V i = 1 n (cid:80) j ∈ N κi V j ( s N κj ) is already very accurate for calculating the policy gradient for agent i .
Theorem 4 justiﬁes that the policy gradients computed based on the sum of the nearby extended value functions is a close approximation of true policy gradients.
Theorem 4.
Let ˆ A t = r ( t ) + γV ( s ( t +1) )
− V ( s ( t ) ) be the TD residual, and g i
=
E [ ˆ A ∇ θ i log π i ( a | s )] be the policy gradient.
If ˜ A t and ˜ g i are the TD residual and policy gradient when value function
V ( s ) is replaced by ˜ V
i ( s ) = 1 n (cid:80) j ∈ N κ i V j ( s N κi ) , we have:
[Equation]
where r max and g max denote the upper bound of the absolute value of reward and gradient, respectively.
E XPERIMENTS 6. E NVIRONMENTS
We test our algorithm in four environments, namely Figure Eight, Ring Attenuation (Wu et al., 2017a), CACC Catchup, and
CACC Slowdown
(Chu et al., 2020).
Detailed description and visualization of these environments is deferred to Appendix A.
Cooperative Adaptive Cruise Control
The objective of CACC is to adaptively coordinate a platoon of 8 vehicles to minimize the car-following headway and speed perturbations based on real-time vehicle-to-vehicle communication.
CACC consists of two scenarios: Catch-up and Slow-down.
In
CACC Catch-up, vehicles need to catch up to the ﬁrst car.
In CACC Slow-down, every vehicle is faster than the optimal speed, and they need to slow down without causing any collision.
The agents receives a negative reward if the headway or the speed is not optimal.
Also, whenever a collision happens, a huge negative reward of -1000 is given.
Flow environments This task consists of Figure Eight and Ring Attenuation.
The objective of these environments is letting the automated vehicles achieve a target average speed inside the road network while avoiding collisions.
The state of each vehicle is its velocity and position, and the action is the acceleration of itself.
In Ring Attenuation, the objective is to achieve a high speed, while avoiding stop-and-go loops.
Vehicles are rewarded with their speed, but also punished for their accelerations.
In the perspective of a networked system, we assume that the vehicles are connected with the preceding and succeeding vehicle, thus resulting in a loop-structured graph.
6. B ASELINES
We describe the following algorithms for performance comparison:
• CPPO:
Centralized PPO learns a centralized critic V i ( s ) .
This baseline aims to analyze the performance when κ is set to be arbitrarily huge, and is used in (Vinitsky et al., 2018) as a benchmark algorithm for networked system control.
• IC3Net (Singh et al., 2018):
A communication-based multi-agent RL algorithm.
The agents maintain their local hidden states with a LSTM kernel, and actively determines the communication target.
Compared with DPPO, IC3Net uses hidden state and continuous communication, whereas DPPO agents directly observe the states of their neighbors.
• DPPO:
Decentralized PPO learns an independent actor and critic for each agent.
We implement it by using neighbor’s state for extended value estimation.
• DMPO (our method): DMPO is a decentralized and model-based algorithm based on DPPO.
On top of it, we use decentralized graph convolutional kernel as predictive model.
6. R ESULTS
Figure 2 shows the episode reward v.s. number of training samples curves of the algorithms.
We address that in CACC environments, DMPO uses decentralized SAC as base algorithm.
Similar with
DPPO, decentralized SAC uses extended Q -function Q i ( s N κi , a N κi ) for its policy gradient.
From the results, we conclude that our algorithm matches the asymptotic performance of model-free methods.
It also learns the policy faster, resulting in increased sample efﬁciency.
The comparison between DMPO and DPPO can be viewed as an ablation study of model usage.
In ﬁgure eight, DMPO increases sample efﬁciency at the beginning, but as the task becomes difﬁcult, the sample efﬁciency of our method decreased.
In a relatively easy task, ring attenuation, our method increased sample efﬁciency massively, compared with its model-free counterpart.
The comparison between the asymptotic performance of CPPO and DMPO or DPPO can be viewed as an ablation study of extended value function.
From the result in four environments, we observe that the asymptotic performance of CPPO does not exceed that of the algorithms that uses extended value function.
In this way, we conclude that by using extended value function, a centralized algorithm can be decomposed into decentralized algorithm, but the performance would not drop signiﬁcantly.
Figure 3 shows the accuracy of our model in predicting the reward and state during training.
The error is deﬁned as the ratio of MSE loss to variance.
From the ﬁgures, we conclude that neighborhood information is accurate enough for a model to predict the next state in these environments.
However, in CACC Slow-down, local models might fail to learn the reward.
We observe that the errors may increase as the agents explore new regions in the state space
[Figure] [Figure]
C ONCLUSIONS
In this paper
[Figure]
we propose algorithm DMPO, a model-based and decentralized multi- agent RL algorithm.
We then give a theoretical analysis on the algorithm to analyze its performance discrepancy, compared with a model-free algorithm.
By experiments in several tasks in networked systems, we show that although our algorithm is decentralized and model-based, it matches the asymptotic performance of some state-of-art multi-agent algorithms.
From the results, we also conclude that using extended value function instead of centralized value function did not sacriﬁce performance massively, yet it makes our algorithm scalable.
A E XPERIMENT D ETAILS
The code of our algorithm can be found at https://anonymous.4open.science/r/
RL-algorithms-0E72 .
A. E NVIRONMENT D ESCRIPTION
The objective of CACC is to adaptively coordinate a line of vehicles to minimize the car-following headway and speed perturbations based on real-time vehicle-to-vehicle communication.
We conduct experiments on two scenarios: CACC catch-up, CACC slow-down.
The local observation of each agent consists of headway h , velocity v , acceleration a , and is shared to neighbors within two steps.
The action of each agent is to choose appropriate hyper-parameters ( α ◦ , β ◦ ) for each OVM controller Bando et al.
(1995), selected from four levels { (0,0),(0.5,0),(0,0.5),(0.5,0.5) } , where α ◦ , β ◦ denotes headway gain and relative gain for OVM controller respectively.
The reward function is deﬁned as ( h i,t − h ∗ ) 2 + ( v i,t − v ∗ t ) 2 + 0 .
1 a 2 i,t to punish the gap between the current state to target state and speed perturbations, where the target headway and velocity proﬁle are h ∗ = 20 m and v ∗ t , respectively.
Whenever a collision happens ( h i,t < 1 m ), a large penalty of -1000 is assigned to each agent and the state becomes absorbing.
An additional cost 5 (2 h st − h i,t ) 2+ is provided in training for potential collisions.
In catch-up scenario, initial headway of the ﬁrst vehicle is larger than the target headway, thus the following agents learn how to catch up with the ﬁrst vehicle, where target speed v ∗ t = 15 m/s and initial headway h 1 , 0 > h i, 0 , ∀ i (cid:54) = 1 .
In slow-down scenario, target speed v ∗ t linearly decreases to 15m/s during the ﬁrst 30s and then stays at constant, thus agents learn how to slow down speed cooperatively,
where initial headway h i,
= h ∗ .
Flow environments consists of Figure Eight and Ring Attenuation.
The objective of these environments is letting the automated vehicles achieve a target average speed inside the road network while avoiding collisions.
The ﬁgure eight network, previously presented in (Wu et al., 2017b), acts as a closed representation of an intersection.
In a ﬁgure eight network containing a total of 14 vehicles, we witness the formation of queues resulting from vehicles arriving simultaneously at the intersection and slowing down to obey right-of-way rules.
This behavior signiﬁcantly reduces the average speed of vehicles in the network.
The state consists of velocity and position for the vehicle.
The action is the acceleration of the vehicle a ∈ R
[ a min ,a max ] .
The objective of the learning agent is to achieve high speeds while penalizing collisions.
Accordingly, the local reward function is deﬁned as r i = v des − | v des − v i | , where v des is an arbitrary target velocity.
In Ring Attenuation, the objective is to achieve a high speed, while avoiding accelerationdeceleration loops.
To achieve this, vehicles are rewarded with their speed and punished for their accelerations.
The state and action of each vehicle is the same as Figure Eight.
In the perspective of a networked system, we assume that the vehicles are connected with the preceding and succeeding vehicle, thus resulting in a loop-structured graph.
[Figure]
B H YPERPARAMETERS
We list some of the key hyperparameters for DMPO and DPPO in Table 1 and 2.
[Table]
[Table] [Table]
[Table]
C P ROOF OF T HEOREMS C. R EMARKS ON P ROOFS OF L EMMAS AND T HEOREMS
To bound the returns, we use TV distance to assess the distance between s i , a i at every timestep, based on the fact that | E s i ,a i ∼ p 1 r
( s i , a i ) − E
s i ,a i ∼ p 2 r
( s i , a i ) | ≤ r max · D TV ( p 1 (cid:107) p 2 ) .
To achieve this, we argue that in an INS, s Ti only depends on s T − 1 N 1 i , which is the 1 -hop neighbors’ states at previous timestep.
Then, inductively, s Ti is only dependent on s T − 1 N
i , s T − 2 N 2 i , ..., s T − k N ki , and ultimately, s 0 N Ti .
Then we consider a chain of states x t =
s N N − t i , t = 0 , 1 , ..., N in Lemma 4 to analyze how s ti depends on model errors and policy divergences in a step-by-step manner.
Insipred by Janner et al., by summing the divergence at each timestep t = 0 , 1 , ... , we can generate a overall discounted returns bound of branched rollout in Lemma 5.
Lemma 6 is a simple corollary of Lemma 5, which handles normal rollout scenario.
If the returns are bounded, we can show that if every optimization step increase the return by C , the real return would also almost monotonically increase.
Speciﬁcally, if ˆ η [ π k +1 ] ≥ ˆ η [ π k ] + C , combined with η [ π k +1 ] ≥ ˆ η [ π k ] − C , we have η [ π k +1 ] ≥ ˆ η [ π k ] .
But note that ˆ η [ π k ] increases monotonically, and it’s a lower bound of η [ π k +1 ] .
This means that ˆ η [ π k +1 ] would almost monotonically increase.
C. P ROOF OF T HEOREM
Proof.
η p [ π 1 , ..., π n ] and η ˆ p [ π 1 , ..., π n ] differs in two ways.
First, they are estimated with different transition dynamics.
Second, the latter sample their states and actions with another policy, namely π D .
To deal with these divergences separately, we use an intermediate return η [ π D ] , which is the return of π D in the environment.
Then the difference can be bounded by:
[Equation]
To bound L
[Equation]
we apply Lemma 6 with (cid:15) m i = 0 .
Then:
[Equation]
In L
[Equation]
both the dynamics and policies and different
[Equation]
therefore:
[Equation]
Putting Equation 10 and 11 yields:
[Equation]
And because the binary relation of k -hop neighbor is symmetric
[Equation]
we have:
[Equation]
Then, by averaging Equation 12, we have:
[Equation]
C. P ROOF OF T HEOREM
Proof.
To bound | η p [ π 1 , ..., π n ] − η branch [ π 1 , ..., π n ] | , we need to analyze how do they differ from each other.
η p denote the real returns of these policies in the environment.
η branch is the returns estimated in the branched rollout scheme.
To explicitly illustrate this point, we describe their difference in Table 3:
By Lemma 5, we have:
[Equation]
branch point before after dynamics policies dynamics policies
[Equation]
And because the binary relation of k -hop neighbor is symmetric, we have:
[Equation]
Then, by averaging Equation 15, we have:
[Equation]
γ
[Equation]
C. P ROOF OF C OROLLARY
Proof.
Although p ( s (cid:48) | s, a ) might not satisfy the criteria of an INS, we can construct another transition dynamic ˜ p
= (cid:81) ni
=1 p i ( s i | s N i , a i ) , which is the product of marginal transitions, thus being an INS.
Recall that by deﬁnition, if p is a transition dynamic of a ξ -dependent system, sup s,a D TV ( p ( s (cid:48) | s, a ) (cid:107) ˜
p
( s (cid:48) | s, a )) ≤ ξ .
Then we divide the difference into two parts:
[Equation]
The ﬁrst part is the difference of policy returns, estimated in two environments.
Since there are no policy divergences, and the transition dynamics only differs in a global manner, utilizing Lemmayields that:
[Equation] [Equation]
Then, we have:
[Equation]
As for L 2 , because ˜ p is an INS, we can directly apply Theorem 2:
[Equation]
Summing Equation 20 and Equation 21 completes the proof.
C. P ROOF OF T HEOREM
Proof.
In (Qu et al., 2020b)
[Equation]
it was proven that if s N κi and a N κi are ﬁxed, then no matter how other states and actions changes, Q -function will not change signiﬁcantly:
[Equation] [Equation]
As value function is the expectation of Q-function
[Equation]
23) we have,
[Equation]
24) which concludes the proof.
C. P ROOF OF T HEOREM
Proof.
The difference of the gradients can be written as
[Equation]
Because for any function b ( s ) that depends only on s
[Equation]
E [ b ( s ) log π θ i
i ( a i | s ¯ N i )]
= 0 .
Therefore, L 2 in
Equation 25 becomes:
[Equation]
For L 1 , note that r j ( s j
[Equation]
a j ) + γV j ( s (cid:48) )
− V j ( s ) =
− V j ( s ) + r j ( s j , a j ) + (cid:80) κ − 2 t =1 E γ t r j ( s tj , a tj ) + γ κ − 1 V j ( s κ − 1 ) .
And in an INS, s tj , a tj , t = 0 , 1 , ..., κ − 2 is not affected by policy
π i if j / ∈ N κi , we have that
[Equation]
Put Equation 26 and 27 together, we have
[Equation]
D U SEFUL L EMMAS
Lemma 1.
(TVD of Joint Distribution) Consider two distributions p 1 ( x, y )
=
p 1 ( x ) p 1 ( y | x ) and p 2 ( x, y ) =
p 2 ( x ) p 2 ( y | x ) .
The total variation distance between them can be bounded as:
[Equation]
Proof.
D
[Equation]
Lemma 2.
Suppose there are two chains of distributions { x t 1 , t ≥ 0 } , { x t 2 , t ≥ 0 } .
At time t , the states of both chains share an identical state space x t 1 , x t 2 ∈ X t .
Suppose these two chains satisfy a Markov-like property: p t +1 ( x t +1 | x t , ..., x 1 , x 0 ) =
p t +1 ( x t +1 | x t ) .
Then, the TVD of distributions of two chains at time t can be decomposed as:
[Equation]
Proof.
We prove this lemma by induction.
When T = 0 , it’s easy to see that this lemma is true.
Assume it is true for T
= k .
We have:
[Equation] [Equation]
| (cid:88)
[Equation] [Equation]
] |
[Equation]
(cid:88)
[Equation]
| x
[Equation]
) p
[Equation]
x
[Equation]
[Equation] [Equation]
[Equation] [Equation]
( x
[Equation]
x
[Equation]
=
[Equation]
=
[Equation]
x
[Equation]
Then this theorem holds for all n ∈ N .
Lemma 3.
Consider two distributions with pdf/pmf p ( x ) and q ( x ) , where x = ( x 1 , ..., x n ) ∈ R n .
Suppose p and q can be factorized as: p ( x )
= (cid:81) ni
[Equation]
1 p i ( x i ) , q ( x ) = (cid:81) ni
=
q i ( x i ) .
Then we have:
[Equation] [Equation]
Also, if the distance is measured by KL-divergence, we have:
[Equation]
[ p ( x ) (cid:107) q
( x )]
= n (cid:88) i =
D KL [ p i ( x i ) (cid:107) q i ( x i )] .
Proof.
We prove this result for discrete distributions, yet by replacing sum with integration, this result stays true in continuous case.
[Equation]
In KL-divergence case, we have:
[Equation]
x n
[Equation]
Lemma 4.
( N -step distribution distance)
Suppose the expected TVD between two dynamics transitions is bounded as (cid:15)
m i
= max s Ni ,a i D TV (cid:2) p ( s (cid:48)
i | s N
i , a i ) (cid:107) ˆ p ( s (cid:48)
i | s N
i , a i ) (cid:3) and (cid:15) π i = max s D TV (cid:2)
π i ( a i | s N i ) (cid:107)
ˆ π i ( a i
| s N i ) (cid:3) .
If N ≤ κ , the N -step distribution distance is bounded as:
D TV (cid:2) p Ni ( s i ) (cid:107) ˆ p Ni ( s i ) (cid:3) ≤
N − 1 (cid:88) t = (cid:88)
j ∈ N ti ( (cid:15) π j + (cid:15) m j ) .
Thus,
D TV (cid:2) p N i ( s i , a i ) (cid:107) ˆ p N i
( s i , a i ) (cid:3) ≤ (cid:15)
π i +
N − 1 (cid:88) t = (cid:88)
j ∈ N ti ( (cid:15) π j + (cid:15) m j ) .
Proof.
Consider a chain of regional state x t =
s N N − t i .
Two chains of distributions p t ( x t ) , ˆ p ( x t ) denote the distributions of x t under the environment and our model, respectively.
First, because of the property of an INS, these two chains’ transition dynamics can be decomposed as:
[Equation]
And because p j ( s (cid:48)
j | s V j ) = (cid:80) a j p j ( s (cid:48)
j | s V j , a j ) π j ( a j | s V j ) , by the property of TVD, we know that:
[Equation]
With Lemma 3, we have:
[Equation]
Then, by Lemma 2, we know that
[Equation]
which completes the proof of the ﬁrst part.
And by Lemma 1, the second part holds true.
Lemma 5.
(Returns bound measured in branched rollout) Consider two MDPs p ( s ) and ˆ p ( s ) .
Suppose they both adopt T -branched rollout scheme.
Before the branch, suppose the dynamics distributions are bounded as max s Ni ,a i D TV (cid:0) p prei ( s (cid:48)
i | s N
i , a i ) (cid:107) ˆ p prei ( s (cid:48)
i | s N
i , a i ) (cid:1) =
(cid:15) prem i , and policy divergences are bounded as max s Ni D TV (cid:0) π prei ( a i | s N i ) (cid:107)
ˆ π prei ( a i | s N i ) (cid:1) =
(cid:15) preπ i .
After the branch, (cid:15) postm i and (cid:15) postπ i are deﬁned similarly.
Then the (local) T -step returns are bounded as:
[Equation]
Proof.
We prove this by estimating the state-action distribution divergence at each timestep.
For notation simplicity, we denote (cid:15)
preN
ki = (cid:15)
preN
ki = (cid:80) j ∈ N ki
(cid:15) prem j + (cid:15) preπ j ) , and (cid:15)
postN ki analogously.
If we divide the chains into two parts: pre-branch and post-branch, and apply Lemma 4 on both parts, we have:
For t ≤ T :
[Equation]
and for t > T :
[Equation]
Then, the difference of discounted distribution can be bounded as:
[Equation]
We can convert this bound into the returns bound:
[Equation]
Lemma 6.
(Returns bound measured in full length rollout) Consider two MDPs p ( s ) and ˆ p ( s ) .
Suppose they both run their rollouts until the end of every trajectory, and the dynamic distributions are bounded as max s Ni ,a i D TV (cid:0) p
i ( s (cid:48)
i | s N
i , a i ) (cid:107) ˆ p
i ( s (cid:48)
i | s N
i , a i ) (cid:1)
=
(cid:15) m i , while policy divergences are bounded as max s Ni D TV (cid:0) π i ( a i | s N i ) (cid:107)
ˆ π i ( a i
| s N i ) (cid:1) =
(cid:15)
π i .
Then the (local) returns are bounded as:
[Equation]
Proof.
We can think it as a special case of branched rollout, where (cid:15) post = (cid:15)
pre = (cid:15) for every subscript, and T = 0 .
From this perspective, applying the result of Lemma 5 completes the proof.