Mingling Foresight with Imagination: Model-Based Cooperative Multi-Agent Reinforcement Learning
Anonymous Author(s)
AffiliationAddress email
Abstract
Recently, model-based agents have achieved better performance than model-freeones using the same computational budget and training time in single-agent environments.
However, due to the complexity of multi-agent systems, it is tough tolearn the model of the environment.
The significant compounding error may hinderthe learning process when model-based methods are applied to multi-agent tasks.
This paper proposes an implicit model-based multi-agent reinforcement learningmethod based on value decomposition methods.
Under this method, agents caninteract with the learned virtual environment and evaluate the current state valueaccording to imagined future states in the latent space, making agents have theforesight.
Our approach can be applied to any multi-agent value decompositionmethod.
The experimental results show that our method improves the sampleefficiency in different partially observable Markov decision process domains
Introduction
In recent years, reinforcement learning has made remarkable achievements in game AI, robots, andautonomous driving.
It is primarily due to the progress of model-free reinforcement learning (MFRL)research.
Unlike model-based reinforcement learning (MBRL), MFRL does not require dynamicsof the environment and learns from the data generated by directly interacting with the environment.
Therefore, although most environment models are unavailable in the real world, MFRL can mastercomplex tasks through large-capacity value function approximators.
However, MFRL requires largeamounts of training data that can be costly and risky in reality, which means that the sample efficiencyis the main bottleneck of the current MFRL algorithms.
So in the problem that the dynamics model isknown, we should prioritize using MBRL, which can even obtain an analytical solution to optimalcontrol problems.
Multi-agent systems are more complex than single-agent tasks.
The strategy of each agent changesduring the learning process in multi-agent systems, which makes the environment unstable from theperspective of each agent.
One method [42, 11, 36, 19, 22] is for agents to use channels to transmitinformation to each other, in this way to reach a consensus between agents.
In addition, many currentstudies on multi-agent reinforcement learning have focused on the paradigm of centralized trainingwith decentralized execution (CTDE)
The agent only obtains global information during trainingcan significantly alleviate the instability problem.
The first branch [28, 12, 16] that emerged isusing the Actor-Critic structure’s advantages to build the centralized critic and decentralized actorframework.
The value decomposition method [43, 37, 41, 29, 47, 46] is also the most popular CTDEapproach lately, and it has obtained excellent performance in the decentralized partially observable
Markov decision process (Dec-POMDP) domains [31].
Nevertheless, most of the above methods aremodel-free and require access to an impractically large number of trajectories.
Some work combines model-based and model-free methods.
They usually learn environment modelsand solve control problems simultaneously, and have achieved good performance in some single-agentscenarios, such as MuJoCo [13] and Atari [20].
However, in multi-agent problems, the training ofthe world model faces great challenges because of the complexity of the environments.
All currentwork on model-based multi-agent reinforcement learning can only be applicable for simple scenariossuch as matrix games.
In this paper, we propose M odel- B ased V alue D ecomposition (MBVD), amethod that introduces the idea of model learning into value decomposition.
When humans makedecisions, they not only rely on the current state but also consider the future state obtained afterseveral interactions with the environment following their current strategies.
The phenomenon iscalled "long-term vision" or foresight.
Enlightened by this ability of humans, we make agents havethe foresight to cooperate by obtaining the aggregated latent states in the future.
Finally, we evaluatedthe performance of MBVD in several different domains, including StarCraft II [39], Google Research
Football [25] and Multi-Agent MuJoCo [35].
We clarified that MBVD has high sample efficiency,which exceeds the performance of other baselines in most scenarios.
To the best of our knowledge,our study is the first attempt to apply the model-based ideas to Dec-POMDP problems.
Related Work2.1 Single-Agent Model-based RL
The model-based approaches are divided into three different categories.
The first branch is representedby Dyna-Q [44, 26, 20, 7, 50], the method of alternating the two processes, including learningenvironment models and improving policies.
The focus of this idea is that the world model generatestrajectories used by the training of reinforcement learning.
Analytic-gradient algorithms can calculatethe analytic gradient related to the reinforcement learning optimization goal through the parameterizedworld model, thereby directly improving the policy.
This method is also called Policy Search with
Backpropagation through Time [14, 9, 15, 8, 13].
The last method rolls out the learned modelsover multiple time steps to predict the value of states.
VPN [30] plans and calculates the path withthe largest accumulated reward in rollouts and goes back to the current moment to improve targets.
MVE [10] uses an update method similar to n-step q-learning to calculate the target state value.
However, a fundamental limitation is that the above algorithms rely heavily on the world model.
So ifthe learned environment model is inaccurate, it will lead to the collapse of the entire learning process.
Especially the last method suffers from the limitation because, in addition to learning the dynamicsmodel of the environment, model-augmented value expansion algorithms [40, 48, 18, 1, 53, 3] alsoneed to learn the reward function of the environment to calculate the accurate value function.
It isimpractical in sparse reward environments.
2.2 Multi-Agent Model-based RL
So far, there is relatively little work on multi-agent model-based RL.
[51] obtained the samplecomplexity of multi-agent model-based RL through theoretical proof when the dynamics model wasavailable.
Nevertheless, it only applies to simple infinite-horizon zero-sum discounted Markov gamesand has not been implemented.
M 3 –UCRL [34] combines model-based RL with mean-field gametheory, which can be used in cooperation problems like stylized swarm motion.
However, due tothe limitation of the mean-field theory, M 3 -UCRL can only be suitable for the scenarios of a largepopulation of agents.
Both [33] and [52] construct the environment model that includes a transitionfunction and a prediction model for the opponents’ actions, and then train their policies with theopponent-wise rollouts.
These two methods have been evaluated in the multi-particle environment,and the results demonstrate that they can reduce sample complexity.
However, it is not feasibleto build the opponent models without access to opponents’ information.
CPS [2] approximates afactored sparse Q-function, which is similar to the value decomposition method.
Besides, CPS learnsthe environment model and maintains a replay buffer with priority, but it cannot solve the partiallyobservable problems.
Similarly, in the multi-agent model-based RL field, we also face compoundingerrors caused by imperfect learned models.
MBVD focuses on learning a standard forward dynamics model and aggregating the informationcontained in the environment model rollouts.
Then the current state value, which is with respect tothe imagined rollouts, can be obtained.
We believe the imagined information can enable all agents tounderstand the current situation better.
Preliminaries3.1 Dec-POMDP
Partially observable stochastic games (POSGs) are one of the most general games, and the DecPOMDP [31] is an important subclass of POSGs.
Formally, the Dec-POMDP is defined as a tuple
G = ⟨ S, U, A, P, r, Z, O, n, γ ⟩ .
Each agent a ∈ A := { 1 , . . .
, n } selects the appropriate actionu a ∈ U at each time step, with only access to the local observation z a ∈ Z provided by theobservation function
O ( s, a ) : S ×
A → Z , where s ∈ S is the global state of the environment.u ∈
U ≡ U n denotes the combined action of all agents.
The state transition function, commonlyknown as the environmental dynamics, is written as P ( s ′ | s, u ) :
S × U × S → [0 , 1] .
Noted thatall agents in Dec-POMDPs have the same reward function: r ( s, u ) : S × U
→ R .
And γ is thediscount factor.
3.2 Value Decomposition
When there are multiple agents in the system, it is impossible to train each agent separately or treatall agents as one entity for joint training in cooperative multi-agent reinforcement learning.
Theappearance of value decomposition methods promotes the collaboration between agents and solvesthe credit assignment problem.
In the Dec-POMDP, we make the following assumption:
[Equation]
where τ ∈ T n represents the joint action-observation histories of all agents, Q tot is the global actionvalue function, and Q n is the individual ones.
The assumption is known as the Individual-Global-Max(IGM) [41] principle, which states that a task may only be decentralized if there is consistency betweenthe local greedy actions and global ones.
Many multi-agent reinforcement learning algorithms haveperformed well by observing the IGM principle, like VDN [43] and QMIX [37].
3.3 Environment Models
The most notable feature of MBRL is that while training the policy, it also learns the world model, aparametric model used to simulate the environment.
Two different models can be learned unsupervisedfrom local observations and actions, namely auto-regressive and state-space models.
Auto-regressive models [21, 5] are intuitive and straightforward.
However, there are two reasons forthe high computational complexity of the auto-regressive models: calculated items of the generationprocess cannot be reused, and auto-regressive models need to render high-dimensional observationsexplicitly.
On the contrary, state-space models [32, 24] first abstract the environment to find a compactlatent state space S containing all vital information.
In the multi-agent system, each ˆ s t ∈ S is anabstract representation of the local observations z t of all agents.
So the observation function can beexpressed by p ( z t | ˆ s 0: t , u 0: t − 1 )
=
p ( z t | ˆ s t ) .
Using imagined rollouts obtained by the transitionfunction on the low-dimensional latent state space, state-space models can significantly reduce theamount of calculation.
The factorization of the predictive distribution is as follows:
[Equation]
where p init (ˆ s 0 ) denotes the prior distribution of the initial state
ˆ s 0 , which is usually a constant.p (ˆ s t | ˆ s t − 1 , u t − 1 , z t ) is the true posterior of the latent state, and p ( u t | z t ) means the joint policy πof all agents.
Model-Based Value Decomposition
This section will elaborate on MBVD, a novel model-based multi-agent reinforcement learningalgorithm based on value decomposition.
We will introduce the framework and flowchart of MBVDfirst and then describe the detailed implementation of MBVD.
[Figure]
4.1 The MBVD Framework
To maintain the applicability of algorithms, the framework of the reinforcement learning part in
MBVD is consistent with other value decomposition methods, including the agent network and themixing network.
We focus on the model learning part of MBVD, called the imagination module.
Inspired by amortized variational inference, we employ neural networks to approximate the intractableposterior p (ˆ s t | ˆ s t − 1 , u t − 1 , z t ) .
The approximate posterior is defined by q θ (ˆ s t | ˆ s t − 1 , u t − 1 , z t ) ,where θ is the parameters.
So we can derive the evidence lower bound (ELBO) as follows:
[Equation]
and the ELBO can be broken down as:
[Equation]
[ q θ (ˆ s t | ˆ s t − 1 , u t − 1 , z t )
∥ p (ˆ s t | ˆ s t − 1 , u t − 1 )] } .
[Equation]
The first term log [ p ( u t | z t )] is the joint policy and we can ignore it.
Furthermore, the termlog [ p ( z t | ˆ s t )] can be viewed as the observation model.
In addition, we propose a new parameterizedfunction p
Prior ϕ (ˆ s t | ˆ s t − 1 , u t − 1 ) to approximate the dynamics model p (ˆ s t | ˆ s t − 1 , u t − 1 ) , which isunavailable.
MBVD follows the idea of planning in latent spaces, so we use the Variational Autoencoder(VAE)
[23] to maximize the ELBO.
The reason is that generative models can find a low-dimensionalspace by reconstructing the original data, which is consistent with abstracting the decision-relatedstate from the real observations.
For the last term in Equation 1, we regard the two items in KullbackLeibler (KL) divergence as the posterior and the prior.
Intuitively, according to Equation 1, we needto minimize the KL divergence between the posterior q θ ( · ) , which incorporates information about thecurrent observations z , with the prior p Prior ϕ ( · ) that tries to predict the posterior without access to thecurrent observations.
Then we use two VAEs as the prior and the posterior, respectively, as shown in
Figure 1(a).
In this way, we obtain the prior latent state and the posterior latent state:
[Equation]
However, we use a modified version of the posterior model in the implementation process.
Sincethe hidden output h at of the recurrent neural network in the agent network can be regarded as the integration of all past information of the individual agent, the posterior q θ (ˆ s t | ˆ s t − 1 , u t − 1 , z t ) can berewritten as q θ (ˆ s t | h t ) .
The advantages of minimizing the KL loss are that we can not only guidethe prior to predicting the latent posterior state distribution without the information of the currentstep but also control how much information the posterior abstracts from the original observation.
It isworth noting that the learned prior model can predict forward in the latent space based on the actionsof all agents, even if the actions are different from reality.
So the imagination module in MBVD weproposed is an action-conditional environment model.
From another perspective, we can view theprior model as the transition function, and the decoder q θ ( z t | ˆ s t ) of the posterior can be regarded asthe observation function.
Through the above framework, we have built the imagination module thatis general and model-agnostic for the multi-agent value decomposition methods.
In addition to learning the transition function and observation function, in some complex environmentssuch as StarCraft II, the imagination module also needs to predict the feasible action set A to chooseimagined actions more reasonably.
Since the posterior can access the actual information of thecurrent step, we can perform multiple auxiliary tasks by using the latent state ˆ s t ∼ q θ (ˆ s t | h t ) asthe intermediate variable to predict the additional environmental signals.
The reward function r ( · )is crucial in reinforcement learning because it steers the agents’ behavior.
However, predictingthe reward function in noisy, sparse-reward, or complex environments is often difficult.
So theimagination module in MBVD does not utilize the reward signal in the environment.
MBVD adoptsthe implicit model-based idea without reward prediction.
The imagination model we offered has all the essential components of the environment, except theaction-value model, which can evaluate the expected returns.
We leave the implementation of theaction-value model to the reinforcement learning process.
With the help of the imagined statesgenerated from the simulated model, the agents seem to have the foresight, which means that theagents estimate the action value based on long-term consequences rather than direct results.
Thus, wecan discern the role of the imagination module in MBVD.
4.2 The MBVD Flow Diagram
The generation process of imagined rollouts can be found in Figure 1(b).
At time t , τ t refers to thetrajectory history of all agents.
Since the posterior encoder can be regarded as the inverse functionof the observation function, we can infer the latent state
ˆ s t from τ t .
Furthermore, according to thecurrent joint policy π of all agents (implemented by the agent network), we can get the actionsˆ u t ∼ π ( · | τ t ) that all agents should perform to maximize the estimate of the state-action value.
Then the learned prior model is used to derive the next latent state
ˆ s Prior t +1 based on ˆ s t and ˆ u t .
Toget the joint action ˆ u t +1 at the next step in the imagined rollouts, we need to obtain the trajectoryhistory τ ′ t +1 ∼ q θ (cid:0) · | ˆ s Prior t +1 (cid:1) corresponding to the latent state
ˆ s t +1 through the observation function.
We perform one step forward in the imagined rollout in the above way.
Then we can roll out theenvironment model over multiple time steps by feeding the following imagined variables into themodel.
As mentioned above, we use all individual hidden outputs h instead of the trajectories τ toaccelerate the training speed in the implementation.
We use the latest policies of all agents as the rollout policies.
Here is a reasonable and intuitiveexplanation: when people make a strategic decision, they always use their current policies as theimagined policies for their decision-making, rather than previous strategies that have led to poorperformance.
Besides, for the stability of reinforcement learning, we turn the random process involvedin the imagination module during reinforcement learning into a deterministic form.
For example, theimagined actions ˆ u are selected from greedy policies.
Moreover, we directly take the mean of theapproximating distribution as the inferred latent state ˆ s and trajectory history ˆ τ .
However, we stilluse the reparameterization trick when learning the imagination module.
One k -step rollout starting with ˆ s t can be written as (cid:8) (ˆ s t , τ t , ˆ u t ) , (cid:0) ˆ s Prior t +1 ,
τ ′ t +1 , ˆ u t +1 (cid:1) , . . .
,(cid:0) ˆ s Prior t + k , τ ′ t
+ k , ˆ u t + k (cid:1)(cid:9) .
Since the latent state ˆ s is inferred from all agents’ historical trajectoriesand actions, ˆ s contains the information of τ and u .
Furthermore, to ensure that the parameters ofthe imagination module do not increase with the rollout horizon k , the set of imagined latent states{ ˆ s t , ˆ s Prior t +1 , . .
, ˆ s Prior t + k } is fed into the GRU [6] to get the aggregated rollout state ˆ s Rollout t .
Finally, weconcatenate the aggregated rollout state
ˆ s Rollout t with the real global state s t and input them into themixing network of the value decomposition framework.
We believe that the imagined states containinformation about the possible states of the future, which can help all agents evaluate the current statemore accurately.
MBVD does not require a pre-trained environment model, which means the simulated model and theaction-value model are all trained from scratch simultaneously.
As an end-to-end efficient MBRLalgorithm, MBVD can be extended to any value decomposition method with the mixing network.
4.3 Overall Learning Objective
Next, we will elaborate on the training objectives of MBVD.
MBVD involves two processes: thereinforcement learning process, whose objective is to minimize the td-error; the other is learningthe imagination module, which includes the optimization of the prior and the posterior.
These twoprocesses are carried out simultaneously.
For the convenience of description, we define the parametersof the value decomposition framework as ψ .
MBVD can be seen as the value decomposition method with an imagination module, so the lossfunction for reinforcement learning is consistent with the original value decomposition method.
Themost obvious difference is that the global action-value function Q tot is calculated with respect tothe actual state s and the latent rollout state ˆ s Rollout .
It is important to note that MBVD generatesthe aggregated rollout state ˆ s
Rollout conditioned on the past state s in the replay buffer, so we do notchange the off-policy update paradigm.
The loss function for reinforcement learning can be obtained:
[Equation]
where y tot = r t + γ max u t +1 Q tot (cid:0) τ t +1 , u t +1 , s t +1 , ˆ s Rollout t +1 ; ψ − ) , and ψ − represents the parameters of the target network.
The loss function of the posterior can be divided into reconstruction loss and KL divergence loss.
The posterior model infers the current latent state after the observations of all agents are given,which requires that the posterior model extract helpful information from the original input.
It canbe achieved by narrowing the difference between the model output τ ′ and the model input τ .
Thereconstruction loss function of the prior model is similar, and both of them can be computed by:
[Equation]
where MSE means the mean square error.
Furthermore, in addition to the KL term in Equation 1,we throw in the KL divergence between the prior distribution and the standard Gaussian distribution
N (0 , 1) as a regular term to avoid the overly complex prior distribution of the latent state ˆ s Prior .
Sothe KL loss is as follows:
[Equation]
We use the KL balancing mechanism to optimize the KL term in the actual implementation.
KLbalancing enables the prior to minimize KL loss at a faster learning rate than the posterior, which canbe expressed as:
[Equation]
where α ∈ [0 , 1] is a configurable parameter, Detach( · ) can stop the backpropagation from gradientsof certain variables or functions.
In addition, we have other optional training objectives for auxiliary tasks in complex scenarios.
Inthis paper, we make predictions of the feasible action set only in StarCraft II:
[Equation]
where BCE denotes the binary cross-entropy error.
Thus, the total loss function can be written as:
[Equation]
By minimizing the total loss function L , we can guide MBVD to accelerate reinforcement learningwith the help of the imagination module.
Experiments
This section will consider whether the value decomposition method can benefit from the implicitmodel-based method.
We compare the performance of MBVD with other popular baselines, including
VDN, QMIX, MAVEN [29], Weighted QMIX [38], ROMA [46], and RODE [47].
Then by conductingthe ablation experiments, we can clarify that every component in the learned model is indispensable,and different horizons of the rollout have a significant impact on performance.
Finally, we willvisualize the imagined rollouts, which will more intuitively and powerfully show the foresight of
MBVD.
Note that the implementation of agents in MBVD in all experiments is based on QMIX.
Thedetails of all experiments can be found in Appendix B.
[Figure]
5.1 Performance on StarCraft II
SMAC is a multi-agent micro-management experimental platform based on the real-time strategygame StarCraft II, which contains a wealth of scenarios corresponding to different challenges.
Toverify the performance improvement brought by the imagination module, we pay more attention tothe performance comparison between MBVD and QMIX.
We select some representative scenarios.
The median performance and the 25-75% percentiles are illustrated in Figure 3.0.
2.0 T (mil)
M ed i an T e s t
W i n ( % ) 2s3z 0.
2.0 T (mil)
M ed i an T e s t
W i n ( % ) 3s5z 0.
2.0 T (mil)
M ed i an T e s t
W i n ( % ) 1c3s5z 0.
[Figure]
i
[Figure]
n
[Figure]
s t
[Figure]
i
[Figure]
MBVD reaches state-of-the-art performance in most scenarios and exceeds the basic algorithm
QMIX.
Especially in hard or super hard scenarios such as 2c_vs_64zg , MMM2 , and 3s5z_vs_3s6z , thesuperiority of MBVD is more significant.
In 5m_vs_6m , due to the uncertainty of the environment,other baselines will encounter a performance bottleneck.
However, MBVD can predict the future stateto achieve the best performance in 5m_vs_6m .
Even in some easy scenarios, MBVD still has highsampling efficiency.
It is impossible for some complex QMIX-style variants.
MBVD can robustlyreduce the sampling complexity in both easy and hard scenarios.
5.2 Performance on Google Research Football
The Google Research Football environment provides a novel reinforcement learning environmentwhere multiple agents can be trained to play football.
To verify the effectiveness of our proposedmethod, we carry out MBVD and other baselines on the Football Academy, which is a diverse setof mini scenarios of varying difficulty.
We select three representative official scenarios, and theexperimental results are delivered in Figure 4.
[Figure]
e s t
[Figure]
i n
[Figure]
Compared with QMIX, which has a relatively simple structure, algorithms such as RODE and ROMAdo not perform well.
A possible reason is that the number of agents in these scenarios is not large,and the role assignments may hinder the learning.
Conversely, MBVD based on QMIX achieves thehighest sample efficiency in three scenarios with different difficulty levels.
The experimental resultsobtained in Google Research Football also fully demonstrate the generalization capability of MBVD.
5.3 Performance on Multi-Agent Discrete MuJoCo
Many model-based reinforcement learning algorithms, including the work mentioned above, tendto be compared on MuJoCo.
[35] recently proposed a novel benchmark for continuous cooperativemulti-agent robotic control in the multi-agent field, called Multi-Agent MuJoCo.
A given singlerobotic agent is viewed as a body graph containing many disjoint sub-graphs, and each sub-graphcontains one or more joints that can be controlled.
In this paper, each joint is regarded as an agent thatmakes local decisions conditioned on partial observations.
Since this paper focuses on the impactsof the world model, we propose a discrete variant of Multi-Agent MuJoCo.
Detailed environmentsettings can be found in Appendix B.3.
We selected four representative scenarios, and Figure 5describes the episode return of each algorithm.
Since most of the relatively complex algorithmscannot converge in this environment, we only choose MBVD, QMIX, and VDN for comparison.
[Figure]
From the results, we can intuitively see the performance improvement brought by the imaginationmodule because we see MBVD as a variant of QMIX that has an additional implicit foresight module.
MBVD outperformed QMIX in all given scenarios, which also means MBVD we proposed can beapplied to different environments.
5.4 Ablation Studies
We carry out ablation studies to test the contribution of the components of the imagination moduleand investigate how the rollout horizon k affects the performance of MBVD.
For the first study,we proposed two variants of QMIX that input the additional information to the mixing network,
QMIX-RS and QMIX-LS.
Both of them aggregate the information of the next k steps from the actualtrajectories rather than imagined rollouts.
However, the difference is that QMIX-RS uses the realstates and QMIX-LS the latent states.
To answer the second problem, we run MBVD with differentrollout horizons on the 2c_vs_64zg scenario of SMAC and compare their performance.
[Figure]
As shown in Figure 6(a), both QMIX-RS and QMIX-LS failed to solve the task, which means that thetransition function that can generate the imagined rollouts is critical.
Besides, we can also infer thecontribution of the observation function from that the better performance of QMIX-LS than QMIXRS.
For the horizon of the rollouts, we can conclude from Figure 6(b).
When using short-horizonrollouts, the agents cannot fully interact with the imagined model, resulting in the same performanceas vanilla QMIX.
However, as k increases significantly, MBVD loses the monotonic improvementsbecause of the compounding error.
The effect of k holds the same for the other scenarios, but theoptimal choice of k in each task is different.
If not explicitly stated, we will set k to 3 in this paperfor convenience.
5.5 Visualization
To intuitively explain agents’ foresight ability in MBVD, we visualized the latent state sequencegenerated by the interaction between agents and the imagined model.
For the trajectories of anepisode in the 2c_vs_64zg scenario, we visualized the t-sne embeddings of the latent states predictedat each step in the rollout and compared them with the real latent state embedding.
From Figure 7, wefind that the difference between the embedding of the imagined latent states and real ones graduallyincreases as the horizon length grows, but there are similarities between them.
It also reveals that ourproposed imagination module can capture and learn the dynamics of the multi-agent environment.
[Figure]
Conclusion
Due to the high complexity of multi-agent systems, MBRL algorithms are difficult to be appliedto them.
This paper proposes MBVD, a novel and implicit model-based cooperative multi-agentreinforcement learning method.
By learning the world model and imagining the future latent statesafter making several decisions under the current policy to estimate the current state value, agentsin MBVD obtain the foresight ability.
Through experiments and visualization, we have proved theefficiency and generalization of MBVD.
This is the first study on Dec-POMDPs from the perspectiveof model-based reinforcement learning.
The defect in this study is that the rollout horizon is manually chosen.
In our future research, weintend to concentrate on how to choose the appropriate rollout horizon.
The issue is an intriguing onewhich could be usefully explored in further research.
Checklist
For all authors...(
a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and
scope?
[Yes b) Did you describe the limitations of your work?
Yes](
c) Did you discuss any potential negative societal impacts of your work?
[N/A]
d) Have you read the ethics review guidelines and ensured that your paper
conforms to
them? [Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix A.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
[Yes] See Appendix B.
(c) Did you report error bars (e g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix B.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re [Table] using/curating?
[N/A]
[Table] (e) Did you discuss whether the data you are using/curating contains personally identifiable [Table]
information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table] (a) Did you include the full text of instructions given to participants and screenshots, if [Table] applicable?
[N/A]
[Table] (b) Did you describe any potential participant risks, with links to Institutional Review [Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table] (c) Did you include the estimated hourly wage paid to participants and the total amount [Table]
spent on participant compensation?
[N/A]
[Table]
A Derivation of the ELBO
The complete derivation of the ELBO in Equation 1 is as follows.
[Equation]
where the inequality in Equation 3 is obtained via Jensen’s inequality.
B Experimental Setup
In order to clarify the generalization of our proposed method, we used three different Dec-POMDPdomains, SMAC, Google Research Football, and Multi-Agent Discrete MuJoCo, as experimentalplatforms.
All experiments in this paper are carried out with five different seeds on Nvidia GeForce
RTX 3090 and Intel(R)
Xeon(R)
Platinum 8280.
We use the official codes for other baselinealgorithms, and the hyperparameters are consistent with their original work.
Next, we will introducethe settings of the three environments, respectively.
B.1 SMAC
In SMAC, each agent can only obtain entity information within the visible range.
The goal of trainingis to guide the allied agents to defeat the enemy units, so the reward function is related to the healthvalue of all enemy agents.
Besides, agents can obtain the current set of available actions.
We set thedimension of the latent state in SMAC as the product of 16 and the number of allied agents, and othertraining hyperparameters for MBVD follow that of QMIX.
For MBVD, each independent experimenttakes 8 to 30 hours, which is the same as the time spent by QMIX.
We used StarCraft version SC2.4.6.2.69232 instead of the relatively easy version SC2.4.10.
Theresults for different versions are not directly comparable since the underlying dynamics differ.
Table 1provides an overview of the SMAC scenarios.
The recognized difficulties of the scenarios are alsodetermined based on the version SC2.4.6.2.69232 of StarCraft.
[Table]
B.2 Google Research Football
In Google Research Football, we need to train our players to kick the ball into the opponent’sgoal.
We chose three official scenarios in the Football Academy: academy_3_vs_1_with_keeper ,academy_pass_and_shoot_with_keeper , and academy_run_pass_and_shoot_with_keeper .
The initialpositions of all players and the ball in the three scenarios are shown in Figure 8.
We control all theagents in red (except the goalkeeper on the far left) against the agents in blue, which are controlledby built-in AI.
Our players have 19 discrete actions, including moving, passing, shooting ,and soon.
Unlike SMAC, the agents in Google Research Football cannot know which actions are feasible.
The global state of the environment includes the two-dimensional position coordinates and movingdirections of all agents on the field, as well as the three-dimensional position coordinates and movingdirections of the football.
The physical meaning of the local observation is the same as that of theglobal state, except that the absolute positions of all entities are replaced with relative ones.
[Figure]
For the reward function, in addition to the reward for scoring a goal, we also added an additionalreward contribution for moving the ball close to the opponent’s goal, similar to the official CHECKPOINT reward function in the Football Engine.
To increase the game’s difficulty and speed up the agent’s training, we set the episode to end when the ball returns to the left half.
Besides, scoringgoals and reaching the maximum time step will also cause the episode to be terminated.
Due to the small number of agents in Google Research Football scenarios, we adjusted the latentstate dimension to the product of 8 and the number of our players.
In addition, since there is noinformation about the feasible action set in this environment, we ignore the L FA item in Equation 2.
All experiments in Google Research Football were completed within two days.
B.3 Multi-Agent Discrete MuJoCo
[Figure]
In the original Multi-Agent MuJoCo, the action space of each joint is [ − 1 , 1] .
To accommodatealgorithms like QMIX, we discretize the action space into K equally spaced atomic actions.
The setof atomic actions for any joint is A = (cid:110)
j K − 1 − 1 (cid:111)
K − 1 j =0 .
We treat each joint as an agent, and jointsare connected by adjacent edges.
A configurable parameter l ≥ 0 determines the maximum graphdistance to the agent at which joints are observable.
The agent observation is then given by a fixedorder concatenation of the representation vector of each observable joint.
Furthermore, all featuresin the state are normalized.
With the above modifications, we get a benchmark for cooperativemulti-agent robotic control with discrete action spaces.
In this paper, we set K = 31 and l = 1 .
[Figure]
B.4 Hyperparameters
[Figure]
In this paper, we use the QMIX-style framework with its default hyperparameters suggested by theoriginal paper for the reinforcement learning process of MBVD.
Table 2 presents the hyperparametersof MBVD.
We use n to represent the number of our agents in the environment.
[Figure]
of optimizer
[Table]
many episodes to update target networks
Reduce global norm of gradients
[Table]
[Table] C Additional Experimental Results [Figure] C.1 Results of Other Scenarios in SMAC
[Figure]
We give the performance of all algorithms on other official SMAC maps in Figure 10.
The version of
StarCraft II in the paper is SC2.4.6.2.69232.
The reason why we use this difficult version is that thedifficulty of each map is originally delineated according to this version.
We do not think there willbe a significant change in the ranking of algorithm performance even in SC2.4.10.
In 3s_vs_5z andcorridor , MBVD performs worse than some other baselines, which we believe is caused by its basicalgorithm QMIX.
As can be seen from the 3s_vs_5z scenario, MBVD can still improve the sampleefficiency of QMIX.
In both 6h_vs_8z and corridor maps, QMIX fails to solve tasks, which is why
MBVD performs poorly in both scenarios.
However, MBVD based on QMIX performs better thanother baselines in most scenarios and can be applied to almost all value decomposition methods toimprove the sample efficiency of the original algorithm.
[Figure]
C.2 Additional Ablation Studies
[Figure]
We perform ablation studies on other SMAC maps and show the results.
The experiments areperformed in the easy map 1c3s5z and the super hard map MMM2 , respectively.
We still explore therole of the imagination module first.
In Figure 11, QMIX-RS and QMIX-LS still perform poorly,especially in MMM2 , which clarifies that the inconsistency between the current policy and the policythat generates imagined states is detrimental to the reinforcement learning process.
[Figure]
Next, we continue to explore how horizon length affects the performance of MBVD.
The performanceof MBVD under different k values in these two scenarios is shown in Figure 12.
In the easy scenario1c3s5z , the sample efficiency of MBVD under smaller k values is higher; in the super hard scenario
MMM2 , the opposite is true.
Therefore, we conclude that the optimal value of k is different indifferent scenarios.
Longer rollout horizons in easy scenarios will introduce more instability early intraining.
In hard scenarios, small values of k can make MBVD underutilize its imagination.
[Figure]
D Discussion on the Choice of the Rollout Horizon
As we describe in ablation studies, the choice of horizon length k affects the performance of MBVD.
Short horizons are easier to be predicted but do not provide enough information to be useful fordecision-making.
Long horizons can carry more valuable information, but the generated imaginedrollouts are inaccurate because of compounding errors.
There has been some work on model-basedreinforcement learning that attempts to break this trade-off.
For bias that might exist in samplingand environmental models, MBPO [17] suggests a branched rollout.
And MBPO can determine thelongest tolerable rollout length according to the lower bound of return.
BMPO [27] uses the newlyintroduced bidirectional models to significantly reduce model compounding error.
In addition, mostof the model-based value expansion (MVE) methods [10] involve adaptive selection of horizons.
They must use a dynamics model to simulate the short-term horizon and Q-learning to estimate thelong-term value beyond the simulation horizon.
The automatic selection of k would be made possibleby explicitly estimating uncertainty in the dynamics model or ensemble models.
STEVE [4] suggestsinterpolating the estimated values of various rollout steps, and the weight for each rollout step ischosen by considering the ensemble predictions’ variance.
AdaMVE [49] mitigates the detrimentaleffects of the compounding error by selecting the rollout horizon for any state based on the learnedmodel error function.
RAVE [54] uses probabilistic models to capture uncertainty (including aleatoricand epistemic uncertainty) and uses the lower confidence bound for value estimation to avoidoptimistic estimation.
DMVE [45] exploits the fact that the uncertainty of the model and the noveltyof data are highly relevant in deep learning.
So DMVE selects horizons based on the top k minimumreconstruction errors of the auto-encoder.
To sum up, in order to further solve the problem of adaptivehorizon selection, we can try to select an appropriate k value with the uncertainty of the imaginedstate as a reference.