Multi-Agent Reinforcement Learning is A Sequence Modeling Problem
Abstract
Large sequence models (SM) such as GPT series and BERT have displayed outstanding performance and generalization capabilities on natural language process, vision and recently reinforcement learning.
A natural follow-up question is how to abstract multi-agent decision making also as an sequence modeling problem and benefit from the prosperous development of the SMs.
In this paper, we introduce a novel architecture named Multi-Agent Transformer (MAT) that effectively casts cooperative multi-agent reinforcement learning (MARL) into SM problems wherein the objective is to map agents’ observation sequence to agents’ optimal action sequence.
Our goal is to build the bridge between MARL and SMs so that the modeling power of modern sequence models can be unleashed for MARL.
Central to our MAT is an encoder-decoder architecture which leverages the multi-agent advantage decomposition theorem to transform the joint policy search problem into a sequential decision making process; this renders only linear time complexity for multiagent problems and, most importantly, endows MAT with monotonic performance improvement guarantee.
Unlike prior arts such as Decision Transformer fit only precollected offline data, MAT is trained by online trial and error from the environment in an on-policy fashion.
To validate MAT, we conduct extensive experiments on StarCraftII, Multi-Agent MuJoCo, Dexterous Hands Manipulation, and Google Research Football benchmarks.
Results demonstrate that MAT achieves superior performance and data efficiency compared to strong baselines including MAPPO and
HAPPO.
Furthermore, we demonstrate that MAT is an excellent few-short learner on unseen tasks regardless of changes in the number of agents.
See our project page at https://sites.google.com/view/multi-agent-transformer (1) .
Introduction
Multi-agent reinforcement learning (MARL) [37] is a challenging problem for its difficulty which arises not only from identifying each individual agent’s policy improvement direction, but also from combining agents’ policy updates jointly which should be beneficial for the whole team.
Recently, such difficulty in multi-agent learning has been eased owing to the introduction of centralized training for decentralized execution (CTDE) [10], which allows agents to access the global information and opponents’ actions during the training phase.
This framework enables successful developments of methods that directly inherit single-agent algorithms.
For examples, COMA replaces the policy-gradient (PG) estimate with a multi-agent PG (MAPG) counterpart [10], MADDPG extends deterministic policy gradient into multi-agent settings with a centralized critic [18, 29], QMIX leverages deep Q networks for decentralized agents and introduces a centralized mixing network for Q value decomposition [24, 31, 21], and MAPPO endowing all agents with the same set of parameters and then training by trust-region methods [38].
These methods, however, cannot cover the whole complexity of multi-agent interactions; in fact, some of them are shown to fail in the simplest cooperative task [15].
To resolve this issue, the multi-agent advantage decomposition theorem was proposed [15, Theorem 1] which captures how different agents contribute to the return and provides an intuition behind the emergence of cooperation through a sequential decision making process scheme.
Based on it, HATRPO and HAPPO algorithms [15] were derived which, thanks to the decomposition theorem and sequential update scheme, established new state-of-the-art methods for MARL.
However, their limitation is that the agents’ policies are unaware of the purpose to develop cooperation and still rely on a carefully handcrafted maximization objective.
Ideally, a team of agents should be aware of the jointness of their training by design, thereby following a holistic and effective paradigm—an ideal solution that is yet to be proposed.
In recent years, sequence models (SM) have made a substantial progress in natural language processing (NLP) [22].
For example, GPT series [3] and BERT models [8], built on autoregressive
SMs, have demonstrated remarkable performance on a wide range of downstream tasks and achieved strong performance on few-shot generalization tasks.
Although SM are mostly used in language tasks due to its natural fitting with the sequential property of languages, the sequential approaches are not confined to NLP only, but is instead a widely applicable general foundation model [2].
For example, in computer vision (CV), one can split an image into sub-images and align them in a sequence as if they were tokens in NLP tasks [8, 9, 11].
Although the idea of solving CV tasks by
SM is straightforward, it serves as the foundation to some of the best-performing CV algorithms [33, 36, 34].
Furthermore, sequential methods are starting to spawn powerful multi-modal visual language models such as Flamingo [1], DALL-E [23], and GATO [25] in the recent past.
Coming with effective and expressive network architectures such as Transformer [35], sequence modeling techniques have also attracted tremendous attention from the RL community, which results in a series of successful offline RL developments based on the Transformer architecture [5, 13, 25].
These methods show great potentials in tackling some of the most fundamental RL training problems, such as long-horizon credit assignment and reward sparsity [32].
For example, by training autoregressive models on pre-collected offline data in a purely supervised way, Decision
Transformer [5] bypasses the need for computing cumulative rewards through dynamic programming, but rather generates future actions conditioning on the desired returns, past states and actions.
Despite their remarkable successes, none of these methods have been designed to model the most difficult (also unique to MARL) aspect of multi-agent systems—the agents’ interactions.
In fact, if we were to simply endow all agents with a Transformer policy and train them independently, their joint performance still could not be guaranteed to improve [15, Proposition 1].
Therefore, while a myriad of powerful SMs are available, MARL—an area that would greatly benefit from SM—has not truly taken advantage of their performance benefit.
The key research question to ask is then
How can we model MARL problems by sequence models ?
In this paper, we take several steps to provide an affirmative answer to the above research question.
Our goal is to enhance MARL studies with powerful sequential modeling techniques.
To fulfill that, we start by proposing a novel MARL training paradigm which establishes the connection between cooperative MARL problems and sequence modeling problems.
Central to the new paradigm are the multi-agent advantage decomposition theorem and sequential update scheme , which effectively transform multi-agent joint policy optimization into a sequential policy search process.
As a natural outcome of our findings, we introduce Multi-Agent Transformer (MAT), an encoder-decoder architecture that implements generic MARL solutions through SM.
Unlike Decision Transformer [5],
MAT is trained online based on trials and errors in an on-policy fashion; therefore, it does not require collecting demonstrations upfront.
Importantly, the implementation of the multi-agent advantage decomposition theorem ensures MAT to enjoy monotonic performance improvement guarantee during training.
MAT establishes a new state-of-the-art baseline model for cooperative MARL tasks.
We justify such a claim by evaluating MAT on the benchmarks of StarCraftII, Multi-Agent MuJoCo,
Dexterous Hands Manipulation, and Google Research Football; results show that MAT achieves superior performance over strong baselines, such as MAPPO [38], HAPPO [15] and QMIX [24].
Finally, we show that MAT possesses great potentials in task generalizations, which holds regardless of the agent number in new tasks.
Preliminaries
In this section, we first introduce the cooperative MARL problem formulation and the multi-agent advantage decomposition theorem, which serves as the cornerstone of our work.
We then review existing MARL methods that relate to MAT, and finally familiarize the reader with the Transformer.
2.1 Problem Formulation
Cooperative MARL problems are often modeled by Markov games ⟨N , O , A , R, P, γ ⟩ [17].
N = { 1 , . . . , n } is the set of agents, O = (cid:81) ni
=
O i is the product of local observation spaces of the agents, namely the joint observation space, A = (cid:81) ni
=
A i is the product of the agents’ action spaces, namely the joint action space, R :
O × A → [ − R max , R max ] is the joint reward function,
P : O × A × O
→ R is the transition probability function, and γ ∈ [0 , 1) is the discount factor.
At time step t ∈ N , an agent i ∈ N observes an observation o it ∈
O i (2) ( o = ( o 1 , . . .
, o n ) is a “joint” observation) and takes an action a it according to its policy π i , which is the i th component of the agents’ joint policy π .
At each time step, all agents take actions simultaneously based on their observation with no sequential dependencies.
The transition kernel P and the joint policy induce the (improper) marginal observation distribution
ρ π ( · ) ≜ (cid:80) ∞ t [-=
 -]=0 γ t Pr ( o t = o | π ) .
At the end of each time step, the whole team receives a joint reward R ( o t , a t ) and observe o t +1 , whose probability distribution is
P ( ·| o t , a t ) .
Following this process infinitely long, the agents earn a discounted cumulative return of
R γ ≜ (cid:80) ∞ t =
γ t R ( o t , a t ) .
2.2 Multi-Agent Advantage Decomposition Theorem
The agents evaluate the value of actions and observations with Q π ( o , a ) and V π ( o ) , defined as
[Equation]
The jointness of the objective causes difficulties associated with the credit assignment problem— having received a shared reward, individual agents are unable to deduce their own contribution to the team’s success or failure [4].
Indeed, applying traditional RL methods which simply employ the above value functions leads to obstacles in training, such as the growing variance of multi-agent policy gradient (MAPG) estimates [14].
Hence, to tackle these, notions of local value functions [19] and counterfactual baselines [10] have been developed.
In this paper, we work with the most general notions of this kind— the multi-agent observation-value functions [15].
That is, for arbitrary disjoint, ordered subsets of agents i 1:
m = { i 1 , . .
, i m } and j 1: h = { j 1 , . . .
, j h } , for m, h ≤ n , we define the multi-agent observation-value function by
[Equation]
which recovers the original state-action value function in Equation (1) when m = n , and the original observation-value function when m =
( i.e. , when the set i 1: m is empty).
Based on Equation (2), we can further measure the contribution of a chosen subset of agents to the joint return and define the multi-agent advantage function by
[Equation]
The above quantity describes how much better/worse than average the joint action a will be if agents i 1: m take the joint action a i 1: m , once j 1: h have taken a j 1: h .
Again, when h = 0 , the advantage compares the value of a i 1: m to the baseline value function of the whole team.
This value-functional representation of agents’ actions enables studying interactions between them, as well to decompose the joint value function signal, thus helping alleviate the severity of the credit assignment problem [24, 30, 20].
The insights of Equation (3) is accomplished by means of the following theorem.
Theorem 1 (Multi-Agent Advantage Decomposition [14]) .
Let i 1: n be a permutation of agents.
Then, for any joint observation o = o ∈ O and joint action a = a i 1: n ∈ A , the following equation always holds with no further assumption needed,
[Equation]
Importantly, this theorem provides an intuition guiding the choice of incrementally improving actions.
Suppose that agent i 1 picks an action a i 1 with positive advantage, A i 1 π ( o , a i 1 ) > 0 .
Then, imagine that for all j = 2 , . . .
, n , agent i j knows the joint action a i 1: j − 1 of its predecessors.
In this case, it can choose an action a i j for which the advantage
A i j π ( o , a i 1: j − 1 , a i j ) is positive.
Altogether, the theorem assures that the joint action a i 1: n has positive advantage.
Furthermore, notice that the joint action has been chosen in n steps, each of which searched an individual agent’s action space.
Hence, the complexity of this search is additive, (cid:80) ni
=1 |A
i | , in the sizes of the action spaces.
If we were to perform the search directly in the joint action space, we would browse a set of multiplicative size, | A | = (cid:81) ni
=1 |A i | .
Later, we will build upon this insight to design a SM that optimizes joint policies efficiently, agent by agent, without the necessity of considering the joint action space at once.
2.3 Existing Methods in MARL
We now briefly summarize two state-of-the-art MARL algorithms.
Both of them build upon Proximal
Policy Optimization (PPO)
[28]—a RL method famous for its simplicity and its performance stability.
MAPPO [38] is the first, and the most direct, approach for applying PPO in MARL.
It equips all agents with one shared set of parameters and use agents’ aggregated trajectories for the shared policy’s update; at iteration k + 1 , it optimizes the policy parameter θ k +1 by maximizing the clip objective of n
[Equation]
where the clip operator clips the input value (if necessary) so that it stays within the interval [1 − ϵ, 1+ ϵ ] .
However, enforcing parameter sharing is equivalent to putting a constraint θ i = θ j , ∀ i, j ∈ N on the joint policy space, which can lead to an exponentially-worse sub-optimal outcome [15].
This motivates a more principled development of heterogeneous-agent trust-region methods, e.g., HAPPO.
HAPPO [15] is currently one of the SOTA algorithm that fully leverages Theorem (1) to implement multi-agent trust-region learning with monotonic improvement guarantee.
During an update, the agents choose a permutation i 1: n at random, and then following the order in the permutation, every agent i m picks π i m new =
π i m that maximizes the objective of
[Equation]
where r ( π i m ) =
π i m ( a i m | o ) /π i m old ( a i m | o ) .
Note that the expectation is taken over the newly-updated previous agents’ policies, i.e, π i 1: m − 1 new ; this reflects an intuition that, under Theorem (1), the agent i m reacts to its preceding agents i 1: m − 1 .
However, one drawback of HAPPO is that agent’s policies has to follow the sequential update scheme in the permutation, thus it cannot be run in parallel.
2.4 The Transformer Model
Transformer [35] was originally designed for machine translation tasks (e.g., input English, output
French).
It maintains an encoder-decoder structure, where the encoder maps an input sequence of tokens to latent representations and then the decoder generates a sequence of desired outputs in an auto-regressive manner wherein at each step of inference, the Transformer takes all previously generated tokens as the input.
One of the most essential component in Transformer is the scaled dot-product attention, which captures the interrelationship of input sequences.
The attention function is written as Attention ( Q , K , V ) = softmax (cid:0) QK T √ d k (cid:1) V , where the Q , K , V corresponds to the vector of queries, keys and values, which can be learned during training, and the d k represent the dimension of Q and K .
Self-attentions refer to cases when Q , K , V share the same set of parameters.
Inspired by the attention mechanism, UPDeT [12] handles various observation sizes by decoupling each agent’s observations into a sequence of observation-entities, matching them with different actiongroups, and modeling the relationship between the matched observation-entities with a Transformerbased function for better representation learning in MARL problems.
Apart from this, based on the sequential property described in the Theorem (1) and the principle behind HAPPO [15], it is intuitive to think about another Transformer-based implementation for multi-agent trust-region learning.
By treating a team of agents as a sequence , the Transformer architecture allows us to model teams of agents with variable numbers and types, while avoiding drawbacks of MAPPO/HAPPO.
We will describe in more details how a cooperative MARL problem can be solved by a sequence model.
The Surprising Connection Between MARL and Sequence Models
To establish the connection between MARL and sequence models, Theorem (1) provides a new angle of understanding the MARL problem from a SM perspective.
If each agent knows its predecessors’ actions with an arbitrary decision order, the sum of agents’ local advantages A i j π ( o , a i 1: m − 1 , a i m ) will be exactly equal to the joint advantages
A i 1: n π ( o , a i 1: n ) .
This orderly decision setting across agents simplifies the update of their joint policy, where maximizing each agent’s own local advantage is equivalent to maximizing the joint advantage.
As such, agents do not need to worry about interference from other agents anymore during the policy update; the local advantage functions have already captured the relationship between agents.
This property revealed by Theorem (1) inspires us to propose a multi-agent sequential decision paradigm for MARL problems as show in Figure (1), where we assign agents with an arbitrary decision order (one permutation for each iteration); each agent can access its predecessors’ behaviors, based on which it then takes the optimal decision.
This sequential paradigm motivates us to leverage a sequential model, e.g., Transformer, to explicitly capture the sequential relationship between agents described in Theorem (1).
Underpinned by Theorem (1), sequence modeling reduces the complexity growth of MARL problems with the number of agents from multiplicative to additive, thus rendering linear complexity.
With the help of the Transformer architecture, we can model policies of heterogeneous agents with an unified network but treat each agent discriminatively with different position, and thus ensuring high sample efficiency while avoiding the exponentially-worse outcome that MAPPO is facing.
Besides, in order to guarantee the monotonic improvement of joint policies, HAPPO has to update each policy one-by-one during training, by leveraging previous update results of π i 1 , ..., π i m − 1 to improve π i m , which becomes critical in computational efficiency at large size of agents.
By contrast, the attention mechanism of Transformer architectures allows for batching the ground truth actions a i 0 t , ..., a i n − 1 t in the buffer to predict a i 1 t , ..., a i n t and update policies simultaneously, which significantly improves the training speed and makes it feasible for large size of agents.
Furthermore, in cases that the number and the type of agents are different, SM can incorporates them into an unified solution through its capability on modeling sequences with flexible sequence length, rather than treat different agent numbers as different tasks.
To realize the above idea, we introduce a practical architecture named
Multi-Agent Transformer in the next section.
The Multi-Agent Transformer
To implement the sequence modeling paradigm for MARL, our solution is Multi-Agent Transformer (MAT).
The idea of applying the Transformer architecture comes from the fact that the mapping between the input of agents’ observation sequence ( o i 1 , . . .
, o i n ) and the output of agents’ action sequence ( a i 1 , . .
. , a i n ) are sequence modeling tasks similar to machine translations.
As eluded by
Theorem (1), the action a i m depends on all previous agents’ decisions a i 1: m − 1 .
Hence, our MAT in Figure (2) consists of an encoder , which learns representations of the joint observations, and a decoder which outputs actions for each individual agent in an auto-regressive manner.
The encoder , whose parameters we denote by ϕ , takes a sequence of observations ( o i 1 , . .
, o i n ) in arbitrary order and passes them through several computational blocks.
Each such block consists of a self-attention mechanism and a multi-layer perceptron (MLP), as well as residual connections to prevent gradient vanishing and network degradation with the increase of depth.
We denote the output encoding of the observations as (ˆ o i 1 , . . .
, ˆ o i n ) , which encodes not only the information of agents ( i 1 , . .
, i n ) but also the high-level interrelationships that represent agents’ interactions.
In order to learn expressive representations, in the training phase, we make the encoder to approximate the value functions, whose objective is to minimize the empirical Bellman error by
[Equation]
where ¯ ϕ is the target network’s parameter, which is non-differentiable and updated every few epochs.
The decoder , whose parameters we denote by θ , passes the embedded joint action a i 0: m − 1 , m = { 1 , . . .
n } (where a i 0 is an arbitrary symbol indicating the start of decoding) to a sequence of decoding blocks.
Crucially, every decoding block comes with a masked self-attention mechanism, where the masking makes sure that, for every i j , attention is computed only between the i th r and the i th j action heads wherein r < j so that the sequential update scheme can be maintained.
This is then followed [Figure] [Figure] by a second masked attention function, which computes the attention between the action heads and observation representations.
Finally, the block finishes with an MLP and skipping connections.
The output to the last decoder block is a sequence of representations of the joint actions, { ˆ a i 0 : i − 1 } mi =1 .
This is fed to an MLP that outputs the probability distribution of i m ’s action, namely, the policy π i m θ ( a i m | ˆ o i 1: n , a i 1: m − 1 ) .
To train the decoder [Figure] we minimize the following clipping PPO objective of
[Equation]
where ˆ A t is an estimate of the joint advantage function.
One can apply generalized advantage estimation (GAE) [27] with ˆ V t = 1 n (cid:80) nm =1 V (ˆ o i m t ) as a robust estimator for the joint value function.
Notably, the action generation process is different between the inference and the training stage.
In the inference stage, each action is generated auto-regressively, in the sense that a i m will be inserted back into the decoder again to generate a i m +1 (starting with a i 0 and ending with a i n − 1 ).
While during the training stage, the output of all actions, a i 1: n can be computed in parallel simply because a i 1: n −have already been collected and stored in the replay buffer.
The attention mechanism, which lies in the heart of MAT, encodes observations and actions with a weight matrix calculated by multiplying the embedded queries, ( q i 1 , . . .
, q i n ) , and keys, ( k i 1 , . .
, k i n ) , where each of the weight w ( q i r , k i j ) =
⟨ q i r , k i j ⟩ .
The embedded values ( v i 1 , . .
. , v i n ) are multiplied with the weight matrix to output representations.
While the unmasked attention in the encoder uses a full weight matrix to extract the interrelationship between agents, i.e., ˆ o i 1: n , the masked attentions in the decoder capture a i 1: m with triangular matrices where w ( q i r , k i j ) = 0 for r <
j (see an visual illustration in Appendix A).
With the properly masked attention mechanism, the decoder can safely output the policy π i m +1 [-✓ ( a i m +
-]θ
[Equation] 1: m ) , which finishes the implementation of Theorem (1).
The monotonic improvement guarantee.
An MAT agent i m optimizes a trust-region objective that is conditioned on new decisions of agents i 1: m − 1 by means of conditioning its policy ratio on them (see Equation (5)).
As such, it increases the joint return monotonically like if it followed the sequential update scheme of HAPPO [15, Theorem 2].
However, as oppose to that method, the MAT model does not require i m to wait until its predecessors make their updates, nor it uses their updated action distribution for importance sampling calculations.
In fact, as actions of all agents are outputs of MAT, their clipping objectives can be computed in parallel (during training), thus dominating [Table]
86.9 (5.6) 100.0 (2.7) 0.6 (0.8) 90.2 (9.8) 2e 27m vs 30m
Super Hard 100.0 (0.7) 95.3 (2.2) 93.1 (3.2) 0.0 (0.0) 39.2 (8.8) 1e
MMM Super Hard 93.8 (2.6) 91.2 (5.3) 81.8 (10.1) 0.3 (0.4) 88.3 (2.4) 1e 6h vs 8z Super Hard 98.8 (1.3) 93.8 (4.7) 88.4 (5.7) 0.0 (0.0) 9.7 (3.1)
[Figure]
vs 3s6z
Super Hard
[Figure]
[Figure]
) 84.3 (19.4) 82.8 (21.2) 68.8 (21.2) 2e
[Figure] to right respectively. HAPPO on the time complexity. Lastly, to assure that the limiting joint policy is such that none of the agents is incentivized to change its policy (Nash equilibrium), MAT requires permutating the sequential order of updates at every iteration, which is inline with the discovery in HAPPO [15,
Theorem 3].
Experiments and Results
MAT provides a new solution paradigm for cooperative MARL problems.
The key insights of MAT are the sequential update scheme, which is inspired by Theorem (1), as well as the encoder-decoder architecture, which provides a highly-efficient implementation for a sequence modeling perspective.
Importantly, MAT inherits the monotonic improvement guarantee, and agents’ policies can be learned in parallel during training.
We firmly believe MAT will become a game changer for MARL studies.
To evaluate if MAT meets our expectations, we test MAT on the StarCraftII Multi-Agent Challenge (SMAC) benchmark [26] where MAPPO with parameter sharing [38] has shown superior performance, and the Multi-Agent MuJoCo benchmark [7] where HAPPO [15] shows the current state-of-the-art performance.
SMAC and MuJoCo environments are common benchmarks in the MARL field.
On top of them, we also test MAT on the Bimanual Dexterous Hands Manipulation (Bi-DexHands) [6] which provides a list of challenging bimanual manipulation tasks (see Figure (3)), and the Google
Research Football [16] benchmark with a series of cooperation scenarios in football game.
We apply the same hyper-parameters of baseline algorithms from their original paper to ensure their best performance, and adopt the same hyper-parameter tuning process for our methods with details in Appendix B. To ensure fair comparisons to CTDE methods, we also introduce a CTDE-variant of MAT called MAT-Dec , which essentially adopts a fully decentralized actor for each individual agent (rather than using the decoder proposed in MAT) while keeping the encoder fixed.
The critic’s loss for MAT-Dec is L ( ϕ )
= 1 T (cid:80) T − 1 t =0 (cid:2) R ( o t , a t )
+ γ 1 n (cid:80) nm =1 V ¯ ϕ (ˆ o i m t +1 ) − 1 n (cid:80) nm =1 V ϕ
(ˆ o i m t ) (cid:3) 2 , and we apply the local advantage estimation
A t (ˆ o i m t , a i m ) to guide the subsequent policy update. [Figure]
5.1 Performance on Cooperative MARL Benchmarks
According to Table (1) and Figure (4) [Figure] MAPPO significantly outperforms HAPPO in SMAC with higher sample efficiency.
This verified the homogeneity of SMAC agents and the heterogeneity of multi-agent MuJoCo agents, that are also discovered by Kuba et al.
Take the SMAC task 25m as an example, all the marines are equivalent and interchangeable so that agents can learn from their teammate’s experience.
Sharing parameters in this settings means leveraging 25 times more examples to train each agents comparing with separated network of HAPPO, and thus enjoying higher learning efficiency.
On the other hand, with the heterogeneous settings of multi-agent MuJoCo, training a "foot" agent with experience from a "thigh" agent can surely harm its performance since they represent different functions on the Cheetah.
However, MAT outperforms MAPPO and HAPPO in almost all tasks in Table (1) and Figure (4), indicating its modeling capability on both homogeneous and heterogeneous-agent tasks.
MAT also enjoys the superior performance over MAT-Dec, which emphasize the importance of the decoder architecture in the MAT design.
On the Bi-DexHands tasks,
MAT outperforms MAPPO and HAPPO methods by a large margin.
We save the Google Football results to Figure (5), where the conclusion stays the same.
5.2 MAT as Excellent Few-short Learners
Since Transformer-based models often demonstrate strong generalization performance on few-short tasks [3, 8], we believe MAT can possess strong generalization ability on unseen MARL tasks as well.
To validate such an assumption, we design zero-shot and few-shot experiments on SMAC and multi-agent MuJoCo tasks.
For SMAC tasks, we pre-train agents on eight tasks involving five types of units ( 3m , 8m vs 9m , 10m vs 11m , 25m , 3s vs 3z , 2s3z , 3s5z , MMM ) with 10M examples in total and then apply them on six separate and much harder tasks ( 5m vs 6m , 8m , 27m vs 30m , 2s vs 1sc , 1c3s5z , MMM2 ) including seven types of units.
This setting is designed to evaluate the generalization ability of MAT when training on simple tasks but transferring to more diverse and complex downstream tasks.
In terms of multi-agent MuJoCo, we reuse the models trained on the complete HalfCheetah robot as the pre-trained agent and then directly apply it to six new tasks, each with a different leg being disfunctioned (see Figure (3a)).
We investigate the generalization capability of pre-trained models on each downstream task with 0% (zero-shot), 1%, 5%, 10% few-short new examples, respectively.
Note that common MARL baselines such as HAPPO assume fixed number of agents during training, thus it cannot directly handle the cases with changing number of agents.
We summarize the zero-shot and few-shot results of each algorithm in Table (2) and (3), where the bold number indicates the best performance.
We also provide the performance of MAT if it was given the same amount of data but is trained from scratch, the "MAT-from scratch", as the control group to demonstrate the effectiveness of pre-training process.
As both tables suggest, bold numbers [Table] [Table] are mainly located in the area of MAT, which justify MAT’s strong generalisation performance as a few-short learner.
Surprisingly, we find that the few-shot MAT with only 10% data show even higher rewards than its counterpart that is purely trained on HalfCheetah with the same disabled joints ( back foot , back shin and back thigh ) and 100% full amount data, we believe it is because the pre-train process offers initial weights that are not only closer to optimum but also less likely to stuck in bad local optima than random initialization.
Conclusion
In the past five years, large sequence models have achieved remarkable successes on solving visual language tasks.
In this paper, we take the initial effort to build the connection between multi-agent reinforcement learning (MARL) problems and generic sequence models (SM), with the ambition that
MARL researchers can hereafter benefit from the prosperous development on the sequence modeling side.
Specifically, we contribute by unifying a general solution to cooperative MARL problems into a
Transformer like encoder-decoder model.
The proposed Multi-Agent Transformer (MAT) leverages the multi-agent advantage decomposition theorem, which essentially transforms the joint policy optimization process into a sequential decision making process that can be simply implemented by an auto-regressive model.
We have demonstrated MAT’s strong empirical performance on three challenging benchmarks against current state-of-the-art MARL solutions including MAPPO and
HAPPO.
Based on the established connection between MARL and SM, in the future, we plan to bring multi-agent learning tasks into large multi-modal SM, chasing for more generally intelligent models as the most recent success of GATO has already demonstrated [25]. Acknowledgment
The SJTU team is partially supported by “New Generation of AI 2030”
Major Project (2018AAA0100900), Shanghai Municipal Science and
Technology
Major
Project (2021SHZDZX0102), Shanghai Sailing Program (21YF1421900), and National Natural Science Foundation of China (62076161, 62106141).
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[No] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Appendix B (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Section(d)
Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See Section(b)
Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Appendices A Algorithm Details A.1 Pseudo Code of Multi-Agent Transformer.
Algorithm do 4: do 5: Collect Phase} 6: Generate feeding
Input do 9:
10: end
Execute joint actions a i t
. . . , a i t in environments and collect the reward R a
Insert ( o t , a t , R ( o t , a t
) in to B
13: end for {
Sample a random minibatch
B
Generate
Calculate L
Equation
Compute the advantage function
A based
V ϕ
(ˆ o i 1 ) , .
. , V ϕ
(ˆ o i n ) with
Input
Calculate L Decoder ( θ ) with Equation (5).
Update the encoder and decoder by minimising L Encoder ( ϕ )+ L Decoder ( θ ) with gradient descent.
21: end for
A.2 Dynamic Process and Source Code of MAT
Please refer to https://sites.google.com/view/multi-agent-transformer
B Hyper-parameter Settings for Experiments
During experiments, the implementations of baseline methods are consistent with their official repositories, all hyper-parameters left unchanged at the origin best-performing status.
The hyperparameters adopted for different algorithms and tasks are listed in Table 4-12.
In particular, the ppo epochs and ppo clip across different SMAC scenarios are unified to 10 and 0 .
05 respectively for pre-training and fine-tuning in few-shot experiments.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
C Details of Experimental Results
[Figure]
[Figure]
[Table]
[Table]
[Figure]
2c
[Figure]
w i n r a t e
[Figure]
capability for homogeneous-agent tasks (agents are interchangeable)
[Figure]
[Figure]
[Figure]
[Figure]
n r a t
[Figure]
[Figure]
D Ablation studies
In this section, we conduct ablation studies to investigate the importance of different components.
Since agents are shuffled at every iteration, the position encoding in vanilla Transformer might not be very relevant in MARL tasks, and has been already discarded in our implementation.
Instead, we bind each observation with corresponding one-hot agent id and build an ablation experiment to investigate their effect.
The results in Figure (14) confirmed our choice, where the agent id encoding significantly outperforms position encoding with regard to episodic rewards and stability.
Further, applying position encoding on top of agent id encoding can not enjoy extra performance boost.
[Figure]
Besides
[Figure]
we also compared the implementation with different model architectures,
i
[Figure]
e. encoderdecoder, encoder only, decoder only
[Figure]
and other sequence models (GRU) to verify the necessity of different architecture components.
We build this ablation on both the homogeneous and heterogeneous scenarios as shown in Figure (15)
[Figure]
where the complete Transformer architecture achieves the best performance, emphasizing the advantage of the Transformer and the necessity of encoder-decoder architectures.
[Figure]
[Figure]