Multi-Agent Reinforcement Learning is A Sequence Modeling Problem
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Pre-trained sequence models (SM) such as GPT series and BERT have displayedoutstanding performance and generalization capabilities on natural language process, vision and recently single-agent reinforcement learning. A natural follow-upquestion is how to abstract multi-agent decision making also as an SM problemand beneﬁt from the prosperous development of the sequence models.
In this paper,we introduce a novel architecture named Multi-Agent Transformer (MAT) thateffectively casts cooperative multi-agent reinforcement learning (MARL) into SMproblems wherein the objective is to map agents’ observation sequence to agents’optimal action sequence.
Our goal is to build the bridge between MARL and SM so that the modeling power of modern sequence models can be unleashed for MARL.
Central to our MAT is an encoder-decoder architecture which leverages themulti-agent advantage decomposition theorem to transform the joint policy searchproblem into a sequential decision making process; this renders only linear timecomplexity for multi-agent problems and, most importantly, endows MAT withmonotonic performance improvement guarantee.
Unlike prior arts such as Decision Transformer ﬁt only pre-collected ofﬂine data, MAT is trained by online trial anderror from the environment in an on-policy fashion.
To validate MAT, we conductextensive experiments on StarCraftII, Multi-Agent MuJoCo, and Dexterous Hands Manipulation benchmarks.
Results demonstrate that MAT achieves superior performance and data efﬁciency compared to strong baselines including MAPPO and HAPPO.
Furthermore, we demonstrate that MAT is an excellent few-short learneron unseen tasks regardless of changes in the number of agents.
Introduction
Multi-agent reinforcement learning (MARL) [37] is a challenging problem for its difﬁculty whicharises not only from identifying each individual agent’s policy improvement direction but also fromcombining agents’ policy updates jointly which should be beneﬁcial for the whole team.
Recently,such difﬁculty in multi-agent learning has been eased owing to the introduction of centralized trainingfor decentralized execution (CTDE) [10], which allows agents to access the global informationand opponents’ actions during the training phase.
This framework enables successful developments of methods that directly inherit single-agent algorithms. For examples, COMA replaces thepolicy-gradient (PG) estimate with a multi-agent PG (MAPG) counterpart [10], MADDPG extendsdeterministic policy gradient into multi-agent settings with a centralized critic [18, 29], QMIX leverages deep Q networks for decentralized agents and introduces a centralized mixing network for Qvalue decomposition [24, 31, 21], and MAPPO endowing all agents with the same set of parametersand then training by trust-region methods [38].
These methods, however, cannot cover the wholecomplexity of multi-agent interactions; in fact, some of them are shown to fail in the simplest cooperative task [15].
To resolve this issue, the multi-agent advantage decomposition theorem was proposed [15, Theorem 1] which captures how different agents contribute to the return and provides an intuitionbehind the emergence of cooperation through a sequential decision making process scheme.
Based onit, HATRPO and HAPPO algorithms [15] were derived which, thanks to the decomposition theoremand sequential update scheme, established new state-of-the-art methods for MARL.
However, theirlimitation is that the agents’ policies are unaware of the purpose to develop cooperation and still relyon a carefully handcrafted maximization objective.
Ideally, a team of agents should be aware of thejointness of their training by design, thereby following a holistic and effective paradigm—an idealsolution that is yet to be proposed.
In recent years, sequence models (SM) have made a substantial progress in natural language processing (NLP) [22].
For example, GPT series [3] and BERT models [8], built on autoregressive
SMs, have demonstrated remarkable performance on a wide range of downstream tasks and achievedstrong performance on few-shot generalization tasks.
Although SM are mostly used in languagetasks due to its natural ﬁtting with the sequential property of languages, the sequential approachesare not conﬁned to NLP only, but is instead a widely applicable general foundation model [2].
Forexample, in computer vision (CV), one can split an image into sub-images and align them in asequence as if they were tokens in NLP tasks [8, 9, 11].
Although the idea of solving CV tasks by
SM is straightforward, it serves as the foundation to some of the best-performing CV algorithms[33, 36, 34].
Furthermore, sequential methods are starting to spawn powerful multi-modal visuallanguage models such as Flamingo [1], DALL-E [23], and GATO [25] in the recent past.
Coming with effective and expressive network architectures such as Transformer [35], sequencemodeling techniques have also attracted tremendous attention from the RL community, whichresults in a series of successful ofﬂine RL developments based on the Transformer architecture[5, 13, 25].
These methods show great potentials in tackling some of the most fundamental RLtraining problems, such as long-horizon credit assignment and reward sparsity [32].
For example,by training autoregressive models on pre-collected ofﬂine data in a purely supervised way, Decision
Transformer [5] bypasses the need for computing cumulative rewards through dynamic programming,but rather generates future actions conditioning on the desired returns, past states and actions.
Despitetheir remarkable successes, none of these methods have been designed to model the most difﬁcult(also unique to MARL) aspect of multi-agent systems—the agents’ interactions.
In fact, if we wereto simply endow all agents with a Transformer policy and train them independently, their jointperformance still could not be guaranteed to improve [15, Proposition 1].
Therefore, while a myriadof powerful SMs are available, MARL—an area that would greatly beneﬁt from SM—has not trulytaken advantage of their performance beneﬁt.
The key research question to ask is then
How can we model MARL problems by sequence models ?
In this paper, we take several steps to provide an afﬁrmative answer to the above research question.
Our goal is to enhance MARL studies with powerful sequential modeling techniques.
To fulﬁll that,we start by proposing a novel MARL training paradigm which establishes the connection betweencooperative MARL problems and sequence modeling problems.
Central to the new paradigm arethe multi-agent advantage decomposition theorem and sequential update scheme , which effectivelytransform multi-agent joint policy optimization into a sequential policy search process.
As a natural outcome of our ﬁndings, we introduce Multi-Agent Transformer (MAT), an encoder-decoderarchitecture that implements generic MARL solutions through SM.
Unlike Decision Transformer [5],
MAT is trained online based on trials and errors in an on-policy fashion; therefore, it does not requirecollecting demonstrations upfront.
Importantly, the implementation of the multi-agent advantagedecomposition theorem ensures MAT to enjoy monotonic performance improvement guarantee duringtraining.
MAT establishes a new state-of-the-art baseline model for cooperative MARL tasks.
Wejustify such a claim by evaluating MAT on the benchmarks of StarCraftII, Multi-Agent MuJoCo, and
Dexterous Hands Manipulation; results show that MAT achieves superior performance over strongbaselines, such as MAPPO [38], HAPPO [15] and QMIX [24].
Finally, we show that MAT possessesgreat potentials in task generalizations, which holds regardless of the agent number in new tasks.
Preliminaries
In this section, we ﬁrst introduce the cooperative MARL problem formulation and the multi-agentadvantage decomposition theorem, which serves as the cornerstone of our work.
We then reviewexisting MARL methods that relate to MAT, and ﬁnally familiarize the reader with the Transformer.
2.1 Problem Formulation
Cooperative MARL problems are often modeled by Markov games h N , O , A , R, P,  i [17].
N ={ 1 , . . . , n } is the set of agents, O =
Q ni =
O i is the product of local observation spaces of theagents, namely the joint observation space, A =
Q ni =
A i is the product of the agents’ actionspaces, namely the joint action space, R : O ⇥ A !
[  R max , R max ] is the joint reward function, 95 P :
O ⇥ A ⇥ O !
R is the transition probability function, and  2 [0 , 1) is the discount factor.
At 96 time step t 2 N , an agent i 2 N observes an observation o it
2 O i (1) ( o = ( o 1 , . . .
, o n ) is a “joint”observation) and takes an action a it according to its policy ⇡ i , which is the i th component of the agents’joint policy ⇡ .
At each time step, all agents take actions simultaneously based on their observationwith no sequential dependencies.
The transition kernel P and the joint policy induce the (improper)marginal observation distribution ⇢ ⇡ ( · ) ,
P 1 t =
t Pr ( o t = o | ⇡ ) .
At the end of each time step, thewhole team receives a joint reward R ( o t , a t ) and observe o t +1 , whose probability distribution is P ( ·| o t , a t ) .
Following this process inﬁnitely long, the agents earn a discounted cumulative return of R  ,
P 1 t =
t R ( o t , a t ) .
2.2 Multi-Agent Advantage Decomposition Theorem
The agents evaluate the value of actions and observations with Q ⇡ ( o , a ) and V ⇡ ( o ) , deﬁned as
[Equation]
⇥ [Equation] The jointness of the objective causes difﬁculties associated with the credit assignment problem—having received a shared reward, individual agents are unable to deduce their own contribution tothe team’s success or failure [4].
Indeed, applying traditional RL methods which simply employ theabove value functions leads to obstacles in training, such as the growing variance of multi-agentpolicy gradient (MAPG) estimates [14].
Hence, to tackle these, notions of local value functions [19]and counterfactual baselines [10] have been developed.
In this paper, we work with the most generalnotions of this kind— the multi-agent observation-value functions [15].
That is, for arbitrary disjoint,ordered subsets of agents i 1:
m = { i 1 , . .
, i m } and j 1: h = { j 1 , . . .
, j h } , for m, h  n , we deﬁne the 114 multi-agent observation-value function by
[Equation]
which recovers the original state-action value function in Equation (1) when m = n , and the originalobservation-value function when m =
( i.e. , when the set i 1: m is empty).
Based on Equation (2),we can further measure the contribution of a chosen subset of agents to the joint return and deﬁne themulti-agent advantage function by
[Equation]
The above quantity describes how much better/worse than average the joint action a will be if agentsi 1: m take the joint action a i 1: m , once j 1: h have taken a j 1: h .
Again, when h = 0 , the advantagecompares the value of a i 1: m to the baseline value function of the whole team.
This value-functionalrepresentation of agents’ actions enables studying interactions between them, as well to decomposethe joint value function signal, thus helping alleviate the severity of the credit assignment problem[24, 30, 20].
The insights of Equation (3) is accomplished by means of the following theorem.
Theorem 1 (Multi-Agent Advantage Decomposition [14]) .
Let i 1: n be a permutation of agents.
Then,for any joint observation o =
o 2 O and joint action
a = a i 1: n 2 A , the following equation alwaysholds with no further assumption needed,
[Equation]
Importantly, this theorem provides an intuition guiding the choice of incrementally improving actions.
Suppose that agent i 1 picks an action a i 1 with positive advantage, A i 1 ⇡ ( o , a i 1 ) > 0 .
Then, imagine that for all j = 2 , . . .
, n , agent i j knows the joint action a i 1: j  1 of its predecessors.
In this case, itcan choose an action a i j for which the advantage
A i j ⇡ ( o , a i 1: j  1 , a i j ) is positive.
Altogether, thetheorem assures that the joint action a i 1: n has positive advantage.
Furthermore, notice that the jointaction has been chosen in n steps, each of which searched an individual agent’s action space.
Hence,the complexity of this search is additive,
P ni =1 |A
i | , in the sizes of the action spaces.
If we were to 135 perform the search directly in the joint action space, we would browse a set of multiplicative size, [Equation]
ni =1 |A
i | .
Later, we will build upon this insight to design a SM that optimizes joint policies 137 efﬁciently, agent by agent, without the necessity of considering the joint action space at once.
2.3 Existing Methods in MARL
We now brieﬂy summarize two state-of-the-art MARL algorithms.
Both of them build upon Proximal
Policy Optimization (PPO)
[28]—a RL method famous for its simplicity and its performance stability.
MAPPO [38] is the ﬁrst, and the most direct, approach for applying PPO in MARL.
It equips allagents with one shared set of parameters and use agents’ aggregated trajectories for the shared policy’supdate; at iteration k + 1 , it optimizes the policy parameter ✓ k +1 by maximizing the clip objective of n
[Equation]
where the clip operator clips the input value (if necessary) so that it stays within the interval [1  ✏ , 1+ ✏ ] .
However, enforcing parameter sharing is equivalent to putting a constraint ✓ i = ✓ j , 8 i, j 2 N onthe joint policy space, which can lead to an exponentially-worse sub-optimal outcome [15].
Thismotivates a more principled development of heterogeneous-agent trust-region methods, e.g., HAPPO.
HAPPO [15] is currently one of the SOTA algorithm that fully leverages Theorem (1) to implementmulti-agent trust-region learning with monotonic improvement guarantee.
During an update, theagents choose a permutation i 1: n at random, and then following the order in the permutation, everyagent i m picks ⇡ i m new =
⇡ i m that maximizes the objective of
[Equation]
i , where r ( ⇡ i m ) =
⇡ i m ( a i m | o ) / ⇡ i m old ( a i m | o ) .
Note that the expectation is taken over the newly-updatedprevious agents’ policies, i.e, ⇡ i 1: m  1 new ; this reﬂects an intuition that, under Theorem (1), the agent i mreacts to its preceding agents i 1: m  1 .
However, one drawback of HAPPO is that agent’s policies
hasto follow the sequential update scheme in the permutation, thus it cannot be run in parallel.
2.4 The Transformer Model
Transformer [35] was originally designed for machine translation tasks (e.g., input English, output
French).
It maintains an encoder-decoder structure, where the encoder maps an input sequence oftokens to latent representations and then the decoder generates a sequence of desired outputs inan auto-regressive manner wherein at each step of inference, the Transformer takes all previouslygenerated tokens as the input.
One of the most essential component in Transformer is the scaleddot-product attention, which captures the interrelationship of input sequences.
The attention functionis written as Attention ( Q , K , V ) =
softmax
QK T p d k
V , where the Q , K , V corresponds to the vectorof queries, keys and values, which can be learned during training, and the d k represent the dimensionof Q and K .
Self-attentions refer to cases when Q , K , V share the same set of parameters.
Inspired by the attention mechanism, UPDeT [12] handles various observation sizes by decouplingeach agent’s observations into a sequence of observation-entities, matching them with different actiongroups, and modeling the relationship between the matched observation-entities with a Transformerbased function for better representation learning in MARL problems.
Apart from this, based on thesequential property described in the Theorem (1) and the principle behind HAPPO [15], it is intuitiveto think about another Transformer-based implementation for multi-agent trust-region learning.
Bytreating a team of agents as a sequence , the Transformer architecture allows us to model teamsof agents with variable numbers and types, while avoiding drawbacks of MAPPO/HAPPO.
We willdescribe in more details how a cooperative MARL problem can be solved by a sequence model.
The Surprising Connection Between MARL and Sequence Models
To establish the connection between MARL and sequence models, Theorem (1) provides a new angleof understanding the MARL problem from a SM perspective.
If each agent knows its predecessors’actions with an arbitrary decision order, the sum of agents’ local advantages A i j ⇡ ( o , a i 1: m  1 , a i m ) willbe exactly equal to the joint advantages
A i 1: n ⇡ ( o , a i 1: n ) .
This orderly decision setting across agentssimpliﬁes the update of their joint policy, where maximizing each agent’s own local advantage isequivalent to maximizing the joint advantage.
As such, agents do not need to worry about interferencefrom other agents anymore during the policy update; the local advantage functions have alreadycaptured the relationship between agents.
This property revealed by Theorem (1) inspires us topropose a multi-agent sequential decision paradigm for MARL problems as show in Figure (1), wherewe assign agents with an arbitrary decision order (one permutation for each iteration); each agent canaccess its predecessors’ behaviors, based on which it then takes the optimal decision.
This sequentialparadigm motivates us to leverage a sequential model, e.g., Transformer, to explicitly capture thesequential relationship between agents described in Theorem (1).
Underpinned by Theorem (1), sequence modeling reduces the complexity growth of MARL problemswith the number of agents from multiplicative to additive, thus rendering linear complexity.
Withthe help of the Transformer architecture, we can model policies of heterogeneous agents with anuniﬁed network but treat each agent discriminatively with different position, and thus ensuring highsample efﬁciency while avoiding the exponentially-worse outcome that MAPPO is facing.
Besides,in order to guarantee the monotonic improvement of joint policies, HAPPO has to update each policyone-by-one during training, by leveraging previous update results of ⇡ i 1 , ..., ⇡ i m  1 to improve ⇡ i m ,which becomes critical in computational efﬁciency at large size of agents.
By contrast, the attentionmechanism of Transformer architectures allows for batching the ground truth actions a i 0 t , ..., a i n  1 t inthe buffer to predict a i 1 t , ..., a i n t and update policies simultaneously, which signiﬁcantly improves thetraining speed and makes it feasible for large size of agents.
Furthermore, in cases that the numberand the type of agents are different, SM can incorporates them into an uniﬁed solution through itscapability on modeling sequences with ﬂexible sequence length, rather than treat different agentnumbers as different tasks.
To realize the above idea, we introduce a practical architecture named Multi-Agent Transformer in the next section.
The Multi-Agent Transformer
To implement the sequence modeling paradigm for MARL, our solution is Multi-Agent Transformer(MAT).
The idea of applying the Transformer architecture comes from the fact that the mappingbetween the input of agents’ observation sequence ( o i 1 , . . .
, o i n ) and the output of agents’ actionsequence ( a i 1 , . .
. , a i n ) are sequence modeling tasks similar to machine translations.
As eluded by
Theorem (1), the action a i m depends on all previous agents’ decisions a i 1: m  1 .
Hence, our MATin Figure (2) consists of an encoder , which learns representations of the joint observations, and adecoder which outputs actions for each individual agent in an auto-regressive manner.
The encoder , whose parameters we denote by  , takes a sequence of observations ( o i 1 , . .
, o i n ) inarbitrary order and passes them through several computational blocks.
Each such block consists of aself-attention mechanism and a multi-layer perceptron (MLP), as well as residual connections toprevent gradient vanishing and network degradation with the increase of depth.
We denote the outputencoding of the observations as (ˆ o i 1 , . . .
, ˆ o i n ) , which encodes not only the information of agents( i 1 , . .
, i n ) but also the high-level interrelationships that represent agents’ interactions.
In order tolearn expressive representations, in the training phase, we make the encoder to approximate the valuefunctions, whose objective is to minimize the empirical Bellman error by
[Equation]
where ¯  is the target network’s parameter, which is non-differentiable and updated every few epochs.
The decoder , whose parameters we denote by ✓ , passes the embedded joint action a i 0: m  1 , m ={ 1 , . . .
n } (where a i 0 is an arbitrary symbol indicating the start of decoding) to a sequence of decodingblocks.
Crucially, every decoding block comes with a masked self-attention mechanism, where themasking makes sure that, for every i j , attention is computed only between the i th r and the i th j actionheads wherein r <
j so that the sequential update scheme can be maintained.
This is then followedby a second masked attention function, which computes the attention between the action heads andobservation representations.
Finally, the block ﬁnishes with an MLP and skipping connections.
Theoutput to the last decoder block is a sequence of representations of the joint actions, { ˆ a i 0 : i  1 } mi =1 .
This is fed to an MLP that outputs the probability distribution of i m ’s action, namely, the policy⇡ i m ✓ ( a i m | ˆ o i 1: n , a i 1: m  1 ) .
To train the decoder, we minimize the following clipping PPO objective of
[Equation]
where r i m t ( ✓ )
= ⇡ i m ✓ ( a i m t | ˆ o i 1: n t , ˆ a i 1: m  1 t ) /
⇡ i m ✓ old ( a i m t | ˆ
o i 1: n t , ˆ a i 1: m  1 t ) and
ˆ A t is an estimate of the jointadvantage function.
One can apply generalized advantage estimation (GAE) [27] with ˆ V t =1 n
P n m =1 V (ˆ o i m t ) as a robust estimator for the joint value function.
Notably, the action generation process is different between the inference and the training stage.
In the inference stage, eachaction is generated auto-regressively, in the sense that a i m will be inserted back into the decoderagain to generate a i m +1 (starting with a i 0 and ending with a i n  1 ).
While during the training stage,the output of all actions, a i 1: n can be computed in parallel simply because a i 1: n  1 have already beencollected and stored in the replay buffer.
[Figure]
The attention mechanism, which lies in the heart of MAT, encodes observations and actionswith a weight matrix calculated by multiplying the embedded queries, ( q i 1 , . . .
, q i n ) , and keys,( k i 1 , . .
, k i n ) , where each of the weight w ( q i r , k i j ) =
h q i r , k i j i .
The embedded values( v i 1 , . .
. , v i n ) are multiplied with the weight matrix to output representations.
While the unmasked attention in the encoder uses a full weight matrix to extract the interrelationship betweenagents, i.e., ˆ o i 1: n , the masked attentions in the decoder capture a i 1: m with triangular matrices wherew ( q i r , k i j ) = 0 for r <
j (see an visual illustration in Appendix A).
With the properly maskedattention mechanism, the decoder can safely output the policy ⇡ i m +1 ✓ ( a i m +
| ˆ o i 1: n , a i 1: m ) , whichﬁnishes the implementation of Theorem (1).
The monotonic improvement guarantee.
An MAT agent i m optimizes a trust-region objectivethat is conditioned on new decisions of agents i 1: m  1 by means of conditioning its policy ratio onthem (see Equation (5)).
As such, it increases the joint return monotonically like if it followed thesequential update scheme of HAPPO [15, Theorem 2].
However, as oppose to that method, the MATmodel does not require i m to wait until its predecessors make their updates, nor it uses their updatedaction distribution for importance sampling calculations.
In fact, as actions of all agents are outputsof MAT, their clipping objectives can be computed in parallel (during training), thus dominating
HAPPO on the time complexity.
Lastly, to assure that the limiting joint policy is such that none ofthe agents is incentivized to change its policy (Nash equilibrium), MAT requires permutating thesequential order of updates at every iteration, which is inline with the discovery in HAPPO [15,
Theorem 3].
To summarize, we list the full pseudocode of MAT in Appendix A and a video that shows the dataﬂow of MAT in https://sites.google.com/view/multi-agent-transformer .
Experiments and Results
MAT provides a new solution paradigm for cooperative MARL problems.
The key insights of MATare the sequential update scheme, which is inspired by Theorem (1), as well as the encoder-decoderarchitecture, which provides a highly-efﬁcient implementation for a sequence modeling perspective.
Importantly, MAT inherits the monotonic improvement guarantee, and agents’ policies can be learnedin parallel during training.
We ﬁrmly believe MAT will become a game changer for MARL studies.
To evaluate if MAT meets our expectations, we test MAT on the StarCraftII Multi-Agent Challenge(SMAC) benchmark [26] where MAPPO with parameter sharing [38] has shown superior performance,and the Multi-Agent MuJoCo benchmark [7] where HAPPO [15] shows the current state-of-the-artperformance.
SMAC and MuJoCo environments are common benchmarks in the MARL ﬁeld.
Ontop of them, we also test MAT on the Bimanual Dexterous Hands Manipulation (Bi-DexHands)
[6]which provides a list of challenging bimanual manipulation tasks (see Figure (3)), and the Google
Research Football [16] benchmark whose results are shown in Appendix C due to space limit.
[Table]
3) 93.8 (4.7) 88.4 (5.7) 0.0 (0.0) 9.7 (3.1) 1e
We apply the same hyper-parameters of baseline algorithms from their original paper to ensure theirbest performance, and adopt the same hyper-parameter tuning process for our methods with detailsin Appendix B. To ensure fair comparisons to CTDE methods, we also introduce a CTDE-variantof MAT called MAT-Dec , which essentially adopts a fully decentralized actor for each individualagent (rather than using the decoder proposed in MAT) while keeping the encoder ﬁxed.
The critic’sloss for MAT-Dec is L (  )
= 1 T
[Equation]
and we apply the local advantage estimation
A t (ˆ o i m t , a i m ) to guide the subsequent policy update.
5.1 Performance on Cooperative MARL Benchmarks
According to Table (1) and Figure (4), MAPPO signiﬁcantly outperforms HAPPO in SMAC withhigher sample efﬁciency.
This veriﬁed the homogeneity of SMAC agents and the heterogeneity ofmulti-agent MuJoCo agents, that are also discovered by Kuba et al.
Take the SMAC task 25mas an example, all the marines are equivalent and interchangeable so that agents can learn from theirteammate’s experience.
Sharing parameters in this settings means leveraging 25 times more examplesto train each agents comparing with separated network of HAPPO, and thus enjoying higher learningefﬁciency.
On the other hand, with the heterogeneous settings of multi-agent MuJoCo, traininga "foot" agent with experience from a "thigh" agent can surely harm its performance since theyrepresent different functions on the Cheetah.
However, MAT outperforms MAPPO and HAPPO inalmost all tasks in Table (1) and Figure (4), indicating its modeling capability on both homogeneousand heterogeneous-agent tasks.
MAT also enjoys the superior performance over MAT-Dec, whichemphasize the importance of the decoder architecture in the MAT design.
On the Bi-DexHands tasks,
MAT outperforms MAPPO and HAPPO methods by a large margin.
We save the Google Footballresults to Appendix C partly because the conclusion stays the same. [Table] [Table]
5.2 MAT as Excellent Few-short Learners
[Table]
Since Transformer-based models often demonstrate strong generalization performance on few-short [Table] tasks [3, 8], we believe MAT can possess strong generalization ability on unseen MARL tasks aswell. To validate such an assumption, we design zero-shot and few-shot experiments on SMAC andmulti-agent MuJoCo tasks.
For SMAC tasks, we pre-train agents on eight tasks involving ﬁve typesof units ( 3m , 8m vs 9m , 10m vs 11m , 25m , 3s vs 3z , 2s3z , 3s5z , MMM ) with 10M examples intotal and then apply them on six separate and much harder tasks ( 5m vs 6m , 8m , 27m vs 30m , 2svs 1sc , 1c3s5z , MMM2 ) including seven types of units.
This setting is designed to evaluate thegeneralization ability of MAT when training on simple tasks but transferring to more diverse andcomplex downstream tasks.
In terms of multi-agent MuJoCo, we reuse the models trained on thecomplete HalfCheetah robot as the pre-trained agent and then directly apply it to six new tasks, eachwith a different leg being disfunctioned (see Figure (3a)).
We investigate the generalization capabilityof pre-trained models on each downstream task with 0% (zero-shot), 1%, 5%, 10% few-short newexamples, respectively.
Note that common MARL baselines such as HAPPO assume ﬁxed number ofagents during training, thus it cannot directly handle the cases with changing number of agents.
We summarize the zero-shot and few-shot results of each algorithm in Table (2) and (3), where thebold number indicates the best performance.
We also provide the performance of MAT if it wasgiven the same amount of data but is trained from scratch, the "MAT-from scratch", as the controlgroup to demonstrate the effectiveness of pre-training process.
As both tables suggest, bold numbersare mainly located in the area of MAT, which justify MAT’s strong generalisation performance as afew-short learner.
Surprisingly, we ﬁnd that the few-shot MAT with only 10% data show even higherrewards than its counterpart that is purely trained on HalfCheetah with the same disabled joints ( backfoot , back shin and back thigh ) and 100% full amount data, we believe it is because the pre-trainprocess offers initial weights that are not only closer to optimum but also less likely to stuck in badlocal optima than random initialization.
Conclusion
In the past ﬁve years, large sequence models have achieved remarkable successes on solving visuallanguage tasks.
In this paper, we take the initial effort to build the connection between multi-agentreinforcement learning (MARL) problems and generic sequence models (SM), with the ambition that
MARL researchers can hereafter beneﬁt from the prosperous development on the sequence modelingside.
Speciﬁcally, we contribute by unifying a general solution to cooperative MARL problems into a
Transformer like encoder-decoder model.
The proposed Multi-Agent Transformer (MAT) leveragesthe multi-agent advantage decomposition theorem, which essentially transforms the joint policyoptimization process into a sequential decision making process that can be simply implemented byan auto-regressive model.
We have demonstrated MAT’s strong empirical performance on threechallenging benchmarks against current state-of-the-art MARL solutions including MAPPO and
HAPPO.
Based on the established connection between MARL and SM, in the future, we plan tobring multi-agent learning tasks into large multi-modal SM, chasing for more generally intelligentmodels as the most recent success of GATO has already demonstrated [25].
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[No] (c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Appendix B
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] See Section
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]