Off-Beat Multi-Agent Reinforcement Learning
Anonymous Author(s)
AffiliationAddress email
Abstract
We investigate model-free multi-agent reinforcement learning (MARL) in environments where off-beat actions are prevalent, i.e. , all actions have pre-set executiondurations.
During execution durations, the environment changes are influencedby, but not synchronised with, action execution.
Such a setting is ubiquitous inmany real-world problems.
However, most MARL methods assume actions areexecuted immediately after inference, which is often unrealistic and can lead tocatastrophic failure for multi-agent coordination with off-beat actions.
In order tofill this gap, we develop an algorithmic framework for MARL with off-beat actions.
We then propose a novel episodic memory, LeGEM, for model-free MARLalgorithms.
LeGEM builds agents’ episodic memories by utilizing agents’ individual experiences.
It boosts multi-agent learning by addressing the challengingtemporal credit assignment problem raised by the off-beat actions via our novelreward redistribution scheme, alleviating the issue of non-Markovian reward.
Weevaluate LeGEM on various multi-agent scenarios with off-beat actions, including
Stag-Hunter Game, Quarry Game, Afforestation Game, and StarCraft II micromanagement tasks.
Empirical results show that LeGEM significantly boosts multi-agentcoordination and achieves leading performance and improved sample efficiency.
Introduction
In Multi-Agent Reinforcement Learning (MARL), multiple agents act interactively and complete tasksin a sequential decision-making manner with Reinforcement Learning (RL).
It has made remarkableadvances in many domains, including autonomous systems [8, 19, 72] and real-time strategy (RTS)video games [58].
By the virtue of the centralised training with decentralised execution (CTDE) [33]paradigm, which aims to tackle the scalability and partial observability challenges in MARL, many
CTDE-based MARL methods are proposed [13, 49, 41, 62, 47, 63, 23, 35].
With these methods, anagent executes actions only via feeding its individual observations independently and optimizes itsown policy with access to global trajectories centrally.
Despite the recent successes of MARL, learning effective multi-agent coordination policies forcomplex multi-agent systems remains challenging.
One key challenge is the off-beat actions, i.e. , allactions have pre-set execution durations 1 and during the execution durations, the environment changesare influenced by, but not synchronised with, action execution (an illustrative scenario is shown in
Fig. 1).
However, Dec-POMDP [32], which underpins many CTDE-based MARL methods, hingeson the assumption that actions are executed momentarily after inference, leading to catastrophicfailure for centralized training on various off-beat multi-agent scenarios (OBMAS).
To fill this gap,we study MARL in settings where off-beat actions are prevalent.
Such setting is very common inmany real-world problems.
For example, in the traffic light control problem, traffic lights in the conjunctions of the road network have pre-set execution time which is set asynchronously
[Figure]
The problem of off-beat actions in MARL has yet to be investigated and tackled.
Training MARLpolicies in OBMAS is challenging: (i) Each agent’s actions can have a variety of execution durations,which augments the order of complexity of OBMAS during decentralized execution, resultingin failure of the coordination; (ii) The action durations are unknown to agents during individualexecutions, and communication is constrained and not always feasible, making it non-trivial to modelthe environment; (iii) During training, both the temporal credit assignment with TD-learning [51]and the inter-agent credit assignment with value decomposition methods [41] cannot perform welldue to the displaced rewards in multi-agent replay.
With off-beat actions, the nonstationarity issue,which mainly stems from rewards’ time dependency on the agents’ past actions, is exacerbated.
While actions durations are ubiquitous, existing works only focus on single-agent settings, i.e. , delay,in RL.
Many approaches [59, 39, 66] augment the state space with the queuing actions to be executedinto the environment.
However, such state-augmentation trick leads to exponentially increasingtraining samples with the growing action duration, making training intractable [11].
Chen et al.
[10]extend the delayed MDP [39] and propose Delayed Markov Game for MARL.
However, on onehand, such state-augmentation treatment is confined to short delays, e.g. , one timestep delay; onthe other hand, the delayed timestep of the actions is privileged information, which is not availablein many scenarios.
Recent works on macro-actions [67, 68] introduce asynchronous actions bydesigning macro-actions with prior environment knowledge.
Macro-actions are different from optionsin hierarchical RL (HRL) [52, 3] in that the later is not manually designed but learned.
The keydifference between macro-actions and off-beat actions is that macro-actions are high-level actionswhile off-beat actions are primitive actions.
Unfortunately, the inter-agent credit assignment is stilla challenge of HRL in OBMAS and the asynchronous 2 nature of off-beat actions undermines thetemporal credit assignment of centralized training , causing poor sample efficiency and unsatisfactoryperformance (more discussions can be found in the related works section in Sec. 7).
We aim to address the aforementioned issues.
We first propose off-beat Dec-POMDP.
We then instantiate a new class of episodic memory, LeGEM, for model-free MARL algorithms.
LeGEM boostsmulti-agent learning by addressing the challenging temporal credit assignment problem raised by theoff-beat actions via our novel levelled graph-based temporal recency reward redistribution scheme.
Specifically, each agent maintains LeGEM and during centralized training, each agent searches thepivot timestep given observations from its graph
The pivot timestep is the timestep wherein the off- beat reward relates to the given node.
The pivot timesteps of each agent are ranked in which the final pivot timestep will be chosen by recency and later used for reward redistribution and target estimation in TD-learning.
We evaluate our method on Stag-Hunter Game, Quarry Game, Afforestation Game, and StarCraft II micromanagement tasks.
Empirical results show that our method significantly boosts multi-agent coordination and achieves leading performance as well as improved sample efficiency
Preliminaries
Dec-POMDP.
A cooperative MARL problem can be modeled as a decentralised partially observable
Markov decision process (Dec-POMDP) which can be formulated as a tuple ⟨S , U , P , R, O, N , γ ⟩ ,where s ∈ S denotes the state of the environment.
Each agent i ∈ N := { 1 , ..., N } chooses anaction u
i ∈ U at each timestep, forming a joint action vector,
u :=
[ u i ]
Ni =1 ∈ U N .
The Markoviantransition function can be defined as P ( s ′ | s , u ) :
S ×U N ×S (cid:55)→ [0 , 1] , transiting one state of currenttimestep to the state of next timestep conditioned on current state and joint action.
Every agent sharesthe reward and the reward function is R ( s , u ) : S × U N (cid:55)→ R .
γ ∈ [0 , 1) is the discount factor.
Due to partial observability , each agent has individual partial observation o ∈ O , according to theobservation function O ( s , i ) : S × N (cid:55)→ O .
The goal of each agent is to optimize its own policyπ i ( u i | τ i ) : T × U
(cid:55)→ [0 , 1] given its action-observation-reward history τ i ∈ T :=
( O × U ) .
Multi-Agent Reinforcement Learning.
MARL aims to learn optimal policies for all the agentsin the team.
With TD-learning and a global Q value proxy Q tot for the optimal Q ∗ , { Q i } Ni =1 areoptimized via minimizing the loss [65, 31]: θ ∗ = arg min θ ∗ L ( θ ) :=
E D ′ ∼D
[( y tot t − Q tot θ ( s t , u t )) 2 ] ,where y tot t =
r t + γ max u ′ Q tot¯ θ ( s t +1 , u ′ ) and θ is the parameters of the agents.
¯ θ is the parameterof the target Q tot and is periodically copied from θ .
D ′ is a sample from the replay buffer D .
Off-Beat Dec-POMDP
We introduce our formulation for OBMAS.
We first define the off-beat actions 3 for multi-agentscenarios; then we propose the Off-Beat Dec-POMDP.
All the proofs can be found in Appx.
A.
Definition 1 (Off-Beat Actions) .
Off-beat action ˜ u
∈ U characterizes OBMAS where the action ˜ u i taken by agent i has execution duration m ˜ u
i ∼ A ( m | ˜
u
i , i ) , A ∈ A , m ∈ { 0 , 1 , 2 , · · · , M } and M ≤ T , where T is the maximum duration and A is the action duration distribution.
It is a distribution and takes ˜ u
i and the index of the agent as parameters.
A can be either stochastic or deterministic.
The joint off-beat action is ˜ u
= [ ˜ u i ]
Ni =1 .
The execution duration is decided at the time the action was committed to the environment.
Thus, the execution duration of an action ˜ u t initiated at timestep t is m t = { m t ˜ u ti } Ni =1 .
properties
Definition 2 (Off-Beat Dec-POMDP) .
Off-Beat Dec-POMDP extends Dec-POMDP, such that
(1) state space is S ; (2) joint action space is U N ; (3) action duration space is A N ; (4) transition function is P ( s ′ | s , ˜ u , m ) :
S ×
U N × S × A N (cid:55)→ [0 , 1] , and m is the action durations of the joint action; (5) the reward function is R ( s , ˜ u , m ) :
S × U N × A N (cid:55)→ R ; (6) we call
a reward r as off-beat reward when any its m ˜ u
i ≥ 1 , m ˜ u i ∈ m , and r ̸ = 0 .
In OBMAS, at each timestep t , the environment receives actions that agent initiates for executionin the environment.
The initiated actions ˜ u t are instantaneous actions inferred by agents’ policiesgiven individuals’ observations.
The joint reward is the consequence of the committed joint actionsof current timestep and previous timesteps, depending on the actions’ duration.
The asynchronicityis an inherent feature of the environment, which is different from asynchronicity incurred by communication delays in many video games (asynchronous gameplay 6 ).
We discuss some properties of
Off-Beat Dec-POMDP below.
html
Remark 1.
When the durations for all actions are identical, off-beat Dec-POMDP reduces to
Delayed Dec-POMDP and there is no off-beat actions in it.
Remark 2.
There exists ˜ u that is synchronous since duration of agents’ actions can be m = 0 .
When m of all actions is zero , off-beat Dec-POMDP reduces to Dec-POMDP.
In Delayed Dec-POMDP, actions have the same delayed timesteps, which is different from offbeat actions where actions have different action durations or delays.
In order to investigate theproblem, we consider the deterministic setting of the transition function and the reward function.
Remark 3 (Non-episodic Reward) .
In our formulation, the reward is not episodic reward [16].
Remark 4 (Non-Markovian Reward) .
With off-beat actions, the Markovian property of the rewardfunction R ( s , ˜ u , m ) does not hold.
With off-beat actions, the shared rewards can be readily displaced, causing non-Markovian rewards.
Solving Off-Beat Dec-POMDP is challenging as discussed in Sec.
We propose our methods totackle aforementioned challenges.
The Journey is the Reward: A Collective Mental Time Travel Method
We propose two methodological elements for Off-Beat MARL.
The first, LeGEM, presented inthis section, is a form of episodic memory that facilitates discovery of a pivotal timestep for offbeat rewards; and the second, presented in Sec. 5, is redistribution of the off-beat reward to the pivottimestep when the relevant off-beat actions were initialised.
4.1 LeGEM: A Levelled Graph Episodic Memory for Off-Beat MARL
Human learning relies on retrospecting our detailed memory of the past [55, 48].
For example,while exploring a new scenic area, we do not just remember a multitude of specific spots there,but can recall the paths that connect them with junctions and turns.
However, there is no MARLmethod that can explicitly recall the past and identify key states that lead to future rewards.
Such“ mental time travel ” [24] ability is vital for tackling the challenges in OBMAS.
Inspired by the recentprogress in RL with episodic memory [18, 5, 17] that is based on the memory prosthesis proposedby neuroscientists [55, 48], we propose our method of episodic memory representation for MARL.
Unlike previous episodic memory methods that train a parameterized memory by either augmentingthe policy inputs for execution [18] or regularizing the TD learning [17] for RL, our method utilizesthe levelled graph data structure [4], a well established structure for data storage and retrieval, torepresent an agent’s individual episodic memory.
Level =  +
Sub-graph1 Sub-graph
Graph
We propose our novel episodic memory, Levelled Graph Episodic
Memory method (LeGEM), via the levelled graph data structure.
LeGEM memorizes each agent’s past trajectories which are partialobservations and the unilateral action of the agent.
During training,each agent i collects its individual trajectories τ i .
We then define τ iof agent i as τ i =
[( o 0 i , ˜ u
i , r 0 ) , · · · , ( o T − 1 i , ˜ u
T − 1 i , r T − 1 )] , where T isthe length of the trajectory and the triplet ( o ti , ˜ u ti , r t ) represents theobservation, action and reward of timestep t .
Note that r t is globallyshared between agents.
We define agent i ’s LeGEM as a directed graphϕ ti ∈ Φ i where Φ i is the set of graphs of agent i and ϕ ti is the t -th graphof Φ i , t ∈ { 0 , · · · , T − 1 } .
Each ϕ ti consists of a tuple of (Ψ , Ξ) whereΨ is the set of nodes and Ξ represents the set of edges that connectnodes in the graph.
To model an agent’s behaviour explicitly and makethe trajectories of agents easy to represent, we create T graphs for eachagent and let Φ i = { ϕ ti } T − 1 t =0 where T is the maximum level of all graphs and the maximum lengthof the episode as well.
The maximum level of ϕ t i is t + 1 .
The node contains key, visit count andpointers connecting the precursors (node at the previous level) and the successors (node at the nextlevel).
Unlike many parameterized episodic memory using state/observation as the key [18, 24], weresort to afterstate [36].
That is, we use agent i ’s observation o ti and action at timestep t , ˜ u ti , to definethe key ( o ti , ˜ u ti ) .
We provide an example to showcase the relationship between sub-graph and thegraph in Fig.
For complex and continuous state scenarios, for example StarCraft II scenarios, we
← ψ ←
Find the
Get the
Get the it ←
Get e t
; 15 Return ωi
; 4 vc
VisitCount
; 5 foreach do e
← UL Λ l j vc ; e t ← LU e it [ j ] ← e i,j, ↑ t ; else e it [ j ] ← t ;
Return use SimHash [9] to discretize the key ( o ti , ˜ u ti ) .
This technique has been widely used in commercialsearch engines and RL [54].
Visit count indicates the total visits made by agent i to the node.
It initialvalue is 1 .
Note that nodes are bidirectional since it is helpful for searching (see Sec. 4.2).
Given a τ i with the length of T , if the node is already in the graph at level t , we then increase thevisit count by 1.
Otherwise, we create a new node for level t of the graph and update its pointers.
Meanwhile, sub-graphs will be also created and updated.
The process of updating LeGEM is in Alg.3.
We provide an example of Alg. 3 in Fig.
9, Appx.
B.1.
It is worth noting that τ i is generated viathe interaction of the agent with the environment, and there is no extra interaction needed to collect τ i .
The generated trajectories are saved in the experience replay and later sampled for MARL training.
4.2 Multi-Agent Collective Mental Time Travel with LeGEM
With structured agent’s past experiences, it can be used to search the pivot timestep when actionsthat triggered the rewarded state were executed.
For example, with LeGEM, we can find the pivottimestep, e t = 5 , when agent 1 shoots the arrow in Fig.
Fact 1.
(Action-Reward Association)
When an off-beat reward r t exists in the trajectory τ
i ( i ∈{ 1 , · · · , N } ) , r t ∈ τ i , off-beat action u t ′ exists ( t ′ < t ) in the trajectory set { τ j } Nj =1 , where { τ j } Nj =1constitutes the global trajectory of all agents.
As the reward function and transition function are deterministic in our setting, Fact 1 holds.
Intuitively,once we find an off-beat reward in a trajectory, we are sure that the action which triggered the rewardcan be found in the trajectory.
With more experiences collected by the agents, such pattern is obviousand significant.
It motivates us to propose a method to leverage the association property of theoff-beat action-reward data and search the pivot timestep for timesteps when off-beat rewards occur,which can further help to redistribute the reward backward to mitigate the temporal credit assignmentissue (c.f.
Sec. 5).
Therefore, we first propose a search method to search the pivot timestep and thenpropose a proximal ranking method to estimate the pivot timestep that invokes the future reward.
Collective Mental Time Travel.
The displaced rewards in the replay buffer hinder multi-agentlearning.
It is essential for each agent to search the pivot timestep when the potential off-beat actionthat triggered the rewarded state was committed to the environment.
Therefore, we propose twosearch schemes to find the pivot timestep for all agents given an off-beat reward.
Scheme I: For agent i , given r t ∈ τ i , episode return r l,i of τ i , ϕ l i =
Φ i [ l ] and ϕ l,ω i =
Φ l, Ω i [ ω ] ,agent i searches from the node (the key is ( o ti , ˜ u ti ) and o ti ∈
τ i , u ti ∈ τ i ) at level t in sub-graph ϕ l,ωi tofind the pivot timestep e t for r t .
Concretely, we propose our bi-directional search method.
The firstone is called Low-Up (LU) search, which traverses from the given node at level t upwards to the nodeat level 0 .
The second one is named Up-Low (UL) search which traverses from the node at level 0downwards to the given node at level t .
LU traversing ends when the pattern of increasing visit count
(𝑜 !" ,𝑢 !"
#$ ) 𝑞 !
𝑄 !"!
[𝑞 ! ] !
%&’#$
Env
DecentralizedExecution:ThePolicy [𝑢 !" ] !
$(
Centralized Training: The Loss actions
Online interaction 𝒔 " ,𝒖 " ,𝑟 " ,𝒔 ")$ "%&* , 𝑒 " "%&*#$ experiences agent 𝑖 ... ... update the graphs
Search for each agent 𝑖
ReplayBuffer save 𝜏 ! !"
#$%& ’"& ( samples 𝑒 ) !,)"
#+%& !"
#,…,$%& 𝜏 ! !"
#$%&
Centralized Training: LeGEM concatenate the results agent 𝑖 sampled trajectories
Rank 𝜅 !
across agents
Graphs policy 𝜏 = 𝒔 " ,𝒖 " ,𝑟 " ,𝒔 ")$ "%&+
TD Learning
CentralMixer
Identify 𝜏 , ,%&-#$ !
Search agent 𝑖
Getpaths 𝜅 !
= 𝑒 "! "
%& *#$ for each 𝑑 allagents 𝜋 !
Agent 𝑖 reward redistribution ends and the corresponding level is the candidate pivot timestep.
On the contrary, UL traversingends when the pattern of decreasing visit count ends and the corresponding level is the candidatepivot timestep.
In Alg. 2, we first get visit count (Line 4) and then apply UL traversing (Line 6)and LU traversing (Line 7).
We summarize the results (Line 14) by select the pivot timestep thathas the maximum count.
UL traversing has a higher searching priority than its counterpart.
Thereason is that there exists pattern that the visit count is decreasing from the node at level 0 andsuch pattern ends at the pivot timestep.
In practise, it works well in scenarios whose trajectories aresingle-off-beat-reward trajectories (there is only one off-beat reward) and the accuracy of Scheme I isover 90% in grid world scenarios.
For scenarios, especially complex scenarios, whose trajectories aremultiple-off-beat-reward trajectories, we apply Scheme II.
We put Alg. 4, Alg. 5, Alg. 6 and Alg. 7 in
Appx.
B.1 as these algorithms are intuitive and easy to understand literally.
The time complexity is
O ( n · m ) (a slight notation abuse) where n is the size of each Λ t,li and m ( 1 ≤ m ≤ n ) is the averagedistance between the level of the given node to the level of the node at the pivot timestep.
is
Given a node at level t , agents collectively search from the node to find the pivot time step (Line 13in Alg. 1).
The visit count is vital for search methods.
In MARL, we use ϵ -greedy [31] for agents toexplore the environment and collect individual trajectories.
The collected trajectories will be used tobuild the memory and train the policy.
We apply annealing to ϵ (in Appx. E).
Ranking the Pivot Timesteps.
With our two search schemes, we can search the pivot timestepsfor each global trajectory τ = { ( s t , ˜ u t , r t , s t +1 ) } T − 1 t =0 .
We define the pivot timesteps κ of eachglobal trajectory τ as κ = { e t } T − 1 t =0 , 0 ≤ e t ≤ t , where e t indicates the pivot timestep of t whenr t is the consequence of actions committed before timestep t .
We first get e t by aggregating all thesearching outcomes (Line 13 in Alg. 1).
Then, each agent gets κ i = { e it } T − 1 t =0 .
In order to subserve 214 the inter-agent credit assignment [13, 41], κ can be collectively calculated via proximity:
[Equation]
The pseudo code is shown in Alg.
For each sampled global trajectory τ , we extract τ i for eachagent in Line 7; then we get e t for each agent and aggregate κ in line 14 and line 15, respectively.
Reward Redistribution for Off-Beat Multi-Agent Reinforcement Learning
Searching in LeGEM leverages the collective intelligence [25, 15] in OBMAS.
We utilize TD learning 219 to train MARL policies.
The TD error is the difference between the TD target and the prediction.
TD targets can be estimated with n -step target, TD( λ ) and other techniques [12, 56].
Unfortunately, 221 current n -step target and TD( λ ) methods are far from accurate estimating TD targets.
They evenincur underestimation with off-beat trajectories.
In essence, to train MARL policies in OBMAS, one 223 should accurately estimate the TD target where the reward plays the key role [46, 70].
We resolve theaforementioned conundrum by redistributing rewards to their pivot timesteps.
The key idea is that we 225 can pull the outcome of one joint off-beat action back to the timestep when it was committed to theenvironment, which can dramatically enhance learning despite the long-term reward delays incurredby off-beat actions.
We utilize e t to update the reward of the transit ( s e t , ˜ u
e t , r e t , s e t +1 ) :
[Equation]
where 1 ( · ) is the indicator function.
Such update rule is conducted iteratively from t = 0 to t =
T − 1 .β is a very small positive hyperparameter.
To stabilize learning and circumvent the overestimation 230 of the TD target, r t is also updated after Eqn. 2 via r t = (1 − 1 ( e t < t ) · (1 − β )) · r t .
It alsoavoids aggregated biased/wrong estimation of TD target being back propagated in Bellman Equation.
Formally, we define the reward redistribution operator as Π Φ , i.e. , e t
= Π
Φ ρ ( r t , s , ˜ u ) , and then 233 define the Off-Beat Bellman operator Γ :
[Equation]
With the Off-Beat Bellman operator Γ , we propose its contraction property.
Proposition 1.
Γ : Q (cid:55)→ Q is a γ -contraction.
Therefore, we can utilize ˆ r e t for centralized training in TD-learning:
[Equation]
, where ˆ y
[Equation]
=
[Equation]
˜
[Equation]
Our method can be easily incorporated into any model-free MARL method for OBMAS.
We presentthe pseudo code of incorporating our method into model-free MARL methods in Alg.
9, Appx.
E.
We also provide a pictorial view of our framework in Fig.
3 to show the whole pipeline.
Experiments
We perform experiments on various multi-agent scenarios with off-beat actions.
We introduceoff-beat actions in Stag-Hunter Game, Quarry Game, Afforestation Game and StarCraft II micromanagement tasks [44] and use them as testbeds in our experiments.
We aim to answer the followingquestions: Q1:
Can our LeGEM improve the multi-agent coordination of many MARL methodsin OBMAS?
Q2:
Can our LeGEM outperform previous parameterized episodic memory (EM) for
MARL?
Q3:
Can bootstrapping method of RL help?
Q4:
Can our LeGEM outperform the multi-agentexploration and multi-agent risk-sensitive (Ex-Risk) methods?
[Figure]
6.1 Experiment Setup
[Figure]
Categories
Methods
[Table]
Baselines and scenarios.
We list all baselines in table 1, including the corresponding researchquestions to be answered.
We implement our method on PyMARL [44] and use 10 random seedsto train each method on all environments.
We do not use macro-action methods [67, 68] as thebaseline because it is hard to make a fair comparison between macro-actions methods and our method.
As discussed in Sec. 1, macro-actions rely on manually designed macro-actions, i.e. , designingthe macro-actions by utilizing the simulator settings and domain knowledge, which is differentfrom learning options [52, 3].
Designing macro-actions is not feasible in scenarios where domainknowledge and simulator settings are unknown, such as the OBMAS scenarios.
In OBMAS, the agenthas no idea of the durations of other agents’ actions, which is challenging for designing macro-actions.
We conduct experiments on Stag-Hunter Game, Quarry Game, Afforestation Game (Fig. 5) and
StarCraft II micromanagement tasks [44] where off-beat action are introduced.
Training settings.
We use opensourced code of baselines publicly by the corresponding authors on
Github in all experiments.
We resort to mean-std values as our performance evaluation measurementin all figures where the bold line and the shaded area indicate the mean value and one standarddeviation of the episode return, respectively.
Readers can refer to Appx.
C, D, E and F for moreinformation on our environment, baselines, training method, training platform and empirical results.
[Figure] [Figure]
6.2 Experiment Results
The Effectiveness of LeGEM.
We answer Q1 .
With LeGEM, MARL methods get enhancedperformance as shown in Fig.
Without LeGEM, all methods perform poorly in Stag-Hunter Game;
IQL and VDN’s final final results are even 0 .
By incorporating LeGEM, all of them can get convergedperformance and improved sample efficiency.
We are also interested in finding if LeGEM couldreinforce the performance of simple methods.
As depicted in Fig.
7, with LeGEM, both VDN and
QMIX outperforms QPLEX, which is a state-of-the-art MARL method armed with various advancedtechniques, including attention network [57], dueling network [64] and advantage function.
Performance of Episodic Memory method.
We answer Q2 by presenting the performance curvesof EMC in Fig.
EMC is an episodic memory MARL method with curiosity-driven exploration.
Itutilizes the episodic memory from RL [74, 17].With LeGEM, QMIX outperforms EMC.
EMC evenfails to converge in Stag-Hunter Game.
[Table]
Performance of n -step return and TD( λ ) methods.
To answer Q3 , we use n -step return and
TD( λ ) to estimate the TD-target.
As shown in Table.
2, with n -step return, both QMIX and VDNfail to learn good policies even with n = 15 .
Surprisingly, with TD( λ ), QMIX can achieve goodperformance with λ ∈ { 0 . 8 , 0 .
However, we cannot find such outcome on VDN and thereis no guarantee of good results on using TD( λ ).
[Figure]
Performance of Multi-Agent Exploration and RiskSensitive MARL methods.
We also provide results ofexploration methods for MARL and risk-sensitive MARLmethod to answer Q4 .
MAVEN utilizes mutual informationto learn latent space for exploration and RMIX aims tolearning risk-sensitive policies for MARL.
In Fig.
7, RMIXeven fails to learn.
Mainly because the potential loss ofreward is displaced by off-beat actions.
Overall, MAVENis stabler than EMC and RMIX.
QMIX-LeGEM is stablein all scenarios and outperforms MAVEN.
With LeGEM,even simple method such VDN can perform well and outperforms many MARL methods with complex and advanced components.
Indeed, explorationin OBMAS is beneficial for multi-agent learning.
However, the key challenge of temporal creditassignment can not be easily addressed merely with exploration.
SMAC.
We also conduct experiments on SMAC [44].
We train MARL methods and our methodon 2m_vs_1z where are two agents combating with one opponent.
To overcome the issue of high dimension continuous state space, We utilize simhash [9] to calculate the hash value of thekey.
We only select the attack action and set the action duration with 9.
As illustrated in Fig.
8,incorporated with our novel episodic memory, QMIX, IQL and VDN illustrate enhanced performance,demonstrating the superiority of our method on complex multi-agent scenarios.
Related Works
Action Delay in RL.
Conventionally, the execution of actions in RL is instantaneous and the executionduration is neglected.
Katsikopoulos et al.
[20] propose the Delayed MDP where actions have delaysand Walsh et al
[59] propose a model-based method for the Delayed MDP.
To optimize the delayed
MDP, many RL approaches [59, 39, 66, 69] augment the state space with the queuing actions tobe executed into the environment.
However, this state-augmentation trick is intractable [11].
Chenet al.
[10] extend the delayed MDP [39] and propose a Delayed Markov Game.
However, thestate-augmentation treatment is confined to short delays and neglects the off-beat actions in multiagent scenarios.
Recently, Bouteiller et al.
[6] apply replay buffer correction method.
However, thedelayed timestep is privileged information.
It is not available for agents in many scenarios.
Simplyapplying this single-agent trajectory correction in MARL cannot attain satisfactory performance dueto off-beat actions; devising inter-agent trajectory correction methods for OBMAS is non-trivial.
Credit Assignment in RL.
Credit assignment [50, 52] tackles
long-horizon sequential decisionmaking problem by distributing the contribution of each single step over the temporal interval.
TDlearning [51] is the most established credit assignment method, which is the basis of many RL methods.
RUDDER [2] redistributes the episodic return to key timesteps in the episode [14, 42, 40].
Klissarovet al.
[22] propose a reward propogation method via graph convolutional neural network [21].
Anotherline of works utilize episodic memory (EM) [37, 5, 73, 27, 74] to recall key events and aggregateinformation of the past for decision-making or learning.
However, simply applying EM of RL to
MARL cannot perform well in OBMAS due to the non-stationarity and the displaced rewards.
Multi-Agent RL.
Many MARL methods focus on factorizing the global Q value to train agents’policies via CTDE [13, 49, 41, 47, 60, 63, 35].
However, these existing works assume actions areexecuted synchronously.
Messias et al.
[30] propose an event-driven, asynchronous formulation ofthe multi-agent POMDP.
However, the assumption of free communication [61] is limited and theasynchronous execution [34] in the paper is confined to the design of events and did not proposemethods on solving challenging credit assignment issue in OBMAS.
Recently, Amato et al.
[1]and Xiao et al.
[67, 68] propose macro-action methods, which are similar to hierarchical methods.
Macro-actions are manually designed via abstracting primitive actions.
However, macro-actionmethods mainly focus on macro-action selection during multi-timestep decision-making and assumethe environment can use manually pre-defined methods for state transition.
Unfortunately, the aboveworks either focus on synchronous actions or defining specific asynchronous execution componentswith human knowledge.
Learning coordination in OBMAS remains a challenge.
Conclusion
In this paper, we investigate model-free MARL with off-beat actions.
To address challenges in
OBMAS, we first propose Off-Beat Dec-POMDP.
Then, we propose a new class of episodic memory,
LeGEM, for model-free MARL algorithms.
LeGEM addresses the challenging temporal creditassignment problem raised by off-beat actions in TD-learning via the novel reward redistributionscheme.
We evaluate our method on various OBMAS scenarios.
Empirical results show that ourmethod significantly boosts the multi-agent coordination and achieves leading performance as well asimproved sample efficiency.
Limitations and Future Work.
Searching from a graph-structured episodic memory takes muchoverhead in LeGEM.
Scaling up LeGEM to complex OBMAS is our future direction.
Recently, thereis a growing interest in model-based planing [45].
Leveraging LeGEM for model-based planningis also our future work.
Our paper focuses on Dec-POMDP-based MARL methods.
We leave it tofuture work for investigating off-beat actions in frameworks like Markov Game [26] and MMDP [7].
We are also interested in finding the merit of our method in real-world problem in our future work,such as scheduling [29] with off-beat settings.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] See Sec. 4.
(b) Did you describe the limitations of your work?
[Yes] See Sec. 8.
(c) Did you discuss any potential negative societal impacts of your work?
[No] Our method
does not have negative social impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See the
statements.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appx.
A.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] See the
supplementary files.
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Appx.
E.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Sec. 6 and Appx.
E.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appx.
E.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cited, see Sec. 6.
(b) Did you mention the license of the assets?
[Yes] See Appx.
E.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
See the supplementary files.
d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]