Off-Policy Correction For Multi-Agent Reinforcement Learning
Abstract
Multi-agent reinforcement learning (MARL) provides a framework for problems involving multiple interacting agents.
Despite apparent similarity to the singleagent case, multi-agent problems are often harder to train and analyze theoretically.
In this work, we propose MA-Trace, a new on-policy actor-critic algorithm, which extends V-Trace to the MARL setting.
The key advantage of our algorithm is its high scalability in a multi-worker setting.
To this end, MA-Trace utilizes importance sampling as an off-policy correction method, which allows distributing the computations with no impact on the quality of training.
Furthermore, our algorithm is theoretically grounded – we prove a ﬁxed-point theorem that guarantees convergence.
We evaluate the algorithm extensively on the StarCraft Multi-Agent
Challenge, a standard benchmark for multi-agent algorithms.
MA-Trace achieves high performance on all its tasks and exceeds state-of-the-art results on some of them.
Introduction
Reinforcement learning has witnessed impressive development in recent years.
Famously, superhuman performance has been achieved in games Go [30], StarCraft II [36], Dota 2 [4] and other applications.
These successes are the result of rapid algorithmic development.
Research in directions like trustregion optimization [29], principle of maximum entropy [13], importance sampling [7], distributional RL [3] or bridging the sim-to-real gap [2] are among these which brought signiﬁcant progress.
Multiagent reinforcement learning (MARL), a framework for problems involving multiple interacting agents, is similar to the standard, single-agent setting.
However, it is inherently harder.
The challenges are both theoretical (e.g., partial observability and lack of the Markov property) and practical (MARL algorithms often suffer from inferior stability and scalability).
In this work, we take a step towards amending this situation.
We propose MA-Trace, a new on-policy actor-critic algorithm, which adheres to the centralized training and decentralized execution paradigm [19, 11, 25].
The key component of MA-Trace is the usage of importance sampling.
This mechanism, based on V-Trace [7], provides off-policy correction for training data.
As we demonstrate empirically, it allows distributing the computations efﬁciently in a multi-worker setup.
Another advantage of MATrace is the fact that it is theoretically grounded.
We provide a ﬁxed-point theorem that guarantees convergence.
The on-policy algorithms directly optimize the objective; thus, they tend to be more stable and robust to hyperparameter choices than off-policy methods [1, 35, 33].
However, it is often impractical to train an on-policy algorithm in the distributed setting.
When data collection is performed using many workers, the communication latency, asynchronicity, and other factors make the behavioral policies lag behind the target one. This results in a shift of the collected data towards off-policy distribution, which hurts the training quality.
V-Trace reduces this shift by utilizing importance weights, thus permitting highly scalable training.
Following that scheme, MA-Trace can be distributed to many workers to vastly reduce the wall-time of training with no negative impact on the results.
We evaluate MA-Trace on StarCraft Multi-Agent Challenge [27] – a standard benchmark for multiagent algorithms.
Our approach achieves competitive performance on all tasks and exceeds state-ofthe-art results on some of them.
Additionally, we provide a comprehensive set of ablations to quantify the inﬂuence of each component on the ﬁnal results.
We conﬁrm that importance sampling is a key factor for MA-Trace’s performance and show that our algorithm scales favorably with the number of actor workers.
Additionally, we provide a few quite surprising ﬁndings, e.g. that an observation-based critic network performs better than a state-based.
For the description of key ideas and videos, visit our webpage: https://sites.google.com/ view/ma-trace/main-page .
The code used for our experiments is available at https://github. com/awarelab/seed_rl .
Our main contributions are the following:
We introduce MA-Trace – a simple, scalable and effective multi-agent reinforcement learning algorithm with theoretical guarantees.
We conﬁrm that the training of MA-Trace can be easily distributed on multiple workers with nearly perfect speed-up and no negative impact on the quality.
We provide extensive experimental validation of the MA-Trace algorithm in StarCraft MultiAgent Challenge, including ablations with regard to importance sampling, centralization of learning, scaling and sharing of parameters.
Related work
For a general overview of multi-agent reinforcement learning (MARL) we refer to [5, 15].
Unsurprisingly, the development of MARL methods is closely coupled with the algorithmic progress in
RL.
A simple approach to multi-agent learning was proposed by Tan [34]: the IQL algorithm uses independent Q learners for each agent, with improvements proposed in [10, 17, 24].
MA-Trace adheres to the centralized training and decentralized execution (CTDE) paradigm. CTDE [16, 23, 9] is based on using the centralized information during training. During execution, the agents act using only their respective observations.
Following this scheme, [9] introduces the RIAL and DIAL algorithms in the context of Q -learning.
CTDE is particularly easy to implement with actor-critic algorithms; the centralized information is imputed only to the critic network (which is not used during the execution). COMA [11] is an example of such an algorithm; additionally, it uses a counterfactual baseline to deal with multi-agent credit assignment explicitly.
Another approach to take advantage of the multi-agent structure is the value decomposition method.
VDN [32] propose a linear decomposition of the collective Q function into agent-local Q functions.
Following this idea, [25] introduced QMIX, which learns a complex state-dependent decomposition by using monotonic mixing hypernetworks.
Extensions of QMIX include MAVEN [21], COMIX [6],
SMIX( λ ) [39], and QTRAN [31] that can represent even general non-monotonic factorizations.
MA-Trace is based on V-Trace [7], a distributed single-agent algorithm.
The idea of extending RL algorithms to the multi-agent setting has been successfully executed multiple times.
Lowe et al.
[19] propose a multi-agent actor-critic algorithm MADDPG, which is based on the DDPG algorithm [18].
Yu et al.
[40] introduce also MASAC, extending SAC [14], and MATD3 building on top of TD3 [12].
Recently [41] showed that MAPPO, a multi-agent version of PPO [29], achieves surprisingly strong results in the most popular benchmarks, comparable with off-policy methods.
Espeholt et al.
[7] propose the V-Trace algorithm to address the problem that in distributed (e.g. multi-node) training the policy used to generate experience is likely to lag behind the policy used for learning.
Munos et al.
[22] considered earlier a similar off-policy corrections for the target of the Q -function.
These corrections are intended to focus on samples generated by behavioral policies close to the target one.
Leaky V-Trace, a more general version of the V-Trace correction, was considered by Zahavy et al.
Vinyals et al. [37] adapt V-Trace importance corrections to large action space to train grandmaster level StarCraft II agents.
These corrections are reﬁnements of the concept of importance sampling; see [33, Sections 5.5, 12.9] for a broader discussion.
Blending all these concepts, DOP [38] utilizes value decomposition and importance sampling to successfully train decentralized agents with policy gradients on off-policy samples.
This is substantially different from our work since in MA-Trace we use importance weights to enable efﬁcient multi-node training.
DOP does not consider distributing the computations, the objective optimized by that algorithm requires providing on-policy samples, which is impossible to satisfy in a highly distributed setting.
Background
Multi-agent reinforcement learning task is formalized by decentralized partially observable Markov decision processes (Dec-POMDP) [23].
A Dec-POMDP is deﬁned as a tuple ( N , S , A , P, r, Z , O, γ, ρ 0 ) .
N is the set of agents { 1 , . . . , n } , S is the state space, A is the set of actions available to agents, P is the transition kernel, r is the reward function, Z is the space of collective observations, O is the set of observation functions { O 1 , . . .
, O n } , γ is the discount factor and ρ 0 is the initial state distribution.
At state s ∈ S , the agents select actions a i ∼ π i ( ·| O i ( s )) , where π i are their respective polices.
Fix a :
= ( a 1 , . .
. , a n ) .
The agents receive rewards according to the reward functions r i
= r i ( s, a ) and the system evolves to the next step generated by P ( s, a ) (might be stochastic).
In the so-called fully cooperative setting, assumed in this work, the rewards are equal, i.e. r 1 = . . .
= r n .
The agents learn a joint policy
[Equation]
with the aim to maximize the expected discounted return
[Equation]
The expected discounted return obtained by policy π starting from state s ∈ S is called the value function
[Equation]
MA-Trace algorithm 4.1 Overview of the algorithm
In this work, we introduce a multi-agent actor-critic algorithm based on V-trace: MA-Trace , see Algorithm 1.
It follows the paradigm of centralized training, decentralized execution.
Each agent compute its action taking its local observation as input.
On the other hand, the critic network operates only during training, so it does not need to obey decentralization requirements.
Furthermore, it can utilize any kind of additional information.
We study two versions of MA-Trace: with the critic V : S → R taking as inputs full states, and with the critic V : Z → R taking as input the joint observation of all agents, denoted respectively as MA-Trace (full) and MA-Trace (obs).
MA-Trace (full) requires collecting states s t in line 6 of Algorithm 1 and using them as input to V φ in linesand 14.
The value function V π corresponding to policy π can be, for example, obtained by repeated application of the Bellman operator.
This requires on-policy data.
The central innovation of V-Trace in the single-player setting and MA-Trace in the multi-player setting is to allow for slightly off-policy data by utilizing importance sampling.
To this end, we use the V-Trace-inspired policy evaluation operator
R , deﬁned as
[Equation]
where c t =
c ( s t , a t ) , ρ t :=
ρ ( s t , a t ) are importance sampling corrections and c : S × A → R + , ρ :
S × A → R + are measurable functions.
In our algorithms we specialize to
[Equation]
where µ is a policy that collected the data and c, ρ are hyperparameters (usually set to 1 . 0 ).
Intuitively speaking, c t controls the speed of training and ρ t balances the learned value function between V π and V µ .
These parameters are further discussed in Corollary 3.
The operator R leads to n -step Monte-Carlo target v t given the state s t
[Equation]
It is a random variable; the clipping with the min function in (4) is instrumental to reducing its variance and thus making it applicable in learning. A key advantage of MA-Trace is a signiﬁcant reduction in the wall-time due to distributed data collection (see line 6 in Algorithm 1).
From the algorithmic standpoint, the major problem to address is that the policy used for collection π θ (cid:48) might be outdated due to communication overheads. This is successfully achieved with the importance correction mechanisms described above. We use the communication model proposed in [8, Figure 1, Figure 3]. It consists of a single learner and actor workers.
The actors are simple loops around the environment, generating observations (and rewards) transmitted to the learner. The learner makes inferences (and sends back actions); moreover, it handles trajectory accumulation and training. Algorithm
Require: d density of training α learning rate 1: for k in 0 , . . . , n − 1 do 2: ω k ← random actor parameters 3: φ ← random critic parameters 4: for epoch in 0 , 1 , 2 , . . . do 5: D ← ∅ 6: add trajectories { τ i } sampled with π θ (cid:48) to D 7: (cid:46) collected by multiple workers possibly remote.
8: for i in 0 , . . . , d − 1 do 9: sample s t ∼ D 10: φ ← φ − α ∇ φ (cid:2) (cid:107) v t − V φ ( s t ) (cid:107) 22 (cid:3) (cid:46) v t is calculated according to (5) 11: for i in 0 , . . . , d − 1 do 12: sample s t ∼ D 13: for k in 1 , . . . , n do 14: g k ← ρ t ∇ ω log( π ω ( a t,k | s t,k ))
( r t,k + γV kφ ( s t +1 ) − V kφ ( s t )) 15: (cid:46) ρ t is calculated according to (4) 16: ω k ← θ k + αg k
4.2 Theoretical analysis of MA-Trace
The operator, R enjoys the ﬁxed point property.
We present a proof of the following Theorem in
Appendix A.
Theorem 1.
Let c, ρ be such that for any s ∈ S , a ∈ A
[Equation]
(6) where s (cid:48) is the state obtained from s after issuing action a .
Assume also that E µ ρ 0 ≥ β ∈ (0 , 1] .
Then the operator R is a C ∞ contraction with a unique ﬁxed point V ˜ π which is a value function of a policy ˜ π given by
[Equation]
The contraction constant is smaller than 1 − (1 − γ ) β < 1 .
Remark 2.
The theorem is an extended version of [7, Theorem 1].
First, we assume the vectorized statement, which is natural for the multi-agent setting.
Second, the condition (6) admits more general importance sampling weights.
We also ﬁx a mathematical inaccuracy present in the original proof of [7, Theorem 1], see Remark 6.
Now we can easily show that the result follows for the importance weights used in our work.
Corollary 3.
Let c t , ρ t be importance sampling weights (4) and 0 ≤ c ≤ ρ .
Assume also that E µ ρ 0 ≥ β ∈ (0 , 1] .
Then the operator R is a C ∞ contraction with a unique ﬁxed point V ˜ π which is a value function of a policy ˜ π given by
[Equation]
The contraction constant is smaller than 1 − (1 − γ ) β < 1 .
Proof.
It is easy to check that c ( s, a ) =
min (cid:16) c,
[Equation]
|
[Equation]
[Equation] , ρ ( s, a ) = min (cid:16) ρ, π [Equation] a | s ) µ
[Equation]
a | s ) (cid:17) yield the importance sampling weights (4).
Moreover, for any s (cid:48) ∈ S
[Equation]
=and therefore [-α ( x, a ) ≥ 
-](6) holds whenever 0 ≤ c ≤ ρ .
Now the result follows by Theorem 1.
Observe that when ρ is inﬁnite, the ﬁxed point V ˜ π corresponds to the target policy π .
On the other hand, when ρ tends to 0, V ˜ π gets close to the value of the behavioral policy µ .
In the general case, when ρ is ﬁnite but positive, the ﬁxed point is the value function of a policy located somewhere between π and µ .
However, ˜ π does not depend on c t – these weights affect the speed of convergence only [7].
Remark 4.
There is a theoretical difference between MA-Trace (full) and MA-Trace (obs), which perhaps is subtle in some cases.
The Markov property is a key element required in the proof of
Corollary 3.
While it is by deﬁnition true for MA-Trace (state) it might fail for MA-Trace (obs) - if the concatenated observations do not provide a sufﬁcient statistic of s t .
For the actor network we use policy gradient updates.
Here we also need importance sampling to correct for using the off-policy behavioral policy µ .
We recall the factorization (1); analogously we denote joint decentralized parameterized policy π ω ( a 1 , . .
. , a K | s ) = (cid:81)
Kk =1 π ω ( a i | s i ) .
The policy gradient theorem suggests the ascent in the direction:
[Equation]
where a t = ( a t, 1 , . .
. , a t,K ) and A π ω is some advantage estimator.
For off-policy data collected with µ we have
[Equation]
where the equality holds if c = + ∞ in (4).
This formula leads to the practical Monte-Carlo estimator used in line 14 of Algorithm 1:
[Equation]
Experiments 5.1 Environment
We evaluate MA-Trace on StarCraft Multi-Agent Challenge (SMAC) [28] (version 4.10), which is a standard benchmark for multi-agent algorithms, based on a popular real-time strategy game
StarCraft II.
It provides 14 micromanagement tasks of varying difﬁculty and structure.
The aim is to win a battle against a built-in AI by using your team of agents.
In easier tasks, often rudimentary coordination is enough. However, harder tasks involve engaging a stronger enemy (e.g., having more units), which requires inventing smart techniques and tricks.
Each unit has a limited line of sight, which makes the environment partially observable.
We provide more details in Appendix F.
5.2 Main result
In Table 1 and Figure [Table] we present the results of the main version of our algorithm – MA-Trace (obs), in which the critic uses stacked observations of agents as described in Appendix F. MA-Trace (obs) reaches competitive results and in some cases exceeds the current state-of-the-art.
We compare with a selection of the state-of-the-art algorithms on SMAC following [41] and [26].
We also demonstarte scalability, which lets MA-Trace can reach good performance even using short training (wall time).
[Table]
5.3 Training details
We use standard feed-forward networks for the actor and critic networks with two hidden layers ofneurons and ReLU activations.
The critic network of MA-Trace (obs) takes stacked observations of agents as input, while MA-Trace (full) utilizes the full state provided by SMAC.
DecMA-Trace have a critic using single-agent observations.
See details in Appendix C.
For each reported version of MA-Trace, we have searched for the best hyperparameters to ensure a fair comparison.
The values of all hyperparameters are listed in Appendix D.2.
To estimate the performance of MA-Trace we run training for 3 days or until convergence.
We report the median win rate of 10 runs (with different random seeds) along with the interquartile range.
Training curves for all the tasks can be found in Appendix G.
5.4 Ablations
Below we present a comprehensive list of ablations to evaluate the design choices of our algorithm.
In each case, we present training curves for tasks, which best illustrate our claims.
For the complete training results and more details, we refer to Appendix E.
Advantage of using importance sampling.
Using the importance weights is the key algorithmic innovation of MA-Trace (and V-Trace), responsible for the strong performance we report.
Indeed, already for 30 actor workers, using the weights is essential.
Otherwise, the algorithm is unstable and suffers from poor asymptotic performance.
See Figure 2 and Appendix E.2.
Training scaling.
The importance sampling enables V-Trace to be truly scalable in multi-node setups.
MA-Trace enjoys the same property.
Importantly, we do not observe any degradation in the training performance when trained in the multi-node setup.
See Figure 3 and Appendix E.3.
Input for the critic network.
We found that MA-Trace (full) performs slightly worse than MATrace (obs).
Usually the differences are small.
However, in two harder tasks, corridor and 6h_vs_8z , MA-Trace (full) learns much slower and often fails.
This is perhaps surprising, as the full state contains additional information (e.g., about invisible opponents).
To deepen the analysis, we ran MA-Trace (obs+full), which uses both the observations and full state as the critic input.
This improves the results, though they are still slightly inferior to MA-Trace (obs); see Figure 4 and Appendix G.
A more detailed discussion of this topic can be found in Appendix E.1.
Centralized vs decentralized.
As noted by [20], centralized training in some cases may suffer from higher variance.
Therefore we compared MA-Trace with its decentralized version (i.e. having indpendent critics for each agent).
The latter typically obtains weaker results and is less stable.
See Figure 5 and details in Appendix B.
Sharing actors networks.
We follow a common approach of sharing the policy network between agents.
In some works, e.g., [26], to preserve individuality, the observations are enriched with agent ID.
This might be beneﬁcial if agents should be assigned different roles within the team.
However, we ﬁnd these beneﬁts rather minor and opt for input provided by the environment (i.e., without ID).
See Figure 7 and details in Appendix C.2.
One can also use separate networks for each agent.
We check that MA-Trace works considerably worse in such a case.
In rare cases, using separate networks is advantageous, but only in the easiest tasks, e.g., 3s5z .
See Figure 6 and details in Appendix C.1.
Limitations and further work
We show that MA-Trace successfully solves SMAC tasks.
Further benchmarking is needed to underpin its quality.
This includes testing on more environments, both fully cooperative (like SMAC) and competitive.
The latter might require further algorithmic developments.
The importance sampling weights successfully reduce distributional shifts arising in distributed training.
An interesting question is whether they can also reduce the shifts introduced by the non-stationarity of the multi-agent environment.
MA-Trace exhibits lower sample efﬁciency than the other methods we used for comparisons.
This, at least partially, can be explained by its on-policy nature.
Adapting the importance correction to accommodate more off-policy data would be an important achievement.
Conclusions
In our work, we introduced MA-Trace, a new multi-agent reinforcement learning algorithm.
We evaluated it on 14 scenarios constituting the StarCraft Multi-Agent Challenge and conﬁrmed its strong performance.
We also included ablations regarding importance sampling, centralization of learning, scaling, and sharing of parameters.
Thanks to the use of importance weights, MA-Trace is highly scalable.
Furthermore, the convergence properties of our algorithm highlighted by Theorem 1 show that it has not only experimental but also mathematical grounding.
Acknowledgments and Disclosure of Funding
We thank Konrad Staniszewski for discussions and experiments in the prequel of this project. The work of Michał Zawalski was supported by the Polish National Science Center (NCN) grant UMO-2019/35/O/ST6/03464. The work of Henryk Michalewski was supported by the Polish National
Science Center grant UMO-2018/29/B/ST6/02959.
The work of Piotr Miło´s was supported by the
NCN grant UMO-2017/26/E/ST6/00622.
This research was supported by the PL-Grid Infrastructure.
Some experiments were performed using the Entropy cluster funded by NVIDIA, Intel, the Polish
National Science Center grant UMO-2017/26/E/ST6/00622 and ERC Starting Grant TOTAL.
Our experiments were managed using https://neptune.ai .
We would like to thank the Neptune team for providing access to the team version and technical support.
Broader societal impact We develop a generic optimization algorithm for multi-agent reinforcement learning; thus, we do not foresee any negative societal impact arising explicitly from this work.
A The proof of the ﬁxed point theorem
Let C K ∞ be the space of functions S (cid:55)→ R K endowed with the inﬁnity norm (cid:107) f (cid:107) C K ∞ =
sup s ∈S | f ( s ) | ∞ .
C K ∞ is Banach if S is ﬁnite or we additionally assume that functions are continuos (this encompasses the case when S is discrete).
For an easy reference we recall the statement of Theorem 1.
Let c : S ×
A → R + , ρ : S ×
A → R + be measurable functions and set c t
= c ( x t , a t ) , ρ t = ρ ( x t , a t ) .
Deﬁne the operator R : C K ∞ (cid:55)→ C K ∞ by
[Equation]
Theorem 5.
Let c, ρ be such that for any s ∈ S , a ∈ A
[Equation]
where s (cid:48) is the state obtained from s after issuing action a .
Assume also that E µ ρ 0 ≥ β ∈ (0 , 1] .
Then the operator R is C K ∞ contraction with a unique ﬁxed point V ˜ π which is a value function of a policy ˜ π given by
[Equation]
The contraction constant η is smaller than 1 − (1 − γ ) β < 1 .
Remark 6.
We note that the proof [7, Theorem 1] has some glitches.
A careful examination reveals that the analysis of the ﬁrst displayed equation on page 12 in [7] is not correct.
We ﬁx it by making introducing ﬁltration and analyzing the measurability of explicitly and argue that such an approach makes the proof more clear
Proof.
Let ˜ c t := (cid:81)
tu =
c u with the convention ˜ c t = 1 for t ≤ 0 .
We use the same convention for ρ s .
It is convenient to express R in the following form
[Equation]
Fix V 1 , V 2 ∈ C K ∞ and put C K ∞ (cid:51)
∆ V :=
V 1 − V 2 .
For s ∈ S we write
[Equation]
We deﬁne ﬁltration {F t } + ∞ t =0 by
F t := σ (( x 0 , a 0 , . . . , a t − 1 , x t )) .
We rewrite in terms of conditional expectation
[Equation]
Clearly, all terms except for ρ t +1 are F t +1 measurable, thus
[Equation]
Recall (10) and observe that ρ t − c t E µ ( ρ t +1 |F t +1 ) = α
[Equation]
s t , a t )
=:
α t +1 .
Let us also put by convention α
= 1 − E µ ρ 0 .
Thus shifting indices we get
[Equation]
By our assumptions we have ˜ c t , α t ≥ 0 and thus we get
[Equation]
We are left with rather straightforward calculations (assume by convention that ρ − 1 = 1 ).
[Equation]
The last inequality follows by dropping all summands except t = 0 .
Now, we are to determine the unique ﬁxed point.
Recall (11), we calculate
[Equation]
where c = (cid:80) b ∈A ρ ( s, b ) µ ( b | s ) is the normalizing constant and the second equality hold by the
Bellman equation.
B Centralized training and decentralized execution
In a multi-agent problem, the agents take actions based on their local observations x i , according to their policies π i .
Decentralized training is a simple approach of learning π i , using any (single-agent)
RL algorithm separately for each agent.
To better utilize the structure of a multi-agent problem, a paradigm of centralized training and decentralized execution was introduced.
The key idea is to centralize the learning process of all the agents – use shared knowledge provided by observations and any other information available during training to more effectively optimize the decentralized policies.
It is now considered a leading paradigm and is usually chosen off-the-shelf.
Recently the authors of [20] suggested that this should be reinvestigated.
Sharing the knowledge in centralized training is indeed beneﬁcial, but the paradigm of independent training also has advantages.
The multi-agent problems lack the strong theoretical guarantees associated with the standard case.
For example, if we train all the agents simultaneously, the environment changes during training from every agent’s perspective.
Such non-stationarity can destabilize standard algorithms.
According to [20], decentralized learning partially mitigates this issue – for every agent, a separate value network is trained; thus it averages much of the stochasticity in the environment, producing more stable estimates.
On the other hand, because of the limited information, these values are less accurate.
Therefore choosing between centralized and decentralized training is a tradeoff.
To address these issues, we compared MA-Trace with its decentralized version, DecMA-Trace.
However, in the StarCraft Multi-Agent Challenge, the decentralized version learns much slower and fails to reach good performance; see Figure 8.
C Networks’ architectures
In our experiments, we use feed-forward networks with two hidden layers of 64 neurons and ReLU activations (without normalization).
This is smaller than the networks used in benchmark [40] and [26], but we found this sufﬁcient to obtain good results.
In this work, we focus on algorithmic aspects rather than tuning architectures.
However, for completeness, we include a discussion of some most popular design choices.
C.1 Sharing parameters
Our default (and most effective) scheme uses a single shared network for the actors and a separate one for the critic.
We experimented with sharing feature extractors between the agents and the critic, which performed worse.
We also checked the performance when training separate networks for all the components.
Interestingly, with this approach, the learning is much faster on the easy tasks, such as 3s5z or 2c_vs_64zg , but completely fails on the hardest ones; see Figure 9.
C.2 ID experiments
Sharing the agents’ networks, as described in Section C.1, can lead to poor results if agents are not homogenous (for example, the behavior of shooting units should differ from melee units).
This might be circumvented by adding units’ characteristics (which is a default in SMAC) or enriching the observations with one-hot encoded ID.
In Figure 10 we present the results of experiments in which we use these two mechanisms.
We observe some improvements, which are, however, relatively minor.
C.3 Agents with memory
The units in StarCraft Multi-agent Challenge have limited sight range, thus the environment is partially observable.
A common approach to mitigate such an issue, is to give the agents memory, e.g. by using recurrent networks or frame stacking.
The authors of [26] show that using recurrent policy is required to achieve good results in the hardest tasks.
However, our experiments show that memoryless policies can be successfully trained to solve all the tasks (possibly except 3s_vs_5z , see
Section F.1).
To implement a simple memory it is enough to pass a few previous observations concatenated (“framestack”).
We experimented with both observation-based and state-based critics.
Figures [-
 and12-]11 and 12 show the progress of training memory-equipped agents.
One can observe hardly any difference in most tasks.
On the easy task 2c_vs_64zg , using memory leads to faster training.
What is particularly interesting, on the corridor , task the stacked versions learn much faster.
This may be related to the strategy learned by MA-Trace, in which at some point a group of units has to hide and wait (see
Section F.2 for detailed description).
Execution of this strategy is probably easier when using at least short memory.
However, on some other tasks, such as 5m_vs_6m , training with memory is much less stable and effective.
D Training details D.1 SEED RL
We base our code on SEED RL [8], which is an open-source framework for distributed learning, licensed under the Apache License, Version 2.0.
This library provides an implementation of V-Trace for single-agent environments.
D.2 Hyperparameters
In our experiments the parameters crucial to the ﬁnal performance were: learning rate and entropy cost .
For other hyperparameters, we use the default values provided by SEED.
The most relevant, common for all the experiments, are listed in Table 2.
We searched for the best learning rate in the set [3 . 5 · 10 − 3 , 2 . 5 · 10 − 3 , 1 . 5 · 10 − 3 , 10 − 3 , 7 · 10 − 4 , 5 · 10 − 4 , 3 . 5 · 10 − 4 ] .
We experimented with adding learning rate schedules with warmup and decay, though no such scheme appeared to be beneﬁcial.
Another hyperparameter that has a strong inﬂuence on the ﬁnal results is entropy cost .
By default,
SEED sets it to a constant value of 2 . 5 · 10 − 4 and allows to use annealing.
We found that aggressive exploration at the beginning is crucial to reach good results.
We annealed entropy cost from 1 towards 10 − 5 .
The speed of adjustment was tuned in the set [10 , 5 , 2 . 5 , 1 , 0 . 5] .
In Table 3, we show the best parameters for our experiments, found by the above grid-search.
Unlike in some other works, we found no advantage of using gradient clipping; thus, we leave the gradients not clipped.
D.3 Infrastructure used
The typical conﬁguration of a computational node used in our experiments was: the Intel Xeon
E5-2697 2 .
60 GHz processor with 128 GB memory.
On a single node, we ran one experiment with 30 workers.
A typical experiment was run for about 20 h. For the ﬁnal evaluation, we extended the training to 3 days, which is usually equivalent to about 3 · 10 8 environment steps.
We did not use
GPUs; we found that with the relatively small size of the network it offers only slight wall-time improvement while generating substantial additional costs.
[Table]
[Table]
E Ablations E.1 Critic comparison
During centralized training, the critic network in MA-Trace algorithm uses any information available.
A natural choice is to aggregate the observations available to the agents, and we denote this version as MA-Trace (obs).
This might not be a sufﬁcient statistic of the (Markov) state of the environment.
SMAC provides additional access to such a state description, which we use in MA-Trace (full).
Intuitively, using complete information should be advantageous.
As shown in Figure 16, on most tasks, both the versions do not differ much.
On 3s5z_vs_3s6z there is a small advantage on the full-state side.
However, MA-Trace (full) shows little progress on 6h_vs_8z and corridor , as opposed to MA-Trace (obs).
Therefore we consider MA-Trace (obs) to be our default version.
Such behavior is a bit counter-intuitive.
We speculate that some information available in the agents’ observation is not easily accessible (computable) for the full state.
To verify this, we compared the two versions with another, MA-Trace (obs+full), which uses both the aggregated observations and the full state.
As we can see in Figure 16, it trains similarly to MA-Trace (obs), without signiﬁcant advantage on any task.
Moreover, on the hardest tasks, the training progresses a bit slower.
We leave a precise explanation of this behavior as future work.
E.2 Importance sampling ablation
We claim that the strong performance of MA-Trace is due to using importance weights.
To verify this statement, we compared MA-Trace with its ablation without importance weights.
The training curves for all the tasks are shown in Figure 13.
Without the corrections, the training is unstable and reaches good performance only on the easiest tasks.
The difference gets even larger when using more compute workers.
One notable exception is the corridor task, in which the ablated version trains faster than MA-Trace, though eventually both reach similar results.
E.3 Scaling experiments
MA-Trace utilizes importance sampling to reduce the shifts arising naturally in distributed training (when behavioral policies lag behind the target policy).
This allows our algorithm to be highly scalable.
Our experiments conﬁrm that MA-Trace scales almost linearly on at least 70 workers.
As shown in Section E.2, the importance weights indeed enable stable training, even when distributed to many workers.
What is equally important, our experiments show that scaling does not affect the learned policies signiﬁcantly.
F StarCraft Multi-Agent Challenge
To evaluate MA-Trace, we use StarCraft Multi-Agent Challenge (SMAC)
It is a standard benchmark for multi-agent algorithms, used e.g. by [26, 11, 40] and many others.
The challenge is based on a popular real-time strategy game StarCraft II and consists of 14 micromanagement tasks.
Each task is a small arena in which two teams, controlled by the player by the built-in AI, ﬁght against each other.
The goal in every task is to defeat (kill) all the enemy units.
Units belong to one of the three races: Protoss, Terran or Zerg.
Additionally, they are divided on a number of classes with unique characteristic, such as speed, shooting range, ﬁre power etc.
In each turn they can move or attack an enemy in their shooting range.
A unit is considered defeated if its health drops to 0 .
A defeated unit remain inactive.
SMAC provides a variety of different tasks.
In the easier tasks, the opponents control the same forces.
Therefore to win such a game, it is enough to coordinate slightly better than the built-in AI.
In the harder tasks, however, the computer starts with a stronger squad.
This can be a minor advantage, such as in the task 10 marines vs 11 marines , or quite a big difference.
In particularly hard scenarios, such as corridor , it is unreasonable for the player to engage in an open ﬁght, so it is essential to develop a long-term strategy to obtain a positional advantage.
For the hardest tasks, the authors of the challenge consider the so-called microtricks sufﬁcient to win consistently.
However, it appears that MA-Trace in some cases develop unexpected strategies, see e.g. Section F.2.
What is particularly interesting, our algorithm manages to learn some techniques associated with professional human players, such as focusing ﬁre, withdrawing low-health units, hit-and-run, sacriﬁcing a unit to deceive the opponent and others.
Units in the game have limited sight range, which makes the environment partially observable.
The observations received by individual agents contain information about all the visible units (including themselves) – their health, energy, position, class, and other relevant features.
All the units beyond the sight range are marked as dead (i.e., defeated and invisible units are not distinguished).
It is possible that the aggregated observations do not provide full information, for there can be enemy units hidden beyond the sight range of any ally.
Therefore, to facilitate centralized training, SMAC provides additional access to the full state of the environment.
Learning from binary reward (win/lose) is prohibitively hard in most tasks.
Therefore SMAC provides dense rewards to enable training.
The team receives additional points for damage dealt and defeating a unit.
This scheme is arguably natural and leads to successful training.
However, in some cases, it might reinforce undesired behaviors, see, e.g., Section F.1.
F.1 3s_vs_5z task
MA-Trace masters all the tasks except 3s_vs_5z , see Table 1.
In that scenario, we control 3 Stalkers ﬁghting against 5 Zealots.
Stalkers can attack the enemy from a distance; however, they are no match for Zealots in close combat.
A strategy to gain an advantage is to shoot the enemies while they are away and ﬂee when they get close.
All the units in this task are Protoss, so they all have protective shields.
The shields absorb some amount of damage, until they are down.
However, as opposed to regular health, the shields regenerate slowly with time.
As dealing damage yields rewards, it might be beneﬁcial to keep enemy alive inﬁnitely.
Apparently, MA-Trace learns to this strategy.
Figure 14 shows an example of the winning rate and episode rewards.
As we can see, after a short training, our algorithm wins almost every time.
Further, it discovers that the reward scheme can be exploited – at some point, the mean return increases fast, while the actual win rate decreases and becomes unstable.
F.2 Corridor task
Another super-hard scenario (according to [26]) is corridor .
In this task, we control a team of[-
-]
Zealots against 24 enemy Zerglings.
Though Zealots are far more powerful in combat, they are outnumbered; thus, the open ﬁght is clearly an unreasonable strategy.
However, the ﬁghting arena contains a narrow passage.
The authors of SMAC suggest that a winning strategy is to gather the forces in that passage, where the number of enemy units should lose its importance (possibly inspired by the Battle of Thermopylae).
However, MA-Trace develops an alternative interesting strategy.
Firstly, our forces split into two groups.
One (strong) hides in the corner, where it easily defeats a few enemies, while the other (one or two units) attracts majority of the enemies to the other side and sacriﬁce itself.
After defeating the second group, the enemies pass to the far side of the arena and wait, unaware of the hidden group.
Then the strong group attack them from behind and defeat the Zerglings one by one.
The strategy is outlined in Figure 15.
[Figure]
G Full training data
In this section, we provide the main practical results of our work – complete training data for all standard versions of MA-Trace on all the SMAC tasks.
They were trained for three days or until convergence.
We report the median win rates and interquartile ranges.