RPM: G ENERALIZABLE B EHAVIORS FOR M ULTIA GENT R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Despite the recent advancement in multi-agent reinforcement learning (MARL), the MARL agents easily overfit the training environment and perform poorly in the evaluation scenarios where other agents behave differently.
Obtaining generalizable policies for MARL agents is thus necessary but challenging mainly due to complex multi-agent interactions.
In this work, we model the problem with Markov Games and propose a simple yet effective method, ranked policy memory (RPM), to collect diverse multi-agent trajectories for training MARL policies with good generalizability.
The main idea of RPM is to maintain a look-up memory of policies.
In particular, we try to acquire various levels of behaviors by saving policies via ranking the training episode return, i.e. , the episode return of agents in the training environment; when an episode starts, the learning agent can then choose a policy from the RPM as the behavior policy.
This innovative self-play training framework leverages agents’ past policies and guarantees the diversity of multi-agent interaction in the training data.
We implement RPM on top of
MARL algorithms and conduct extensive experiments on Melting Pot.
It has been demonstrated that RPM enables MARL agents to interact with unseen agents in multi-agent generalization evaluation scenarios and complete given tasks, and it significantly boosts the performance up to 402% on average.
I NTRODUCTION
Recent years have witnessed considerable progress in Multi-Agent Reinforcement Learning (MARL) research (Yang & Wang, 2020).
In MARL, each agent acts decentrally and interacts with other agents to complete particular tasks or achieve specific goals via reinforcement learning (RL).
However, generalization (Hupkes et al., 2020) remains a critical issue in MARL research.
Generalization may have different meanings from different perspectives and research directions.
Such as generalization at the representation level studied in supervised learning and generalization to different test environments rather than the training environments for a single agent (Kirk et al., 2021).
In this work, we study the the generalization ability of agents to collaborate or compete with other agents with unseen policies during training.
Such a setup is critical to real-world MARL applications (Leibo et al., 2021).
Unfortunately, current MARL methods mostly neglect generalization issues and could be fragile; for example, a co-player changing its policy may cause the trained agent to fail to cooperate.
In this work, we aim to train MARL agents that can adapt to new scenarios where other agents’ policies are unseen during training for MARL.
To understand the difficulties and why it is crucial to tackle generalization, we illustrate a two-agent stag-hunt game as an example in Figure 1.
The agents are trained to obtain policies maximizing the group reward by shooting arrows to the stag.
As a result, they may perform well in evaluation scenarios similar to the training environment, as shown in Figure 1 (left) and (middle), respectively.
However, these agents may fail when evaluated in scenarios different from the training scenarios.
As shown in Figure 1 (right), the learning agent (called the focal agent following the convention in (Leibo et al., 2021)) is supposed to work together with another agent (called the background agent following the naming in (Leibo et al., 2021)) are pre-trained to be selfish ( i.e. , only capture the hare).
In this case, the focal agent will fail to capture the stag without the help from its teammates and the optimal policy to capture the hare.
However, background agents are unseen to the focal agent during training.
Therefore, without generalization, the agents trained as Figure 1 (left) cannot achieve an optimal policy in the new evaluation scenario.
[Figure]
We model the problem with Markov games (Littman, 1994) and propose a simple yet effective method called ranked policy memory (RPM) to attain generalizable policies in multi-agent systems during training.
The core idea of RPM is to maintain a look-up memory of policies during training for the agents.
In particular, we first evaluate the trained agents’ policies after each training update.
We then rank and save the trained agents’ policies by the training episode returns.
In this way, we obtain various levels, i.e. , performances, of policies.
When starting an episode, the agent can access the memory and load the randomly sampled policy to replace the current behavior policy.
The new ensemble of policies enables agents in the self-play framework to collect diversified experiences in the training environment for training.
These diversified experiences contain many novel multiagent interactions that enhance the extrapolation capacity of MARL, boosting the generalization performance.
We note that an easy extension to having different behavior properties as the keys in
RPM could potentially further enrich the generalization but it is left for future work.
We implement RPM on top of state-of-the-art MARL algorithm, MAPPO (Yu et al., 2021).
To verify its effectiveness, we use Melting Pot (Leibo et al., 2021) as our testbeds.
We then conduct large-scale experiments with the Melting Pot benchmark, which is a well-recognized benchmark for MARL generalization evaluation.
The experiment results demonstrate that RPM significantly boosts the performance of generalized social behaviors up to 402% on average and outperforms many baselines in a variety of multi-agent generalization evaluation scenarios.
Our code, pictorial examples, and videos are available at this link: https://sites.google.com/view/rpm-iclr2023/ .
P RELIMINARIES 2. M ARKOV G AMES AND N OTATIONS
We consider the Markov Games (Littman, 1994) represented by a tuple G = ⟨N , S , A , O , P, R, γ, ρ ⟩ .
N is a set of agents with the size |N| =
N ; S is a set of states; A = × Ni =
A i is a set of joint actions with A i denoting the set of actions for an agent i ; O = ×
Ni =
O i is the observation set, with O i denoting the observation set of the agent i ; P : S ×
A → S is the transition function and
R = ×
Ni =1 r
i is the reward function where r i : S ×
A → R specifies the reward for the agent i given the state and the joint action; γ is the discount factor; the initial states are determined by a distribution ρ : S → [0 , 1] .
Given a state s ∈ S , each agent i ∈ N chooses its action u i and obtains the reward r ( s, u ) with the private observation o i ∈
O i , where u = { u i } Ni =1 is the joint action.
The joint policy of agents is denoted as π θ =
{ π θ i } Ni =
where π θ i : S
×
A i → [0 , 1] is the policy for the agent i .
The objective of each agent is to maximize its own total expected return R i =
(cid:80) ∞ t =
γ t r ti .
2. M ULTI -A GENT R EINFORCEMENT L EARNING
In MARL, multiple agents act in the environment to maximize their respective returns with RL (Sutton & Barto, 2018).
Each agent’s policy π i is optimized by maximizing the following objective:
[Equation]
where J ( π i ) is a performance measure for policy gradient RL methods (Williams, 1992; Lillicrap et al., 2016; Fujimoto et al., 2018).
Each policy’s Q value Q i is optimized by minimizing the 𝜋 ! !
𝜋 ! "
𝜋 !
# 𝜋 !
$ 𝜋 !
𝜋 ! & 𝝅 𝜽
Substrate
MARL Training
I n t e r a c t i o n
Collect data
Update 𝜋 ! !
𝜋 ! "
𝜋 !
# 𝜋 ! $ 𝜋 " !
#$% &
Substrate
Data Buffer
I n t e r a c t i o n
Collect data 𝜋 " !
𝜋 " "
Scenario
Multi-Agent Training
Multi-Agent Evaluation policies background agents following regression loss (Watkins & Dayan, 1992; Mnih et al., 2015) with TD-learning:
[Equation]
where y it =
r it + γ max u ′ Q
i ¯ θ i (cid:0) s t
+1 , u ′ , s it , u i, ′ (cid:1) .
θ i are the parameters of the agents.
¯ θ i is the parameter of the target Q i and periodically copied from θ .
D ′ is a sample from the replay buffer D .
P ROBLEM F ORMULATION
We introduce the formulation of MARL for training and evaluation in our problem.
Our goal is to improve generalizabiliby of MARL policies in scenarios where policies of agents or opponents are unseen during training while the physical environment is unchanged.
Following Leibo et al.
(2021), the training environment is defined as substrate .
Each substrate is an N -agent partially observable
Markov game G .
Each agent optimizes its policy π θ i via the following protocol.
Definition 1 (Multi-Agent Training) .
There are N agents act in the substrate, which is denoted as G .
Each agent receives partial environmental observation not known to other agents and aims to optimizes its policy π θ i by optimizing its accumulated rewards: (cid:80)
∞ t =
γ t r it .
The performance of the joint policy π θ = { π θ i } Ni =1 is measured by the mean individual return: ¯ R ( π θ )
= 1 N (cid:80) Ni =1 R ( π θ i ; G ) .
R ( π θ i ; G ) measures the episode return of policy π θ i in game G for agent i .
In order to evaluate the trained MARL policies in evaluation scenario G ′ , we follow the evaluation protocol defined by Leibo et al.
Definition 2 (Multi-Agent Evaluation) .
There are M (1 ≤ M ≤ N − 1) focal agents that are selected from N agents.
The focal agents are agents to be evaluated in evaluation scenarios.
They are paired with N − M background agents whose policies π ϕ =
{ π ϕ j } N − M j = were pre-trained with pseudo rewards in the same physical environment where the policies π θ are trained.
To measure the generalized performance in evaluation scenarios, we use the mean individual return of focal agents as the performance measure: ¯ R ( { π θ } Mi =1 )
= 1 M (cid:80) Mi
=1 R ( π θ i ; G ′ ) .
We show an example of our formulation in Figure 2.
Note that the focal agents cannot utilise the interaction data collected during evaluation to train or finetune their policies.
Without training the policies of focal agents with the collected trajectories during evaluation, the focal agents should behave adaptively to interact with the background agents to complete challenging multi-agent tasks.
It is also worth noting that the ad-hoc team building (Stone & Kraus, 2010; Gu et al., 2021) is different from our formulation both in the training and evaluation.
We discuss the differences in the related works section (Paragraph 3, Section 6).
M ETHODOLOGY
We propose a Ranked Policy Memory (RPM) method to provide diversified multi-agent behaviors for self-play to improve generalization of MARL.
Then, we incorporate RPM into MAPPO (Yu et al., 2021) for training MARL policies.
[Figure]
𝜋 !
[Figure]
𝜋
[Figure]
Sampled
[Figure]
𝜋 ! "
[Figure]
𝜋 $
[Figure]
Pretrained policy
Background
[Figure]
data
[Figure]
MARL
[Figure]
Multi
[Figure]
Agent Evaluation
[Figure]
4. RPM: R ANKED P OLICY M EMORY
In MARL, the focal agents need adaptively interact with background agents to complete given tasks.
Formally, we define the objective for optimizing performance of the focal agents without exploiting their trajectories in the evaluation scenario for training the policies { π θ j } Mj =1 :
[Equation]

[Equation]
t
[Equation]
γ
[Equation]
M
[Equation]
j
[Equation]
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) G
[Equation]
  .
To improve the generalization performance of MARL, it is crucial for agents in the substrate to cover as much as multi-agent interactions, i.e. , data, that resemble the unseen multi-agent interactions in the evaluation scenario.
However, current training paradigms, like independent learning (Tampuu et al., 2017) and centralized training and decentralized execution (CTDE) (Oliehoek et al., 2008), cannot give diversified multi-agent interactions, as the agents’ policies are trained at the same pace.
To address this issue, we propose to gather massive diversified agent-agent interaction data for multi-agent learning.
The diversified agent-agent interaction data are generated by agents that have different ranks of policies.
Concretely, we maintain a look-up memory during training, where each entry is a key and the corresponding value is a list.
We take the training episode return of the agents’ policies as the key, and use the list to store policies of the agents evaluated.
When agents in the substrate start a new episode, there is a probability p to replace all agents’ behavior policies with the sampled policies from the memory.
This method is termed Ranked Policy Memory (RPM).
Below we introduce how to build it and how to sample policies from it.
RPM Building & Updating.
We denote an RPM with Ψ , which consists of | R max | entries, i.e. , ranks, where | R max | is the maximum training episode return (the episode return in the substrate).
While agents are acting in the substrate, the training episode return R of all agents (with policies { π iθ } Ni =1 ) is returned.
Then { π iθ } Ni =1 are saved into Ψ by appending agents’ policies into the corresponding memory slot, Ψ [ r e ] .
add( { π ie } Ni =1 ) .
To avoid there being too many entries in the policy memory caused by continuous episode return values, we discretize the training episode return.
The discretized entry κ covers a range of [ κ, κ + ψ ) , where ψ > 0 and it can be either an integer or a float number.
For the training episode return R , the corresponding entry κ can be calculated by:
[Equation]
where 1 {·} is the indicator function, and ⌊·⌋ is the floor function.
Intuitively, discretizing R saves memory and memorize policies of similar performance in to the same rank.
Therefore, diversified policies can be saved to be sampled for agents.
RPM Sampling.
The memory Ψ stores diversified policies with different levels of performance.
We can sample various policies of different ranks and assign each policy to each agent in the substrate to collect multi-agent trajectories for training.
These diversified multi-agent trajectories can resemble trajectories generated by the interaction with agents possessing unknown policies in the evaluation scenario.
At the beginning of an episode, we first randomly sample N keys with replacement and then randomly sample one policy for each key from the corresponding list.
All agents’ policies will be replaced with the newly sampled policies for multi-agent interactions in the substrate, thus generating diversified multi-agent trajectories.
MARL with RPM.
We showcase an example of the workflow of RPM in Figure 3.
There are three agents in training.
Agents sample policies from RPM.
Then all agents collect data in the substrate for training.
The training episode return is then used to update RPM.
During evaluation, agents
1 and 2 are selected as focal agents and agent 3 is selected as the background agent.
We present the pseudo-code of MARL training with RPM in Algorithm 1.
In Lines 4-5, the π θ b is updated by sampling policies from RPM.
Then, new trajectories of D are collected in Line 6.
π θ is trained in
Line 7 with MARL method by using the newly collected trajecotries and π θ b is updated with the newly updated π θ .
RPM is updated in Line 8.
After that, the performance of π θ is evaluated in the evaluation scenario G ′ and the evaluation score ¯ R is returned in Line 9.
[Table]
do
[Table]
if
[Table]
sampling then
[Table]
GatherTrajectories
[Table]
MARLTrainig
[Table]
D )
[Table]
Evaluate
[Table]
Discussion.
RPM leverages agents’ previously trained models in substrates to cover as many patterns of multi-agent interactions as possible to achieve generalization of MARL agents when paired with agents with unseen policies in evaluation scenarios.
It uses the self-play framework for data collection.
Selfplay (Brown, 1951; Heinrich et al., 2015; Silver et al., 2018; Baker et al., 2019) maintains a memory of the opponent’s previous policies for acquiring equilibria.
RPM differs from other self-play methods in four aspects: (i) self-play utilizes agent’s previous policies to create fictitious opponents when the real opponents are not available.
By playing with the fictitious opponents, many fictitious data are generated for training the agents.
In RPM, agents load their previous policies to diversify the multi-agent interactions, such as multi-agent coordination and social dilemmas, and all agents’ policies are trained by utilizing the diversified multi-agent data.
(ii) Self-play does not maintain explicit ranks for policies while RPM maintains ranks of policies.
(iii) Self-play was not introduced for generalization of MARL while RPM aims to improve the generalization of MARL.
In
Section 5, we also present the evaluation results of a self-play method.
4. MARL T RAINING
We incorporate RPM into the MARL training pipeline.
We take MAPPO (Yu et al., 2021) for instantiating our method, which is a multi-agent variant of PPO (Schulman et al., 2017) and outperforms many MARL methods (Rashid et al., 2018; 2020; Wang et al., 2021a) in various complex multi-agent domains.
In MAPPO, a central critic is maintained for utilizing the concealed information of agents to boost multi-agent learning due to non-stationarity.
RPM introduces a novel method for agents to collect experiences/trajectories τ
= { τ i } Ni =1 .
Each agent optimizes the following objective:
[Equation]
where η ti ( θ ti ) =
π θti
[Equation]
π θ old i ( u ti | τ ti ) denotes the important sampling weight.
The clip ( · ) clips the values of θ i that are outside the range [1 − ϵ, 1 + ϵ ] and ϵ is a hyperparameter.
A t i is a generalized advantage estimator (GAE) (Schulman et al., 2015).
To optimize the central critic V ψ ( { o ti , u ti } Ni =1 ) , we mix agents’ observation-action pairs and output an N -head vector where each value corresponds to the agent’s value:
[Equation]
where y t = (cid:104)(cid:80)
k − 1 l =
γ l r t + l i + γ k V ¯ ψ ( { o t + k i , u t +
[Equation]
i } Ni =1 )
[ i ] (cid:105) N i
=1 is a vector of k -step returns, and D ′ is a sample from the replay buffer D .
In complex scenarios, e.g. , Melting Pot, with an agent’s observation as input, its action would not impact other agents’ return, since the global states contain redundant information that deteriorates multi-agent learning.
We present the whole training process, the network architectures of the agent and the central critic in Appendix D.
E XPERIMENTS
In this section, to verify the effectiveness of RPM in improving generalization of MARL, we conduct extensive experiments on Melting pot and present the empirical results.
We first introduce Melting
Pot, baselines and experiments setups.
Then we present the main results to show the superiority
[Figure]
✗ ✗ ✓ ✗ ✗ ✗
[Figure]
✓ ✓ ✓ ✓ ✗ ✓
[Figure]
✓ ✗ ✓ ✓ ✗ ✓
[Figure]
✗ ✗ ✓ ✗ ✗ ✗
[Figure]
✓ ✓ ✓ ✗ ✓ ✓
[Figure]
✗ ✗ ✓ ✓ ✗ ✗
[Figure]
✓ ✗ ✗ ✗ ✗ ✓
[Figure]
✗ ✗ ✓ ✗ ✗ ✗ of RPM.
To demonstrate that ψ is important for RPM, we conducted ablation studies.
We finally showcase a case study to visualize RPM.
To sum up, we answer the following questions: Q1 :
Is RPM effective in boosting the generalization performance of MARL agents?
Q2 :
Does the value of ψ matter in RPM for training?
Q3 :
Does RPM gather diversified policies and trajectories?
5. E XPERIMENTAL S ETUP
[Figure]
Melting Pot.
To demonstrate that RPM enables MARL agents to learn generalizable behaviors, we carry out extensive experiments on DeepMind’s Melting Pot (Leibo et al., 2021).
Melting Pot is a suite of testbeds for the generalization of MARL methods.
It proposes a novel evaluation pipeline for the evaluation of the MARL method in various domains.
That is, all MARL agents are trained in the substrate; during evaluation, some agents are selected as the focal agents (the agents to be evaluated) and the rest agents become the background agents (pretrained policies of MARL models will be plugged in); the evaluation scenarios share the same physical properties with the substrates.
Melting Pot environments possess many properties, such as temporal coordination and free riding as depicted in Table 1.
MARL agent performing well in these environments means its behaviors demonstrate these properties.
In Figure 5, the agent’s observation is shown in the green box to the lower left of the state ( i.e. , the whole image).
The agent is in the lower middle of the observation.
The neural network architecture of the agent’s policy is shown on the left.
More information about the setting of substrates, neural network architectures, MARL training can be found in Appendix D.
Baselines.
Our baselines are MAPPO (Yu et al., 2021), MAA2C (Papoudakis et al., 2021),
OPRE (Vezhnevets et al., 2020), heuristic fictitious self-play (HFSP) (Heinrich, 2017; Berner et al., 2019) and RandNet (Lee et al., 2019).
MAPPO and MAA2C are MARL methods that achieved outstanding performance in various multi-agent scenarios (Papoudakis et al., 2021).
OPRE was proposed for the generalization of MARL.
RandNet is a general method for the generalization of RL by introducing a novel component in the convolutional neural network.
HFSP is a general self-play method for obtaining equilibria in competitive games, we use it by using the policies saved by RPM.
Training setup.
We use 6 representative substrates (Figure 4) to train MARL policies and choose one evaluation scenario from each substrate as our evaluation testbed.
The properties of the environments
[Figure]
are listed in Table 1.
We train agents in Melting Pot substrates for 200 million frames with 3 random seeds for RPM and 4 seeds for baselines.
Our training framework is a distributed framework where there are 30 CPU cores (actors) to collect experiences and 1 GPU for the learner to to learn policies.
We implement our actors with Ray (Moritz et al., 2018) and the learner with EPyMARL (Papoudakis et al., 2021).
We use mean-std to measure the performance of all methods.
The bold lines in all figures are the mean and the shades stand for the standard deviation.
Due to a limited computation budget, it is redundant in computation to compare our method with other methods such as QMIX (Rashid et al., 2018) and MADDPG (Lowe et al., 2017) as MAPPO outperforms them.
All experiments are conducted on NVIDIA A100 GPUs.
5. R ESULTS
To answer Q1 , we present the evaluation results of 6 Melting Pot evaluation scenarios in Figure 6.
We can find that our method can boost MARL in various evaluation scenarios which have different properties as shown in Table 1.
In Chicken Game (eval, ‘eval’ stands for the evaluation scenario of the substrate Chicken Game), RPM outperforms its counterparts with a convincing margin.
HFSP attains over 20 evaluation mean returns.
RandNet gets around 15 evaluation mean returns.
MAA2C and
OPRE perform nearly random (the red dash lines indicate the random result).
In Pure Coordination (eval), Rational Coordination (eval) and Prisoners’ Dilemma (eval), most of the baselines perform poorly.
In Stag Hunt (eval) and Clean Up (eval), MAPPO and MAA2C also perform unsatisfactorily.
We can also find that HFSP even gets competitive performance in Stag Hunt (eval) and Clean Up (eval).
However, HFSP performs poorly in Pure Coordination (eval), Rational Coordination (eval) and Prisoners’ Dilemma (eval).
Therefore, vanilla self-play method cannot directly be applied to improve the generalization of MARL methods.
To sum up, RPM boosts the performance up to around 402% on average compared with MAPPO on 6 evaluation scenarios.
5. A BLATION S TUDY
[Figure]
To investigate which value of ψ has the greatest impact on RPM performance, we conduct ablation studies by (i) removing ranks and sampling from the checkpoint directly; (ii) reducing the number of ranks by changing the value of ψ .
As shown in
Figure 7, without ranks (sampling policies without ranks randomly), RPM cannot perform well in all evaluation scenarios.
Especially in Pure Coordination (eval) where the result is low and has large variance.
In
[Figure]
RPM, choosing the right interval ψ can improve the performance as shown in the results of Pure Coordination (eval) and Prisoners’ Dilemma (eval), showing that the value of ψ is important for RPM.
We summarize the results and values of ψ in Table 2 and Table
[Table]
[Table]
[Table]
5. C ASE S TUDY
We showcase how RPM helps to train the focal agents to choose the right behaviors in the evaluation scenario after training in the substrate.
To illustrate the trained performance of RPM agents, we use the RPM agent trained on Stag Hunt and run the evaluation on Stag Hunt (eval).
In Stag Hunt, there are 8 agents in this environment.
Each agent collects resources that represent ‘hare’ (red) or ‘stag’ (green) and compares inventories in an interaction, i.e. , encounter.
The results of solving the encounter are the same as the classic Stag Hunt matrix game.
In this environment, agents are facing tension between the reward for the team and the risk for the individual.
In Stag Hunt (eval) (Figure(a)).
One focal agent interacts with seven pretrained agents.
All background agents were trained to play the ‘stag’ strategy during the interaction 1 .
The optimal policy for the focal agent is also to play ‘stag’.
However, it is challenging for agents to detect other agents’ strategy since such a behavior may not persist in the substrate.
Luckily, RPM enables focal agents to behave correctly in this scenario.
To answer Q3 , we present the analysis of RPM on the substrate Stag Hunt and its evaluation scenario
Stag Hunt (eval) in Figure 8.
We can find that in Figure 8 (b), the number of the keys in RPM is growing monotonically during training and the maximum number of the keys in RPM is over 20, showing that agents trained with RPM discover many novel patterns of multi-agent interaction and new keys are created and the trained models are saved in RPM.
Meanwhile, the evaluation performance is also increasing in Stag Hunt (eval) as depicted in Figure 8 (c).
In Figure 8 (d), it is interesting to see that the distribution of the keys of RPM is expanding during training.
In the last 25 million training steps, the last distribution of RPM keys covers all policies of different levels of performance, ranging from 0 to 14.
By utilizing RPM, agents can collect diversified multi-agent trajectories for multi-agent training.
Figure 8 (e) demonstrates the final histogram of RPM keys after training.
There are over 600 trained policies that have small value of keys.
Since at the early stage of training, agents should explore the environment, it is reasonable to find that a large number of trained policies of RPM keys have low training episode returns.
After 50 million training steps,
RPM has more policies that have higher training episode returns.
Note that the maximum training episode return of RPM keys is over 14 while the maximum mean evaluation return of RPM shown in
Figure 8 (c) is around 14.
In our experiments, we find that training policies with good performance in the substrate is crucial for improving generalization performance in the evaluation scenarios.
When MARL agents perform poorly, i.e. , execute sub-optimal actions, the evaluation performance will be also inferior or even random, making it is hard to have diversified policies.
We show the results in Appendix E.
R ELATED W ORKS
Recent advances in MARL (Yang & Wang, 2020; Zhang et al., 2021) have demonstrated its success in various complex multi-agent domains, including multi-agent coordination (Lowe et al., 2017;
Rashid et al., 2018; Wang et al., 2021b), real-time strategy (RTS) games (Jaderberg et al., 2019;
Berner et al., 2019; Vinyals et al., 2019), social dilemma (Leibo et al., 2017; Wang et al., 2018;
Jaques et al., 2019; Vezhnevets et al., 2020), multi-agent communication (Foerster et al., 2016; Yuan et al., 2022), asynchronous multi-agent learning (Amato et al., 2019; Qiu et al., 2022), open-ended environment (Stooke et al., 2021), autonomous systems (Hüttenrauch et al., 2017; Peng et al., 2021) and game theory equilibrium solving (Lanctot et al., 2017; Perolat et al., 2022).
Despite strides made in MARL, training generalizable behaviors in MARL is yet to be investigated.
Generalization in RL (Packer et al., 2018; Song et al., 2019; Ghosh et al., 2021; Lyle et al., 2022) has achieved much progress in domain adaptation (Higgins et al., 2017) and procedurally generated environments (Lee et al., 2019;
Igl et al., 2020; Zha et al., 2020) in recent years.
However, there are few works of generalization in MARL domains (Carion et al., 2019;
Vezhnevets et al., 2020; Mahajan et al., 2022; McKee et al., 2022).
Recently, Vezhnevets et al.
(2020) propose a hierarchical MARL method for agents to play against opponents it hasn’t seen during training.
However, the evaluation scenarios are only limited to simple competitive scenarios.
Mahajan et al.
(2022) investigated the generalization in MARL empirically and proposed theoretical findings based on successor features (Dayan, 1993;
Barreto et al., 2018).
However, no technique to achieve generalization in MARL has been proposed in (Mahajan et al., 2022).
Ad-hoc team building (Stone & Kraus, 2010; Gu et al., 2021) models the multi-agent problem as a single-agent learning task.
In ad-hoc team building, one ad-hoc agent is trained by interacting with agents that have fixed pretrained policies and the non-stationarity issue is not severe.
However, in our formulation, non-stationarity is the main obstacle to MARL training.
In addition, there is only one ad-hoc agent evaluated by interacting agents that are unseen during training while there can be more than one focal agent in our formulation as defined in Definition 2, thus making our formulation general and challenging.
There has been a growing interest in applying self-play to solve complex games (Heinrich et al., 2015; Silver et al., 2018; Hernandez et al., 2019; Baker et al., 2019); however, its value in enhancing the generalization of MARL agents has yet to be examined.
C ONCLUSION , L IMITATIONS AND F UTURE W ORK
In this paper, we consider the problem of achieving generalizable behaviors in MARL.
We first model social learning with Markov Game.
In order to train agents that are able to interact with agents that possess unseen policies.
We propose a simple yet effective method, RPM, to save policies of different levels.
We save policies by ranking the training episode return.
Empirically, RPM significantly boosts the performance of MARL agents in a variety of Melting Pot evaluation scenarios.
RPM’s performance is dependent on the appropriate value of ψ .
Several attempts may be needed to determine the correct value of ψ for RPM.
We are interested in discovering more broad measures for ranking policies that do not explicitly consider the training episode return.
Recently, there is a growing interest in planning in RL, especially with model-based RL.
We are interested in exploring the direction of applying planning and opponent/teammate modelling for attaining generalized behaviors with MARL for future work.
In multi-agent scenarios, agents are engaged in complex interactions.
Devising novel self-play method is our future direction for improving generalization of
MARL methods.
E THICS S TATEMENT
We addressed the relevant aspects in our conclusion and have no conflicts of interest to declare.
R EPRODUCIBILITY S TATEMENT
We provide detailed descriptions of our experiments in the appendix and list all relevant parameters in Table 4 and Table 5 in Appendix D.
The code can be found at this anonymous link: https: //sites.google.com/view/rpm-iclr2023/ .
A E NVIRONMENTS
This section introduces Melting Pot in-depth, including substrates and evaluation scenarios.
A. G ENERAL S ETTINGS
[Figure]
Melting Pot (Leibo et al., 2021) is a suite of testbeds for MARL evaluation.
It proposes a novel evaluation pipeline for evaluating the MARL method in various domains.
That is, all MARL agents are trained in the substrate; during evaluation, some agents are selected as the focal agents (the agents to be evaluated), and the rest agents become the background agents (pretrained policies of MARL models will be plugged in); the evaluation scenarios share the same physical properties with the substrates.
Melting Pot environments possess many properties, such as temporal coordination and free riding.
MARL agent performing well in these environments means its behaviors demonstrate these properties.
In each substrate, episodes last 1000 or 2000 steps.
The agents have a partial observability window of 11 ×sprites.
The agent can observe 9 rows in front of itself, 1 row behind, and 5 columns to either side.
Sprites are 8 × 8 pixels.
Thus, in RGB pixels, the size of each observation is 88 × 88 × 3 .
All agents use RGB pixel representations as their inputs.
In Figure 9, the agent’s observation is shown in the green box to the lower left of the state ( i.e. , the whole image).
The agent is in the lower middle of the observation.
The neural network architecture of the agent’s policy is shown on the left.
We introduce the neural network architecture design in Appendix C, MARL training and hyperparameters in Appendix D.
A. S UBSTRATES AND E VALUATION S CENARIOS
We introduce substrates and evaluation scenarios used in the experiments.
In all substrates and scenarios, agents’ movement actions are: forward , backward , strafe left , strafe right
We the environments in Figure 10, readers’ convenience.
Chicken Game.
In this environment, there are 8 agents in the substrate.
Agents move around the environments 2 and collect resources of 2 different colors.
Each agent carries an inventory ρ
= ( ρ 1 , ρ 2 ) with the count of resources picked up since the last respawn.
Due to partial observability, agents can only observe their inventory 3 .
The more resources of a given type an agent picks up, the more committed the agent becomes to the pure strategy corresponding to that resource 4 .
The agent can zap the other agent via its zapping beam for interaction.
When an interaction occurs, a traditional matrix game is started.
Here, in this environment, it is a Chicken Game (Sugden, 2005) where both agents trying to exploit the other leads to the worst payoff, i.e. rewards, for both.
Gathering red resources makes the agent’s strategy towards committing ‘hawk’ while collecting green resources pushes it toward playing ‘dove’.
The payoff matrix for row and column players is:
[Equation]
Chicken Game (eval).
The task and the payoff matrix in this scenario are the same as in Chicken
Game.
In this scenario, one focal agent is joining seven background agents.
Unlike the focal agent that can play any strategy, the background agents were pretrained with pseudo rewards to play ‘dove’.
The best strategy for the focal agent is to play ‘hawk’.
Stag Hunt.
Similar to Chicken Game, there are 8 agents in this environment.
Each agent collects resources that represent ‘hare’ (red) or ‘stag’ (green) and compares inventories in an interaction, i.e. , encounter.
The results of solving the encounter are the same as the classic Stag Hunt matrix game.
In this environment, agents are facing tension between the reward for the team and the risk for the individual.
The matrix for the interaction is:
[Equation]
Stag Hunt (eval).
In this environment, one agent interacts with seven pretrained agents.
All background agents were trained to play the ‘stag’ strategy during the interaction.
The optimal policy for the focal agent is also to play ‘stag’.
Clean Up.
There are seven agents in the environment.
Agents are rewarded (+1) for collecting apples.
In the environment, there are an orchard and a river.
Agents should clean the river frequently to reduce pollution for the irrigation of the orchard.
Apples in the orchard grow at a rate inversely related to the river’s cleanliness.
When the cleanliness rate reaches a certain threshold, apples stop growing.
Agents can take clean action to clean a small amount of pollution from the river.
However, such action only works in a small region around the agent in the river.
So, agents should move to clean the river without any rewards.
Consequently, agents should maintain the public good of orchard regrowth by cleaning the river.
This creates a tension between the short-term individual incentive to maximize agents’ reward by staying in the orchard and the long-term group interest in a clean river.
Clean Up (eval).
In this evaluation scenario, three focal agents join four background agents.
All background agents have been trained to behave altruistically, i.e. , always cleaning the river without consuming apples.
Thus, the optimal policy for the focal agent is to collect as many apples as possible without moving out of the orchard to clean the river.
Pure Coordination.
In this environment, eight agents cannot be identified as individuals because all agents look the same.
Agents gather resources of three different colors.
So, the size of the agent’s inventory is 3.
To maximize the reward, all agents should collect the same colored resource when the encounter occurs.
The matrix for the interaction is:
[Equation]
Pure Coordination (eval).
In this evaluation scenario, there are seven focal agents and one background agent.
The background agent has been trained to target one particular resource out of three colors of resources.
Focal agents should observe other agents to see the resources other agents are collecting and then decide the right color to pick.
This scenario aims to evaluate that agents’ coordination is not disrupted by the presence of unfamiliar other agents who has a special preference for one particular colored resource.
Prisoners’ Dilemma.
Eight agents collect colored resources that represent ‘defect’ (red) or ‘cooperate’ (green).
Agents compare their inventories in an encounter where a classic Prisoner’s Dilemma matrix game is resolved.
Agents face tension between the reward for the group and the reward for the individual.
The matrix for the interaction is:
[Equation]
Prisoners’ Dilemma (eval).
In this evaluation scenario, one focal agent joins seven background agents.
All background agents will play cooperative strategies, i.e. , collecting ‘cooperate’ resources and rarely collecting ‘defect’).
The optimal policy for the focal agent is to identify such a pattern and then collect ‘defect’ resources.
Rational Coordination.
The environment setting is the same as Pure Coordination, except that different colored resources are of different values.
Agents should find the optimal color to maximize the group reward.
The matrix for the interaction is:
[Equation]
Rational Coordination (eval).
In this evaluation scenario, there are seven focal agents and one background agent.
The background agent has been trained to target one particular resource out of three colors of resources.
This scenario is similar to Pure Coordination (eval) since it aims to evaluate that agents’ coordination is not disrupted by the presence of unfamiliar other agents who has a special preference for one particular colored resource.
However, this scenario is more challenging than Pure Coordination (eval) .
While focal agents’ choices are better than miscoordination, some choices are better than coordinating for the focal agents.
B B ASELINES
We introduce baselines trained and evaluated in the experiment in detail.
Baselines are MAPPO (Yu et al., 2021), MAA2C (Papoudakis et al., 2021), OPRE (Vezhnevets et al., 2020), RandNet (Lee et al., 2019) and HFSP (Heinrich et al., 2015; Baker et al., 2019).
B. MAPPO
MAAPO is an extension of PPO (Schulman et al., 2017) for multi-agent RL.
Following the
CTDE (Oliehoek et al., 2008) training and execution paradigm, agents take actions independently during execution and agents’ policies are trained via sharing information ( e.g. ,) with other agents.
In MAPPO, there are N policies { π i } Ni =1 for each agent i .
A central critic is maintained by feeding all agents’ observations and actions { o ti , u ti } Ni =1 .
Although the global state contains all agents’ observations, it contains redundant information that deteriorates the central critic learning with TDlearning (Sutton, 1984).
Note that all baselines that have a central critic takes all agents’ observations and actions { o ti , u ti } Ni =1 as the input.
B. MAA2C
MAA2C is a multi-agent RL variant of A2C (Mnih et al., 2016).
MAA2C adopts the same training and execution paradigm used in MAPPO.
Similar to A2C, TD error is used as the advantage in
MAA2C for training agents’ policies via maximizing policy gradient loss.
B. OPRE
We build OPRE (Vezhnevets et al., 2020) on top of MAPPO.
The key idea behind OPRE is to re-use the same latent space to factorise the policy via creating a hierarchical policy structure:
[Equation]
where o ′ is { o ti , u ti } Ni =1 and η ( u i | o ≤ t i , z ) is a mixture component of the policy, i.e. , an option.
o ≤ t i can be represented via recurrent neural networks (Hochreiter & Schmidhuber, 1997; Cho et al., 2014).
Note that the ‘option’ here differs from the option in hierarchical RL (Sutton et al., 1999; Bacon et al., 2017).
In OPRE, the option has no explicit probability distribution of entering an option and no explicit probability distribution of exiting the current option.
The behavior policy is defined as:
[Equation]
Then p ( z | o ′ ) can be trained via KL ( q || p ) together with the policy and the central critic in an end-toend manner.
We use the default hyperparameters used in OPRE in our experiments.
The number of options is 16 .
B. R AND N ET
Lee et al.
(2019) proposed RandNet for improving the generalization of RL in unseen environments, especially environments with new textures and layouts.
RandNet utilizes a single-layer convolutional neural network (CNN) as a random network, where its output has the same dimension with the input.
To reinitialize the parameters of the random network, RandNet utilizes the following mixture of distributions: P ( ϕ ) =
[Equation]
I ( ϕ
[Equation]
I )
+
[Equation]
N (cid:16) 0 ; (cid:113)
[Equation]
(cid:17) where I is an identity kernel, α ∈ [0 , 1] is a positive constant, N stands for the normal distribution.
n in and n out are the number of input and output channels, respectively.
We use RandNet in the policy network and the critic network of
MAPPO.
We use the default hyperparameters used in RandNet in our experiments.
B. HFSP
Self-play (Brown, 1951; Heinrich et al., 2015; Silver et al., 2018; Baker et al., 2019) has been studied for obtaining equilibria via creating fictitious plays by sampling agents’ past policies.
HFSP is a heuristic fictitious self-play method.
HFSP uses the MARL framework of MAPPO.
Like RPM, it maintains a memory to save all the policies after each training step.
HFSP agents have a probability of 0 .
7 to sample the lasted policies and a probability of 0 .
3 to sample previous policies.
RPM can be considered as a ranked self-play by sampling policies with a hierarchy.
C A RCHITECTURES
We first introduce the neural network architecture of the policy, the critic and the training pipeline for all methods, and the hyperparameters used in the neural network architectures.
RPM and all baselines use the same network architecture.
[Figure]
Actor Network.
The actor network consists of a convolutional neural network (CNN) with two layers.
The two CNN layers use the ReLU activation function.
The first and the second layer haveand 32 output channels, 8 and 4 kernel shapes and 8 and 1 strides, respectively.
An MLP follows the two CNN layers with two layers with 64 neurons each.
The MLP uses the ReLU action function.
It is then followed by a GRU (Cho et al., 2014) with 128 units.
The input of the GRU is the concatenation of the output of the MLP and the features (such as the agent’s position, orientation and inventory).
The output of the GRU is fed into the MLP, and it outputs the policy
π i for agent i .
Critic Network.
The critic network is shared by all agents.
The critic network consists of a CNN with two layers.
The two CNN layers use the ReLU activation function.
The first and the second layer have 16 and 32 output channels, 8 and 4 kernel shapes and 8 and 1 strides, respectively.
The
CNN is then followed by a concatenation of all agents’ actions and features (such as agent’s position, orientation and inventory).
The concatenation is then fed into an MLP with two layers withneurons.
The MLP uses the ReLU action function.
The MLP outputs the value, a vector with the dimension of N .
We take all agents’ observations as a batch and feed them into the CNN.
We then flatten the CNN’s output and feed it with agents’ actions and features as inputs to the MLP network to get the value vector for all agents.
Training.
Our training framework is a distributed framework with 30 CPU cores to collect experiences and 1 GPU for the learner to learn policies, similar to the framework used in IMPALA (Espeholt et al., 2018).
To improve the efficiency and save memory, we use parameter sharing (Rashid et al., 2018; Wang et al., 2021a; Yu et al., 2021), i.e. , all agents share a policy network.
We adopt the CTDE framework to train the policies and the critic.
D T RAINING S ETTINGS
We implement our method with Python and PyTorch.
The learner is implemented with EPyMARL (Papoudakis et al., 2021) and the actors that collect experiments are implemented with Ray (Moritz et al., 2018).
We train agents in Melting Pot substrates for 200 million frames with 3 random seeds for RPM and 4 seeds for baselines.
We randomly sample policies from RPM.
The discount factor γ = 0 .
and we follow the default hyper-parameters used in the original papers of all methods in our research.
We carry out experiments on NVIDIA A100 Tensor Core GPU.
We resort to mean-std values as our performance evaluation measurement.
We use Adam as our optimizer.
We list some important hyper-parameters in Table.
[Table] [Table]
E R ESULTS
We depict the episode return within the substrate.
During training, the MARL methods are evaluated in the substrate.
Figure 12 demonstrates that despite the environments being distinct, RPM also demonstrates leading performance.
Once the agents in the substrate achieve a satisfactory episode return, the trained policy will be saved at the appropriate rank.
In turn, it improves the performance of RPM in the evaluation scenario by collecting diverse data on multi-agent interactions
[Figure]
F E XTENDED R ELATED W ORKS
Recent advances in MARL (Yang & Wang, 2020; Zhang et al., 2021) have demonstrated its success in various complex multi-agent domains, including multi-agent coordination (Lowe et al., 2017;
Rashid et al., 2018; Wang et al., 2021b), real-time strategy (RTS) games (Jaderberg et al., 2019;
Berner et al., 2019; Vinyals et al., 2019), social dilemma (Leibo et al., 2017; Wang et al., 2018;
Jaques et al., 2019; Vezhnevets et al., 2020), multi-agent communication (Foerster et al., 2016; Yuan et al., 2022), asynchronous multi-agent learning (Amato et al., 2019; Qiu et al., 2022), open-ended environment (Stooke et al., 2021), autonomous systems (Hüttenrauch et al., 2017; Peng et al., 2021) and game theory equilibrium solving (Lanctot et al., 2017; Perolat et al., 2022).
Despite strides made in MARL, training generalizable behaviors in MARL is yet to be investigated.
Generalization in RL (Packer et al., 2018; Song et al., 2019; Ghosh et al., 2021; Lyle et al., 2022) has achieved much progress in domain adaptation (Higgins et al., 2017) and procedurally generated environments (Lee et al., 2019;
Igl et al., 2020; Zha et al., 2020) in recent years.
However, there are few works of generalization in MARL domains (Carion et al., 2019;
Vezhnevets et al., 2020; Mahajan et al., 2022; McKee et al., 2022).Recently, Vezhnevets et al.
(2020) propose a hierarchical MARL method for agents to play against opponents it hasn’t seen during training.
However, the evaluation scenarios are only limited to simple competitive scenarios.
Mahajan et al.
(2022) investigated the generalization in MARL empirically and proposed theoretical findings based on successor features (Dayan, 1993;
Barreto et al., 2018).
However, no technique to achieve generalization in MARL has been proposed in (Mahajan et al., 2022).
Several recent studies (Strouse et al., 2021;
Lupu et al., 2021; Tang et al., 2021) have used populationbased training to enhance multi-agent interaction by improving multi-agent diversity.
Fictitious
Co-Play (FCP) proposed in (Strouse et al., 2021) aims to learn policies for a two-agent cooperative game.
It is a two-stage method.
In the first stage, n agents are trained independently with different random seeds via self-play.
It needs n seeds and much more extra computation.
In the second stage, an
FCP agent is trained by interacting with the trained policies of n agents.
Another extra run of training is also needed.
However, our method, RPM, is an end-to-end training method for social dilemmas, competitive, cooperative and even mixed environments with more than two agents.
It needs only one run training (i.e., end-to-end one step) by utilizing fictitious self-play to sample n policies for each agent without maintaining n populations of agents.
Lupu et al.
(2021) considered the problem of zeroshot coordination and proposed a method to achieve diversity via population-based training (PBT) method.
In contrast, our work aims to achieve the generalization of coordination, competition and social dilemmas in multi-agent systems via our novel ranked policy memory method.
The proposed method in (Lupu et al., 2021) cannot be applied to competition and social dilemma scenarios.
Besides that, PBT needs much more computation, which could be computationally expensive for large-scale multi-agent scenarios.
Lupu et al.
(2021) is also pointed out that “self-play (SP) agents control their own trajectory distribution during training, each policy typically only performs well on this exact distribution.
" We believe that is the key issue of the self-play method.
The issue can be alleviated by introducing ranks, and each agent loads its previous policy for multi-agent self-play.
The RPG (Tang et al., 2021) method is a population-based training method without self-play.
It is highly dependent on the randomized reward function.
For simple grid world scenarios in (Tang et al., 2021), conducting randomized reward function perturbations is not challenging.
However, it is non-trivial to find proper reward function perturbations for complex scenarios.
Ad-hoc team building (Stone & Kraus, 2010; Gu et al., 2021) models the multi-agent problem as a single-agent learning task.
In ad-hoc team building, one ad-hoc agent is trained by interacting with agents that have fixed pretrained policies and the non-stationarity issue is not severe.
However, in our formulation, non-stationarity is the main obstacle to MARL training.
In addition, there is only one ad-hoc agent evaluated by interacting agents that are unseen during training while there can be more than one focal agent in our formulation as defined in Definition 2, thus making our formulation general and challenging.
There has been a growing interest in applying self-play to solve complex games (Heinrich et al., 2015; Silver et al., 2018; Hernandez et al., 2019; Baker et al., 2019); however, its value in enhancing the generalization of MARL agents has yet to be examined.
Meta-learning in MARL (Al-Shedivat et al., 2018;
Kim et al., 2021) aims to address the nonstationarity issue in MARL, a well-known issue that has been extensively studied.
To address the issue, the two works adopted the learning-to-learn framework.
Typically, Al-Shedivat et al.
(2018) considers the problem of continuous adaptation in non-stationary environments where agents have few-shot interactions, i.e. , the agent must learn from only a limited amount of experience that it can collect before its environment changes.
The MAML framework was used to address the issue.
Kim et al.
(2021) proposed a novel meta-multiagent policy gradient theorem that directly accounts for the non-stationary policy dynamics inherent to multiagent learning settings.
The proposed metamultiagent policy gradient theorem explicitly models the learning procedure of the other agent (peer learning) in two-agent settings via considering the sequential dependence of the future parameters of other agents on the meta-agent i’s parameter.
While the previous work (Al-Shedivat et al., 2018) ignored it.
However, it would be difficult to train 3+ meta-agents simultaneously in complex scenarios due to the infinite recursion problem associated with the meta-learning framework in (Kim et al., 2021).
The two works consider two-agent settings.
However, solving the non-stationarity issue in scenarios where there are more than two agents is more challenging than in two-agent settings.
The performance of the two works in these scenarios is yet to be investigated.
Furthermore, adapting the framework of meta-learning in MARL to improve the generalization in MARL is non-trivial.