L EARNING M ETA R EPRESENTATIONS FOR A GENTS IN M ULTI -A GENT R EINFORCEMENT L EARNING
A BSTRACT
In multi-agent reinforcement learning, the behaviors that agents learn in a single Markov Game (MG) are typically confined to the given agent number.
Every single MG induced by varying population sizes may possess distinct optimal joint strategies and game-specific knowledge, which are modeled independently in modern multi-agent algorithms.
In this work, we focus on creating agents that generalize across population-varying MGs.
Instead of learning a unimodal policy, each agent learns a policy set that is formed by effective strategies across a variety of games.
We propose Meta Representations for Agents (MRA) that explicitly models the game-common and game-specific strategic knowledge.
By representing the policy sets with multi-modal latent policies, the common strategic knowledge and diverse strategic modes are discovered with an iterative optimization procedure.
We prove that as an approximation to a constrained mutual information maximization objective, the learned policies can reach Nash Equilibrium in every evaluation MG under the assumption of Lipschitz game on a sufficiently large latent space.
When deploying it at practical latent models with limited size, fast adaptation can be achieved by leveraging the first-order gradient information.
Extensive experiments show the effectiveness of MRA on both training performance and generalization ability in hard and unseen games.
I NTRODUCTION
Behaviors of agents learned in a single Markov Game (MG) highly depend on the environmental settings, especially the number of agents, i .
e ., population size Suarez et al.
; Long* et al.
Many multi-agent reinforcement learning (MARL) algorithms
Sukhbaatar et al.
; Foerster et al.
; Lowe et al.
(2017) are developed in games with fixed population sizes.
However, the algorithms may suffer from generalization issues, i .
e ., the policies learned in a single MG are brittle to the change of game setting Suarez et al.
Recent works have experimentally shown the benefit of knowledge transfer between MGs with different population sizes Agarwal et al.
Long* et al.
(2020), which is required to perform between successive games.
Unfortunately, the resulting agents are still confined to particular training games, with less ability for extrapolation.
In this work, we are concerned with learning multi-agent policies that generalize across Markov
Games constructed by varying the population from the same underlying environment.
The created agents are expected to behave well in both training and novel (or unseen) evaluation MGs.
However, optimizing one unimodal policy even for maximizing the performance of the entire training set is still challenging Teh et al.
Effective policies in population-varying games, e .
g ., the ones that achieve Nash Equilibrium in each game, may behave dramatically different due to the game-specific strategic knowledge of themselves.
Such discrepancy will hamper the performance in individual games Brunskill & Li (2013).
In this regard, it is desirable to learn sets of policies that contain the optimal strategies for each training MG, while transferring knowledge to unseen MGs is still challenging nevertheless.
To cope with this challenge, we explicitly model the MG-specific and MG-common strategic knowledge.
In unseen games, although the optimal game-specific knowledge that leads to optimal policies is unobtainable, the common strategic knowledge and various strategic modes can still be captured during training by imposing knowledge variations , i .
e ., the suboptimal game-specific knowledge.
By learning to make the smartest decisions under multiple imagined variations instead of only fitting the best response, the strategic knowledge is learned in an unsupervised manner and agents can effectively generalize to novel MGs.
Since games induced by varying populations possess distinct optimal policy behaviors characterized by different (egocentric) strategic relationships, we model the game-specific knowledge as such relationship.
For example, the optimal game-specific knowledge for PacMan agents in a ghost-dominant game is to focus on ghost agents for survival.
However, this leads to PacMan agents that are unaware of eating more food if evaluated in other games, e .
g ., a PacMan-dominant game.
By additionally impelling the PacMan to pay more attention to other competing PacMan, although the resulting policy may be suboptimal in the training ghost-dominant game, the common knowledge can be learned, which involves eluding ghosts while moving towards food that is with less competition.
We propose Meta Representations for Agents (MRA) to discover the underlying strategic structures in the games.
By meta-representing the policy sets with multi-modal latent policies, the gamecommon strategic knowledge and diverse policy modes are captured with an iterative optimization procedure.
We prove that as an approximation to a constrained mutual information maximization objective, the latent policies can reach Nash Equilibrium in every evaluation MG under Lipschitz game assumption and on a large latent space.
When with practical limited-size latent models, fast adaptation is achieved by leveraging first-order gradient information.
We further empirically validate the benefits of MRA, which is capable of boosting training performance and extrapolating over a variety of unseen MGs.
P RELIMINARIES
Game: An N-agent Markov Game is defined by state set S , action sets {A 1 , . . . ,
A N } , and observation sets {O 1 , . . .
, O N } .
For every agent i , o i ∈
O i is an observation of the global state s ∈ S .
State transition and per-agent reward function are defined as P ( S , A 1 , . . .
, A N , S ′ )
and
R i : S
×
A
i (cid:1) [0 , 1] , respectively.
The joint strategy is denoted as π =
( π 1 , . . .
, π N ) =
( π i , π - i ) , where π i is the strategy of agent i and π - i is the joint strategy excluding it.
In this work, we consider role-symmetric MGs Suarez et al.
; Muller et al.
(2020), where homogeneous agents are with the same reward function and action space.
The type number of homogeneous agents is denoted as h , e .
g .,
h
= 2 for Pac-Man and ghosts in any Pac-Man game.
Relational Representation is an opponent modeling framework to capture the strategic relationship between agents and output embedding e for further policy or value function learning Long* et al.
(2020); Agarwal et al.
; Iqbal & Sha (2018).
Consider the observation o i of agent i with entities o i = (cid:2) o is , o i 1 , . .
, o i j , . . .
, o i N (cid:3) , where o is is agent
i ’s self properties ( e .
g ., its speed), o ij is agent i ’s observation on agent j ( e .
g ., distance from agent j ), and the observed environment information ( e .
g ., landmark locations) is concatenated to these entities.
Then with self-attention
Vaswani et al.
(2017) generating the pair-wise relation g i,j , i .
e ., the j -th entity of agent i ’s egocentric relational graph
g i , the representation embedding e i for agent i is formulated as:
[Equation]
o ij ) , where g i,j =
[Equation]
[Equation] ( o [Equation] Here, V ( · ) , Q ( · ) and K ( · ) denote linear transformations.
The observation embedding with an arbitrary number of agents can thus be represented with a fixed length.
Nash Equilibrium: A core concept in game theory is Nash Equilibrium (NE).
When every agent in game m acts according to the joint strategy π at state s , the value v i,m π ( s ) of agent i is the expectation of i ’s γ -discounted cumulative reward:
[Equation]
In this work, the bold symbol is joint over all agents, and variables with superscript i are of agent i .
Denote the value of the best response for agent i as v ∗ i,m π - i , which is the best policy of agent i when π - i is executed, i .
e ., v ∗ i,m π - i = max
π i v i,mπ i , π - i .
Then π reaches NE if ∀ i ∈ { 1 , ..., N } , v i,m π = v ∗ i,m π - i .
A common metric to measure the distance to a Nash Equilibrium is N ASH C ONV , which represents how much each player gains by deviating to their best response (unilaterally) in total.
And it can be approximately calculated in small games Johanson et al.
; Lanctot et al.
We denote the
N ASH C ONV of π in game m as D m ( π ) .
Then the joint strategy π reaches NE in m if D m ( π ) = 0 .
[Table] [Table]
L EARNING M ETA R EPRESENTATIONS FOR A GENTS 3. P ROBLEM S TATEMENT
In a single stochastic game, achieving Nash Equilibrium gives reasonable solutions and is of great importance Hu & Wellman (2003); Yang et al.
; P ´ erolat et al.
To enable generalization in different MGs, the most straightforward way is to learn a joint strategy set Π that contains effective joint strategies for every MG, e .
g ., the ones that achieve NE.
Denote the set of all training
MGs as M and the set of evaluation MGs as M ′ .
The goal is then to obtain an optimal discrete (finite) or continuous (infinite) joint strategy set Π ∗ such that:
[Equation]
For a satisfiable Π , we first need its size | Π | to be sufficiently large to contain at least one effective strategy for every m ′ ∈ M ′ .
Then Π should be improved with respect to the worst-performing m ′ , i .
e ., the game with no effective strategy contained in Π , to achieve low regret D m ′ for all m ′ .
In other words, Π is updated to include the joint strategy π that minimizes D m ′
( π ) .
Formally,
[Equation]
However, minimizing L ( Π ) over the unseen games in M ′ is impractical in general.
We cope with such intractability by introducing a heuristic algorithm and showing that the resulting objective is indeed equivalent to equation 5 and can thus lead to the optimal Π ∗ in equation 4.
3. R ELATIONAL R EPRESENTATION WITH L ATENT V ARIABLE P OLICIES
Instead of learning independent policies to form Π , we adopt hierarchical latent variable policies to represent the multimodality.
In this way, the MG-common and MG-specific strategic knowledge can be explicitly modeled.
Specifically, in the relational representation framework, the specific strategic knowledge for population-varying MGs is captured by the egocentric relational graph g since agents optimally behave in each game by learning per-game optimal relationship.
Besides, agents take different actions when incorporating different strategic relationships, i .
e ., multiple policy modes are obtained with varied g .
Therefore, we treat g as a higher-level latent variable that is dynamically generated by g = ϕ
( o, z ) .
Here, z is a lower-level latent sampled from a learned distribution p ( z | m ; ψ ) .
Then the common knowledge that how agents optimally behave under different g can be distilled into the policy parameter θ , which includes the transform V in equation 1 and the successive policy network parameters.
An agent takes action a ∼ π ( ·| o, g ; θ ) , where g = ϕ
( o, z ) and z ∼ p ( z | m ; ψ ) .
Although the instantiation can be applied to MARL multi-task learning setups, the design of latent policies by itself does not immediately suggest an algorithm that generalizes to novel MGs.
As such, we present the key ingredient of the Meta Representations for Agents (MRA) algorithm as follows.
3. G ENERALIZATION BY S TRATEGIC K NOWLEDGE D ISCOVERY
Our core idea to enable generalization is to discover the underlying strategic structures in the underlying games.
Although the effective policies in the evaluation games are never known during training, agents can still learn the common strategic knowledge and different behavioral modes solely in the training games in an unsupervised manner with the imposed suboptimal game-specific knowledge.
In particular for population varying games, the agents are assigned different strategic relationships, i .
e ., each agent pays additional attention to some agents while ignoring others.
Instead of learning only one optimal joint policy, training with multiple strategic relationships enables the unsupervised discovery of behavior modes, some of which offer appreciable returns in evaluation
MGs.
Thus, when evaluating in novel games, the desired policy behaviors can be quickly generated by adaptation.
In the extreme case that sufficiently many strategic modes are captured with an extremely large latent space, the desired policy for evaluation games can be directly found.
Specifically, agents optimally behave in each game m ∈ M with the optimal policy parameter θ ∗ and the (per-game) optimal relational graph g ∗ .
By imposing knowledge (or relation ) variations in m , i .
e ., multiple suboptimal g at a certain observation, agents learn how the best decisions to accomplish the task are made, i .
e ., learn θ ∗ that achieves the highest average return.
With the discovery of distinct strategic modes, the MG-common knowledge contained in θ ∗ is obtained.
Thus, when agents are in the novel MGs m ′ ∈ M ′ , their policies can effectively adapt by learning the optimal relational graph in m ′ , or achieve zero-shot transfer (without adaptation) if the latent space is large.
This gives the objective of θ i that maximizes the average return of all knowledge variations and all training MGs:
[Equation]
Notably, equation 6 differs from the objective of multi-task learning where the average training return is maximized by learning the optimal θ ∗ and a single optimal relational graph in each game.
In order to perform well in all m ′ ∈ M ′ , the strategic modes captured during training should cover as many behaviors as possible.
This requires both a large latent space size | Z | and diverse actions.
For fixed | Z | , we introduce a diversity-driven objective that encourages high mutual information between g and a for behavior diversity, as well as between m and g to encourage game-specific knowledge learning.
With high dependence between g and a , distinct variations g can generate diverse actions a .
[Equation]
where I ( g ; a | o )
=
H ( a | o ) − H ( a | o, g ) is the mutual information between g and a conditioned on observation o .
With a slight abuse of notation, m denotes the basic information of game m , e .
g ., the populations of each role.
Iterative optimization of equation 6 and equation 7 is then performed.
3. F AST A DAPTATION WITH L IMITED L ATENT S PACE S IZE
Despite the diversity-inducing objective, we also need a large latent space | Z | .
For Π parameterized by Θ = { ψ, ϕ, θ } , denote Π Θ = Π .
Specially, achieving zero-shot transfer requires | Z | = | Π Θ | ≥ | Π ∗ | .
However, if the settings of games in M ′ are not restricted, the size | Π ∗ | is unbounded.
For practical limited-size latent models and unrestricted M ′ , fast adaptation ability is thus desired.
The generated graph g , together with observations, actions are stored in the replay buffer.
Then various knowledge variations, i .
e ., ( o, g ) pair, are sampled to update θ .
Compared with only | Z | variations generated in an on-policy manner, this leads to a much larger size of variations.
Then we use similar techniques from Reptile Nichol et al.
(2018) to achieve fast adaptation.
The optimization of θ is now to perform K policy gradient steps on each individual MG, instead of on the average return (over m ) in a joint training way.
In game m , the objective for θ in the k -th mini-batch changes from equation 6 to L km ( θ i ) =
E g i (cid:104) v i,mπ i ( ·|· ,g i ; θ i ) , π - i (cid:105) .
Let U Km ( θ ) denote the policy parameter after K gradient steps with learning rate β .
Then θ is updated by θ (cid:0) θ + α ∆ θ , where α is a hyperparameter and ∆ θ =
U Km ( θ ) − θ .
By doing so, the first-order gradient information can be leveraged to update θ towards the instance-specific adapted policy parameter.
Specifically, the expected policy parameter update E k [∆ θ ] over mini-batches in game m is
[Equation]
where ∇L km ( θ ) is the gradient at the initial θ .
The derivation is in Appendix B.
The second term of the RHS in equation 8 is with direction that increases the inner product between gradients of different mini-batches j, k .
That is, θ is optimized not only to maximize the return under all relation variations, but also towards the place that gradients of different variations point to the same direction, i .
e ., the place that is easy to optimize from.
With this property, when the optimal strategic modes of evaluation MGs are not discovered during training, θ can still fast adapt to effective policies.
A NALYSIS
In this section, we provide a theoretical analysis of MRA.
We show how the tractable objective in equation 6 and equation 7 can be derived out of the primary optimization problem in equation 4.
To begin with, we introduce the Markov state transition operator
P π
i , π - i m in MG m , defined as
[Equation]
i , [Equation] Here, x : S (cid:1) R is an L 1 Lebesgue integrable function.
The norm of the operator is defined as ∥ Λ ∥ op = sup {∥ Λ x ∥ L 1 ( S ) :
∥ x ∥ L 1 ( S ) ≤ 1 } , where ∥·∥ L 1 ( S ) is the L 1 -norm over the state space S .
Then we make the assumption of Lipschitz Game.
Assumption 1.
(Lipschitz Game).
For any Markov Game m ∈ ( M ∪ M ′ ) , there exists a Lipschitz coefficient ι m > 0 such that for all agent in m and s ∈ S :
[Equation]
Similar assumptions also appear in many previous works Liu et al.
; Zhang et al.
We note that Assumption 1 is reasonable since the Lipschitz coefficient ι m can be interpreted as the influence of agents
Radanovic et al.
; Dimitrakakis et al.
(2019), which measures how much the policy changing of an agent can affect the game environment.
Then we define a distance metric that measures the discrepancy between M and M ′ by comparing and computing the distance to NE in the games of the two sets.
Let N m denote the total number of agents in game m , and h i,m denote the homogeneous agent set of agent i in m .
Definition 1.
For two sets of MGs M and M ′ , define the distance ς between M and M ′ by
[Equation]
We also define the ϵ -range joint strategy set ˆΠ to guide the policy learning of agents during training.
Definition 2.
For the training MG set M and ϵ > 0 , the ϵ -range joint strategy set ˆΠ is defined as:
[Equation]
, where [Equation] By bounding ϵ that characterizes a large set ˆΠ , equation 5 can be shown to be equivalent to a constrained mutual information maximization objective.
Formally, we provide the following theorem.
The variables and parameters are per agent, e .
g .
, π θ is joint over π θ i .
The superscript is omitted for clarity.
Theorem 1.
If | Π Θ | ≥ | ˆΠ | and ϵ satisfies ϵ ≥ ς − min ι m ,ι m ′ ςγ ( ι m ′ − ι m ) γι m ′ +1 − γ , then with the optimal parameters Θ ∗ = { ψ ∗ , ϕ ∗ , θ ∗ } given by
[Equation]
for every evaluation Markov Game m ′ ∈ M ′ , there exists a joint strategy π ∈ Π Θ ∗ that reaches
Nash Equilibrium (i.e., Π Θ ∗ =
Π ∗ satisfies equation 4).
Theorem 1 suggests a general paradigm of diversity-driven learning that is effective when ˆΠ satisfies certain properties.
In practical MGs, however, the unknown Lipschitz coefficient and the hardness of calculating ς pose challenges to compute the satisfying ϵ .
An approximation to the optimal parameters in equation 12 is to perform iterative optimization following equation 6 and equation 7.
With fixed ϕ and ψ , the objective of θ in equation 6 (greedily) maximizes the expected value over variations in order to minimize the distances to Nash of different policy modes.
In other words, the distance D m ( π ) of the corresponding joint strategies is minimized to satisfy D m ( π ) ≤ ϵ in the long run.
Then the optimization of ϕ and ψ follows to maximize I ( g ; a | o )
+ I ( m ; g | o ) .
By iteratively improving the mutual information and updating θ towards the ϵ -range ˆΠ , the obtained solutions are close to the optimal parameters in equation 12.
Besides, the condition | Π Θ | ≥ | ˆΠ | in the theorem supports the intuition that a sufficiently large policy set (or latent space) is required for zero-shot transfer.
However, as an approximation to the theorem, MRA also has some limitations, which we discuss and provide potential improvements in Section 7.
P RACTICAL A LGORITHM
We have shown that the iterative optimization of MRA can arise from a theoretically justified objective.
In this section, we present practical implementations of the two optimization procedures.
We provide the pseudocode in Appendix D. Implementation details can be found in Appendix E.
M AXIMIZATION OF E XPECTED V ALUE OVER V ARIATIONS
The policy parameter θ i in objective equation 6 is optimized by introducing a centralized critic
Q ζ i for each agent i Lowe et al.
Denote the target network with delayed policy and critic parameters as ¯ θ , ¯ ζ , and replay buffer as D .
The parameterized critic Q ζ i is optimized to minimize:
[Equation]
Then the gradient of the policy parameter θ i of agent i during training is given by:
[Equation]
When evaluation in a novel MG, θ i and ϕ i is fine-tuned to greedily maximize agents’ individual rewards.
Denote ω
i
=
θ i , ϕ i } .
Then the gradient of ω i is given by:
[Equation]
5. M UTUAL I NFORMATION M AXIMIZATION
In an iteration [Equation] several update steps of actor and critic are followed by mutual information
I ( g i ; a i | o i )
+ I ( m ; g i | o i ) maximization.
According to the definition of mutual information and the non-negativeness of KL divergence, the following bound holds.
And ϕ is optimized to optimize equation 16 by gradient ascent.
All the derivations below are provided in Appendix C.
[Equation]
For |M| training games [Equation] the I ( m ; g i |
o i ) term can be simplified as:
[Equation]
[Equation] To calculate the RHS of equation [Equation] we introduce an auxiliary inference network ξ .
Denote the game’s one-hot index as x .
Then the auxiliary network outputs the prediction ˆ x , i .
e ., p (ˆ x | o, g ; ξ ) .
By minimizing the cross-entropy loss in equation 18, ψ and ξ are simultaneously optimized.
Gumbelsoftmax trick Jang et al.
(2016) is used for discrete z .
[Equation]
E XPERIMENTS
Experiments are conducted in three environments which cover both competitive and mixed games and are built based on the particle-world framework Lowe et al.
Models are trained withrandom seeds.
Details of environment settings and hyperparameters are listed in Appendix E.
6. B ENEFITS OF M ETA R EPRESENTATIONS
We first conduct experiments to show the benefits of the proposed meta-representations.
We aim to answer the question that if the MG-common strategic knowledge in various training MGs can be extracted by MRA, and if it can benefit individual training games.
In Fig.
1, we compare MRA and the following methods: (1) the MADDPG algorithm Lowe et al.
(2017) with relational representations ( MADDPG ); (2) the Reptile algorithm Nichol et al.
(2018) ( Reptile ); (3) baseline with the same network architectures as MRA, but agents learn their policies only in a single MG ( baseline ).
For MRA and Reptile, the size of training MG set |M| for the three environments is 4 , 4 , 3 , respectively.
Further specifications are in
Appendix E. We note that the actual executed episodes of MRA in one MG are |M| times smaller than that for baseline and MADDPG, which reveals the efficiency of the proposed meta-representation.
Although the Pacman-like world is not a zero-sum game, we still provide cross-comparison results in Appendix E for completeness.
[Figure]
6. P ERFORMANCE C OMPARISON IN M ULTIPLE G AMES
In this part, we compare the performance of MRA with multi-task and meta-learning methods, including EPC Long* et al.
(2020) and RL 2 Duan et al.
The curriculum learning EPC is implemented by initializing 3 parallel sets of agents and mix-andmatch the top 2 sets to the successive MG.
For RL 2 , each trial contains a cycle of all the |M| MGs.
We also compare another MRA variant, uni-MRA , that samples z from a uniform distribution.
In resource occupying environment with 8 resources, the results are shown in Fig.
Due to the discrepancy between effective policies in different MGs, the game-common strategic knowledge is not well exploited by EPC.
RL 2 agents are also observed to perform poorly in some
MGs, which verifies the benefits of explicitly modeling the MG-common knowledge and MGspecific knowledge when population varies.
Since no game-specific information is conditioned in uni-MRA, we observe that some MGs dominate the others.
Detailed settings and curves in other environments are provided in Appendix E.
6. G ENERALIZATION E VALUATION
If the learned policies both (1) adapt better and faster; and (2) perform well in novel games with no additional training ( i .
e ., zero-shot transfer), then the algorithm is considered to generalize well.
Now we evaluate the generalization ability of MRA and other methods based on these two metrics.
[Figure]
Better Adaptation: We first show that MRA have better adaptation ability benefited from the distilled common knowledge and first-order gradient information.
Comparisons are conducted between
MRA , EPC , RL 2 , MADDPG and MAAC Iqbal & Sha (2018).
MADDPG and MAAC are trained from scratch, while the others are fine-tuned from the parameters trained in multiple MGs.
The common knowledge can provide agents a good policy initialization and overcome the random exploration bottleneck.
For example, the random exploration often results in Pac-Man agents being killed if ghost agents dominate the game.
And all agents end up with almost random behaviors.
However, with common knowledge guiding the Pac-Man to take reasonable actions, agents will get useful information and learn to accomplish the task.
In the Pac-Man game where 8 ghost agents chase 2 Pac-Man agents, the benefits of the common knowledge are reflected in Fig. 3(c).
The random exploration leads to worse performance of MAAC and MADDPG agents.
Besides, in Fig.
3(a) when reward shaping is removed, MRA agents can still adapt with fewer episodes and has better asymptotic performance.
We also show in Fig. 3(b) that the complexity brought by the large populations, e .
g ., 48, can be successfully handled by
MRA.
The results verify the benefits of common knowledge compared with the transfer knowledge in EPC.
Zero-shot Transfer: MRA also has better zero-shot transferability than EPC and RL 2 .
The results in resource occupying are reported in Fig.
Performance of MRA is calculated by taking expectations over the latent.
However, its return will get higher if enumerated trial-and-error is taken, i .
e ., choose the best policy mode by trying every z .
C ONCLUSION & D ISCUSSIONS
In this paper, we propose meta representations for agents (MRA) that can generalize in Markov
Games with varying populations.
With latent variable policies and relational representations, the diverse strategic modes are captured.
As an approximation to a theoretically justified objective, MRA effectively discovers the underlying strategic structures in the games that facilities generalizable knowledge learning.
Experimental results also verify the benefits of MRA.
Our work also opens some new problems.
Theorem 1 requires the computation of ς and ϵ as well as an extremely large latent space, both of which are impractical.
Although approximations that MRA makes are reasonable, obtaining optimal Π ∗ will not always be guaranteed.
Possible improvements include: bound ς by imposing restrictions on the evaluation MG set, or enlarge the latent space size by e .
g .
, adopting continuous latent variables, which we would like to explore as future work.
With role-symmetric game settings, MRA has benefits in many research problems, including dealing with population complexity, overcoming the multi-agent random exploration bottleneck, and adapting faster with the meta-represented agents.
A fruitful avenue for future work is to augment MRA by e .
g ., adapting roles Wang et al.
(2020), to apply to other game settings.
Besides, achieving NE may not indicate the global optimality in general-sum MGs, and metrics such as social optimum can be investigated.
For population-varying MGs we model game-specific knowledge as strategic relationship.
Although it may lose the universality in broader scopes compared with general meta-RL algorithms, we hope the idea of explicit strategic knowledge modeling can inspire algorithms that adjust with the task of interest.
A P ROOFS
In this section, we provide three propositions in A.1, A.2 and A.3.
The proof of Theorem 1 in A. is built upon these propositions.
A. P ROPOSITION
Before giving Proposition 1, we provide a useful lemma of the Markov state transition operator
P π i , π - i m .
The following Lemma 1 is given in (Liu et al., 2021), which is concerned with Fictitious
Self-Play.
We modify it to be suitable for multi-agent general-sum Markov Games with our notations.
Lemma 1.
(Liu et al., 2021).
The Markov state transition operator satisfies:
[Equation]
Proof.
To begin with
[Equation]
we have the following inequality of the operator
P π
i , π - i m
[Equation]
(cid:17)
[Equation]
i
[Equation]
(cid:17)
[Equation]
(cid:17)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
Then we have:
[Equation]
Thus,:
[Equation]
Now we state our first proposition.
Let the distance κ ( π i ) between policy
π i and the best response π ∗ i in the action space be defined as:
[Equation]
Proposition 1.
For the joint strategy π , D m ( π ) is bounded by:
[Equation]
Proof.
First, a state visitation measure ρ π ∗ i , π - i s,m of the joint strategy ( π ∗ i , π - i ) in MG m is defined as follows:
[Equation]
where δ s is a Dirac delta function.
By converting the value of strategy to the integration of reward over state measure [Equation] we have:
[Equation]
We then bound the two terms separately.
For the first term
[Equation]
we have from Lemma 1 that:
[Equation]
Then by the definition of L m
[Equation]
we have:
[Equation] [Equation]
Thus, the first term satisfies:
[Equation]
i , [Equation] m
[Equation]
The L + ∞ -norm over s of the second term is bounded by:
[Equation]
Finally, from equation 25 and equation
[Equation]
we have the following bound for D m ( π ) :
[Equation]
P ROPOSITION
Proposition 2.
For the training MG set M and the evaluation MG set M ′ , if ϵ satisfies
[Equation]
then for every evaluation Markov Game m ′ ∈ M ′ , the joint strategy that achieves Nash Equilibrium in m ′ is guaranteed to be contained in the ϵ -range joint strategy set ˆΠ .
Proof.
For Definition 1 of the distance ς between the training MG set M and the evaluation MG set
M ′ , we have the following equivalent logic statement:
[Equation]
In an evaluation MG ˜ m ′ ∈ M ′ , let i and i ′
be the agent index defined as follows:
[Equation]
Intuitively, the above agent i is the agent in ˜ m ′ that being replaced by the trained policy π i ′ in the policy set leads to the largest distance to the Nash Equilibrium.
And agent i ′ is the corresponding agent that π i ′ achieves an NE in a particular training MG.
With i and i ′ denied in equation 29, the bound in Proposition 1 can be specified as follows:
[Equation]
The second equality holds since D ˜ m ′ ( π ′ )
= 0 , and the distance from the joint strategy ( π i ′ , π ′ - i ) to a Nash Equilibrium is equal to the distance to π ′ .
The last inequality holds due to the bound in
Proposition 1 and the definition of i and i ′ .
This implies that for any MG ˜ m
′ we have:
[Equation]
With the maximum influence ι m ′ over the evaluation MG m ′ ∈ M ′ , we obtain:
[Equation]
Since D m ( π ) = 0 and D m ′ ( π ′ )
= 0 , the best response in MG m with other agent’s strategies fixed as π - i ′ is π i ′ .
And in MG m ′ , the best response with other agent’s strategies fixed as π ′ - i is π ′ i .
Thus we get:
[Equation]
For ϵ that satisfies equation 28, we obtain:
[Equation]
For i and i ′ defined in equation
[Equation]
we have the following inequality by noticing that D m ( π ) = 0 :
[Equation]
This indicates that if we choose ϵ to satisfy equation 28, then the policy π ′ i that achieves Nash
Equilibrium in MG ˜ m ′ is guaranteed to be included in the policy set.
Since all the above inequalities hold for any ˜ m ′ ∈ M ′ , the policy that achieves NE in all MGs in the evaluation MG set M ′ is guaranteed to be included in the policy set.
This completes the proof.
A. P ROPOSITION
Proposition 3.
If | Π Θ | ≥ | ˆΠ | and ϵ satisfies ϵ ≥ ς
[Equation]
m
[Equation]
γ , then the optimal Π that maximizes the objective:
[Equation]
for every evaluation Markov Game m ′ ∈ M ′ , there exists a joint strategy π ∈ Π Θ ∗ that reaches
Nash Equilibrium (i.e., Π Θ ∗ =
Π ∗ satisfies equation 4).
Proof.
Since | Π | ≥ | ˆΠ | , the optimal | Π | that maximizes L ( Π ) in equation 36 must satisfy:
[Equation]
In other words, for every policy ˆ π in the joint strategy set ˆΠ , i .
e ., ˆ π ∈ ˆΠ , there exists a learned policy π ∈ Π , such that π = ˆ π .
Note that the above statement is true only when | Π | ≥ | ˆΠ | .
For ϵ that satisfies equation 28, from Proposition 2 we know that for every evaluation MG m ′ ∈ M ′ , the strategy that achieves Nash Equilibrium are guaranteed to be contained in ˆΠ .
Thus, the optimal policy set Π that results from optimizing equation 36 also contains the strategies that are NE in every MG m ′ ∈ M ′ .
So the optimal policy set Π satisfies equation 4, which completes the proof.
A. P ROOF OF T HEOREM
Proof.
Since | Π Θ | ≥ | ˆΠ | , we can simplify the objective in equation 36 by updating the joint strategy π in a fixed-size set Π .
That is, maximizing equation 36 is equivalent to:
[Equation]
From the non-negativeness of KL divergence, we have:
[Equation]
The equality holds when π = ˆ π .
Thus, we have:
[Equation]
If π ∈ ˆΠ is constrained, then the equality holds and max π E ˆ
[Equation]
π )
This leads to:
[Equation]
The above equation states that the strategy π is learned to fit ˆ π .
And to cover all the ˆ π ∈ ˆΠ , the entropy of strategies in Π should also be maximized.
Then we get the following equality:
[Equation]
where the first equality holds following the definition of mutual information and by noticing that policy π is meta-represented by Ω .
The last equality holds since the size of the meta-represented policy set | Π Θ | is sufficiently large and satisfying | Π Θ | ≥ | ˆΠ | .
Combining equation 39 and equation 40.
we have:
[Equation]
Then by equation 37 and equation 41, we have that the solution of the following objective is equivalent to the solution of equation 36:
[Equation]
Thus, for every evaluation Markov Game m ′ ∈ M ′ , there exists a joint strategy π ∈ Π Θ that reaches Nash Equilibrium ( i .
e ., Π Θ ∗ =
Π ∗ satisfies equation 4).
B F AST A DAPTATION WITH F IRST -O RDER G RADIENT
Reptile Nichol et al.
(2018) is a meta-learning algorithm that uses first-order gradient information for fast adaptation.
For parameter θ that maximizes objective L km ( θ ) in the k -th mini-batch of game m , θ is updated by θ (cid:0) θ + α ∆ θ , where ∆ θ = U Km ( θ ) − θ
and U Km ( θ ) denotes the updated θ after K gradient steps with learning rate β , and α is a hyperparameter.
Denote the k -th step parameter as θ k , then the update ∆ θ of K gradient steps is as follows:
[Equation]
where the last equation holds since ∇L km ( θ j )
=
∇L km ( θ 1 ) + O ( β ) .
For the initial parameter θ = θ 1 , the term K − 1 (cid:80) k
=1 ∇L km ( θ 1 ) maximizes the overall performance at θ in all the K mini-batches in an MG m .
The key difference from the joint training objective is the term ∇ 2 L km ( θ 1 ) ∇L jm
( θ 1 ) .
When the expectation are taken under mini-batch sampling in m , denote E k as the expectation over the mini-batch defined by J k .
Omitting the higher-order term O (cid:0) β
[Equation]
, we have:
[Equation]
Thus, updating θ by θ (cid:0) θ + α ∆ θ not only maximizes the average performance in K mini-batches of all MGs, but also maximizes the inner product between gradients of different mini-batches, i .
e ., ∇L km ( θ ) ∇L jm ( θ ) .
Thus, the generalization ability is improved and fast adaptation is achieved.
do sample latent
; execute maximize while execute where while
C D ERIVATION OF M UTUAL I NFORMATION C ALCULATION
The two mutual information terms in equation 12, i .
e .
, I ( g ; a | o ) and I ( m ; g | o ) can be calculated as follows:
[Equation]
Similarly, we have:
[Equation]
D P SEUDOCODE
The high level algorithmic frameworks of the training process and evaluation process are shown in
Algorithm 3 and Algorithm 2, respectively.
The complete pseudocode for training MRA is provided in Algorithm 3, and the pseudocode of the
MRA adaptation process is given in Algorithm 4.
do A mini-batch of B samples of ( o b , a b , o ′ b , g b , r b ) is sampled from D m ; for do
Update all agents’ critic parameter ζ i by minimizing:
[Equation]
, a
[Equation]
g ib
[Equation]
ζ
[Equation]
2 , where B i π Q = r ib +
[Equation]
The k -th step of policy parameter θ ik is updated by gradient ascent:
[Equation]
end for
Update all agents’ θ i by
[Equation]
i
[Equation]
Sample n latent code z
[Equation]
and approximate
[Equation]
Update all agents’ ϕ i by maximizing:
[Equation]
Update ψ and auxiliary network ξ simultaneously by minimizing:
[Equation]
Update all agents’ delayed parameters ¯ θ i and ¯ ζ i ; end if end for end for end while
Input
Output: Adapted parameters.
Initialize replay buffer D ;
Each agent i samples lower-level latent code z i ∼ p ψ ( z | m ′ ) ; while total episode number not reach do
Reset game and receive initial observation o ; for time steps in an episode do
Each agent i executes action a i ∼ π (cid:0) ·| o i , ϕ
i
( o i , z i ); θ i (cid:1) simultaneously and get reward r i , next observation o ′ i ;
Push ( o , a , o ′ , r ) to replay buffer D m ′ ; o (cid:0) o ′
A mini-batch of B samples of ( o b , a b , o ′ b , r b ) is sampled from D m ;
Calculate the detached relational graph g i
= ϕ
i ( o ib , z ib )
Update all agents’ critic parameter ζ i by minimizing:
L (cid:0) ζ i (cid:1) =
1 B (cid:80) b (cid:0) B i π Q − Q (cid:0) o b , a b , g i ; ζ i (cid:1)(cid:1) 2 , where B i π Q = r ib + γ E a ′ ∼ ¯ π (cid:2) Q (cid:0) o ′ b , a ′ , g i ; ¯ ζ i (cid:1)(cid:3) ;
Update all agents’ parameter ω
i =
( θ i , ϕ i ) by gradient ascent: ∇ ω
i J = 1 B (cid:80) b ∇ ω
i log π (cid:0) a i | o ib , ϕ i ( o ib , z ib );
θ ik (cid:1) Q (cid:0) o b , a b , g i ; ζ i (cid:1) ;
Update all agents’ delayed parameters ¯ θ i and ¯ ζ i ; end for end while
E A DDITIONAL E XPERIMENTS E. T RAINING S ETUPS AND H YPERPARAMETERS
Treasure Collection:
Each agent is with the goal to collect more treasures in an episode.
Treasures disappear and re-generate at a random location when touched by agents.
Resource Occupying:
Agents receive rewards for occupying varisized resource landmarks: higher reward if one agent is occupying a larger resource with fewer other agents in it.
Pacman-like World: It is similar to predatory-prey games, but with additional food dots.
Pac-Man agents are with goals to collect food and elude ghosts, and ghosts are with goals to touch Pac-Man.
The treasures in the treasure collection environment and the food dots in the Pacman-like world are randomly initialized in the position range of [ − 1 , +1] , and regenerated when touched by collector agents and PacMan agents, respectively.
The sizes of the resource landmarks are pre-defined as { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 , 0 . 6 } and fixed in each episode.
When evaluating the benefits of MRA in Section 6.1 of the main paper, the training Markov Games in the three environments are as follows.
In treasure collection environment, there are 4 training
MGs in total.
The agent populations in the 4 MGs are 3 , 6 , 12 , 24 , respectively, which we denote as { 3 , 6 , 12 , 24 } .
The 4 training MGs in the resource occupying environment is { 6 , 9 , 12 , 15 } .
In the
PacMan-like world, there are 3 training MGs in total: { (4 , 2) , (6 , 3) , (8 , 4) } , where (4 , 2) in the first
MG denotes that there are 4 PacMan agents and 2 ghost agents.
We adopt the same set of hyperparameter for experiments.
12 rollouts are executed in parallel when training.
The maximum length of the replay buffer is 1 e 6 .
Episode length is set to 20 .
The dimension of the latent code z is 6.
The critic also adopts a self-attention network in a similar way with MAAC
Iqbal & Sha (2018).
And the number of gradient steps of policy and critic parameters in each update, i .
e ., K , is set as 10 .
And α = 1 works well in experiments.
Batch size is set to 1024 and Adam is used as the optimizer.
The initial learning rate is set to 0 .
In all experiments, we use one
E. C ROSS -C OMPARISON R ESULTS
We provide the cross-comparison results in the PacMan-like world.
The comparisons are conducted between the MRA agents trained in multiple MGs and the agents trained in a single MG.
The score is summed in each episode, averaged across homogeneous agents on 40 runs and normalized.
[Table] [Table]
The cross-comparison results are shown in Table 1 and Table 2.
We can see that the agents created by the proposed MRA outperform the single-MG counterparts for both PacMan agents and ghosts agents, validating the effectiveness of the proposed method.
E. I MPLEMENTATION V ARIANTS
In role-symmetric games, the parameters θ, ϕ, ζ are shared by homogeneous agents.
And ϕ can be implemented as the option architecture Sutton et al.
, i .
e ., g corresponds to the z -th option sampled from the categorical distribution.
We then conduct experiments on different instantiations of the ϕ ( o, z ) function in MRA.
We denote the default implementation architecture, where different relational graph from different attention heads is controlled by the lower-level latent variable z , as the option Sutton et al.
(1999) architecture.
Specifically, we apply the multi-head self-attention architecture Vaswani et al.
And the head number is set as the dimension of the categorical distribution p ( z ) , and g corresponds to the graph at the z -th head when sampling z .
The other two variants we consider for implementing ϕ
( o, z ) are the ones that are discussed in
Florensa et al.
The first variant is to concatenate z to each entity of the observation decomposition
o i = (cid:2) o is , o i 1 , · · · , o ij , · · · , o iN (cid:3) .
The same relational representation is also adopted to generate the relational graph g .
The second variant is to perform the outer product between each observation entity and z .
We refer to the two variants as “concat” and “bilinear”, respectively.
The performance of the three implementations is evaluated in the 6-resource occupying environment.
The number of training MGs is set to 3 , and the numbers of agents in the 3 games are { 6 , 9 , 12 } .
The results in Figure 5 show that all the three implementations can obtain agents that effectively act in all the 3 scenarios.
And the default option architecture achieves better performance than the other two variants.
The possible reason is that the lower-level latent code z in the option architecture can explicitly control the structural factors and can thus learn the common knowledge more quickly and better.
E. A BLATION S TUDY ON THE S IZE OF T RAINING MG S ET
The information in all the training MGs determines the common knowledge that agents can learn.
We provide ablation study on the number of training MGs.
In the resource occupying environment, we train the agents in three settings, each of which is with different size of training MG set: 2 , 3 and 4 .
Specifically, the population size of the three settings are: { 6 , 12 } , { 6 , 9 , 12 } and { 6 , 9 , 12 , 15 } .
The curves in MGs with { 6 , 9 , 12 } populations are shown in Figure 6.
We observe that when the size of the training MG set is greater than 2 , the benefits of the metarepresentation are obvious.
The knowledge that agents learn from few training MGs, e .
g ., 1 ortraining MGs, is limited, and the random exploration bottleneck still exists.
However, the performance can be significantly improved by leveraging the information from more training MGs, e .
g .,or 4 training MGs, where the common knowledge is more likely to be distilled and thus guiding the exploration.
E. A DDITIONAL C URVES
When training in multiple treasure collection MGs, the curves of MRA and other related multi-task or meta-learning approaches are shown in Fig.
We provide the training phase curves of the approximated mutual information I ( g ; a | o ) and the inference loss of MG index output by auxiliary network ξ .
The curves are shown in Figure 8.
E. V ISUALIZATIONS
The screenshots of the three environments in the experiments are shown in Figure 9.
We visualize the trajectories of one agent in a resource occupying MG in Figure 10.
Green dots and blue dots are agents and resources, respectively.
When agents are only trained in this MG with different random seeds, different behaviors are obtained.
This indicates that agents trained in single MGs are confined to environmental settings.
Agents only learn the best responses and fit an
NE.
However, if the agents are only aware of some of the successful behaviors, the generalization
[Figure] [Figure]
will be constrained.
On the contrary, MRA has large capacities to represent multiple strategies by incorporating different relational graph with the distilled common knowledge, which leads to diverse reasonable behaviors.
[Figure]
We also visualize some instances of the learned relation variations, i .
e ., different relational graph g under observation o , as well as how agents make the smartest decisions under different variations in
Fig.
The common knowledge learned by the agent can be interpreted as ”moving to less-agent resources”.
Specifically, in Fig.
11(a)
the black agent makes decisions to move left by focusing on the topmost red agents which are occupying a resource.
By focusing on the leftmost red agents in Fig. 11(b), the black agent makes decisions to move up.
Although such behavior might not be optimal, since the topmost resource is smaller than the leftmost resource, this variation helps agents learn common knowledge and optimally behave in an unseen MG by incorporating the optimal relation mapping in that game.
[Figure]