Generating Diverse Cooperative Agents by Learning Incompatible Policies
Abstract
Effectively training a robust agent that can cooperate with unseen agents requires diverse training partner agents.
Nonetheless, obtaining cooperative agents with diverse behaviors is a challenging task.
Previous work proposes learning a diverse set of agents by diversifying the state-action distribution of the agents.
However, without information about the task’s goal, the diversified behaviors are not motivated to find other important, albeit non-optimal, solutions, resulting in only local variations of a solution.
In this work, we propose to learn diverse behaviors by looking at policy compatibility while using state-action information to induce local variations of behaviors.
Conceptually, policy compatibility measures whether policies of interest can collectively solve a task.
We posit that incompatible policies can be behaviorally different.
Based on this idea, we propose a novel objective to learn diverse behaviors.
We theoretically show that our novel objective can generate a dissimilar policy, which we incorporate into a population-based training scheme.
Empirically, the proposed method outperforms the baselines in terms of the number of discovered solutions given the same number of agents.
1. Introduction
Understanding the interaction between artificial intelligence (AI) and humans is crucial for the development of better AI systems (Lavin et al., 2021). However, developing and evaluating in the real world is difficult and expensive. Consequently, AI researchers resort to agent-based modeling (ABM) simulations that treat both parties as agents (Helbing, 2012).
ABM maintains the essence of human-AI interac-
AI tion (HAI) while being computationally cheaper than its real-world counterpart, allowing efficient investigation of
HAI without the inconvenience of real-world experiments (Macal & North, 2005).
One of the most important topics in HAI is the cooperation between AI and humans.
For an AI agent to be considered good at collaboration, it must be able to react and adapt to various (human) partner’s behaviors while completing the task at hand (Carroll et al., 2019).
Still, cooperating with unseen agents (e.g., humans) in multi-agent systems is a challenging problem.
Current state-of-the-art multiagent reinforcement learning techniques can produce highly competent agents in cooperative settings (Lowe et al., 2017;
Sunehag et al., 2017; Mahajan et al., 2019; Peng et al., 2021).
Nonetheless, those agents are often overfitted to their training partners and cannot coordinate with unseen agents effectively (Lanctot et al., 2017;
Carroll et al., 2019; Bard et al., 2020; Hu et al., 2020).
The problem of working with unseen partners (i.e., ad-hoc teamwork problem (Stone et al., 2010)) has been tackled in many different ways (Albrecht & Stone, 2018; Grover et al., 2018; Carroll et al., 2019; Shih et al., 2020; Rahman et al., 2021; Strouse et al., 2021; Xie et al., 2021; Mirsky et al., 2022; Parekh et al., 2022; Wang et al., 2022).
These methods allow the agents to learn how to coordinate with unseen agents and, sometimes, humans.
However, the success of these methods relies heavily on the quality of the pool of training partner agents.
It has been shown that the diversity of the training partners is crucial to the generalization of cooperative agents (Charakorn et al., 2020; 2021; McKee et al., 2021).
In spite of its importance, obtaining a diverse set of partners is still an open problem.
The simplest way to generate training partners is to use handcrafted policies (Leibo et al., 2021; Papoudakis et al., 2021;
Xie et al., 2021;
Wang et al., 2022) or agents produced from multiple runs of self-play training process (Grover et al., 2018; Charakorn et al., 2020; Strouse et al., 2021).
These methods, however, are not scalable nor guaranteed to produce diverse behaviors.
Many prior works recognize the importance of diverse training partner agents in the cooperative domain and propose techniques aiming to generate diverse agents by changing the state visitation and action distributions (Lucas & Allen, 2022), or joint trajectory dis- tribution of the agents (Mahajan et al., 2019; Lupu et al., 2021).
However, Lupu et al.
(2021) discuss a potential drawback of using such information from trajectories to diversify the behaviors.
Specifically, agents that make locally different decisions do not necessarily exhibit different high-level behavior.
To avoid this potential pitfall, we propose an alternative way to diversify behaviors using information about the task’s objective.
In contrast to previous work that uses joint trajectory distribution as a representation of behavior, we use policy compatibility instead.
In multi-agent systems, policy compatibility evaluates whether policies of interest can collaboratively complete a task.
Since cooperative environments commonly require all agents to coordinate on the same solution , if the policies have learned different solutions, then they cannot coordinate effectively.
Consequently, if an agent discovers a solution that is incompatible with all other agents in a population, then the solution must be unique relative to the population.
Motivated by this intuition, we introduce a novel training objective that regularizes agents in a population to find solutions that are compatible with their partner agents while not compatible with any other agents in the population.
We call this method “Learning Incompatible Policies” (LIPO).
We theoretically show that optimizing the proposed objective will yield a dissimilar policy, which we extend further for a population-based training scheme.
Furthermore, we utilize a mutual information objective to diversify the local behaviors of each joint policy.
Empirically, LIPO can discover more solutions than previous methods, given the same population size.
2. Preliminaries
Our main focus lies in fully cooperative environments which are modeled as decentralized partially observable Markov decision processes (Dec-POMDP)
(Bernstein et al., 2002).
In this work, we start our investigation in the two-player variant.
A two-player Dec-POMDP is defined by a tuple ( S , A 1 , A 2 , Ω 1 , Ω 2 , T, O, r, γ, H ) where S is the state space, A ≡ A 1 ×
A 2 and Ω ≡ Ω 1 × Ω 2 are the joint-action the joint-observation space of player 1 and player 2.
The transition probability from state s to s ′ after taking a joint action ( a 1 , a 2 ) is given by T ( s ′ | s, a 1 , a 2 ) .
O (
o 1 , o 2 | s ) is the conditional probability of observing a joint observation ( o 1 , o 2 ) under state s .
All players share a common reward function r ( s, a 1 , a 2 ) , γ is the reward discount factor and H is the horizon length.
Player 1 and 2, with potentially different observation and action spaces, are controlled by policy π 1 and π 2 .
At each timestep t , the players observe o t =
( o 1 t , o 2 t ) ∼
O ( o 1 t , o 2 t | s t ) under state s t ∈ S and produces a joint action a t = ( a 1 t , a 2 t ) ∈
A sampled from the joint policy π ( a t | τ t ) =
π 1 ( a 1 t | τ 1 t ) π 2 ( a 2 t | τ 2 t ) where τ 1 t and τ 2 t contain a trajectory history until timestep t from the perspective of each agent.
All players receive a shared reward r t = r
( s t , a 1 t , a 2 t ) .
The return of a joint trajectory τ =
( o 0 , a 0 , r 0 , ..., r H − 1 , o H ) ∈ T ≡ (Ω ×
A × R ) H can be written as G ( τ ) =
(cid:80)
Ht
=
γ t r t .
The expected return of a joint policy ( π 1 , π 2 ) is J ( π 1 , π 2 ) =
E τ ∼ ρ ( π 1 ,π 2 ) G ( τ ) =
(cid:82) τ P ( τ | π 1 , π 2 ) G ( τ ) where ρ ( π 1 , π 2 ) is the distribution over trajectories of the joint policy ( π 1 , π 2 ) and P ( τ | π 1 , π 2 ) is the probability of τ being sampled from a joint policy ( π 1 , π 2 ) .
Since we will be working with multiple joint policies, we use subscript to denote different joint policies e.g., π A
= ( π 1 A , π 2 A ) is a different joint policy from π B =
( π 1 B , π 2 B ) .
Furthermore, we will use i, j ∈ { 1 , 2 } where i ̸ =
j as superscripts to refer to different player roles.
Finally, policies ( π 1 A , π 2 A ) from a joint policy π A aim to maximize J ( π 1 A , π 2 A ) .
Since we are interested in creating dissimilar policies for any Dec-POMDP, it is useful to have an environment-agnostic measure that captures the similarity of policies.
First, we consider a measure that can compute the similarity between policies of the same role i , e.g., π iA and π iB .
We can measure this with the probability of a joint trajectory τ produced by either π iA or π iB .
However, in the two-player setting, we need to pair these policies with a reference policy π j ref .
Specifically, π iA and π iB are considered similar if they are likely to produce the same trajectories when paired with a reference policy π j ref .
We define similar policies as follows:
Definition 2.1 (Similar policies) .
Considering two policies of the same role i , π iA and π iB , and a reference policy π j ref of a different role j , π iA is similar to π iB up to ϵ if and only if | 1 − P (
[Equation]
|
[Equation]
iA ,
[Equation]
| ≤
[Equation]
∀
[Equation]
∈ T , where 0 ≤ ϵ ≤ 1 .
Next, we consider an alternate view on measuring similarity between policies using policy compatibility (Section 3).
Policy compatibility measures the change in performance of a joint policy π B before and after one of its policies π iB is substituted by another policy π iA .
We define compatibility between a policy π iA and a joint policy π B as follows:
Definition 2.2 (Compatible policies) .
Given a policy π iA and a joint policy π B , we define the compatibility ratio between π iA and π B as: C ( π iA , π B ) :
=
J ( π iA ,π jB ) J ( π 1 B ,π 2 B ) .
We say that π iA is compatible with π B
if and only if C ( π iA , π B ) ≥ 1 − ϵ .
Note that LIPO can be applied to environment with more than two players with a slight modification.
Specifically, a policy π j would represent the joint policy of all players except player i , π j ( a jt | τ jt ) =
Π k ̸
= i
π k ( a kt | τ kt ) .
3. Learning Incompatible Policies (LIPO)
The goal of LIPO is to learn several joint policies that are dissimilar to each other and, therefore, create a pool of diverse partner agents.
We theoretically show that policy compatibility can be used to identify whether two policies are different.
Based on this observation, we propose a novel training objective that produces behaviorally diverse policies.
Finally, we incorporate a mutual information objective that encourages each policy to learn local variations.
3.1. Identifying Distinct Policies with Policy Compatibility
In this section, we motivate our objective by looking at two joint policies: π A = ( π 1 A , π 2 A ) and π B = ( π 1 B , π 2 B ) .
The goal is for π A to learn a different solution from π B via the compatibility criterion.
Importantly, the compatibility criterion can be computed without direct access to the trajectory distribution, which can be difficult to estimate.
Under the following assumptions, we can simplify the setting such that a simple relationship between similarity measure and compatibility criterion emerges.
Assumption 3.1.
A common ϵ is used for Def. 2.1 and 2.
Assumption 3.2.
P ( τ | π iB , π jB ) > 0; ∀ τ ∈ T
Assumption 3.3.
G ( τ ) > 0; ∀ τ ∈ T
By reasoning about the expected return under different pairs of policies, we derive our main result.
(The proof can be found in Appendix A.)
Theorem 3.4.
If π iA is similar to π iB with π jB as a a reference, then π iA is compatible with π B .
Corollary 3.5 (Contrapositive of Theorem 3.4) .
If π iA is not compatible with π B , then π iA is not similar to π iB .
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure] [Equation] [Figure] The result from Corollary 3.5 shows that we can find a policy π iA that is not similar to π iB by decreasing its compatibility ratio with π B until they are incompatible, i.e.,
C ( π iA , π B ) < 1 − ϵ .
As a result, we can create π A that is not similar to π B by C ( π 1 A π B ) and C ( π 2 A π B ) .
Additionally, we can ensure that π A learns a meaningful solution by maximizing J ( π 1 A , π 2 A ) .
Assuming that π B has learned a solution and is fixed, the optimization objective of π A can be written as
[Equation]
where λ XP > 0 is a hyper-parameter controlling importance of the compatibility term.
Since the denominators of the compatibility ratios do not depend on π A , therefore, the objective becomes
[Equation]
In practice, we use the parameter sharing technique for better sample efficiency and faster convergence (Tan, 1993;
Foerster et al., 2018; Rashid et al., 2018).
Assuming that a policy π iA is a neural network parameterized by θ iA , this means that for a joint policy ( π 1 A , π 2 A ) , we have θ 1 A
= θ
2 A .
Still, π 1 A and π 2 A can behave differently as they observe different parts of the environment and have a different player indicator concatenated with their local observations.
We denote the expected joint return of self-play (SP) trajectories – where a policy would interact with a clone of itself – as
J SP ( π A ) :
=
J ( π 1 A , π 2 A ) and expected joint return of crossplay (XP) trajectories – where policy for each agent will be chosen from a different joint policy – as J XP ( π A , π B ) :
=
J ( π 1 A , π 2 B ) + J ( π 1 B , π 2 A ) .
By incorporating the parameter sharing technique, we can rewrite Eq.
(1) as
[Equation]
3.2. Learning a Population of Diverse Policies
We propose to use a population of agents P = { π A | 1 ≤
A ≤ N } where each policy is represented by a neural network with a set of parameters θ A and N is the population [Figure] MI-based [Figure]
Self-play
TrajeDi
LIPO [Figure] size. Ultimately, we want each member of the population to have a different behavior relative to the rest of the population .
We can write such an objective by expanding the cross-play term in Eq.
(2) to cover the entire population.
For a policy π A in a population P , its objective becomes
[Equation]
Intuitively, π A should behave differently from every policy in the population.
So, we use the max operation as an aggregation function in Eq.
We refer to J LIPO as the compatibility gap between a policy π A and a population P .
Fig.
2 shows graphical interpretation of this objective.
We can see that the compatibility gap objective only uses the expected return ( J SP and J XP ), and therefore it is insensitive to the state and action information from trajectories.
We argue that this distinction of LIPO helps the agents discover diverse solutions in situations in which previous methods might be ineffective (Sec. 4.1).
Fig.
3 shows the difference between the training objective of our method and other works.
3.3. Inducing Variations in Each Policy
In the previous section, we propose to find a dissimilar policy π A by maximizing the compatibility gap.
It is important to note that π A could also have different local variations to its solution that still satisfy the compatibility gap,
J LIPO ( π A , P ) . For example, it is possible that there exists different behaviors that are fully compatible.
We propose to capture such behavioral variations by using a mutual information objective (Kumar et al., 2020; Osa et al., 2022; Lucas & Allen, 2022).
Specifically, we condition the policy on a latent variable z such that a policy π A has the form of
[Equation]
where p ( z 1 , z 2 ) is a pre-defined prior distribution.
We can promote solution variety by maximizing I ( { o 1 , a 1 } ; z 1 ) and
I ( { o 2 , a 2 } ; z 2 ) , where I ( · ; · ) is the mutual information between two random variables.
Intuitively, this objective encourages each policy to observe different observation and perform different actions given different values of the latent variable.
However, maximizing I ( { o i , a i } ; z i ) directly is intractable, instead we optimize the lower bound of the mutual information (Kingma & Welling, 2013; Mahajan et al., 2019; Osa et al., 2022):
[Equation]
where q ϕ A ( z i |
o
i , a i ) is an approximation of the true posterior p ( z i |
o i , a i ) parameterized by ϕ A .
Thus, maximizing
I ( { o 1 , a 1 } ; z 1 ) and I ( { o 2 , a 2 } ; z 2 ) is an optimization problem that can be written as
[Equation]
In previous work, shared z (i.e., z 1 = z 2 ) allows all policies to collectively switch between different modes of behavior (Mahajan et al., 2019).
However, LIPO uses independently sampled z as it utilizes z for a different purpose.
Specifically, LIPO maximizes J LIPO to learn diverse solutions and optimizes the MI objective to learn variations of each solution.
That is, the MI objective does not impact the diversity between different policies but increases variations of a learned solution of each individual policy.
Importantly, different variations of a policy π A must be compatible with each other to still maximize J SP ( π A ) .
3.4. Implementation
while do for A do B sp ← GetEpisodeRollouts( π
Compute J SP ( π A ) using B sp B xp ← GetCrossPlayRollouts( π A , P , E XP )
Compute ˜ J XP ( π A , P ) using B xp (Eq. 4)
Compute L MI (Eq. 6) using B sp and B xp θ A ← θ A − ∇ θ A [ − J SP + λ XP ˜ J XP + λ MI L MI ] ϕ
A ← ϕ A − λ MI ∇ ϕ
A L MI Function GetEpisodeRollouts( π A , π B , E ) :
B ← {} for episode ∈ { 1 , .., E 2 } do z 1 , z 2 ∼ p ( z 1 , z 2 )
τ AB ∼ ρ ( π 1 A ( ·|· , z 1 ) , π 2 B ( ·|· , z 2 ))
τ BA ∼ ρ ( π 1 B ( ·|· , z 1 ) , π 2 A ( ·|· , z 2 ))
B ← B ∪ { τ AB , τ BA } return B
Function GetCrossPlayRollouts( π A , P , E XP ) :
B xp ← {} for π B ∈ P − A do
B ← GetEpisodeRollouts( π A , π B , E XP |P − A | )
B xp ← B xp ∪ B return B xp
We implement LIPO on top of the multi-agent version of
PPO (MAPPO) (Schulman et al., 2017; Yu et al., 2021), which is an on-policy algorithm.
For simplicity, we use a feed-forward architecture for all neural networks.
Each member A ∈ { 1 , ..., N } has a joint policy ( π 1 A , π 2 A ) and a discriminator q ϕ A .
MAPPO is used for both maximizing
J SP and minimizing J XP .
We train all joint policies in the population concurrently.
In practice, we modify the MI objective (Eq. 5) such that it is differentiable with respect to the policy π iA .
Specifically, we modify the variational posterior q ϕ
A such that, instead of a sampled action a i , it takes the whole distribution π iA ( ·| o i , z i ) as an input, i.e., q ϕ A ( z | o, π iA ( ·| o i , z i )) .
In contrast to previous MI-based approaches (Eysenbach et al., 2018; Sharma et al., 2019; Jiang & Lu, 2021; Lucas & Allen, 2022), we can optimize I ( { o i , a i } ; z i ) directly without computing an auxiliary reward (Mahajan et al., 2019; Osa et al., 2022).
The loss function of the modified MI objective is
[Equation]
We set z as a discrete variable and use the uniform distribution for p ( z 1 ) and p ( z 2 ) .
At the beginning of each episode, each policy is given an independently sampled z that will be used until the end of the episode.
The overall objective of a policy π A in a population P becomes
[Equation]
In each training iteration, LIPO collects self-play and crossplay trajectories of all policies combinations to compute
J SP , ˜ J XP and L MI .
Algorithm 1 shows the pseudocode for the training process of LIPO.
4. Experiments
In this section, we study the effectiveness of LIPO under two cooperative environments to answer the following questions: (i) Can LIPO discover diverse solutions?
(ii) How does the
MI objective affect the behavior of LIPO agents?
In this work, we compared LIPO with cooperative MARL methods that do not require domain knowledge.
Our baselines include:
Multiple runs of self-play (Multi SP):
A simple but effective way to produce multiple agents that
could
learn
different
solutions by training multiple SP agents that have different neural network initializations and random seeds (Charakorn et al ., 2020; Strouse et al
Specifically, each run produces a joint policy π A that maximizes J SP ( π A ) using MAPPO.
Self-play with MI (SP MI ): A single run of SP agent trained with added MI objective I ( z | o i , a i ) .
SP MI uses a shared z for both policies and consider each z as a different joint policy.
We train SP MI using the same training procedure as LIPO by setting N = 1 , λ XP = 0 and z 1 =
z 2 .
MAVEN (Mahajan et al
): An algorithm designed specifically for learning diverse solutions in cooperative multi-agent environments.
A joint policy is represented as π ( ·| τ, z ) and each mode of behavior is represented by the latent variable z .
Similar to SP MI , MAVEN uses a shared z for all policies.
Multi SP MI and Multi MAVEN: A population containing joint policies from multiple runs of a corresponding method.
Like Multi SP, each run has different neural network initializations and random seeds.
TrajeDi (Lupu et al., 2021)
A method that produces a population of diverse agents that also maximize the expected return in cooperative environments.
The diversity measure of this method is based on the Jensen-Shannon divergence (JSD) between the trajectory distribution of each policy.
More details of all environments and baselines can be found in the appendix.
[Figure] 4.1. Can LIPO discover diverse solutions?
We start our investigation in (i) One-Step Cooperative Matrix Game (CMG) and (ii) Point Mass Rendezvous (PMR).
One-Step Cooperative Matrix Game (CMG): A game of CMG is defined by a tuple ( M, { k m } , { r m } ) where M is the number of solution.
For m ∈ { 1 , ..., M } , k m is the number of compatible actions and r m is the reward of a solution m .
The game is stateless and terminate immediately after both players simultaneously choose an action.
By choosing the same solution, both player get a reward r m associated with the chosen solution.
We consider two setups of CMG: sub-optimal solutions (CMG-S) and hard-to-find solutions (CMG-H).
We set ( M = 32 , k m = 8 , r m = 0 .
5 ∗ (1 + m − 1 M − 1 ) ) and ( M = 32 , k m = m , r m = 1 ) for CMG-S and CMG-H respectively.
An example payoff matrix is shown in Fig. 4a.
Point Mass Rendezvous (PMR): The environment is based on the Multi-Agent Particle Environment (Lowe et al., 2017;
Terry et al., 2020).
The goal of this environment is for the two agents to navigate to a landmark together.
Doing so will reward both agents as long as they stay on top of a landmark.
We consider each landmark as a solution in this environment.
There are two modes in this environment: Circle (PMR-C) and Line (PMR-L).
In Circle , landmarks are distributed evenly on a circumference of a circle.
Thus, all landmarks are equally optimal.
In Line , landmarks are placed in a vertical line.
In this scenario, closer landmarks are easier to be found, while further landmarks are harder to be discovered.
The population size of MI-based methods ( SP MI and
MAVEN) is equal to the number of dimensions of the latent variable | z | .
In methods that require multiple runs (Multi SP MI and Multi MAVEN), the population size is n × | z | where n is the number of runs.
For Multi SP MI and
Multi MAVEN, we use | z | = 8 in all environments.
For population-based methods (Multi SP, TrajeDi, and LIPO), the population size is the number of joint policies in the population.
Results: Fig. 5 shows the number of learned solutions as the population size increases.
Ideally, if the population size increases by one, one more solution should be discovered, as depicted by the dashed line in the figure.
In all environments,
LIPO can discover more solutions than the baselines given the same population size.
LIPO is also better at discovering sub-optimal solutions as illustrated in CMG-S and PMR-L (Fig. 5a,5d).
Also, we have experimented with large λ MI for the baselines.
This helps the baselines discover more solutions.
However, if λ MI is too large, we find that the baselines fail to produce capable policies.
Hyperparameters of all methods can be found in the appendix.
4.2. How does the MI objective effect the behavior of LIPO agents?
Here, we investigate further into how the MI objective affects the behaviors of produced policies.
Fig. 6 shows the behaviors of policies produced by LIPO with and without the MI objective in PMR-C. We can see the effect of the
MI objective in the distributions of the trajectories, which exhibit larger variations given a small MI regularization λ MI = 0 .
With or without the MI regularization, LIPO discovers all the landmarks with N = 4 .
5. Related Work
Mutual information objectives have been used in reinforcement learning to learn diverse behaviors.
In the single-agent domain, (Kumar et al., 2020; Osa et al., 2022) propose to optimize the RL and MI objective simultaneously.
The
MI objective helps the agent learn diverse solutions for a specific task and increases generalization to unseen environment instances.
A similar idea has also been applied to the cooperative multi-agent domain.
For instance, MAVEN (Mahajan et al., 2019) optimizes RL and an MI objective to encourage the agents to explore in a committed manner and discover diverse solutions.
Also, Any-play (Lucas & Allen, 2022) uses a similar objective to produce training partners with many solutions for an adaptive agent.
In contrast, our approach uses the MI objective to regularize each policy to learn local variations of each solution.
Population-based training is another technique that has been proposed to learn a population of diverse solutions.
DvD (Parker-Holder et al., 2020) introduces a metric distance between policies using policy-embedding and learns diverse solutions by maximizing the distance between policies while also maximizing the expected return.
In the multi-agent cooperative domain, (Canaan et al., 2019; 2020) use a Quality
Diversity (QD) algorithm (Mouret & Clune, 2015; Pugh et al., 2016) to produce a population of agents with a different behavioral niche attached to each agent. QD, however, [Figure] [Figure]
Population
[Figure] D i s c o v e r e d s o l u t i o n s
[Figure]
[Figure]
[Figure]
D i s c
o
v e r e d s o l u t i o n s (c) Number of learned solutions in PMR
[Figure]
[Figure] [Figure] requires domain knowledge to encode different types of behaviors. RPG (Tang et al., 2021) obtain diverse policies by randomizing the factorized reward function, which is not applicable in general.
Finally, TrajeDi (Lupu et al., 2021) produces a diverse population of agents, based on the trajectory distribution, that shares a common best response policy.
Like TrajeDi, ours does not require domain-specific knowledge.
However, LIPO uses state-action information for learning local variations and utilizes expected returns of different pairs of policies to promote behavioral diversity.
The idea of using the empirical return in multi-agent games as a diversity metric has been explored in the context of finding diverse solutions in non-transitive competitive games (Liu et al., 2021;
Balduzzi et al., 2019; Perez-Nieves et al., 2021).
In particular, Liu et al.
share some similar ideas with this work.
They propose to use both expected return when encountering different opponents and state-action information to promote diversity in a population of agents in zero-sum games.
LIPO can be thought of as an alternative approach designed specifically for cooperative environments.
6. Discussion and Conclusion
Discussion An agent trained with LIPO are incentivized to act adversarially toward agents that behave differently from itself.
This behavior might not be desirable for certain downstream tasks.
For example, agents produced by LIPO might not be suitable for interacting with humans as they would refuse to conform with the user, which is the same downside found in Multi SP agents (Carroll et al., 2019;
Bard et al., 2020; Hu et al., 2020).
However, we believe that training an adaptive agent with these agents, which is the main motivation of this work, would have an opposite effect, in that the adaptive agent would try to comply with what its current partner is doing.
In previous work, adaptive agents trained with Multi SP have shown such a behavior (Charakorn et al., 2020; Strouse et al., 2021).
[Figure] Limitations Although LIPO can be fully parallelized, it requires more computation than the baselines to get an accurate approximation of ˜ J XP , which makes it harder to scale up to bigger population size.
Instead of collecting all policy pairs, sampling a portion of policy pairs to approximate ˜ J XP could reduce computation cost and training time.
Additionally, LIPO requires an additional hyperparameter λ XP .
If λ XP is too big, it is possible that a joint policy would focus on minimizing ˜ J XP and struggle to maximize J SP , which will result in an incompetent joint policy.
An adaptive mechanism that selects a suitable value for λ XP at different stages of training could help increase training stability.
Future work Evaluating the effectiveness of produced policies in different downstream tasks could give additional metrics for "goodness" of the populations.
For example, they can be used as training partners for an adaptive agent or training a neural network for agent modeling.
These tasks will be included in our future work.
Furthermore, more complex cooperative environments will be included in future work to properly challenge LIPO and understand its potential failure modes.
Conclusion We propose LIPO, a method that can create a population of diverse agents in cooperative multi-agent environments.
Unlike previous work that uses state-action information from trajectories, LIPO utilizes the concept of compatible policies to create diverse policies.
This alternative view of producing diverse agents makes LIPO more robust to the state and action space of the environments.
On top of learning diverse solutions, LIPO uses the MI objective to learn local variations of each solution.
Empirically, LIPO can produce more solutions than the baselines in various settings. Acknowledgement
We thank Natchaya Sricom for drawing Fig.
1 and
A. Proof for Theorem 3.
Theorem.
If π iA is similar to π iB with π jB as a a reference, then π iA is compatible with π B .
Proof.
Let r ( τ ) =
P ( τ | π iA ,π jB ) P ( τ | π iB ,π jB ) .
Because π iA is similar to π iB , then 1 − ϵ ≤ r ( τ ) ≤ 1 + ϵ ; ∀ τ (Def. 2.1).
From the definition of the expected return of a policy pair in Section 2, we have
[Equation]
Assume that Def. 2.1 and 2.2 share a common ϵ , and
P ( τ | π iB , π jB ) > 0 and G ( τ ) > 0 for all τ ∈ T .
Then, we can use importance sampling to write J ( π iA , π jB ) in relation to J ( π i B , π j B ) :
[Equation]
Because 1 − ϵ ≤ r ( τ ) ≤ 1 + ϵ , loose upper and lower bounds of J ( π iA , π jB ) can be written as:
[Equation]
This means that π iA is compatible with π B (Def. 2.2).
∴ If π i A is similar to π i B , then π i A is compatible with π B .
B. Additional Environments Details B.1. Point Mass Rendezvous (PMR)
PMR is based on the the Multi-Agent Particle Environment (Lowe et al., 2017; Terry et al., 2020).
The observation of each agent is relative positive of the landmarks and the other agent to itself.
In PMR-C, the start positions of the agents are {(0.3,0), (-0.3,0)} and the landmarks positions are {(1.59, 1.59), (1.59, -1.59), (-1.59, 1.59), (-1.59, -1.59)}.
For
PMR-L, the start and the landmark positions are {(1,0),(0,1)} and {(0,2.25),(0,0.75),(0,-0.75),(0,-2.25)}.
An episode will be terminated after 50 timesteps.
The agents are incentivized to go to the same landmark and stay close together with the reward function
[Table]
where d ( · , · ) is the euclidean distance between two points, p i is the 2-d coordinate of agent i , c is the average coordinate of all agents, and L is the set of all landmarks.
C. Implementation Details
[Table] [Equation] timesteps
[Equation]
All methods are implemented on top of MAPPO except
MAVEN.
The critic, policy, and discriminator q ϕ are 3-layer feed-forward neural networks where each hidden layer has 64 units.
For a fair comparison, we use the same or more environment steps in the policy update of the baselines compared to LIPO.
Common hyperparameters of methods based on MAPPO are shown in Table.
For a fair comparison, the baselines collect additional data to have the same or more timesteps than LIPO.
C.1. MAPPO
MAPPO is the base MARL algorithm for all baselines except MAVEN.
The policy parameters are shared among all policies.
The critic takes a state of the environment and outputs an expected return of a given global state.
The global state is provided by the environment and only used during training.
For the full training objectives of MAPPO, we refer the reader to Appendix A of Yu et al.
C.2. Multi SP
A population of Multi SP is obtained by simply running multiple runs of the MAPPO algorithm.
Each run collects a total of T timesteps.
C.3. SP MI
SP MI is based on the MAPPO algorithm.
In addition to policy and critic networks, a discriminator network q ϕ is implemented using the same feed-forward architecture.
It takes a local observation o i and action distribution π i ( ·| o i ) as inputs and outputs the discrete probability of the latent variable.
The latent variable of all policies is shared during an episode.
It collects a total of 2 | z | T timesteps and produces | z | joint policies.
C.4. MAVEN
We use the same network architecture presented in (Mahajan et al., 2019) with recurrent neural networks.
However, we do not use the hierarchical policy but sample z from the uniform distribution.
The latent variable of all policies is shared during an episode.
Similar to SP MI , it collects a total of 2 | z | T timesteps and produces | z | joint policies.
C.5. Multi SP MI and Multi MAVEN
A population is produced by multiple runs of each base algorithm.
The final population size is n | z | , where n is the number of runs.
Notably, this baseline uses the training data differently from the base algorithms.
Instead of training a long single run, this approach allows the policy to "restart" by using different initialization of neural networks.
For example, let N = n | z | , training a single run with | z | =
N, n = 1 might not discover as many solutions as training n runs with | z | =
Nn .
Empirically, we find that multiple runs of a base algorithm can find more solutions compared to a longer run of the algorithm.
C.6. TrajeDi
TrajeDi is implemented on top of the MAPPO algorithm.
Different from the original implementation, we remove the best-response (BR) policy from the population.
Since the
BR policy might work well with only a specific solution, removing BR potentially increase the number of variations in the population.
Our modified loss is:
[Equation]
where JSD is the proposed diversity objective of TrajeDi.
α and γ are the hyperparameters of TrajeDi.
C.7. LIPO
LIPO use the same implementation as SP MI except LIPO uses independent latent variable for each policy and λ XP > 0 .
Additionally, LIPO uses extra critics for the cross-play trajectories.
In total, LIPO has a self-play critic V π A sp and
N − 1 cross-play critics { V π A ,π B xp | π B ∈ P − A } .
It uses 2 NT timesteps for policy updates and produces
N policy.
Specifically, each joint policy uses T timesteps from self-play trajectories and another T timesteps from max( ˜ J XP ) .
We set λ MI as 0.0 and 0.5 in CMG and PMR respectively.
D. Hyperparameters
We provide the possible search values of each method in
Tables 2, 3, 4, 5, 6 and 7.
The hyperparameters are searched individually for each population size.
We use three random seeds for each set of hyperparameters.
We do not use any validation method; instead, we present the results using the best parameters in the main paper.
E. Additional Results E.1. Sensitivity of λ XP
We provide additional results of LIPO with varying values of λ XP in Fig.
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
[Table]
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
[Table]
[Table]
[Table] [Figure]
i s c
o v e r e d s
[Figure]
l u t i o n s
[Figure]
D i s c
o
v e r e d s o l u t i o n s
[Figure]