A Uniﬁed Diversity Measure for Multiagent Reinforcement Learning
Abstract
Promoting behavioural diversity is of critical importance in multi-agent reinforcement learning, since it helps the agent population maintain robust performance when encountering unfamiliar opponents at test time, or, when the game is highly non-transitive in the strategy space (e.g., Rock-Paper-Scissor).
While a myriad of diversity metrics have been proposed, there are no widely accepted or uniﬁed deﬁnitions in the literature, making the consequent diversity-aware learning algorithms difﬁcult to evaluate and the insights elusive.
In this work, we propose a novel metric called the Uniﬁed Diversity Measure (UDM) that offers a uniﬁed view for existing diversity metrics.
Based on UDM, we design the UDM-Fictitious
Play (UDM-FP) and UDM-Policy Space Response Oracle (UDM-PSRO) algorithms as efﬁcient solvers for normal-form games and open-ended games.
In theory, we prove that UDM-based methods can enlarge the gamescape by increasing the response capacity of the strategy pool, and have convergence guarantee to two-player Nash equilibrium.
We validate our algorithms on games that show strong non-transitivity, and empirical results show that our algorithms achieve better performances than strong PSRO baselines in terms of the exploitability and population effectivity .
Introduction
Diversity is a widely studied topic in machine learning, including autonomous driving [52], gaming AI [46], recommender systems [6], generative models [13], and latent variable models [51].
Specially, in Multi-Agent Reinforcement Learning (MARL) [4, 20, 55, 59], diversity of agents’ strategies is helpful for learning different skills [14], facilitating explorations [42], and discovering sophisticated cooperative policies [27] or an automated agenda towards unexploitable policies in authors. competitive games [15, 29].
Promoting diversity of strategies is also an effective method for solving games with non-transitive dynamics [1, 40, 12, 46].
In general, an arbitrary game, of either the normal-form type [5] or the differential type [2], can always be decomposed into a sum of a transitive part and a non-transitive part, where the former represents a transitive rule (i.e., if A beats B, B beats C, then A beats C), while the latter represents a cyclic rule (i.e., the cycles among Rock, Paper and Scissors).
In fact, many real-world games demonstrate strong non-transitivity [8], and thus each player must use a diverse pool of winning strategies to achieve low exploitability [53], which has been justiﬁed recently by the super-human performance of AIs in sophisticated tasks like StarCraft [48, 43] and DOTA2 [3, 57].
In speciﬁc, encouraging the diversity of strategies is critical in achieving high-level learning performance due to the following two main aspects: (1) in the training process, diversity prevents agents from checking the same strategies repeatedly [40], and helps them explore strategy space sufﬁciently [42]; and (2) at test time, diversity not only prevents agents from being exploited (i.e., a single strategy can always be beaten by another one on non-transitive games [40, 30]), but also helps them learn adaptable strategies and thus maintain robust performances [50, 30].
Due to its great importance, there are a number of studies that investigate the deﬁnition of diversity and propose diversity-driven MARL algorithms.
The majority of work has followed a heuristic approach [40], such as evolutionary computation [16] with a central focus that mimics the natural evolution process.
In evolutionary computation, one idea for encouraging diversity is novelty search [24] that searches for strategies leading to novel outcomes.
Quality-diversity (QD) [44] combines novelty search with ﬁtness objective, resulting two representative methods: Novelty Search with Local Competition [25] and MAP-Elites [7, 37].
Although these methods achieve empirical successes [21, 7], quantifying diversity in evolutionary computation is often hand-crafted and taskdependent [40].
There are also various metrics proposed to model diversity in a more rigorous way, which can be mainly classiﬁed into Behavioral Diversity (BD) and Response Diversity (RD)
A main principle in BD to characterize diversity is to construct metrics over the trajectory or stateaction distribution [28, 14, 27, 42].
However, this kind of methods only focus on the behavioral trajectories and ignore the reward attributes of strategies, which may be problematic in scenarios when a slight difference in strategy behaviors will lead to a huge difference in the corresponding reward [30].
Different from BD, RD directly uses the empirical payoff matrix to construct the diversity of strategies [1, 40, 30], and thus it estimates the diversity from the response when encountering distinct opponents.
Despite of the above great efforts, there are still no consistent formal deﬁnitions for diversity, making it difﬁcult to evaluate the diverse strategies in MARL.
In this paper, we work towards offering a consistent deﬁnition for diversity, and propose a novel population-wide diversity measure called the
Uniﬁed Diversity Measure (UDM) that provides a uniﬁed view for the existing metrics, including
Effective Diversity (ED)
[1], Population Diversity (PD) [42] and Expected Cardinality (EC) [40].
UDM can be interpreted from a geometric perspective through analyzing the meanings of the eigenvalues of a kernel matrix.
Using UDM, we can also analyze the advantages and shortcomings of the existing metrics, and study why ED and PD cannot measure the diversity properly in certain cases.
With our task-agnostic metric, we design the corresponding diversity-promoting objective and develop novel MARL algorithms, i.e., UDM Fictitious Play (UDM-FP) and UDM Policy-Space
Response Oracle (UDM-PSRO), for solving normal-form games and open-ended games.
Theoretically, we prove that our algorithms can enlarge the gamescape [1], i.e., the convex hull of the row vectors in the payoff matrix, by increasing the response capacity of the strategy pool, and can converge on two-player games.
We validate our diversity-aware algorithms on games that show strong non-transitivity including the real-world matrix games, Blotto and non-transitive mixture model, and the empirical results show that our algorithms achieve lower exploitability [23] and higher population effectivity [30] than the baselines.
Preliminary 2.1 Concepts of Games
Normal-Form Games and Open-Ended Meta-Games.
Normal-form games (NFGs) are denoted by ⟨ N, S , G ⟩ , where N is the number of players.
Each player n ∈ [ N ] has a ﬁnite set of pure strategies S n .
Denote the space of joint pure-strategy proﬁles as S = ∏ n ∈ [ N ] S n , and the space of joint strategy proﬁles except the n -th player as S − n .
G ( S ) =
( G 1 ( S ) , · · · , G N ( S ))
∈ R N is a payoff table mapping each joint strategy proﬁle S ∈ S to a vector of reward values for each player.
A mixed strategy (also called policy) of player n is denoted by π n ∈ ∆ S n , where ∆ is a probability simplex.
π ( S ) =
∏ n ∈ [ N ] π n ( S ) represents the probability of joint strategy proﬁle S .
If all players follow π , the expected payoff of player n is denoted as G n ( π ) =
G ( π n , π − n ) = ∑ S ∈ S π ( S ) G n ( S ) .
However, it is inefﬁcient to describe real-world games with NFGs, where the strategy space can be prohibitively large by enumerating all possible game plans [8].
For example, the number of atomic actions in StarCraft is 10 26 at every time-step and the number of time-steps per episode is typically several thousands [47], in which case the number of game plans can be more than (10 26 ) 1000 .
To describe this type of games, meta-games denoted by ⟨ N, S , M⟩ are introduced to consider a policy as a “high-level” strategy, i.e., the meta-strategy .
With slight abuse of notation, the space of meta-strategies is still denoted by S n for each player n ; and thus π n ∈ ∆ S n denotes the meta-policy of player n .
Additionally, meta-games are always open-ended since there could be inﬁnite meta-strategies to play a game.
A meta-game payoff M can be described by a game engine g : S → R n .
For example, in two-player games, the meta-game payoff can be described as M = { g ( S 1 , S 2 ) :
( S 1 , S 2 ) ∈ S 1 × S 2 } , where g ( S 1 , S 2 ) > 0 if S 1 ∈ S 1 beats S 2 ∈ S 2 , while g < 0 and g = 0 refer to losses and ties, respectively.
A game is symmetric if S 1 = S 2 .
A game is transitive if there exists a monotonic rating function h such that g ( S 1 , S 2 )
=
h ( S 1 ) − h
( S 2 ) , ∀ S 1 ∈ S, S 2 ∈ S 2 ; and non-transitive if ∫ S 2 g ( S 1 , S 2 )d S 2 = 0 , ∀ S 1 ∈ S 1 , which means that each strategy can be counterbalanced by another strategy.
Solution Concepts of Games.
In a game, a (possibly mixed) strategy π n is a best response for player n against joint strategy π − n , if π n ∈ BR n ( π − n ) , where BR n ( π − n ) := arg max ˜ π ∈ ∆ S n [ G n (˜ π , π − n )] .
A Nash Equilibrium (NE) [39] is a joint strategy π if π n ∈
BR n ( π − n ) , ∀ n .
An ϵ -NE is a joint strategy π if π n ∈ BR nϵ ( π − n ) , ∀ n , where BR nϵ := { π n : G n ( π n , π − n ) ≥ G n (˜ π , π − n ) − ϵ } , ˜ π ∈ BR n ( π − n ) .
Computing NE is PPAD-hard in Nplayer general-sum games [9]; and thus α -Rank [41] is proposed as an alternative solution concept, which has a polynomial-time solvability in multi-player general-sum games.
A more detailed description of α -Rank can be found in Appendix A.1.
Game Solvers.
A game solver consists of an Oracle function and a (meta-)policy solver , where the Oracle function O computes strategies given joint strategies, such as the best-response Oracle O n ( π ) =
BR n ( π − n ) , and the (meta-)policy solver S computes the (meta)-policy based on the payoff table.
At each time step, the game solver uses the policy solver to compute a policy for each player based on the current payoff; and then based on this joint policy, it uses O ( · ) to ﬁnd a new strategy for each player and adds it in their populations.
Variations of (meta-)game solvers are summarised in Table 1.
[Table]
2.2 Evaluation Metrics Empirical Gamescape.
The empirical gamescape (EGS) [1] in a meta-game is deﬁned as the convex hull of the payoff vectors of all meta-strategies in S n , written as EGS ( S n ) :
= {∑ i α i · m i : α ≥ 0 , α T · 1 = 1 , m i =
M [ i, :] } , which represents the response capacity of a pool of strategies in meta-games.
Exploitability
The exploitability [10] measures the “distance” of a joint strategy π to the NE, deﬁned as Exploit.
( π ) :
= ∑ n ∈ [ N ] [ G n (˜ π , π − n ) − G n ( π )] , ˜ π ∈ BR n ( π − n ) .
Population Effectivity.
The population effectivity (PE) [30] is a fairer evaluation metric to represent the effectiveness of a population (i.e., a strategy pool) than exploitability: PE ( S n ) :
[-=min π − n max 1 T α =1 ,α k ≥∑ | S n | k =
-]= min [Equation]
[Equation]
2.3 Existing Diversity Measures
There are various diversity metrics to measure the diversity of a population.
Effective Diversity.
Effective Diversity (ED)
[1] is a metric proposed to measure the diversity of a pool of effective-strategies (strategies with support under the NE) in two-player zero-sum symmetric games as follows: [Equation]
[Equation]
where ⌊ x ⌋ + = x if x > 0 and ⌊ x ⌋ + = [-
-]0 if x ≤ 0 .
M is the meta-payoff table of S n and π ∗ is the
NE of the M .
Expected Cardinality.
Inspired by the derterminantal point process (DPP) [31, 22], the diversity of a population can be increased through increasing the Expected Cardinality (EC) [40] of the random set Y ⊆ S n drawn from the population:
[Equation]
where | Y | means the cardinality of Y , and L n = MM T .
Form of Euclidean Projection.
To characterize the contribution of a payoff vector to the enlargement of the EGS directly, Form of Euclidean Projection (FEP) [30] is deﬁned as the form of
Euclidean projection:
[Equation]
[Equation]
) where S nnew is the new meta-strategy of player n , and a T new :=
( g n ( S n new , S − n j ))
j .
Note that FEP only measures the contribution of the new meta-strategy to the current population, rather than the diversity of the population.
Population Diversity. Population Diversity
(PD) [42] uses the determinant to measure the diversity by:
[Equation]
where K ni,j = K ( ϕ ni , ϕ nj ) , and ϕ ni = { π ni ( ·| s ) } s ( s means the state in games) is the behavioral embedding of a meta-strategy.
Here we rename PD as Reward-PD (RPD) when it replaces the behavioral embedding with ϕ ni =
M [ i, :] .
Methods
Based on the existing metrics, we now offer a uniﬁed view for them by introducing a novel diversity measure.
Specially, this measure is based on a geometric interpretation of the diversity of strategies, and is capable of offering a comprehensive analysis of these existing diversity metrics.
3.1 A Uniﬁed Diversity Measure
There are various methods to represent a strategy in the existing studies.
In RD, a fundamental way to represent a strategy is through the row vector of the empirical payoff matrix [40, 30], since each row in this matrix embeds the response of the corresponding strategy against different opponents; and in BD, the trajectory or the action-state distribution is often used to characterize the corresponding strategy [42, 30].
To represent a strategy more ﬂexibly, we introduce the strategy feature :
Deﬁnition 1 (Strategy Feature) .
Let S ni ∈ S n denote the i -th (meta-)strategy for player n , then the strategy feature of S ni is deﬁned as a vector : ϕ ni ∈ R 1 × p , p ≤ M =: | S n | , such that ϕ ni
=
ϕ
nj ⇐⇒
S ni = S nj , where ∀ S ni , S nj ∈ S n .
Hence, we can represent the i -th strategy by ϕ ni
= m i
=:
M [ i, :] in RD, or ϕ ni = { π ni ( ·| s ) } s in BD.
Equipped with the strategy feature, we can then deﬁne the diversity kernel to measure the pairwise similarity as follows:
Deﬁnition 2 (Diversity Kernel) .
Consider a ﬁnite population S n consisting of M (meta-)strategies.
The diversity kernel of player n is deﬁned as a positive semi-deﬁnite (PSD) matrix: L nK := [ K ( ϕ ni , ϕ nj )]
M × M , where K : R d × R d → R is a given kernel function, such that | K ( ϕ ni , ϕ nj ) | ≤ C and K ( ϕ ni , ϕ ni ) =
C , C > 0 .
Given a PSD matrix L = BB T , where B ∈ R D 1 × D 2 and D 1 ≥ D 2 .
The geometric interpretation of det( L ) is the squared volume of the parallelepiped spanned by the rows of B [22].
Further more, the determinant of diversity kernel represents the squared volume of a parallelepiped spanned by strategy feature corresponding to the kernel choice [42].
Thus, a population can be considered diverse if its parallelepiped can ﬁll the strategy space as much as possible.
A natural idea to measure the diversity of a population is to use the determinant of the diversity kernel, which is the intuition of PD (RPD) introduced in section 2.3.
However, since the determinant becomes zero with duplicated rows, PD (RPD) cannot deal with the redundant-strategy problem which turns out to be a critical challenge for game evaluation [1] (See more discussions in Appendix A.2.).
To avoid this problem, we use the additive term over eigenvalues rather than the product of eigenvalues (i.e., the determinant) to measure the diversity and propose the Uniﬁed Diversity Measure as follows:
Deﬁnition 3 (Uniﬁed Diversity Measure) .
Consider a function
[Equation]
where R is the convergence domain of f .
Denote the eigenvalues of L nK constructed by S n = {
S n 1 , · · · , S nM } as λ i ≥ 0 .
Then the Uniﬁed Diversity Measure (UDM) of the population S n is deﬁned as follows:
[Equation]
Since the values and the number of the non-zero eigenvalues represent the lengths and the number of the edges of the parallelepiped, respectively, increasing UDM can help enlarge the volume of the parallelepiped, which indicates a more diverse strategy population.
Otherwise, considering only the number of non-zero eigenvalues (i.e., the rank of the diversity kernel) will result in the difference instead of the diversity being measured [40, 56].
For example, in RPS, the meta-strategy [0.99 Rock, 0.01 Paper] is different from [0.98 Rock, 0.02 Paper], but they are not diverse since they both favor playing Rock.
Meanwhile, we offer an equivalent representation of UDM which can be computed more easily:
Proposition 1 (Equivalent Representation of UDM) .
The UDM deﬁned in Deﬁnition 3 has an equivalent representation:
[Equation]
where the deﬁnition of f ( L nK ) is similar to the matrix exponentials.
Proof.
See Appendix A.3.1.
Using f ( λ i ) rather than λ
i directly is vital for UDM.
In the case where f ( λ i )
=
λ i , UDM will degenerate to Tr( L n ) =
∑ i L nii , where L nii =
K ( ϕ ni , ϕ ni ) only captures the modulus of S ni and thus ∑ i L n ii ignores the similarity L n ij between S n i and S n j , i ̸
=
j , making it unable to measure the diversity properly.
Instead, using bounded concave functions f ( λ i ) that involve the pairwise similarity, such as f ( λ i ) = [Equation] +
[Equation] ( − [Equation] i[-) − 11+ γ , γ > 
-]
[Equation] − 11+ [Equation] , γ > 0 (the term 11+ γ is applied to satisfy f (0) = 0 ), can force the player to explore more novel strategies since this function with a supremum gains more marginal beneﬁt from adding a new non-zero eigenvalue rather than increasing an already large eigenvalue.
Given the convexity of f ( x ) , UDM becomes concave due to the following proposition:
Proposition 2 (Convexity of UDM) .
Consider a concave function f ∈ F .
Then UDM is concave if all the eigenvalues of L nK exist in the convergence domain of f .
Proof.
See Appendix A.3.2.
3.2 Unify Existing Metrics into UDM
By mathematically transformations, we argue that some existing diversity metrics, especially the
RD metrics, are the special cases of UDM, which are summarized in Table 2.
[Table]
UDM vs. ED.
Formally, ED is equivalent to the L 1 , 1 norm of the payoff matrix in two-player symmetric zero-sum games [1]:
[Equation]
where π ∗ is the NE on M , ⊙ is the Hadamard product (i.e., element-wise product), and
∥
A ∥ 1 , 1 := ∑ i,j | a i,j | .
Denote M ∗ :
= π ∗ ⊙ M ⊙ π ∗ T , then:
[Equation]
where λ ∗ i are the eigenvalues of M ∗ M ∗ T .
Hence, ED can be covered by UDM with K ( · , · ) = ⟨· , ·⟩ and f ( x ) = x .
Using the strategy feature ϕ i
= m ∗ i
=: M ∗ [ i, :] and function f ( x ) = x in ED encourages the player to amplify its strengths and ignore its weaknesses in ﬁnding a new metastrategy, by focusing only on the strong strategies.
However, it can sometimes be problematic since weak strategies may be useful to tackle niche tasks and discover stronger strategies later during training [40].
A counter example that fails ED is the RPS-X game [38]:
[Equation]
In RPS-X, if the initial population is either { R } , { P } or { S } , then PSRO-rN [1], which is a variation of PSRO equipped with ED, will terminate in { R, P, S } without ﬁnding { X } , since the best response of { R, P, S } is still in { R, P, S } .
However, this population will be exploited by others using strategy
X endlessly at test time.
We show that how our method tackles this problem in Appendix A.2.
UDM vs. PD (RPD).
We consider the case that L nK has full rank here since PD (RPD) does not work when the rank of diversity kernel is not full.
Then PD (RPD) is actually UDM with f ( x ) =
ln( x ) and the linear kernel function:
[Equation]
When the new eigenvalue is less than 1, PD (RPD) would decrease and thus may ignore some weak but useful meta-strategies.
In fact, through the Eq.
(6) and the Jacobi formula [32]: det (exp ( A )) = exp
Tr ( A ) , A ∈ R M × M , we can ﬁnd that UDM also uses a determinant to measure the diversity.
However, UDM is different from PD (RPD) since the exponential of the diversity kernel is nondecreasing with duplicated rows, and thus UDM is well-deﬁned when dealing with the redundantstrategy problem.
We give an example tackles
UDM vs. EC.
EC can be rewritten as:
[Equation]
where λ i are the eigenvalues of L n := MM T .
Hence, EC is a special case of QDM when f ( x ) =
x/ (1 + x ) , K ( · , · ) = ⟨· , ·⟩ and strategy feature ϕ i = m i .
To summarize, the existing diversity metrics are special cases of UDM with explicit feature ϕ ni , kernel function K , and function f .
By exploring novel settings of these components, we can then propose a more reasonable metric to circumvent the shortcomings of existing diversity metrics.
Algorithms
Inspired by [40], we extend the classic FP, PSRO and α -PSRO to their diverse versions by incorporating UDM into their Oracle functions, and prove their convergences in two-player games.
4.1 UDM Fictitious Play
At each iteration t , UDM-FP discovers a new strategy that gains a higher payoff and at the same time enriches the current population.
Formally, UDM-FP only modiﬁes the best response as follows:
[Equation]
(7) where τ t is tunable constant; and the population is updated by adding the new strategy π n ∈
BR nτ t ( π − n t ) so as S nt +1 ← S nt ∪ { π n } .
Intuitively, as t → ∞ , UDM-FP will almost surely converge to generalised weakened ﬁctitious play (GWFP) [26] as long as τ t → 0 , and thus it has the same convergence guarantees with GWFP which converges to the NE on two-player zero-sum games or potential games (A more detailed description of GWFP can be found in Appendix A.1).
So, we have the following proposition:
Proposition 3 (Convergence of UDM-FP) .
If UDM is concave, and UDM-FP uses the update rule:
[Equation]
where α t = o
(1 / log t ) is deterministic and perturbations
U nt +
are the differences between the actual and expected changes in strategies.
Then UDM-FP shares the same convergence property as GWFP: the policy sequence π nt converges to the NE on two-player zero-sum games or potential games.
Proof.
See Appendix [-A.3.
-]A.3.
4.2 UDM Policy-Space Response Oracle
In open-ended (meta-)games, PSRO [23, 33, 49, 58, 35] is a more effective method than the direct search methods since there are inﬁnite (meta-)strategies.
Here we extend our diversity metric to PSRO and develop UDM-PSRO for open-ended (meta-)games.
Suppose that player n has learned a population S nt at t -th iteration.
The goal of UDM-PSRO is to ﬁnd a new (meta-)strategy S θ parameterised by θ ∈ R d , which maximizes the payoffs of player n and the diversity of S nt .
Therefore, the Oracle function is:
[Equation]
(8) where π − n is the (meta-)policy of the player − n .
(Meta-)policy π n is computed by (meta-)policy solver, such as NE, UNIFORM, etc.
UDM-PSRO can enlarge EGS when it adds a new (meta-)strategy via Eq.
(8); and thus the population learned by UDM-PSRO has a great response capacity.
Unlike UDM-PSRO, PSRO-rN can also enlarge EGS, but it needs to assume that both players should use their Nash policies.
Proposition 4 (EGS Enlargement) .
Adding a new (meta-)strategy S θ via Eq.
(8) enlarges EGS.
Formally, we have EGS( S n ) ⊆
EGS( S ∪ { S θ } ) .
Proof.
See Appendix A.3.4.
4.3 UDM α -Policy-Space Response Oracle
UDM-PSRO suffers from the shortcomings of high complexity (i.e., PPAD-hard) in computing the
NE in N -player general-sum games [9, 11] and the equilibrium-selection problems [19, 18].
To avoid these problems, UDM α -PSRO replaces NE with α -Rank [54] in the meta-policy solver.
Suppose that there are L sink strongly-connected components (SSCC) nodes that have only incoming edges but no outgoing edges in the response graph of the game; and denote the α -Rank distribution of l -th SSCC as π ( l ) , which can be considered as the meta-policies for different types of opponents.
Inspired by EC α -PSRO [40], we design an Oracle function that suits α -Rank for UDM α -PSRO: [Equation]
[Equation]
where the diversity kernel L n = [ q i K ( ϕ i , ϕ j ) q j ] M × M is constructed by the strategy feature ϕ
i = m i /
∥ m i ∥ and the quality term
q i :
[Equation]
In fact, Eq.
(9) celebrates two aspects of diversity.
The ﬁrst aspect is from UDM that forces players to ﬁnd more diverse populations; and the second aspect is from L meta-polices π ( l ) , which represents diverse opponents.
The following proposition shows the convergence of UDM on two-player symmetric NFGs.
Proposition 5 (Convergence of UDM α -PSRO) .
In two-player symmetric NFGs, UDM α -PSRO converges to the sub-cycle of the unique SSCC.
Proof.
See Appendix [-A.3.
-]A.3.
Experiments
To validate the diverse game solvers proposed in Section 4, we compare them with the baselines including self-play [17], PSRO [23], Pipeline-PSRO [34], PSRO-rN [1], FEP-PSRO [30], and
EC-PSRO [40].
Their performances are investigated in several games involving both transitive and non-transitive dynamics.
If an algorithm cannot ﬁnd a diverse and effective population, it will easily stuck in local strategy cycles, and thus be exploited by others.
Hence, exploitability and PE are used as evaluation metrics in the experiments.
We focus on RD metrics and thus use the strategy feature ϕ ni
= m i , and use the kernel function K ⟨ x, y ⟩ = ( ⟨ x, y ⟩ + 1) 3 since the number of features is the same with the number of strategies.
The function in UDM-PSRO is a concave function f ( x ) = 1+exp( − x ) − 1 2 ∈ F (see Appendix A.5 for more discussions on the selection of f and K ) .
More experimental settings and results can be found in Appendix A.4.
Real-World Meta-Game.
The properties of some complex real-world meta-games are studied in paper [8], including AlphaStar and AlphaGO.
In Figure 1, we report the performances of different algorithms over the AlphaStar game, which contains the meta-payoffs for 888 RL meta-strategies.
The results show that our method achieves the smallest exploitability, largest population effectivity and the largest diversity, while baselines without diverse objectives are easily exploited since they do not ﬁnd diverse meta-strategies.
Our method performs better since more diverse meta-strategies can help it jump out the current strategic cycles and thus ﬁnd better strategies.
Blotto.
Blotto is a classical resource allocation game that is often used to analyze electoral competition [45].
In this game, each of two players has a budget of coins which will be distributed simultaneously over a ﬁxed number of areas.
A player wins an area where it puts the most coins; and the player that wins the most areas wins the game.
We show the performances of different algorithms in this game with 3 areas and 10 coins in Figure 1; and the results show that our methods perform as well as the diversity-aware baselines, and outperforms baselines without diversity objectives in terms of expolitability and PE. [Figure] o i [Figure]
E x p l
o i t a b ili t y
[Figure] Non-Transitive Mixture Model.
Non-Transitive Mixture Model [40] is a zero-sum two-player game consisting of seven equally-distanced Gaussian humps on the 2D plane.
Each meta-strategy corresponds to a point on the plane, and each point is translated into a 7-dimensional vector S ni with each coordinate being the density in the corresponding Gaussian distribution function.
Since the number of points is inﬁnite, this game is open-ended, and it involves both a transitive component and a non-transitive component, which can be reﬂected by the payoff:
[Equation]
Therefore, a player should stay close to the center of the Gaussian and explore all the Gaussian distributions equally to avoid being exploited.
We report the exploration trajectories when solving this game using different algorithms in Figure 2, which shows that our method can indeed generate diverse trajectories.
We also report the exploitability and PE values for the ﬁnal population generated by different algorithms in Table 3.
It can be found that our method outperforms all the baselines in terms of PE, which is a better metric to evaluate diverse populations [30]
[Table]
[Figure]
[Table]
Conclusion
In this paper, we offer a consistent deﬁnition for the diversity of strategies in MARL, and propose a novel diversity measure that provides a uniﬁed view for existing diversity metric.
With this measure, we develop corresponding diversity-promoting algorithms based on FP and PSRO.
In theory, we prove that our method converges to two-player NE and can enlarge gamescape by increasing the response capacity of the population.
Empirically, we validate our method on games that show strong non-transitive including the matrix games, Blotto and non-transitive mixture model.
The results show that our method outperforms the baselines in terms of the exploitability and population effectivity.
In this paper, we mainly focus on the two-player games due to the expensive computational cost of UDM in n-player cases (see Appendix 5 for more explanations).
Investigating how to reduce the computational cost when extending UDM to n-player, general-sum, or non-symmetric games can be an important future work
Acknowledgments and Disclosure of Funding
The authors would like to thank anonymous reviewers for helpful feedback and discussions.
This work was supported by the SYSU-ByteDance Research Project and the National Natural Science
Foundation of China under Grant
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[N/A] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See Appendix A. (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Appendix A. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]