A Uniﬁed Diversity Measure for Multiagent Reinforcement Learning
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Promoting behavioural diversity is of critical importance in multi-agent reinforcement learning, since it helps the agent population maintain robust performancewhen encountering unfamiliar opponents at test time, or, when the game is highlynon-transitive in the strategy space (e.g., Rock-Paper-Scissor).
While a myriadof diversity metrics have been proposed, there are no widely accepted or uniﬁeddeﬁnitions in the literature, making the consequent diversity-aware learning algorithms difﬁcult to evaluate and the insights elusive.
In this work, we proposea novel metric called the Uniﬁed Diversity Measure (UDM) that offers a uniﬁedview for existing diversity metrics.
Based on UDM, we design the UDM-Fictitious
Play (UDM-FP) and UDM-Policy Space Response Oracle (UDM-PSRO) algorithms as efﬁcient solvers for normal-form games and open-ended games.
In theory, we prove that UDM-based methods can enlarge the gamescape by increasingthe response capacity of the strategy pool, and have convergence guarantee totwo-player Nash equilibrium.
We validate our algorithms on games that showstrong non-transitivity, and empirical results show that our algorithms achieve better performances than strong PSRO baselines in terms of the exploitability andpopulation effectivity .
Introduction
Diversity is a widely studied topic in machine learning, including recommender systems [6], generative models [11] and latent variable models [43].
Specially, in Multi-Agent Reinforcement Learning(MARL) [4, 17, 45], diversity of agents’ strategies is helpful for learning different skills [12], facilitating explorations [36] and discovering sophisticated cooperative policies [24].
Promoting diversityof strategies is also an effective method for solving games with non-transitive dynamics [1, 34, 26].
In general, an arbitrary game, of either the normal-form type [5] or the differential type [2], canalways be decomposed into a sum of a transitive part and a non-transitive part, where the formerrepresents a transitive rule (i.e., if A beats B, B beats C, then A beats C), while the latter representsa cyclic rule (i.e., the cycles among Rock, Paper and Scissors).
In fact, many real-world gamesdemonstrate strong non-transitivity [8], and thus each player must use a diverse pool of winningstrategies to achieve low exploitability [44], which has been justiﬁed recently by the super-humanperformance of AIs in sophisticated tasks like StarCraft [41, 37] and DOTA2 [3, 46].
In speciﬁc, encouraging the diversity of strategies is critical in achieving high-level learning performance due to the following two main aspects: (1) in the training process, diversity preventsagents from checking the same strategies repeatedly [34], and helps them explore strategy spacesufﬁciently [36]; and (2) at test time, diversity not only prevents agents from being exploited (i.e., asingle strategy can always be beaten by another one on non-transitive games [34, 26]), but also helpsthem learn adaptable strategies and thus maintain robust performances [42, 26].
Due to its great importance, there are a number of studies that investigate the deﬁnition of diversity and propose diversity-driven MARL algorithms.
The majority of work has followed a heuristicapproach [34], such as evolutionary computation [13] with a central focus that mimics the natural evolution process.
In evolutionary computation, one idea for encouraging diversity is noveltysearch [21] that searches for strategies leading to novel outcomes.
Quality-diversity (QD) [38] combines novelty search with ﬁtness objective, resulting two representative methods: Novelty Searchwith Local Competition [22] and MAP-Elites [7, 31].
Although these methods achieve empiricalsuccesses [18, 7], quantifying diversity in evolutionary computation is often hand-crafted and taskdependent [34].
There are also various metrics proposed to model diversity in a more rigorous way,which can be mainly classiﬁed into Behavioral Diversity (BD) and Response Diversity (RD)
A main principle in BD to characterize diversity is to construct metrics over the trajectory or stateaction distribution [25, 12, 24, 36].
However, this kind of methods only focus on the behavioraltrajectories and ignore the reward attributes of strategies, which may be problematic in scenarioswhen a slight difference in strategy behaviors will lead to a huge difference in the correspondingreward [26].
Different from BD, RD directly uses the empirical payoff matrix to construct the diversity of strategies [1, 34, 26], and thus it estimates the diversity from the response when encounteringdistinct opponents.
Despite of the above great efforts, there are still no consistent formal deﬁnitions for diversity, making it difﬁcult to evaluate the diverse strategies in MARL.
In this paper, we work towards offering aconsistent deﬁnition for diversity, and propose a novel population-wide diversity measure called the
Uniﬁed Diversity Measure (UDM) that provides a uniﬁed view for the existing metrics, including
Effective Diversity (ED)
[1], Population Diversity (PD) [36] and Expected Cardinality (EC) [34].
UDM can be interpreted from a geometric perspective through analyzing the meanings of the eigenvalues of a kernel matrix.
Using UDM, we can also analyze the advantages and shortcomings ofthe existing metrics, and study why ED and PD cannot measure the diversity properly in certaincases.
With our task-agnostic metric, we design the corresponding diversity-promoting objectiveand develop novel MARL algorithms, i.e., UDM Fictitious Play (UDM-FP) and UDM Policy-Space
Response Oracle (UDM-PSRO), for solving normal-form games and open-ended games.
Theoretically, we prove that our algorithms can enlarge the gamescape [1], i.e., the convex hull of the rowvectors in the payoff matrix, by increasing the response capacity of the strategy pool, and can converge on two-player games.
We validate our diversity-aware algorithms on games that show strongnon-transitivity including the real-world matrix games, Blotto and non-transitive mixture model, andthe empirical results show that our algorithms achieve lower exploitability [20] and higher population effectivity [26] than the baselines.
Preliminary2.1 Concepts of Games
Normal-Form Games and Open-Ended Meta-Games.
Normal-form games (NFGs) are denoted by ⟨ N, S , G ⟩ , where N is the number of players.
Each player n ∈ [ N ] has a ﬁnite set of purestrategies S n .
Denote the space of joint pure-strategy proﬁles as S = ∏ n ∈ [ N ] S n , and the spaceof joint strategy proﬁles except the n -th player as S − n .
G ( S ) =
( G 1 ( S ) , · · · , G N ( S ))
∈
R N is apayoff table mapping each joint strategy proﬁle S ∈ S to a vector of reward values for each player.
Amixed strategy (also called policy) of player n is denoted by π n ∈ ∆ S n , where ∆ is a probability simplex.
π ( S ) =
∏ n ∈ [ N ] π n ( S ) represents the probability of joint strategy proﬁle S .
If all players follow π , the expected payoff of player n is denoted as
[Equation]
However, it is inefﬁcient to describe real-world games with NFGs, where the strategy space canbe prohibitively large by enumerating all possible game plans [8].
For example, the number ofatomic actions in StarCraft is 10 26 at every time-step and the number of time-steps per episodeis typically several thousands [40], in which case the number of game plans can be more than(10 26 ) 1000 .
To describe this type of games, meta-games denoted by ⟨ N, S , M⟩ are introduced toconsider a policy as a “high-level” strategy, i.e., the meta-strategy .
With slight abuse of notation,the space of meta-strategies is still denoted by S n for each player n ; and thus π n ∈ ∆ S n denotesthe meta-policy of player n .
Additionally, meta-games are always open-ended since there couldbe inﬁnite meta-strategies to play a game.
A meta-game payoff M can be described by a gameengine g : S → R n .
For example, in two-player games, the meta-game payoff can be described as
M = { g ( S 1 , S 2 ) :
( S 1 , S 2 ) ∈ S 1 × S 2 } , where g ( S 1 , S 2 ) > 0 if S 1 ∈ S 1 beats S 2 ∈ S 2 , whileg < 0 and g = 0 refer to losses and ties, respectively.
A game is symmetric if S 1 = S 2 .
A game istransitive if there exists a monotonic rating function h such that g ( S 1 , S 2 )
=
h ( S 1 ) − h
( S 2 ) , ∀ S 1 ∈ 93 S, S 2 ∈ S 2 ; and non-transitive if ∫ S 2 g ( S 1 , S 2 )d S 2 = 0 , ∀ S 1 ∈ S 1 , which means that each strategycan be counterbalanced by another strategy.
Solution Concepts of Games.
In a game, a (possibly mixed) strategy π n is a best response for player n against joint strategy π − n , if π n ∈ BR n ( π − n ) , where BR n ( π − n ) :=arg max ˜ π ∈ ∆ S n [ G n (˜ π , π − n )] .
A Nash Equilibrium (NE) [33] is a joint strategy π if π n ∈
BR n ( π − n ) , ∀ n .
An ϵ -NE is a joint strategy π if π n ∈ BR nϵ ( π − n ) , ∀ n , where BR nϵ := 99 { π n : G n ( π n , π − n ) ≥ G n (˜ π , π − n ) − ϵ } , ˜ π ∈ BR n ( π − n ) .
Computing NE is PPAD-hard in Nplayer general-sum games [9]; and thus α -Rank [35] is proposed as an alternative solution concept,which has a polynomial-time solvability in multi-player general-sum games.
A more detailed description of α -Rank can be found in Appendix A.1.
Game Solvers.
A game solver consists of an Oracle function and a (meta-)policy solver , wherethe Oracle function O computes strategies given joint strategies, such as the best-response Oracle 105 O n ( π ) =
BR n ( π − n ) , and the (meta-)policy solver S computes the (meta)-policy based on thepayoff table.
At each time step, the game solver uses the policy solver to compute a policy foreach player based on the current payoff; and then based on this joint policy, it uses O ( · ) to ﬁnd anew strategy for each player and adds it in their populations.
Variations of (meta-)game solvers aresummarised in Table 1.
[Table]
2.2 Evaluation Metrics
Empirical Gamescape.
The empirical gamescape (EGS) [1] in a meta-game is deﬁned asthe convex hull of the payoff vectors of all meta-strategies in S n , written as EGS ( S n ) :
= 113 {
∑ i α i · m i : α ≥ 0 , α T · 1 = 1 , m i =
M [ i, :] } , which represents the response capacity of a poolof strategies in meta-games.
Exploitability.
The exploitability [10] measures the “distance” of a joint strategy π to the NE,deﬁned as Exploit.
( π ) :
= ∑ n ∈ [ N ] [ G n (˜ π , π − n ) − G n ( π )] , ˜ π ∈ BR n ( π − n ) .
Population Effectivity.
The population effectivity (PE) [26] is a fairer evaluation metric torepresent the effectiveness of a population (i.e., a strategy pool) than exploitability: PE ( S n ) :
{+=min π − n max 1 T α =1 ,α k ≥∑ | S n | k =
+}
α k G n ( S nk , π − n ) .
2.3 Existing Diversity Measures
There are various diversity metrics to measure the diversity of a population.
Effective Diversity.
Effective Diversity (ED)
[1] is a metric proposed to measure the diversity of apool of effective-strategies (strategies with support under the NE) in two-player zero-sum symmetricgames as follows:
[Equation]
where ⌊ x ⌋ + = x if x > 0 and ⌊ x ⌋ + =
if x ≤ 0 .
M is the meta-payoff table of S n and π ∗ is the
NE of the M .
Expected Cardinality.
Inspired by the derterminantal point process (DPP) [27, 19], the diversity of a population can be increased through increasing the Expected Cardinality (EC) [34] of therandom set Y ⊆ S n drawn from the population:
[Equation]
where | Y | means the cardinality of Y , and L n = MM T .
Form of Euclidean Projection.
To characterize the contribution of a payoff vector to the enlargement of the EGS directly, Form of Euclidean Projection (FEP) [26] is deﬁned as the form of
Euclidean projection:
[Equation]
where S nnew is the new meta-strategy of player n , and a T new :
= ( g n ( S n new , S − n j ))
j .
Note that FEPonly measures the contribution of the new meta-strategy to the current population, rather than thediversity of the population.
Population Diversity.
Population Diversity (PD) [36] uses the determinant to measure the diversity by:
[Equation]
where K ni,j = K ( ϕ ni , ϕ nj ) , and ϕ ni = { π ni ( ·| s ) } s ( s means the state in games) is the behavioralembedding of a meta-strategy.
Here we rename PD as Reward-PD (RPD) when it replaces thebehavioral embedding with ϕ ni =
M [ i, :] .
Methods
Based on the existing metrics, we now offer a uniﬁed view for them by introducing a novel diversitymeasure.
Specially, this measure is based on a geometric interpretation of the diversity of strategies,and is capable of offering a comprehensive analysis of these existing diversity metrics.
3.1 A Uniﬁed Diversity Measure
There are various methods to represent a strategy in the existing studies.
In RD, a fundamental way torepresent a strategy is through the row vector of the empirical payoff matrix [34, 26], since each rowin this matrix embeds the response of the corresponding strategy against different opponents; andin BD, the trajectory or the action-state distribution is often used to characterize the correspondingstrategy [36, 26].
To represent a strategy more ﬂexibly, we introduce the strategy feature :
Deﬁnition 1 (Strategy Feature) .
Let S ni ∈ S n denote the i -th (meta-)strategy for player n , then thestrategy feature of S ni is deﬁned as a vector : ϕ ni ∈ R 1 × p , p ≤ M
=
| S n | , such that ϕ ni =
ϕ
nj ⇐⇒
S ni = S nj , where ∀ S ni , S nj ∈ S n .
Hence, we can represent the i -th strategy by ϕ ni
= m i
=:
M [ i, :] in RD, or ϕ ni = { π ni ( ·| s ) } s in BD.
Equipped with the strategy feature, we can then deﬁne the diversity kernel to measure the pairwisesimilarity as follows:
Deﬁnition 2 (Diversity Kernel) .
Consider a ﬁnite population S n consisting of M (meta-)strategies.
The diversity kernel of player n is deﬁned as a positive semi-deﬁnite (PSD) matrix: L nK :=[ K ( ϕ ni , ϕ nj )]
M × M , where K : R d × R d → R is a given kernel function, such that | K ( ϕ ni , ϕ nj ) | ≤ Cand K ( ϕ ni , ϕ ni ) =
C , C > 0 .
Given a PSD matrix L = BB T , where B ∈ R D 1 × D 2 and D 1 ≥ D 2 . The geometric interpretationof det( L ) is the squared volume of the parallelepiped spanned by the rows of B [19]. Further more,the determinant of diversity kernel represents the squared volume of a parallelepiped spanned bystrategy feature corresponding to the kernel choice [36].
Thus, a population can be considereddiverse if its parallelepiped can ﬁll the strategy space as much as possible.
A natural idea to measurethe diversity of a population is to use the determinant of the diversity kernel, which is the intuitionof PD (RPD) introduced in section 2.3.
However, since the determinant becomes zero with duplicated rows, PD (RPD) cannot deal with the redundant-strategy problem which turns out to be acritical challenge for game evaluation [1] (See more discussions in Appendix A.2.).
To avoid this problem, we use the additive term over eigenvalues rather than the product of eigenvalues (i.e., thedeterminant) to measure the diversity and propose the Uniﬁed Diversity Measure as follows:
Deﬁnition 3 (Uniﬁed Diversity Measure) .
Consider a function
[Equation]
where R is the convergence domain of f .
Denote the eigenvalues of L nK constructed by S n = 175 {
S n 1 , · · · , S nM } as λ i ≥ 0 .
Then the Uniﬁed Diversity Measure (UDM) of the population S n isdeﬁned as follows:
[Equation]
Since the values and the number of the non-zero eigenvalues represent the lengths and the numberof the edges of the parallelepiped, respectively, increasing UDM can help enlarge the volume of theparallelepiped, which indicates a more diverse strategy population.
Otherwise, considering only thenumber of non-zero eigenvalues (i.e., the rank of the diversity kernel) will result in the differenceinstead of the diversity being measured [34].
For example, in RPS, the meta-strategy [0.99 Rock,0.01 Paper] is different from [0.98 Rock, 0.02 Paper], but they are not diverse since they both favorplaying Rock.
Meanwhile, we offer an equivalent representation of UDM which can be computed more easily:
Proposition 1 (Equivalent Representation of UDM) .
The UDM deﬁned in Deﬁnition 3 has an equivalent representation:
[Equation]
(6) where the deﬁnition of f ( L nK ) is similar to the matrix exponentials.
Proof.
See Appendix A.3.1.
Using f ( λ i ) rather than λ
i directly is vital for UDM.
In the case where f ( λ i )
=
λ i , UDM willdegenerate to Tr( L n ) =
∑ i L nii , where L nii =
K ( ϕ ni , ϕ ni ) only captures the modulus of S ni andthus
∑ i L nii ignores the similarity L nij between S ni and S nj , i ̸
=
j , making it unable to measurethe diversity properly.
Instead, using bounded concave functions f ( λ i ) that involve the pairwisesimilarity, such as f ( λ i )
= 1 1+ γ exp( − λ i ) − 11+ γ , γ >
(the term 11+ γ is applied to satisfy f (0) = 0 ),can force the player to explore more novel strategies since this function with a supremum gains moremarginal beneﬁt from adding a new non-zero eigenvalue rather than increasing an already largeeigenvalue.
Given the convexity of f ( x ) , UDM becomes concave due to the following proposition:
Proposition 2 (Convexity of UDM) .
Consider a concave function f ∈ F .
Then UDM is concave ifall the eigenvalues of L nK exist in the convergence domain of f .
Proof.
See Appendix A.3.2.
3.2 Unify Existing Metrics into UDM
By mathematically transformations, we argue that some existing diversity metrics, especially the
RD metrics, are the special cases of UDM, which are summarized in Table 2.
[Table]
UDM vs. ED.
Formally, ED is equivalent to the L 1 , 1 norm of the payoff matrix in two-playersymmetric zero-sum games [1]:
[Equation]
where π ∗ is the NE on M , ⊙ is the Hadamard product (i.e., element-wise product), and ∥
A ∥ 1 , 1 := 206 ∑ i,j | a i,j | .
Denote M ∗ :
= π ∗ ⊙ M ⊙ π ∗ T , then:
[Equation]
where λ ∗ i are the eigenvalues of M ∗ M ∗ T .
Hence, ED can be covered by UDM with K ( · , · ) = ⟨· , ·⟩and f ( x ) = x .
Using the strategy feature ϕ i
= m ∗ i =: M ∗ [ i, :] and function f ( x ) = x in EDencourages the player to amplify its strengths and ignore its weaknesses in ﬁnding a new metastrategy, by focusing only on the strong strategies.
However, it can sometimes be problematic sinceweak strategies may be useful to tackle niche tasks and discover stronger strategies later duringtraining [34].
A counter example that fails ED is the RPS-X game [32]:
[Equation]
[Equation]
In RPS-X, if the initial population is either { R } , { P } or { S } , then PSRO-rN [1], which is a variationof PSRO equipped with ED, will terminate in { R, P, S } without ﬁnding { X } , since the best responseof { R, P, S } is still in { R, P, S } .
However, this population will be exploited by others using strategy
X endlessly at test time.
We show that how our method tackles this problem in Appendix A.2.
UDM vs. PD (RPD).
We consider the case that L nK has full rank here since PD (RPD) doesnot work when the rank of diversity kernel is not full.
Then PD (RPD) is actually UDM withf ( x ) =
ln( x ) and the linear kernel function:
[Equation]
When the new eigenvalue is less than 1, PD (RPD) would decrease and thus may ignore some weakbut useful meta-strategies.
In fact, through the Eq.
(6) and the Jacobi formula [28]: det (exp ( A )) =exp
Tr ( A ) , A ∈ R M × M , we can ﬁnd that UDM also uses a determinant to measure the diversity.
However, UDM is different from PD (RPD) since the exponential of the diversity kernel is nondecreasing with duplicated rows, and thus UDM is well-deﬁned when dealing with the redundantstrategy problem.
We give an example to illustrate how UDM tackles this issue in Appendix A.2.
UDM vs. EC.
EC can be rewritten as:
[Equation]
where λ i are the eigenvalues of L n := MM T .
Hence, EC is a special case of QDM when f ( x ) =
x/ (1 + x ) , K ( · , · ) = ⟨· , ·⟩ and strategy feature ϕ i = m i .
To summarize, the existing diversity metrics are special cases of UDM with explicit feature ϕ ni ,kernel function K , and function f .
By exploring novel settings of these components, we can thenpropose a more reasonable metric to circumvent the shortcomings of existing diversity metrics.
Algorithms
Inspired by [34], we extend the classic FP, PSRO and α -PSRO to their diverse versions by incorporating UDM into their Oracle functions, and prove their convergences in two-player games.
4.1 UDM Fictitious Play
At each iteration t , UDM-FP discovers a new strategy that gains a higher payoff and at the same timeenriches the current population.
Formally, UDM-FP only modiﬁes the best response as follows:
[Equation]
where τ t is tunable constant; and the population is updated by adding the new strategy π n ∈
BR nτ t ( π − n t ) so as S nt +1 ← S nt ∪ { π n } .
Intuitively, as t → ∞ , UDM-FP will almost surely converge to generalised weakened ﬁctitious play(GWFP)
[23] as long as τ t → 0 , and thus it has the same convergence guarantees with GWFP whichconverges to the NE on two-player zero-sum games or potential games (A more detailed descriptionof GWFP can be found in Appendix A.1).
So, we have the following proposition:
Proposition 3 (Convergence of UDM-FP) .
If UDM is concave, and UDM-FP uses the update rule:
[Equation]
where α t = o
(1 / log t ) is deterministic and perturbations
U nt +
are the differences between theactual and expected changes in strategies.
Then UDM-FP shares the same convergence propertyas GWFP: the policy sequence π nt converges to the NE on two-player zero-sum games or potentialgames.
Proof.
See Appendix A.3.
4.2 UDM Policy-Space Response Oracle
In open-ended (meta-)games, PSRO is a more effective method than the direct search methods sincethere are inﬁnite (meta-)strategies.
Here we extend our diversity metric to PSRO and develop UDMPSRO for open-ended (meta-)games.
Suppose that player n has learned a population S nt at t -thiteration.
The goal of UDM-PSRO is to ﬁnd a new (meta-)strategy S θ parameterised by θ ∈ R d ,which maximizes the payoffs of player n and the diversity of S nt .
Therefore, the Oracle function is:
[Equation]
where π − n is the (meta-)policy of the player − n .
(Meta-)policy π n is computed by (meta-)policysolver, such as NE, UNIFORM, etc.
UDM-PSRO can enlarge EGS when it adds a new (meta-)strategy via Eq.
(8); and thus the population learned by UDM-PSRO has a great response capacity.
Unlike UDM-PSRO, PSRO-rN can alsoenlarge EGS, but it needs to assume that both players should use their Nash policies.
Proposition 4 (EGS Enlargement) .
Adding a new (meta-)strategy S θ via Eq.
(8) enlarges EGS.
Formally, we have EGS( S n ) ⊆
EGS( S ∪ { S θ } ) .
Proof.
See Appendix A.3.4.
4.3 UDM α -Policy-Space Response Oracle
UDM-PSRO suffers from the shortcomings of high complexity (i.e., PPAD-hard) in computing the
NE in N -player general-sum games [9] and the equilibrium-selection problems [16, 15].
To avoidthese problems, UDM α -PSRO replaces NE with α -Rank in the meta-policy solver.
Suppose thatthere are L sink strongly-connected components (SSCC) nodes that have only incoming edges but nooutgoing edges in the response graph of the game; and denote the α -Rank distribution of l -th SSCCas π ( l ) , which can be considered as the meta-policies for different types of opponents.
Inspired by
EC α -PSRO [34], we design an Oracle function that suits α -Rank for UDM α -PSRO:
[Equation]
where the diversity kernel L n = [ q i K ( ϕ i , ϕ j ) q j ] M × M is constructed by the strategy feature ϕ
i =m i /
∥ m i ∥ and the quality term
q i :
[Equation]
In fact, Eq.
(9) celebrates two aspects of diversity.
The ﬁrst aspect is from UDM that forces players toﬁnd more diverse populations; and the second aspect is from L meta-polices π ( l ) , which representsdiverse opponents.
The following proposition shows the convergence of UDM on two-player symmetric NFGs.
Proposition 5 (Convergence of UDM α -PSRO) .
In two-player symmetric NFGs, UDM α -PSROconverges to the sub-cycle of the unique SSCC.
Proof.
See Appendix A.3.
Experiments
To validate the diverse game solvers proposed in Section 4, we compare them with the baselinesincluding self-play [14], PSRO [20], Pipeline-PSRO [29], PSRO-rN [1], FEP-PSRO [26], and
EC-PSRO [34].
Their performances are investigated in several games involving both transitive andnon-transitive dynamics.
If an algorithm cannot ﬁnd a diverse and effective population, it will easilystuck in local strategy cycles, and thus be exploited by others.
Hence, exploitability and PE are usedas evaluation metrics in the experiments.
We focus on RD metrics and thus use the strategy featureϕ ni
= m i , and use the kernel function K ⟨ x, y ⟩ = ( ⟨ x, y ⟩ + 1) 3 since the number of features isthe same with the number of strategies.
The function in UDM-PSRO is a concave function f ( x ) =1 1+exp( − x ) − 12 ∈ F (see Appendix A.5 for more discussions on the selection of f and K ) .
Moreexperimental settings and results can be found in Appendix A.4.
[Figure]
Real-World Meta-Game.
The properties of some complex real-world meta-games are studied inpaper [8], including AlphaStar and AlphaGO.
In Figure 1, we report the performances of differentalgorithms over the AlphaStar game, which contains the meta-payoffs for 888 RL meta-strategies.
The results show that our method achieves the smallest exploitability, largest population effectivityand the largest diversity, while baselines without diverse objectives are easily exploited since theydo not ﬁnd diverse meta-strategies.
Our method performs better since more diverse meta-strategiescan help it jump out the current strategic cycles and thus ﬁnd better strategies.
Blotto.
Blotto is a classical resource allocation game that is often used to analyze electoral competition [39].
In this game, each of two players has a budget of coins which will be distributedsimultaneously over a ﬁxed number of areas.
A player wins an area where it puts the most coins; and the player that wins the most areas wins the game.
We show the performances of differentalgorithms in this game with 3 areas and 10 coins in Figure 1; and the results show that our methods perform as well as the diversity-aware baselines, and outperforms baselines without diversityobjectives in terms of expolitability and PE.
Non-Transitive Mixture Model.
Non-Transitive Mixture Model [34] is a zero-sum two-playergame consisting of seven equally-distanced Gaussian humps on the 2D plane.
Each meta-strategycorresponds to a point on the plane, and each point is translated into a 7-dimensional vector S ni witheach coordinate being the density in the corresponding Gaussian distribution function.
Since thenumber of points is inﬁnite, this game is open-ended, and it involves both a transitive componentand a non-transitive component, which can be reﬂected by the payoff:
[Equation]
Therefore, a player should stay close to the center of the Gaussian and explore all the Gaussiandistributions equally to avoid being exploited.
We report the exploration trajectories when solvingthis game using different algorithms in Figure 2, which shows that our method can indeed generatediverse trajectories.
We also report the exploitability and PE values for the ﬁnal population generatedby different algorithms in Table 3.
It can be found that our method outperforms all the baselines interms of PE, which is a better metric to evaluate diverse populations [26].
[Table]
[Figure]
[Table]
Conclusion
In this paper [Table] we offer a consistent deﬁnition for the diversity of strategies in MARL, and proposea novel diversity measure that provides a uniﬁed view for existing diversity metric.
With this measure [Table] we develop corresponding diversity-promoting algorithms based on FP and PSRO.
In theory,we prove that our method converges to two-player NE and can enlarge gamescape by increasingthe response capacity of the population.
Empirically, we validate our method on games that showstrong non-transitive including the matrix games, Blotto and non-transitive mixture model.
The results show that our method outperforms the baselines in terms of the exploitability and populationeffectivity.
In this paper, we mainly focus on the two-player games due to the expensive computational cost of UDM in n-player cases (see Appendix 5 for more explanations). Investigating howto reduce the computational cost when extending UDM to n-player, general-sum, or non-symmetricgames can be an important future work.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[N/A]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See AppendixA.4(b)
Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Appendix A.
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix A.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data
you’re using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁ-
able information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]