Original: A G ENERALIZED T RAINING A PPROACH FOR M ULTIAGENT L EARNING
Revised: A G ENERALIZED T RAINING A PPROACH FOR M ULTIAGENT L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: This paper investigates a population-based training regime based on game-theoretic principles called Policy-Spaced Response Oracles (PSRO).
Revised: This paper investigates a population-based training regime based on game-theoretic principles called Policy-Spaced Response Oracles (PSRO).

Original: PSRO is general in the sense that it (1) encompasses well-known algorithms such as ﬁctitious play and double oracle as special cases, and (2) in principle applies to general-sum, manyplayer games.
Revised: PSRO is general in the sense that it (1) encompasses well-known algorithms such as ﬁctitious play and double oracle as special cases, and (2) in principle applies to general-sum, manyplayer games.

Original: Despite this, prior studies of PSRO have been focused on two-player zero-sum games, a regime wherein Nash equilibria are tractably computable.
Revised: Despite this, prior studies of PSRO have been focused on two-player zero-sum games, a regime wherein Nash equilibria are tractably computable.

Original: In moving from two-player zero-sum games to more general settings, computation of Nash equilibria quickly becomes infeasible.
Revised: In moving from two-player zero-sum games to more general settings, computation of Nash equilibria quickly becomes infeasible.

Original: Here, we extend the theoretical underpinnings of PSRO by considering an alternative solution concept, α -Rank, which is unique (thus faces no equilibrium selection issues, unlike Nash) and applies readily to general-sum, many-player settings.
Revised: Here, we extend the theoretical underpinnings of PSRO by considering an alternative solution concept, α -Rank, which is unique (thus faces no equilibrium selection issues, unlike Nash) and applies readily to general-sum, many-player settings.

Original: We establish convergence guarantees in several games classes, and identify links between Nash equilibria and α -Rank.
Revised: We establish convergence guarantees in several games classes and identify links between Nash equilibria and α -Rank.

Original: We demonstrate the competitive performance of α -Rank-based PSRO against an exact Nash solver-based PSRO in 2-player Kuhn and Leduc Poker.
Revised: We demonstrate the competitive performance of α -Rank-based PSRO against an exact Nash solver-based PSRO in 2-player Kuhn and Leduc Poker.

Original: We then go beyond the reach of prior PSRO applications by considering 3- to 5-player poker games, yielding instances where α -Rank achieves faster convergence than approximate Nash solvers, thus establishing it as a favorable general games solver.
Revised: We then go beyond the reach of prior PSRO applications by considering 3- to 5-player poker games, yielding instances where α -Rank achieves faster convergence than approximate Nash solvers, thus establishing it as a favorable general games solver.

Original: We also carry out an initial empirical validation in MuJoCo soccer, illustrating the feasibility of the proposed approach in another complex domain.
Revised: We also carry out an initial empirical validation in MuJoCo soccer, illustrating the feasibility of the proposed approach in another complex domain

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Creating agents that learn to interact in large-scale systems is a key challenge in artiﬁcial intelligence.
Revised: Creating agents that learn to interact in large-scale systems is a key challenge in artiﬁcial intelligence.

Original: Impressive results have been recently achieved in restricted settings (e g., zero-sum, two-player games) using game-theoretic principles such as iterative best response computation (Lanctot et al., 2017), self-play (Silver et al., 2018) and evolution-based training (Jaderberg et al., 2019; Liu et al., 2019).
Revised: Impressive results have been recently achieved in restricted settings (e.g., zero-sum, two-player games) using game-theoretic principles such as iterative best response computation (Lanctot et al., 2017), self-play (Silver et al., 2018), and evolution-based training (Jaderberg et al., 2019; Liu et al., 2019).

Original: A key principle underlying these approaches is to iteratively train a growing population of player policies, with population evolution informed by heuristic skill ratings (e.g., Elo (Elo, 1978)) or game-theoretic solution concepts such as Nash equilibria.
Revised: A key principle underlying these approaches is to iteratively train a growing population of player policies, with population evolution informed by heuristic skill ratings (e.g., Elo (Elo, 1978)) or game-theoretic solution concepts such as Nash equilibria.

Original: A general application of this principle is embodied by the Policy-Space Response Oracles (PSRO) algorithm and its related extensions (Lanctot et al., 2017; Balduzzi et al., 2019).
Revised: A general application of this principle is embodied by the Policy-Space Response Oracles (PSRO) algorithm and its related extensions (Lanctot et al., 2017; Balduzzi et al., 2019).

Original: Given a game (e.g., poker), PSRO constructs a higherlevel meta-game by simulating outcomes for all match-ups of a population of players’ policies.
Revised: Given a game (e.g., poker), PSRO constructs a higherlevel meta-game by simulating outcomes for all match-ups of a population of players’ policies.

Original: It then trains new policies for each player (via an oracle ) against a distribution over the existing meta-game policies (typically an approximate Nash equilibrium, obtained via a meta-solver 1 ), appends these new policies to the meta-game population, and iterates.
Revised: It then trains new policies for each player (via an oracle ) against a distribution over the existing meta-game policies (typically an approximate Nash equilibrium, obtained via a meta-solver 1 ), appends these new policies to the meta-game population, and iterates.

Original: In two-player zero sum games, ﬁctitious play (Brown, 1951), double oracle (McMahan et al., 2003), and independent reinforcement learning can all be considered instances of PSRO, demonstrating its representative power (Lanctot et al., 2017).
Revised: In two-player zero sum games, ﬁctitious play (Brown, 1951), double oracle (McMahan et al., 2003), and independent reinforcement learning can all be considered instances of PSRO, demonstrating its representative power (Lanctot et al., 2017).

Original: Prior applications of PSRO have used Nash equilibria as the policy-selection distribution (Lanctot et al., 2017; Balduzzi et al., 2019), which limits the scalability of PSRO to general games: Nash equilibria are intractable to compute in general (Daskalakis et al., 2009); computing approximate
Revised: Prior applications of PSRO have used Nash equilibria as the policy-selection distribution (Lanctot et al., 2017; Balduzzi et al., 2019), which limits the scalability of PSRO to general games: Nash equilibria are intractable to compute in general (Daskalakis et al., 2009); computing approximate

Original: Nash equilibria is also intractable, even for some classes of two-player games (Daskalakis, 2013); ﬁnally, when they can be computed, Nash equilibria suffer from a selection problem (Harsanyi et al., 1988; Goldberg et al., 2013).
Revised: Nash equilibria is also intractable, even for some classes of two-player games (Daskalakis, 2013); ﬁnally, when they can be computed, Nash equilibria suffer from a selection problem (Harsanyi et al., 1988; Goldberg et al., 2013).

Original: It is, thus, evident that the reliance of PSRO on the Nash equilibrium as the driver of population growth is a key limitation, preventing its application to general games.
Revised: It is, thus, evident that the reliance of PSRO on the Nash equilibrium as the driver of population growth is a key limitation, preventing its application to general games.

Original: Recent work has proposed a scalable alternative to the Nash equilibrium, called α -Rank, which applies readily to general games (Omidshaﬁei et al., 2019), making it a promising candidate for population-based training.
Revised: Recent work has proposed a scalable alternative to the Nash equilibrium, called α -Rank, which applies readily to general games (Omidshaﬁei et al., 2019), making it a promising candidate for population-based training.

Original: Given that the formal study of PSRO has only been conducted under the restricted settings determined by the limitations of Nash equilibria, establishing its theoretical and empirical behaviors under alternative meta-solvers remains an important and open research problem.
Revised: Given that the formal study of PSRO has only been conducted under the restricted settings determined by the limitations of Nash equilibria, establishing its theoretical and empirical behaviors under alternative meta-solvers remains an important and open research problem.

Original: We study several PSRO variants in the context of general-sum, many-player games, providing convergence guarantees in several classes of such games for PSRO instances that use α -Rank as a meta-solver.
Revised: We study several PSRO variants in the context of general-sum, many-player games, providing convergence guarantees in several classes of such games for PSRO instances that use α -Rank as a meta-solver.

Original: We also establish connections between Nash and α -Rank in speciﬁc classes of games, and identify links between α -Rank and the Projected Replicator Dynamics employed in prior
Revised: We also establish connections between Nash and α -Rank in speciﬁc classes of games, and identify links between α -Rank and the Projected Replicator Dynamics employed in prior

Original: PSRO instances (Lanctot et al., 2017).
Revised: PSRO instances (Lanctot et al., 2017).

Original: We develop a new notion of best response that guarantees convergence to the α -Rank distribution in several classes of games, verifying this empirically in randomly-generated general-sum games.
Revised: We develop a new notion of best response that guarantees convergence to the α -Rank distribution in several classes of games, verifying this empirically in randomly-generated general-sum games.

Original: We conduct empirical evaluations in Kuhn and Leduc Poker, ﬁrst establishing our approach as a competitive alternative to Nash-based PSRO by focusing on two-player variants of these games that have been investigated in these prior works.
Revised: We conduct empirical evaluations in Kuhn and Leduc Poker, ﬁrst establishing our approach as a competitive alternative to Nash-based PSRO by focusing on two-player variants of these games that have been investigated in these prior works.

Original: We subsequently demonstrate empirical results extending beyond the reach of PSRO with Nash as a meta-solver by evaluating training in 3- to 5-player games.
Revised: We subsequently demonstrate empirical results extending beyond the reach of PSRO with Nash as a meta-solver by evaluating training in 3- to 5-player games.

Original: Finally, we conduct preliminary evaluations in MuJoCo soccer (Liu et al., 2019), another complex domain wherein we use reinforcement learning agents as oracles in our proposed PSRO variants, illustrating the feasibility of the approach.
Revised: Finally, we conduct preliminary evaluations in MuJoCo soccer (Liu et al., 2019), another complex domain wherein we use reinforcement learning agents as oracles in our proposed PSRO variants, illustrating the feasibility of the approach.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: Games We consider K -player games, where each player k ∈ [ K ] has a ﬁnite set of pure strategies
Revised: Games We consider K -player games, where each player k ∈ [ K ] has a ﬁnite set of pure strategies

Original: S k .
Revised: S k .

Original: Let S =
Revised: Let S =

Original: (cid:81)
Revised: (cid:81)

Original: k S k denote the space of pure strategy proﬁles.
Revised: k S k denote the space of pure strategy proﬁles.

Original: Denote by S − k = (cid:81) l (cid:54) =
Revised: Denote by S − k = (cid:81) l (cid:54) =

Original: k S l the set of pure strategy proﬁles excluding those of player k .
Revised: k S l the set of pure strategy proﬁles excluding those of player k .

Original: Let M ( s ) =
Revised: Let M ( s ) =

Original: ( M 1 ( s ) , . . .
Revised: ( M 1 ( s ) , . . .

Original: , M K ( s ))
Revised: , M K ( s ))

Original: ∈
Revised: ∈

Original: R K denote the vector of expected player payoffs for each s ∈ S .
Revised: R K denote the vector of expected player payoffs for each s ∈ S .

Original: A game is said to be zero-sum if (cid:80) k M k
Revised: A game is said to be zero-sum if (cid:80) k M k

Original: ( s )
Revised: ( s )

Original: = 

Revised: =

Original: for all s ∈ S .
Revised: for all s ∈ S .

Original: A game is said to be symmetric if all players have identical strategy sets S k , and for any permutation ρ ,
Revised: A game is said to be symmetric if all players have identical strategy sets S k , and for any permutation ρ ,

Original: strategy proﬁle
Revised: strategy proﬁle

Original: ( s 1 , . . .
Revised: ( s 1 , . . .

Original: , s K ) ∈ S , and index k ∈
Revised: , s K ) ∈ S , and index k ∈

Original: [ K ] , one has
Revised: [ K ] , one has

Original: M k ( s 1 , . . .
Revised: M k ( s 1 , . . .

Original: , s K )
Revised: , s K )

Original: =
Revised: =

Original: M ρ ( k ) ( s ρ (1) , . . .
Revised: M ρ ( k ) ( s ρ (1) , . . .

Original: , s ρ ( K ) ) .
Revised: , s ρ ( K ) ) .

Original: A mixed strategy proﬁle is deﬁned as π ∈ ∆ S , a tuple representing the probability distribution over pure strategy proﬁles s ∈ S .
Revised: A mixed strategy proﬁle is deﬁned as π ∈ ∆ S , a tuple representing the probability distribution over pure strategy proﬁles s ∈ S .

Original: The expected payoff to player k under a mixed strategy proﬁle π is given by M k ( π )
Revised: The expected payoff to player k under a mixed strategy proﬁle π is given by M k ( π )

Original: = (cid:80) s ∈ S
Revised: = (cid:80) s ∈ S

Original: π ( s ) M k
Revised: π ( s ) M k

Original: ( s ) .
Revised: ( s ) .

Original: Nash Equilibrium (NE) Given a mixed proﬁle π , the best response for a player k is deﬁned BR k
Revised: Nash Equilibrium (NE) Given a mixed proﬁle π , the best response for a player k is deﬁned BR k

Original: ( π ) =
Revised: ( π ) =

Original: arg max ν ∈ ∆
Revised: arg max ν ∈ ∆

Original: Sk [ M k
Revised: Sk [ M k

Original: ( ν , π − k )] .
Revised: ( ν , π − k )] .

Original: A factorized mixed proﬁle π ( s ) =
Revised: A factorized mixed proﬁle π ( s ) =

Original: (cid:81)
Revised: (cid:81)

Original: k π k ( s k ) is a Nash equilibrium (NE) if π k ∈ BR k
Revised: k π k ( s k ) is a Nash equilibrium (NE) if π k ∈ BR k

Original: ( π ) for all k ∈ [ K ] .
Revised: ( π ) for all k ∈ [ K ] .

Original: Deﬁne N ASH C ONV ( π ) =
Revised: Deﬁne N ASH C ONV ( π ) =

Original: (cid:80) k M k
Revised: (cid:80) k M k

Original: ( BR k ( π ) , π − k ) − M k
Revised: ( BR k ( π ) , π − k ) − M k

Original: ( π ) ; roughly speaking, this measures “distance” from an NE (Lanctot et al., 2017).
Revised: ( π ) ; roughly speaking, this measures “distance” from an NE (Lanctot et al., 2017).

Original: In prior PSRO instances (Lanctot et al., 2017), a variant of the replicator dynamics (Taylor and Jonker, 1978; Maynard Smith and Price, 1973), called the Projected Replicator Dynamics (PRD), has been used as an approximate Nash meta-solver (see Appendix E for details on PRD).
Revised: In prior PSRO instances (Lanctot et al., 2017), a variant of the replicator dynamics (Taylor and Jonker, 1978; Maynard Smith and Price, 1973), called the Projected Replicator Dynamics (PRD), has been used as an approximate Nash meta-solver (see Appendix E for details on PRD).

Original: α -Rank While NE exist in all ﬁnite games (Nash, 1950), their computation is intractable in general games, and their non-uniqueness leads to an equilibrium-selection problem (Harsanyi et al., 1988;
Revised: α -Rank While NE exist in all ﬁnite games (Nash, 1950), their computation is intractable in general games, and their non-uniqueness leads to an equilibrium-selection problem (Harsanyi et al., 1988;

Original: Goldberg et al., 2013).
Revised: Goldberg et al., 2013).

Original: This limits their applicability as the underlying driver of training beyond the two-player, zero-sum regime.
Revised: This limits their applicability as the underlying driver of training beyond the two-player, zero-sum regime.

Original: Recently, an alternate solution concept called α -Rank was proposed by
Revised: Recently, an alternate solution concept called α -Rank was proposed by

Original: Omidshaﬁei et al.
Revised: Omidshaﬁei et al.

Original: (2019), the key associated beneﬁts being its uniqueness and efﬁcient computation in many-player and general-sum games, making it a promising means for directing multiagent training.
Revised: (2019), the key associated beneﬁts being its uniqueness and efﬁcient computation in many-player and general-sum games, making it a promising means for directing multiagent training.

Original: The α -Rank distribution is computed by constructing the response graph of the game: each strategy proﬁle s ∈ S of the game is a node of this graph; a directed edge points from any proﬁle s ∈ S to σ ∈ S in the graph if (1) s and σ differ in only a single player k ’s strategy and (2) M k ( σ )
Revised: The α -Rank distribution is computed by constructing the response graph of the game: each strategy proﬁle s ∈ S of the game is a node of this graph; a directed edge points from any proﬁle s ∈ S to σ ∈ S in the graph if (1) s and σ differ in only a single player k ’s strategy and (2) M k ( σ )

Original: > M k ( s ) .
Revised: > M k ( s ) .

Original: α -Rank constructs a random walk along this directed graph, perturbing the process by injecting a small probability of backwards-transitions from σ to s (dependent on a parameter, α , whose value is prescribed by the algorithm); this ensures irreducibility of the resulting Markov chain and the existence of a unique stationary distribution, π ∈ ∆ S , called the α -Rank distribution.
Revised: α -Rank constructs a random walk along this directed graph, perturbing the process by injecting a small probability of backwards-transitions from σ to s (dependent on a parameter, α , whose value is prescribed by the algorithm); this ensures irreducibility of the resulting Markov chain and the existence of a unique stationary distribution, π ∈ ∆ S , called the α -Rank distribution.

Original: The masses of π are supported by the sink strongly-connected components (SSCCs) of the response graph (Omidshaﬁei et al., 2019).
Revised: The masses of π are supported by the sink strongly-connected components (SSCCs) of the response graph (Omidshaﬁei et al., 2019).

Original: For more details on α -Rank, see Appendix D and Rowland et al.
Revised: For more details on α -Rank, see Appendix D and Rowland et al.

Original: Oracles We deﬁne an oracle O as an abstract computational entity that, given a game, computes policies with precise associated properties.
Revised: Oracles We deﬁne an oracle O as an abstract computational entity that, given a game, computes policies with precise associated properties.

Original: For instance, a best-response oracle O k ( π ) =
Revised: For instance, a best-response oracle O k ( π ) =

Original: BR k ( π )
Revised: BR k ( π )

Original: [Figure]
Revised: [Figure]

Original: 1: Initialize players’ policyS k S k via random policies 2: for iteration ∈ { 1 , · · · } do 3: payoff tensor policy proﬁles 1a
Revised: computes the best-response policy for any player k , given a proﬁle π .

Original: 1b

Original: Expand the policy space for each player k [ K ] via S k ← 1c computes the best-response policy for any player k , given a proﬁle π . One may also consider approximate-best-response oracles that, e.g., use reinforcement learning to train a player k ’s policy against a ﬁxed distribution over the other players’ policies, π − k .
Revised: One may also consider approximate-best-response oracles that, e.g., use reinforcement learning to train a player k ’s policy against a ﬁxed distribution over the other players’ policies, π − k .

Original: Oracles play a key role in populationbased training, as they compute the policies that are incrementally added to players’ growing policy populations (McMahan et al., 2003;
Revised: Oracles play a key role in populationbased training, as they compute the policies that are incrementally added to players’ growing policy populations (McMahan et al., 2003;

Original: Lanctot et al., 2017; Balduzzi et al., 2019).
Revised: Lanctot et al., 2017; Balduzzi et al., 2019).

Original: The choice of oracle
Revised: The choice of oracle

Original: O also affects the training convergence rate and ﬁnal equilibrium reached (e.g., Nash or α -Rank).
Revised: O also affects the training convergence rate and ﬁnal equilibrium reached (e.g., Nash or α -Rank).

Original: Empirical Game-theoretic Analysis PSRO relies on principles from empirical game-theoretic analysis (EGTA)
Revised: Empirical Game-theoretic Analysis PSRO relies on principles from empirical game-theoretic analysis (EGTA)

Original: (Walsh et al., 2002;
Revised: (Walsh et al., 2002;

Original: Phelps et al., 2004; Wellman, 2006).
Revised: Phelps et al., 2004; Wellman, 2006).

Original: Given a game (e.g., poker),
Revised: Given a game (e.g., poker),

Original: EGTA operates via construction of a higher-level ‘meta-game’, where strategies s correspond to policies (e.g., ‘play defensively’ in poker) rather than atomic actions (e.g., ‘fold’).
Revised: EGTA operates via construction of a higher-level ‘meta-game’, where strategies s correspond to policies (e.g., ‘play defensively’ in poker) rather than atomic actions (e.g., ‘fold’).

Original: A meta-payoff table M is then constructed by simulating games for all joint policy combinations, with entries corresponding to the players’ expected utilities under these policies.
Revised: A meta-payoff table M is then constructed by simulating games for all joint policy combinations, with entries corresponding to the players’ expected utilities under these policies.

Original: Game-theoretic analysis can then be conducted on the meta-game in a manner analogous to the lower-level game, albeit in a much more scalable manner.
Revised: Game-theoretic analysis can then be conducted on the meta-game in a manner analogous to the lower-level game, albeit in a much more scalable manner.

Original: As the theoretical discussion hereafter pertains to the meta-game, we use s ,
Revised: As the theoretical discussion hereafter pertains to the meta-game, we use s ,

Original: M , and π to respectively refer to policies, payoffs, and distributions at the meta-level, rather than the underlying low-level game.
Revised: M , and π to respectively refer to policies, payoffs, and distributions at the meta-level, rather than the underlying low-level game.

Original: In our analysis, it will be important to distinguish between SSCCs of the underlying game, and of the meta-game constructed by PSRO; we refer to the latter as meta-SSCCs.
Revised: In our analysis, it will be important to distinguish between SSCCs of the underlying game, and of the meta-game constructed by PSRO; we refer to the latter as meta-SSCCs.

Original:  P OLICY -S PACE R ESPONSE O RACLES : N ASH AND B EYOND
Revised:  P OLICY -S PACE R ESPONSE O RACLES : N ASH AND B EYOND

Original: We ﬁrst overview Policy-Space Response Oracles (PSRO) prior to presenting our ﬁndings.
Revised: We ﬁrst overview Policy-Space Response Oracles (PSRO) prior to presenting our ﬁndings.

Original: Given an underlying game (e.g., Poker), PSRO ﬁrst initializes the policy space S using randomly-generated policies, then expands the players’ policy populations in three iterated phases: complete , solve , and expand (see Algorithm 1 and Fig. 1).
Revised: Given an underlying game (e.g., Poker), PSRO ﬁrst initializes the policy space S using randomly-generated policies, then expands the players’ policy populations in three iterated phases: complete , solve , and expand (see Algorithm 1 and Fig. 1).

Original: In the complete phase, a meta-game consisting of all match-ups of these joint policies is synthesized, with missing payoff entries in M completed through game simulations.
Revised: In the complete phase, a meta-game consisting of all match-ups of these joint policies is synthesized, with missing payoff entries in M completed through game simulations.

Original: Next, in the solve phase, a meta-solver M computes a proﬁle π over the player policies (e.g., Nash, α -Rank, or uniform distributions).
Revised: Next, in the solve phase, a meta-solver M computes a proﬁle π over the player policies (e.g., Nash, α -Rank, or uniform distributions).

Original: Finally, in the expand phase, an oracle O computes at least one new policy s (cid:48) k for each player k ∈
Revised: Finally, in the expand phase, an oracle O computes at least one new policy s (cid:48) k for each player k ∈

Original: [ K ] , given proﬁle π .
Revised: [ K ] , given proﬁle π .

Original: As other players’ policy spaces
Revised: As other players’ policy spaces

Original: S − k and proﬁle π − k are ﬁxed, this phase involves solving a single-player optimization problem.
Revised: S − k and proﬁle π − k are ﬁxed, this phase involves solving a single-player optimization problem.

Original: [Table]
Revised: [Table]

Original: The new policies are appended to the respective players’ policy sets, andthe algorithmiterates.
Revised: The new policies are appended to the respective players’ policy sets, and algorithm

Original: We use PSRO( M , O ) to refer to the PSRO instance using meta-solver M and oracle O .
Revised: We use PSRO( M , O ) to refer to the PSRO instance using meta-solver M and oracle O .

Original: Notably,
Revised: Notably,

Original: PSRO-based training for two-player symmetric games can be conducted using a single population of policies that is shared by all players (i.e., S k is identical for all k ).
Revised: PSRO-based training for two-player symmetric games can be conducted using a single population of policies that is shared by all players (i.e., S k is identical for all k ).

Original: Thus, we henceforth refer to twoplayer symmetric games as ‘single-population games’, and more generally refer to games that require player-speciﬁc policy populations as ‘multi-population games’.
Revised: Thus, we henceforth refer to twoplayer symmetric games as ‘single-population games’, and more generally refer to games that require player-speciﬁc policy populations as ‘multi-population games’.

Original: Recent investigations of PSRO have solely focused on Nash-based meta-solvers and best-response-based oracles (Lanctot et al., 2017;
Revised: Recent investigations of PSRO have solely focused on Nash-based meta-solvers and best-response-based oracles (Lanctot et al., 2017;

Original: Balduzzi et al., 2019), with theory focused around the two-player zero-sum case.
Revised: Balduzzi et al., 2019), with theory focused around the two-player zero-sum case.

Original: Unfortunately, these guarantees do not hold in games beyond this regime, making investigation of alternative meta-solvers and oracles critical for further establishing PSRO’s generalizability.
Revised: Unfortunately, these guarantees do not hold in games beyond this regime, making investigation of alternative meta-solvers and oracles critical for further establishing PSRO’s generalizability.

Original:  G ENERALIZING PSRO T HEORY
Revised:  G ENERALIZING PSRO T HEORY

Original: This section establishes theoretical properties of PSRO for several useful classes of general games.
Revised: This section establishes theoretical properties of PSRO for several useful classes of general games.

Original: We summarize our results in Table 1, giving a full exposition below.
Revised: We summarize our results in Table 1, giving a full exposition below.

Original: 4. E STABLISHING C ONVERGENCE TO α -R ANK
Revised: 4. E STABLISHING C ONVERGENCE TO α -R ANK

Original: [Table]
Revised: [Table]

Original: − φ φ − ε
Revised: − φ φ − ε

Original: B φ − φ − ε
Revised: B φ − φ − ε

Original: C −φ − φ − ε
Revised: C −φ − φ − ε

Original: D − φ −φ − ε
Revised: D − φ −φ − ε

Original: X ε ε ε ε
Revised: X ε ε ε ε

Original: It is well-known that PSRO(Nash, BR) will eventually return an NE in two-player zero-sum games (McMahan et al., 2003).
Revised: It is well-known that PSRO(Nash, BR) will eventually return an NE in two-player zero-sum games (McMahan et al., 2003).

Original: In more general games, where Nash faces the issues outlined earlier, α -Rank appears a promising meta-solver candidate as it applies to many-player, general-sum games and has no selection problem.
Revised: In more general games, where Nash faces the issues outlined earlier, α -Rank appears a promising meta-solver candidate as it applies to many-player, general-sum games and has no selection problem.

Original: However, open questions remain regarding convergence guarantees of PSRO when using α -Rank, and whether standard BR oracles sufﬁce for ensuring these guarantees.
Revised: However, open questions remain regarding convergence guarantees of PSRO when using α -Rank, and whether standard BR oracles sufﬁce for ensuring these guarantees.

Original: We investigate these theoretical questions, namely, whether particular variants of PSRO can converge to the α -Rank distribution for the underlying game.
Revised: We investigate these theoretical questions, namely, whether particular variants of PSRO can converge to the α -Rank distribution for the underlying game.

Original: A ﬁrst attempt to establish convergence to α -Rank might involve running PSRO to convergence (until the oracle returns a strategy already in the convex hull of the known strategies), using α -Rank as the meta-solver, and a standard best response oracle.
Revised: A ﬁrst attempt to establish convergence to α -Rank might involve running PSRO to convergence (until the oracle returns a strategy already in the convex hull of the known strategies), using α -Rank as the meta-solver, and a standard best response oracle.

Original: However, the following example shows that this will not work in general for the single-population case (see Fig.
Revised: However, the following example shows that this will not work in general for the single-population case (see Fig.

Original: A.5 for a step-by-step illustration).
Revised: A.5 for a step-by-step illustration).

Original: Example 1.
Revised: Example 1.

Original: Consider the symmetric zero-sum game speciﬁed in Table 2.
Revised: Consider the symmetric zero-sum game speciﬁed in Table 2.

Original: As X is the sole sink component of the game’s response graph (as illustrated in Fig. A.5a), the single-population α -Rank distribution for this game puts unit mass on X .
Revised: As X is the sole sink component of the game’s response graph (as illustrated in Fig. A.5a), the single-population α -Rank distribution for this game puts unit mass on X .

Original: We now show that a PSRO algorithm that computes best responses to the α -Rank distribution over the current strategy set need not recover strategy X , by computing directly the strategy sets of the algorithm initialized with the set { C } .
Revised: We now show that a PSRO algorithm that computes best responses to the α -Rank distribution over the current strategy set need not recover strategy X , by computing directly the strategy sets of the algorithm initialized with the set { C } .

Original: The initial strategy space consists only of the strategy C ; the best response against C is D . 2.
Revised: The initial strategy space consists only of the strategy C ; the best response against C is D . 2.

Original: The α -Rank distribution over { C, D } puts all mass on D ; the best response against D is A . 3.
Revised: The α -Rank distribution over { C, D } puts all mass on D ; the best response against D is A . 3.

Original: The α -Rank distribution over { C, D, A } puts all mass on A ; the best response against A is B . 4.
Revised: The α -Rank distribution over { C, D, A } puts all mass on A ; the best response against A is B . 4.

Original: The α -Rank distribution over { C, D, A, B } puts mass ( 1 / 3 , 1 / 3 , 1 / 6 , 1 / 6 ) on ( A, B, C, D ) respectively.
Revised: The α -Rank distribution over { C, D, A, B } puts mass ( 1 / 3 , 1 / 3 , 1 / 6 , 1 / 6 ) on ( A, B, C, D ) respectively.

Original: For φ sufﬁciently large, the payoff that C receives against B dominates all others, and
Revised: For φ sufﬁciently large, the payoff that C receives against B dominates all others, and

Original: since B has higher mass than C in the α -Rank distribution, the
Revised: since B has higher mass than C in the α - Rank distribution, the

Original: best response is
Revised: best response is

Original: C.
Revised: C

Original: Thus, PSRO( α -Rank, BR) leads to the algorithm terminating with strategy set { A, B, C, D } and not discovering strategy X in the sink strongly-connected component.
Revised: Thus, PSRO( α -Rank, BR) leads to the algorithm terminating with strategy set { A, B, C, D } and not discovering strategy X in the sink strongly-connected component.

Original: This conclusion also holds in the multi-population case, as the following counterexample shows.
Revised: This conclusion also holds in the multi-population case, as the following counterexample shows.

Original: Example 2.
Revised: Example 2.

Original: Consider the game in Table 2, treating it now as a multi-population problem.
Revised: Consider the game in Table 2, treating it now as a multi-population problem.

Original: It is readily veriﬁed that the multi-population α -Rank distributions obtained by PSRO with initial strategy sets consisting solely of C for each player are: (i) a Dirac delta at the joint strategy ( C, C ) , leading to best responses of D for both players; (ii) a Dirac delta at ( D, D ) leading to best responses of A for both players; (iii) a Dirac delta at ( A, A ) , leading to best responses of B for both players; and ﬁnally (iv) a distribution over joint strategies of the 4 × 4 subgame induced by strategies A, B, C, D that leads to a best response not equal to X ; thus, the full α -Rank distribution is again not recovered.
Revised: It is readily veriﬁed that the multi-population α -Rank distributions obtained by PSRO with initial strategy sets consisting solely of C for each player are: (i) a Dirac delta at the joint strategy ( C, C ) , leading to best responses of D for both players; (ii) a Dirac delta at ( D, D ) leading to best responses of A for both players; (iii) a Dirac delta at ( A, A ) , leading to best responses of B for both players; and ﬁnally (iv) a distribution over joint strategies of the 4 × 4 subgame induced by strategies A, B, C, D that leads to a best response not equal to X ; thus, the full α -Rank distribution is again not recovered.

Original: 4. A N EW R ESPONSE O RACLE
Revised: 4. A N EW R ESPONSE O RACLE

Original: The previous examples indicate that the use of standard best responses in PSRO may be the root cause of the incompatibility with the α -Rank solution concept.
Revised: The previous examples indicate that the use of standard best responses in PSRO may be the root cause of the incompatibility with the α -Rank solution concept.

Original: Thus, we deﬁne the Preference-based
Revised: Thus, we deﬁne the Preference-based

Original: Best Response (PBR) oracle , which is more closely aligned with the dynamics deﬁning α -Rank, and which enables us to establish desired PSRO guarantees with respect to α -Rank.
Revised: Best Response (PBR) oracle , which is more closely aligned with the dynamics deﬁning α -Rank, and which enables us to establish desired PSRO guarantees with respect to α -Rank.

Original: Consider ﬁrst the single-population case.
Revised: Consider ﬁrst the single-population case.

Original: Given an N -strategy population { s 1 , . . .
Revised: Given an N -strategy population { s 1 , . . .

Original: , s N } and corresponding meta-solver distribution ( π i )
Revised: , s N } and corresponding meta-solver distribution ( π i )

Original: Ni =1 ∈ ∆ N , a PBR oracle is deﬁned as any function satisfying
Revised: Ni =1 ∈ ∆ N , a PBR oracle is deﬁned as any function satisfying

Original: [Equation]
Revised: [Equation]

Original: where the arg max returns the set of policies optimizing the objective, and the optimization is over pure strategies in the underlying game.
Revised: where the arg max returns the set of policies optimizing the objective, and the optimization is over pure strategies in the underlying game.

Original: The intuition for the deﬁnition of PBR is that we would like the oracle to return strategies that will receive high mass under α -Rank when added to the population; objective (1) essentially encodes the probability ﬂux that the vertex corresponding to σ would receive in the random walk over the α -Rank response graph (see Section 2 or Appendix D for further details).
Revised: The intuition for the deﬁnition of PBR is that we would like the oracle to return strategies that will receive high mass under α -Rank when added to the population; objective (1) essentially encodes the probability ﬂux that the vertex corresponding to σ would receive in the random walk over the α -Rank response graph (see Section 2 or Appendix D for further details).

Original: We demonstrate below that the use of the PBR resolves the issue highlighted in Example 1 (see
Revised: We demonstrate below that the use of the PBR resolves the issue highlighted in Example 1 (see

Original: Fig.
Revised: Fig.

Original: A.6 in Appendix A for an accompanying visual).
Revised: A.6 in Appendix A for an accompanying visual).

Original: Example 3.
Revised: Example 3.

Original: Steps 1 to 3 of correspond exactly to those of Example 1.
Revised: Steps 1 to 3 of correspond exactly to those of Example 1.

Original: In step 4, the α -Rank distribution over { C, D, A, B } puts mass ( 1 / 3 , 1 / 3 , 1 / 6 , 1 / 6 ) on ( A, B, C, D ) respectively.
Revised: In step 4, the α -Rank distribution over { C, D, A, B } puts mass ( 1 / 3 , 1 / 3 , 1 / 6 , 1 / 6 ) on ( A, B, C, D ) respectively.

Original: A beats C and D , thus its PBR score is 1 / 3 .
Revised: A beats C and D , thus its PBR score is 1 / 3 .

Original: B beats A and D , thus its PBR score is 1 / 2 .
Revised: B beats A and D , thus its PBR score is 1 / 2 .

Original: C beats B , its PBR score is thus 1 / 3 .
Revised: C beats B , its PBR score is thus 1 / 3 .

Original: D beats C , its PBR score is thus 1 / 6 .
Revised: D beats C , its PBR score is thus 1 / 6 .

Original: Finally, X beats every other strategy, and its PBR score is thus 1 .
Revised: Finally, X beats every other strategy, and its PBR score is thus 1 .

Original: Thus, there is only one strategy maximizing PBR, X , which is then chosen, thereby recovering the SSCC of the game and the correct α -Rank distribution at the next timestep.
Revised: Thus, there is only one strategy maximizing PBR, X , which is then chosen, thereby recovering the SSCC of the game and the correct α -Rank distribution at the next timestep.

Original: In the multi-population case, consider a population of N strategy proﬁles { s 1 , . . .
Revised: In the multi-population case, consider a population of N strategy proﬁles { s 1 , . . .

Original: , s N } and corresponding meta-solver distribution ( π i )
Revised: , s N } and corresponding meta-solver distribution ( π i )

Original: Ni =1 .
Revised: Ni =1 .

Original: Several meta-SSCCs may exist in the multi-population α -Rank response graph.
Revised: Several meta-SSCCs may exist in the multi-population α -Rank response graph.

Original: In this case, we run the PBR oracle for each meta-SSCC separately, as follows.
Revised: In this case, we run the PBR oracle for each meta-SSCC separately, as follows.

Original: Suppose there are (cid:96) meta-SSCCs, and denote by π ( (cid:96) )
Revised: Suppose there are (cid:96) meta-SSCCs, and denote by π ( (cid:96) )

Original: the distribution π restricted to the (cid:96) th meta-SSCC, for all 1 ≤ (cid:96) ≤ L .
Revised: the distribution π restricted to the (cid:96) th meta-SSCC, for all 1 ≤ (cid:96) ≤ L .

Original: The PBR for player k on the (cid:96) th meta-SSCC is then deﬁned by
Revised: The PBR for player k on the (cid:96) th meta-SSCC is then deﬁned by

Original: [Equation]
Revised: [Equation]

Original: Thus, the PBR oracle generates one new strategy for each player for every meta-SSCC in the α -Rank response graph; we return this full set of strategies and append to the policy space accordingly, as in Line 5 of Algorithm 1.
Revised: Thus, the PBR oracle generates one new strategy for each player for every meta-SSCC in the α -Rank response graph; we return this full set of strategies and append to the policy space accordingly, as in Line 5 of Algorithm 1.

Original: Intuitively, this leads to a diversiﬁcation of strategies introduced by the oracle, as each new strategy need only perform well against a subset of prior strategies.
Revised: Intuitively, this leads to a diversiﬁcation of strategies introduced by the oracle, as each new strategy need only perform well against a subset of prior strategies.

Original: This hints at interesting links with the recently-introduced concept of rectiﬁed-Nash BR (Balduzzi et al., 2019), which also attempts to improve diversity in PSRO, albeit only in two-player zero-sum games.
Revised: This hints at interesting links with the recently-introduced concept of rectiﬁed-Nash BR (Balduzzi et al., 2019), which also attempts to improve diversity in PSRO, albeit only in two-player zero-sum games.

Original: We henceforth denote PSRO( α -Rank, PBR) as α -PSRO for brevity.
Revised: We henceforth denote PSRO( α -Rank, PBR) as α -PSRO for brevity.

Original: We next deﬁne α -C ONV , an approximate measure of convergence to α -Rank.
Revised: We next deﬁne α -C ONV , an approximate measure of convergence to α -Rank.

Original: We restrict discussion to the multi-population case here, describing the single-population case in Appendix A.4.
Revised: We restrict discussion to the multi-population case here, describing the single-population case in Appendix A.4.

Original: With the notation introduced above, we deﬁne PBR-S CORE k ( σ ; π , S )
Revised: With the notation introduced above, we deﬁne PBR-S CORE k ( σ ; π , S )

Original: =
Revised: =

Original: (cid:80) i (cid:80) (cid:96) π
Revised: (cid:80) i (cid:80) [Equation]

Original:  ( (cid:96) )
Revised: π ( (cid:96) )

Original: i 1 (cid:2) M k ( σ, s − k i ) >
Revised: [Equation] (cid:2) M k ( σ, s − k i ) >

Original: M k ( s ki , s − k i ) (cid:3) , and
Revised: M k ( s ki , s [Equation] k i ) (cid:3) , and

Original: [Equation]
Revised: [Equation]

Original: where max σ is taken over the pure strategies of the underlying game.
Revised: where max σ is taken over the pure strategies of the underlying game.

Original: Unfortunately, in the multi-population case, a PBR-S CORE of 0 does not necessarily imply α -partial convergence.
Revised: Unfortunately, in the multi-population case, a PBR-S CORE of 0 does not necessarily imply α -partial convergence.

Original: We thus introduce a further measure, PCS-S CORE , deﬁned by PCS-S CORE =
Revised: We thus introduce a further measure, PCS-S CORE , deﬁned by PCS-S CORE =

Original: [Table]
Revised: [Table]

Original: α-PSRO
Revised: α

Original: [Table]

Original: [Table]
Revised: [Table]

Original: , which assesses the quality of the α -PSRO population.
Revised: , which assesses the quality of the α -PSRO population.

Original: We refer readers to Appendix C.3 for pseudocode detailing how to implement these measures in practice.
Revised: We refer readers to Appendix C.3 for pseudocode detailing how to implement these measures in practice.

Original: 4. α -PSRO: T HEORY , P RACTICE , AND C ONNECTIONS TO N ASH
Revised: 4. α -PSRO: T HEORY , P RACTICE , AND C ONNECTIONS TO N ASH

Original: We next study the theoretical properties of PSRO( α -Rank, PBR), which we henceforth refer to as α -PSRO for brevity.
Revised: We next study the theoretical properties of PSRO( α -Rank, PBR), which we henceforth refer to as α -PSRO for brevity.

Original: We consider that α -PSRO has converged if no new strategy has been returned by PBR for any player at the end of an iteration.
Revised: We consider that α -PSRO has converged if no new strategy has been returned by PBR for any player at the end of an iteration.

Original: Proofs of all results are provided in Appendix B.
Revised: Proofs of all results are provided in Appendix B.

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: A PSRO algorithm is said to converge α -fully (resp., α -partially ) to an SSCC of the underlying game if its strategy population contains the full SSCC (resp., a sub-cycle of the SSCC, denoted a ‘sub-SSCC’) after convergence.
Revised: A PSRO algorithm is said to converge α -fully (resp., α -partially ) to an SSCC of the underlying game if its strategy population contains the full SSCC (resp., a sub-cycle of the SSCC, denoted a ‘sub-SSCC’) after convergence.

Original: Deﬁnition 2.
Revised: Deﬁnition 2.

Original: We also adapt PBR to be what we call novelty-bound by restricting the arg max in
Revised: We also adapt PBR to be what we call novelty-bound by restricting the arg max in

Original: Equation (1) to be over strategies not already included in the population with PBR-S CORE > 0 .
Revised: Equation (1) to be over strategies not already included in the population with PBR-S CORE > 0 .

Original: In particular, the novelty-bound version of the PBR oracle is given by restricting the arg max appearing in (2) to only be over strategies not already present in the population.
Revised: In particular, the novelty-bound version of the PBR oracle is given by restricting the arg max appearing in (2) to only be over strategies not already present in the population.

Original: These deﬁnitions enable the following results for α -PSRO in the single- and multi-population cases.
Revised: These deﬁnitions enable the following results for α -PSRO in the single- and multi-population cases.

Original: Proposition 1.
Revised: Proposition 1.

Original: If at any point the population of α -PSRO contains a member of an SSCC of the game, then α -PSRO will α -partially converge to that SSCC.
Revised: If at any point the population of α -PSRO contains a member of an SSCC of the game, then α -PSRO will α -partially converge to that SSCC.

Original: Proposition 2.
Revised: Proposition 2.

Original: If we constrain the PBR oracle used in α -PSRO to be novelty-bound, then α -PSRO will α -fully converge to at least one SSCC of the game.
Revised: If we constrain the PBR oracle used in α -PSRO to be novelty-bound, then α -PSRO will α -fully converge to at least one SSCC of the game.

Original: Stronger guarantees exist for two-players symmetric (i.e., single-population) games, though the multi-population case encounters more issues, as follows.
Revised: Stronger guarantees exist for two-players symmetric (i.e., single-population) games, though the multi-population case encounters more issues, as follows.

Original: Proposition 3.
Revised: Proposition 3.

Original: (Single-population)
Revised: (Single-population)

Original: α -PSRO converges α -partially to the unique SSCC.
Revised: α -PSRO converges α -partially to the unique SSCC.

Original: Proposition 4.
Revised: Proposition 4.

Original: (Multi-population)
Revised: (Multi-population)

Original: Without a novelty-bound oracle, there exist games for which α -PSRO does not converge α -partially to any SSCC.
Revised: Without a novelty-bound oracle, there exist games for which α -PSRO does not converge α -partially to any SSCC.

Original: Intuitively, the lack of convergence without a novelty-bound oracle can occur due to intransitivities in the game (i.e., cycles in the game can otherwise trap the oracle).
Revised: Intuitively, the lack of convergence without a novelty-bound oracle can occur due to intransitivities in the game (i.e., cycles in the game can otherwise trap the oracle).

Original: An example demonstrating this issue is shown in Fig.
Revised: An example demonstrating this issue is shown in Fig.

Original: B.7, with an accompanying step-by-step walkthrough in Appendix B.4.
Revised: B.7, with an accompanying step-by-step walkthrough in Appendix B.4.

Original: Speciﬁcally, SSCCs may be hidden by “intermediate” strategies that, while not receiving as high a payoff as current population-pool members, can actually lead to well-performing strategies outside the population.
Revised: Speciﬁcally, SSCCs may be hidden by “intermediate” strategies that, while not receiving as high a payoff as current population-pool members, can actually lead to well-performing strategies outside the population.

Original: As these “intermediate” strategies are avoided, SSCCs are consequently not found.
Revised: As these “intermediate” strategies are avoided, SSCCs are consequently not found.

Original: Note also that this is related to the common problem of action/equilibrium shadowing, as detailed in
Revised: Note also that this is related to the common problem of action/equilibrium shadowing, as detailed in

Original: Matignon et al.
Revised: Matignon et al.

Original: In Section 5, we further investigate convergence behavior beyond the conditions studied above.
Revised: In Section 5, we further investigate convergence behavior beyond the conditions studied above.

Original: In practice, we demonstrate that despite the negative result of Proposition 4, α -PSRO does signiﬁcantly increase the probability of converging to an SSCC, in contrast to PSRO(Nash, BR).
Revised: In practice, we demonstrate that despite the negative result of Proposition 4, α -PSRO does signiﬁcantly increase the probability of converging to an SSCC, in contrast to PSRO(Nash, BR).

Original: Overall, we have shown that for general-sum multi-player games, it is possible to give theoretical guarantees for a version of PSRO driven by α -Rank in several circumstances.
Revised: Overall, we have shown that for general-sum multi-player games, it is possible to give theoretical guarantees for a version of PSRO driven by α -Rank in several circumstances.

Original: By contrast, using exact NE in
Revised: By contrast, using exact NE in

Original: PSRO is intractable in general.
Revised: PSRO is intractable in general.

Original: In prior work, this motivated the use of approximate Nash solvers generally based on the simulation of dynamical systems or regret minimization algorithms, both of which generally require speciﬁcation of several hyperparameters (e.g., simulation iterations, window sizes for computing time-average policies, and entropy-injection rates), and a greater computational burden than α -Rank to carry out the simulation in the ﬁrst place.
Revised: In prior work, this motivated the use of approximate Nash solvers generally based on the simulation of dynamical systems or regret minimization algorithms, both of which generally require speciﬁcation of several hyperparameters (e.g., simulation iterations, window sizes for computing time-average policies, and entropy-injection rates), and a greater computational burden than α -Rank to carry out the simulation in the ﬁrst place.

Original: Implementing the PBR Oracle Recall from Section 3 that the BR oracle inherently solves a singleplayer optimization problem, permitting use of a single-agent RL algorithm as a BR approximator, which is useful in practice.
Revised: Implementing the PBR Oracle Recall from Section 3 that the BR oracle inherently solves a singleplayer optimization problem, permitting use of a single-agent RL algorithm as a BR approximator, which is useful in practice.

Original: As noted in Section 4.1, however, there exist games where the BR and PBR objectives are seemingly incompatible, preventing the use of standard RL agents for PBR approximation.
Revised: As noted in Section 4.1, however, there exist games where the BR and PBR objectives are seemingly incompatible, preventing the use of standard RL agents for PBR approximation.

Original: While exact PBR is computable in small-scale (e.g., normal-form) games, we next consider more general games classes where PBR can also be approximated using standard RL agents.
Revised: While exact PBR is computable in small-scale (e.g., normal-form) games, we next consider more general games classes where PBR can also be approximated using standard RL agents.

Original: Deﬁnition 3.
Revised: Deﬁnition 3.

Original: Objective A is ‘compatible’ with objective B if any solution to A is a solution to B .
Revised: Objective A is ‘compatible’ with objective B if any solution to A is a solution to B .

Original: Proposition 5.
Revised: Proposition 5.

Original: A constant-sum game is denoted as win-loss if M k ( s ) ∈ { 0 , 1 } for all k ∈ [ K ] and s ∈ S .
Revised: A constant-sum game is denoted as win-loss if M k ( s ) ∈ { 0 , 1 } for all k ∈ [ K ] and s ∈ S .

Original: BR is compatible with PBR in win-loss games in the two-player single-population case.
Revised: BR is compatible with PBR in win-loss games in the two-player single-population case.

Original: Proposition 6.
Revised: Proposition 6.

Original: A symmetric two-player game is denoted monotonic if there exists a function f :
Revised: A symmetric two-player game is denoted monotonic if there exists a function f :

Original: S → R and a non-decreasing function σ : R → R such that M 1 ( s, ν ) =
Revised: S → R and a non-decreasing function σ : R → R such that M 1 ( s, ν ) =

Original: σ ( f ( s ) − f ( ν )) .
Revised: σ ( f ( s ) − f ( ν )) .

Original: BR is compatible with PBR in monotonic games in the single-population case.
Revised: BR is compatible with PBR in monotonic games in the single-population case.

Original: [Figure]
Revised: [Figure]

Original: Finally [Figure] we next demonstrate that under certain conditions, there are strong connections between the
Revised: Finally, we next demonstrate that under certain conditions, there are strong connections between the

Original: PBR objective deﬁned above and the broader ﬁeld of preference-based RL (Wirth et al., 2017).
Revised: PBR objective deﬁned above and the broader ﬁeld of preference-based RL (Wirth et al., 2017).

Original: Proposition 7.
Revised: Proposition 7.

Original: Consider symmetric win-loss games where outcomes between deterministic strategies are deterministic.
Revised: Consider symmetric win-loss games where outcomes between deterministic strategies are deterministic.

Original: A preference-based RL agent (i [Figure] e., an agent aiming to maximize its probability of winning against a distribution π of strategies { s 1 , . . . , s N } ) optimizes exactly the PBR objective (1) .
Revised: A preference-based RL agent (i.e., an agent aiming to maximize its probability of winning against a distribution π of strategies { s 1 , . . . , s N } ) optimizes exactly the PBR objective (1) .

Original: Given this insight [Figure] we believe an important subject of future work will involve the use of preferencebased RL algorithms in implementing the PBR oracle for more general classes of games.
Revised: Given this insight, we believe an important subject of future work will involve the use of preferencebased RL algorithms in implementing the PBR oracle for more general classes of games.

Original: We conclude this section with some indicative results of the relationship between α -Rank and NE.
Revised: We conclude this section with some indicative results of the relationship between α -Rank and NE.

Original: Proposition 8.
Revised: Proposition 8.

Original: For symmetric two-player zero-sum games where off-diagonal payoffs have equal magnitude, all NE have support contained within that of the single-population α -Rank distribution.
Revised: For symmetric two-player zero-sum games where off-diagonal payoffs have equal magnitude, all NE have support contained within that of the single-population α -Rank distribution.

Original: Proposition 9.
Revised: Proposition 9.

Original: In a symmetric two-player zero-sum game, there exists an NE with support contained within that of the α -Rank distribution.
Revised: In a symmetric two-player zero-sum game, there exists an NE with support contained within that of the α -Rank distribution.

Original: For more general games, the link between α -Rank and Nash equilibria will likely require a more complex description.
Revised: For more general games, the link between α -Rank and Nash equilibria will likely require a more complex description.

Original: We leave this for future work, providing additional discussion in Appendix A.3.
Revised: We leave this for future work, providing additional discussion in Appendix A.3.

Original:  E VALUATION
Revised:  E VALUATION

Original: We conduct evaluations on games of increasing complexity, extending beyond prior PSRO applications that have focused on two-player zero-sum games.
Revised: We conduct evaluations on games of increasing complexity, extending beyond prior PSRO applications that have focused on two-player zero-sum games.

Original: For experimental procedures, see Appendix C.
Revised: For experimental procedures, see Appendix C.

Original: Oracle comparisons We evaluate here the performance of the BR and PBR oracles in games where
Revised: Oracle comparisons We evaluate here the performance of the BR and PBR oracles in games where

Original: PBR can be exactly computed.
Revised: PBR can be exactly computed.

Original: We consider randomly generated, K -player, general-sum games with increasing strategy space sizes, | S k | .
Revised: We consider randomly generated, K -player, general-sum games with increasing strategy space sizes, | S k | .

Original: Figure 2 reports these results for the 4- and 5-player instances (see Appendix C.4 for 2-3 player results).
Revised: Figure 2 reports these results for the 4- and 5-player instances (see Appendix C.4 for 2-3 player results).

Original: The asymmetric nature of these games, in combination with the number of players and strategies involved, makes them inherently, and perhaps surprisingly, large in scale.
Revised: The asymmetric nature of these games, in combination with the number of players and strategies involved, makes them inherently, and perhaps surprisingly, large in scale.

Original: For example, the largest considered game in Fig.
Revised: For example, the largest considered game in Fig.

Original: 2 involves 5 players with 30 strategies each, making for a total of more than 24 million strategy proﬁles in total.
Revised: 2 involves 5 players with 30 strategies each, making for a total of more than 24 million strategy proﬁles in total.

Original: For each combination of K and | S k | , we generate 1 e 6 random games.
Revised: For each combination of K and | S k | , we generate 1 e 6 random games.

Original: We conduct 10 trials per game, in each trial running the BR and PBR oracles starting from a random strategy in the corresponding response graph, then iteratively expanding the population space until convergence.
Revised: We conduct 10 trials per game, in each trial running the BR and PBR oracles starting from a random strategy in the corresponding response graph, then iteratively expanding the population space until convergence.

Original: Importantly, this implies that the starting strategy may not even be in an SSCC.
Revised: Importantly, this implies that the starting strategy may not even be in an SSCC.

Original: As mentioned in Section 4.2, α -C ONV and PCS-S CORE jointly characterize the oracle behaviors in these multi-population settings.
Revised: As mentioned in Section 4.2, α -C ONV and PCS-S CORE jointly characterize the oracle behaviors in these multi-population settings.

Original: Figure 2a plots α -C ONV for both oracles, demonstrating that PBR outperforms BR in the sense that it captures more of the game SSCCs.
Revised: Figure 2a plots α -C ONV for both oracles, demonstrating that PBR outperforms BR in the sense that it captures more of the game SSCCs.

Original: Figures 2b and 2c, respectively, plot the PCS-S CORE for BR and PBR over all game instances.
Revised: Figures 2b and 2c, respectively, plot the PCS-S CORE for BR and PBR over all game instances.

Original: The PCS-S CORE here is typically either (a) greater than 95%, or (b) less than 5%, and otherwise rarely between 5% to 95%.
Revised: The PCS-S CORE here is typically either (a) greater than 95%, or (b) less than 5%, and otherwise rarely between 5% to 95%.

Original: For all values of | S k | , PBR consistently discovers a larger proportion of the α -Rank support in contrast to
Revised: For all values of | S k | , PBR consistently discovers a larger proportion of the α -Rank support in contrast to

Original: BR, serving as useful validation of the theoretical results of Section 4.3.
Revised: BR, serving as useful validation of the theoretical results of Section 4.3.

Original: Meta-solver comparisons We consider next the standard benchmarks of Kuhn and Leduc
Revised: Meta-solver comparisons We consider next the standard benchmarks of Kuhn and Leduc

Original: poker (Kuhn, 1950; Southey et al., 2005; Lanctot et al., 2019).
Revised: poker (Kuhn, 1950; Southey et al., 2005; Lanctot et al., 2019).

Original: We detail these domains in Appendix C.2,
Revised: We detail these domains in Appendix C.2,

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: noting here that both are K -player, although Leduc is signiﬁcantly more complex than Kuhn.
Revised: noting here that both are K -player, although Leduc is signiﬁcantly more complex than Kuhn.

Original: We ﬁrst consider two-player instances of Kuhn and Leduc poker, permitting use of an exact Nash meta-solver.
Revised: We ﬁrst consider two-player instances of these poker domains, permitting use of an exact Nash meta-solver.

Original: Figure 3 compares the N ASH C ONV of PSRO( M , BR) for various meta-solver M choices.
Revised: Figure 3 compares the N ASH C ONV of PSRO( M , BR) for various meta-solver M choices.

Revised: Note that the x axis of Figure 3 and Figure 4 is the Total Pool Length (The sum of the length of each player’s pool in PSRO) instead of the number of iterations of PSRO, since Rectiﬁed solvers can add more than one policy to the pool at each PSRO iteration (Possibly doubling pool size at every PSRO iteration).

Revised: It is therefore more pertinent to compare exploitabilities at the same pool sizes rather than at the same number of PSRO iterations.

Original: In Kuhn poker (Fig. 3a)
Revised: In Kuhn poker (Fig. 3a)

Original: [Figure]
Revised: [Figure]

Original: the α -Rank,  Nash 
Revised: the α -Rank, Nash, and the Projected Replicator Dynamics (PRD) metasolvers converge essentially at the same rate towards zero N ASH C ONV , in contrast to the slower rate of the Uniform meta-solver, the very slow rate of the Rectiﬁed PRD solver, and the seemingly constant N ASH C ONV of the Rectiﬁed Nash solver.

Original: [Figure]
Revised: We provide in Appendix C.5 a walkthrough of the ﬁrst steps of the Rectiﬁed Nash results to more precisely determine the cause of its plateauing N ASH C ONV . A high level explanation thereof is that it is caused by Rectiﬁed Nash cycling through the same policies, effectively not discovering new policies.

Original: and the Projected Replicator Dynamics (PRD) meta-solvers converge essentially at the same rate towards zero N ASH C ONV , in contrast to  the slower rate of the Uniform meta-solver.
Revised: We posit these characteristics, antipodal to the motivation behind Rectiﬁed Nash, come from the important fact that Rectiﬁed Nash was designed to work only in symmetric games, and is therefore not inherently well-suited for the Kuhn and Leduc poker domains investigated here, as they are both asymmetric games. We do not add Rectiﬁed PRD results the other experiments, as its performance remained non-competitive.

Original: As noted in Lanctot et al.
Revised: As noted in Lanctot et al.

Original: (2017), PSRO(Uniform, BR) corresponds to Fictitious Play (Brown, 1951) and is thus guaranteed to ﬁnd an NE in such instances of two-player zero-sum games.
Revised: (2017), PSRO(Uniform, BR) corresponds to Fictitious Play (Brown, 1951) and is thus guaranteed to ﬁnd an NE in such instances of two-player zero-sum games.

Original: Its slower convergence rate is explained by the assignment of uniform mass across all policies s ∈ S , implying that PSRO essentially wastes resources on training the oracle to beat even poor-performing strategies.
Revised: Its slower convergence rate is explained by the assignment of uniform mass across all policies s ∈ S , implying that PSRO essentially wastes resources on training the oracle to beat even poor-performing strategies.

Original: While α -Rank does not seek to ﬁnd an approximation of Nash, it nonetheless reduces the N ASH C ONV yielding competitive results in comparison to an exact-Nash solver in these instances.
Revised: While α -Rank does not seek to ﬁnd an approximation of Nash, it nonetheless reduces the N ASH C ONV yielding competitive results in comparison to an exact-Nash solver in these instances.

Original: Notably, the similar performance of α -Rank and Nash serves as empirical evidence that α -Rank can be applied competitively even in the two-player zero-sum setting, while also showing great promise to be deployed in broader settings where Nash is no longer tractable.
Revised: Notably, the similar performance of α -Rank and Nash serves as empirical evidence that α -Rank can be applied competitively even in the two-player zero-sum setting, while also showing great promise to be deployed in broader settings where Nash is no longer tractable.

Original: We next consider signiﬁcantly larger variants of Kuhn and Leduc Poker involving more than two players, extending beyond the reach of prior PSRO results (Lanctot et al., 2017).
Revised: We next consider signiﬁcantly larger variants of Kuhn and Leduc Poker involving more than two players, extending beyond the reach of prior PSRO results (Lanctot et al., 2017).

Original: Figure 4 visualizes the N ASH C ONV of PSRO using the various meta-solvers (with the exception of an exact Nash solver, due to its intractability in these instances).
Revised: Figure 4 visualizes the N ASH C ONV of PSRO using the various meta-solvers (with the exception of an exact Nash solver, due to its intractability in these instances).

Original: In all instances of Kuhn Poker, α -Rank and
Revised: In all instances of Kuhn Poker, α -Rank and

Original: PRD show competitive convergence rates.
Revised: PRD show competitive convergence rates.

Original: In 3-player Leduc poker, however, α -Rank shows fastest convergence, with Uniform following throughout most of training and PRD eventually reaching a similar N ASH C ONV .
Revised: In 3-player Leduc poker, however, α -Rank shows fastest convergence, with Uniform following throughout most of training and PRD eventually reaching a similar N ASH C ONV .

Original: Several key insights can be made here.
Revised: Several key insights can be made here.

Original: First, computation of an approximate
Revised: First, computation of an approximate

Original: Nash via PRD involves simulation of the associated replicator dynamics, which can be chaotic (Palaiopanos et al., 2017) even in two-player two-strategy games, making it challenging to determine when PRD has suitably converged.
Revised: Nash via PRD involves simulation of the associated replicator dynamics, which can be chaotic (Palaiopanos et al., 2017) even in two-player two-strategy games, making it challenging to determine when PRD has suitably converged.

Original: Second, the addition of the projection step in PRD severs its connection with NE; the theoretical properties of PRD were left open in Lanctot et al.
Revised: Second, the addition of the projection step in PRD severs its connection with NE; the theoretical properties of PRD were left open in Lanctot et al.

Original: (2017), leaving it without any guarantees.
Revised: (2017), leaving it without any guarantees.

Original: These limitations go beyond theoretical, manifesting in practice, e.g., in
Revised: These limitations go beyond theoretical, manifesting in practice, e.g., in

Original: Fig.
Revised: Fig.

Original: 4d, where PRD is outperformed by even the uniform meta-solver for many iterations.
Revised: 4d, where PRD is outperformed by even the uniform meta-solver for many iterations.

Original: Given these issues, we take a ﬁrst (and informal) step towards analyzing PRD in Appendix E. For α -Rank, by contrast, we both establish theoretical properties in Section 4, and face no simulation-related challenges as its computation involves solving of a linear system, even in the general-sum many-player case (Omidshaﬁei et al., 2019), thus establishing it as a favorable and general PSRO meta-solver.
Revised: Given these issues, we take a ﬁrst (and informal) step towards analyzing PRD in Appendix E. For α -Rank, by contrast, we both establish theoretical properties in Section 4, and face no simulation-related challenges as its computation involves solving of a linear system, even in the general-sum many-player case (Omidshaﬁei et al., 2019), thus establishing it as a favorable and general PSRO meta-solver.

Original: MuJoCo
Revised: MuJoCo

Original: Soccer While the key objective of this paper is to take a ﬁrst step in establishing a theoretically-grounded framework for PSRO-based training of agents in many-player settings, an exciting question regards the behaviors of the proposed α -Rank-based PSRO algorithm in complex domains where function-approximation-based policies need to be relied upon.
Revised: Soccer While the key objective of this paper is to take a ﬁrst step in establishing a theoretically-grounded framework for PSRO-based training of agents in many-player settings, an exciting question regards the behaviors of the proposed α -Rank-based PSRO algorithm in complex domains where function-approximation-based policies need to be relied upon.

Original: In Appendix F, we take a ﬁrst step towards conducting this investigation in the MuJoCo soccer domain introduced in
Revised: In Appendix F, we take a ﬁrst step towards conducting this investigation in the MuJoCo soccer domain introduced in

Original: Liu et al.
Revised: Liu et al.

Original: We remark that these results, albeit interesting, are primarily intended to lay the foundation for use of α -Rank as a meta-solver in complex many agent domains where RL agents serve as useful oracles, warranting additional research and analysis to make conclusive insights.
Revised: We remark that these results, albeit interesting, are primarily intended to lay the foundation for use of α -Rank as a meta-solver in complex many agent domains where RL agents serve as useful oracles, warranting additional research and analysis to make conclusive insights.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: We discuss the most closely related work along two axes.
Revised: We discuss the most closely related work along two axes.

Original: We start with PSRO-based research and some multiagent deep RL work that focuses on training of networks in various multiagent settings.
Revised: We start with PSRO-based research and some multiagent deep RL work that focuses on training of networks in various multiagent settings.

Original: Then we continue with related work that uses evolutionary dynamics ( α -Rank and replicator dynamics) as a solution concept to examine underlying behavior of multiagent interactions using meta-games.
Revised: Then we continue with related work that uses evolutionary dynamics ( α -Rank and replicator dynamics) as a solution concept to examine underlying behavior of multiagent interactions using meta-games.

Original: Policy-space response oracles (Lanctot et al., 2017) unify many existing approaches to multiagent learning.
Revised: Policy-space response oracles (Lanctot et al., 2017) unify many existing approaches to multiagent learning.

Original: Notable examples include ﬁctitious play (Brown, 1951; Robinson, 1951), independent reinforcement learning (Matignon et al., 2012) and the double oracle algorithm (McMahan et al., 2003).
Revised: Notable examples include ﬁctitious play (Brown, 1951; Robinson, 1951), independent reinforcement learning (Matignon et al., 2012) and the double oracle algorithm (McMahan et al., 2003).

Original: PSRO also relies, fundamentally, on principles from empirical game-theoretic analysis (EGTA)
Revised: PSRO also relies, fundamentally, on principles from empirical game-theoretic analysis (EGTA)

Original: (Walsh et al., 2002;
Revised: (Walsh et al., 2002;

Original: Phelps et al., 2004; Tuyls et al., 2018; Wellman, 2006; Vorobeychik, 2010; Wiedenbeck and Wellman, 2012; Wiedenbeck et al., 2014).
Revised: Phelps et al., 2004; Tuyls et al., 2018; Wellman, 2006; Vorobeychik, 2010; Wiedenbeck and Wellman, 2012; Wiedenbeck et al., 2014).

Original: PSRO allows the multiagent learning problem to be decomposed into a sequence of single-agent learning problems.
Revised: PSRO allows the multiagent learning problem to be decomposed into a sequence of single-agent learning problems.

Original: A wide variety of other approaches that deal with the multiagent learning problem without this reduction are also available, such as Multiagent Deep Deterministic Policy Gradients (MADDPG) (Lowe et al., 2017), Counterfactual Multiagent Policy Gradients (COMA)
Revised: A wide variety of other approaches that deal with the multiagent learning problem without this reduction are also available, such as Multiagent Deep Deterministic Policy Gradients (MADDPG) (Lowe et al., 2017), Counterfactual Multiagent Policy Gradients (COMA)

Original: (Foerster et al., 2018), Differentiable Inter-Agent Learning (DIAL) (Foerster et al., 2016), Hysteretic Deep Recurrent
Revised: (Foerster et al., 2018), Differentiable Inter-Agent Learning (DIAL) (Foerster et al., 2016), Hysteretic Deep Recurrent

Original: Q-learning (Omidshaﬁei et al., 2017), and lenient Multiagent Deep Reinforcement Learning (Palmer et al., 2018). 
Revised: Q-learning (Omidshaﬁei et al., 2017), and lenient Multiagent Deep Reinforcement Learning (Palmer et al., 2018). Several notable contributions have also been made in addressing multiagent learning challenges in continuous-control settings, most recently including the approaches of Iqbal and Sha (2019); Gupta et al. ; Wei et al. ; Peng et al.

Revised: ; Khadka et al.

Revised: We refer interested readers to the following survey of recent deep multiagent RL approaches Hernandez-Leal et al.

Original: α -Rank was introduced by Omidshaﬁei et al.
Revised: α -Rank was introduced by Omidshaﬁei et al.

Original: as a scalable dynamic alternative to Nash equilibria that can be applied in general-sum, many-player games and is capable of capturing the underlying multiagent evolutionary dynamics.
Revised: as a scalable dynamic alternative to Nash equilibria that can be applied in general-sum, many-player games and is capable of capturing the underlying multiagent evolutionary dynamics.

Original: Concepts from evolutionary dynamics have long been used in analysis of multiagent interactions from a meta-game standpoint (Walsh et al., 2002; Tuyls and Parsons, 2007;
Revised: Concepts from evolutionary dynamics have long been used in analysis of multiagent interactions from a meta-game standpoint (Walsh et al., 2002; Tuyls and Parsons, 2007;

Original: Hennes et al., 2013; Bloembergen et al., 2015; Tuyls et al., 2018).
Revised: Hennes et al., 2013; Bloembergen et al., 2015; Tuyls et al., 2018).

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: This paper studied variants of PSRO using α -Rank as a meta-solver, which were shown to be competitive with Nash-based PSRO in zero-sum games, and scale effortlessly to general-sum manyplayer games, in contrast to Nash-based PSRO.
Revised: This paper studied variants of PSRO using α -Rank as a meta-solver, which were shown to be competitive with Nash-based PSRO in zero-sum games, and scale effortlessly to general-sum manyplayer games, in contrast to Nash-based PSRO.

Original: We believe there are many interesting directions for future work, including how uncertainty in the meta-solver distribution, informed by recent developments in dealing with incomplete information in games (Reeves and Wellman, 2004; Walsh et al., 2003; Rowland et al., 2019), can be used to inform the selection of new strategies to be added to populations.
Revised: We believe there are many interesting directions for future work, including how uncertainty in the meta-solver distribution, informed by recent developments in dealing with incomplete information in games (Reeves and Wellman, 2004; Walsh et al., 2003; Rowland et al., 2019), can be used to inform the selection of new strategies to be added to populations.

Original: In summary, we strongly believe that the theoretical and empirical results established in this paper will play a key role in scaling up multiagent training in general settings A F E XPOSITION
Revised: In summary, we strongly believe that the theoretical and empirical results established in this paper will play a key role in scaling up multiagent training in general settings.

Original: Player

Original: A

Original: B

Original: C

Original: D

Original: X

Original: Player

Original: A − φ φ − ε

Original: B φ − φ − ε

Original: C −φ − φ − ε

Original: D − φ −φ − ε

Original: X ε ε ε ε

Original: Player

Original: A

Original: B

Original: C

Original: D

Original: X

Original: Player

Original: A − φ φ − ε

Original: B φ − φ − ε

Original: C −φ − φ − ε

Original: D − φ −φφφ − ε

Original: X ε ε ε ε

Original: Player

Original: A

Original: B

Original: C

Original: D

Original: X

Original: Player

Original: A − φ φφφ − ε

Original: B φ − φ − ε

Original: C −φ − φ − ε

Original: D − φ −φ − ε

Original: X ε ε ε ε

Original: Player

Original: A

Original: B

Original: C

Original: D

Original: X

Original: Player

Original: A − φ φ − ε

Original: B φφφ − φ − ε

Original: C −φ − φ − ε

Original: D − φ −φ − ε

Original: X ε ε ε ε

Original: Player

Original: A
Revised: R

Original: [Equation]

Original: A

Original: B

Original: C

Original: D

Original: X

Original: Player

Original: A − φ φ − ε

Original: B φ − φ − ε

Original: C −φ − φ − ε

Original: D − φ −φ − ε

Original: X εεε εεε εεε εεε

Original: A. E XAMPLE BEHAVIOR OF PSRO(N ASH , BR)
Revised: A. E XAMPLE BEHAVIOR OF PSRO(N ASH , BR)

Original: A ﬁrst attempt to establish convergence to α -Rank might involve running PSRO to convergence (until the oracle returns a strategy already in the convex hull of the known strategies) and then running α -Rank on the resulting meta-game.
Revised: A ﬁrst attempt to establish convergence to α -Rank might involve running PSRO to convergence (until the oracle returns a strategy already in the convex hull of the known strategies), and then running α -Rank on the resulting meta-game.

Original: However, the following provides a counterexample to this approach when using either PSRO(Nash, BR) or PSRO(Uniform, BR).
Revised: However, the following provides a counterexample to this approach when using either PSRO(Nash, BR) or PSRO(Uniform, BR).

Original: Player
Revised: Player

Original: A
Revised: A

Original: B
Revised: B

Original: X
Revised: X

Original: Player
Revised: Player

Original: A  ε
Revised: A  ε

Original: B  − ε
Revised: B  − ε

Original: X − ε ε (a) Example
Revised: X − ε ε (a) Example

Original: Player
Revised: Player

Original: A
Revised: A

Original: B
Revised: B

Original: Player
Revised: Player

Original: AB -
Revised: AB -

Original: X − ε − ε/(
Revised: X − ε − ε/(b)

Original: [Table]
Revised: Example

Original: The ﬁrst game is symmetric, whilst the second is zero-sum.
Revised: The ﬁrst game is symmetric, whilst the second is zero-sum.

Original: Both tables specify the payoff to Playerunder each strategy proﬁle.
Revised: Both tables specify the payoff to Playerunder each strategy proﬁle.

Original: Example 4.
Revised: Example 4.

Original: Consider the two-player symmetric game speciﬁed in Table 3a.
Revised: Consider the two-player symmetric game speciﬁed in Table 3a.

Original: The sink stronglyconnected component of the single-population response graph (and hence the α -Rank distribution) contains all three strategies, but all NE are supported on { A, B } only, and the best response to a strategy supported on { A, B } is another strategy supported on { A, B } .
Revised: The sink stronglyconnected component of the single-population response graph (and hence the α -Rank distribution) contains all three strategies, but all NE are supported on { A, B } only, and the best response to a strategy supported on { A, B } is another strategy supported on { A, B } .

Original: Thus, the single-population variant of PSRO, using M ∈ { Nash , Uniform } with initial strategies contained in { A, B } will terminate before discovering strategy X ; the full α -Rank distribution will thus not be recovered.
Revised: Thus, the single-population variant of PSRO, using M ∈ { Nash , Uniform } with initial strategies contained in { A, B } will terminate before discovering strategy X ; the full α -Rank distribution will thus not be recovered.

Original: Example 5.
Revised: Example 5.

Original: Consider the two-player zero-sum game speciﬁed in Table 3b.
Revised: Consider the two-player zero-sum game speciﬁed in Table 3b.

Original: All strategy proﬁles recieve non-zero probability in the multi-population α -Rank distribution.
Revised: All strategy proﬁles recieve non-zero probability in the multi-population α -Rank distribution.

Original: However, the Nash equilibrium over the game restricted to actions A, B for each player has a unique Nash equilibrium of (1 / 2 , 1 / 2) .
Revised: However, the Nash equilibrium over the game restricted to actions A, B for each player has a unique Nash equilibrium of (1 / 2 , 1 / 2) .

Original: Player 1’s best response to this Nash is to play some mixture of A and B , and therefore strategy X is not recovered by PSRO(Nash, BR) in this case, and so the full α -Rank distribution will thus not be recovered.
Revised: Player 1’s best response to this Nash is to play some mixture of A and B , and therefore strategy X is not recovered by PSRO(Nash, BR) in this case, and so the full α -Rank distribution will thus not be recovered.

Original: A. C OUNTEREXAMPLES : α -R ANK VS . N ASH S UPPORT
Revised: A. C OUNTEREXAMPLES : α -R ANK VS . N ASH S UPPORT

Original: The Game of Chicken
Revised: The Game of Chicken

Original: The Game of Chicken provides an example where the support of α -Rankin the multipopulation case - does not include the full support of Nash Equilibria.
Revised: The Game of Chicken provides an example where the support of α -Rankin the multipopulation case - does not include the full support of Nash Equilibria.

Original: This game has three Nash equilibria: Two pure, (D,C) and (C,D), and one mixed, where the population plays Dare with probability 13 .
Revised: This game has three Nash equilibria: Two pure, (D,C) and (C,D), and one mixed, where the population plays Dare with probability 13 .

Original: Nevertheless, α -rank only puts weight on (C,D) and (D,C), effectively not putting weight on the full mixed-nash support.
Revised: Nevertheless, α -rank only puts weight on (C,D) and (D,C), effectively not putting weight on the full mixed-nash support.

Original: Player
Revised: Player

Original: D
Revised: D

Original: C
Revised: C

Original: Player
Revised: Player

Original: D (0 , 0) (7 , 2)
Revised: D (0 , 0) (7 , 2)

Original: C (2 , 7) (6 , 6)
Revised: C (2 , 7) (6 , 6)

Original: Prisoner’s Dilemma The Prisoner’s Dilemma provides a counterexample that the support of αRank- in the multi-population case - does not include the full support of correlated equilibria.
Revised: Prisoner’s Dilemma The Prisoner’s Dilemma provides a counterexample that the support of αRank- in the multi-population case - does not include the full support of correlated equilibria.

Original: Player
Revised: Player

Original: D

Original: [Table]
Revised: [Table]

Original: Player

Original: D (0 , 0) (3 , − 1)
Revised: D (0 , 0) (3 , − 1)

Revised: [Table]

Original: C ( − 1 , 3) (2 , 2)
Revised: ( − 1 , 3) (2 , 2)

Original: This game has correlated equilibria that include (C,D), (D,C) and (C,C) in their support; nevertheless, α -Rank only puts weight on (D,D), effectively being fully disjoint from the support of the correlated equilibria.
Revised: This game has correlated equilibria that include (C,D), (D,C) and (C,C) in their support; nevertheless, α -Rank only puts weight on (D,D), effectively being fully disjoint from the support of the correlated equilibria.

Original: A. S INGLE - POPULATION α -C ONV
Revised: A. S INGLE - POPULATION α -C ONV

Original: In analogy with the multi-population deﬁnition in Section 4.2, we deﬁne a single-population version of α -C ONV .
Revised: In analogy with the multi-population deﬁnition in Section 4.2, we deﬁne a single-population version of α -C ONV .

Original: We start by deﬁning the single-population version of PBR-Score, given by
Revised: We start by deﬁning the single-population version of PBR-Score, given by

Original: PBR-S CORE ( σ ; π , S ) =
Revised: PBR-S CORE ( σ ; π , [Equation] [Equation] s

Original: [Equation]
Revised: [Equation]

Original: i π i 1 (cid:2) M 1 ( σ, s i ) > M 2 ( σ i , s i ) (cid:3) .
Revised: ) > [Equation] i , s i ) [Equation]

Original: The single-population α -C ONV is then deﬁned as
Revised: The single-population α -C ONV is then deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where max σ is taken over the pure strategies of the underlying game.
Revised: where max σ is taken over the pure strategies of the underlying game.

Original: B P ROOFS B. P ROOF OF P ROPOSITION
Revised: B P ROOFS B. P ROOF OF P ROPOSITION

Original: Proposition 1.
Revised: Proposition 1.

Original: If at any point the population of α -PSRO contains a member of an SSCC of the game, then α -PSRO will α -partially converge to that SSCC.
Revised: If at any point the population of α -PSRO contains a member of an SSCC of the game, then α -PSRO will α -partially converge to that SSCC.

Original: Proof.
Revised: Proof.

Original: Suppose that a member of one of the underlying game’s SSCCs appears in the α -PSRO population.
Revised: Suppose that a member of one of the underlying game’s SSCCs appears in the α -PSRO population.

Original: This member will induce its own meta-SSCC in the meta-game’s response graph.
Revised: This member will induce its own meta-SSCC in the meta-game’s response graph.

Original: At least one of the members of the underlying game’s corresponding SSCC will thus always have positive probability under the α -Rank distribution for the meta-game, and the PBR oracle for this meta-SSCC will always return a member of the underlying game
Revised: At least one of the members of the underlying game’s corresponding SSCC will thus always have positive probability under the α -Rank distribution for the meta-game, and the PBR oracle for this meta-SSCC will always return a member of the underlying game

Original: ’s SSCC.
Revised: ’s SSCC.

Original: If the PBR oracle returns a member of the underlying SSCC already in the PSRO population, we claim that the corresponding meta-SSCC already contains a cycle of the underlying SSCC.
Revised: If the PBR oracle returns a member of the underlying SSCC already in the PSRO population, we claim that the corresponding meta-SSCC already contains a cycle of the underlying SSCC.

Original: To see this, note that if the meta-SSCC does not contain a cycle, it must be a singleton.
Revised: To see this, note that if the meta-SSCC does not contain a cycle, it must be a singleton.

Original: Either this singleton is equal to the full SSCC of the underlying game (in which we have α -fully converged), or it is not, in which case the PBR oracle must return a new strategy from the underlying SSCC, contradicting our assumption that it has terminated.
Revised: Either this singleton is equal to the full SSCC of the underlying game (in which we have α -fully converged), or it is not, in which case the PBR oracle must return a new strategy from the underlying SSCC, contradicting our assumption that it has terminated.

Original: B. P ROOF OF P ROPOSITION
Revised: B. P ROOF OF P ROPOSITION

Original: Proposition 2.
Revised: Proposition 2.

Original: If we constrain the PBR oracle used in α -PSRO to be novelty-bound, then α -PSRO will α -fully converge to at least one SSCC of the game.
Revised: If we constrain the PBR oracle used in α -PSRO to be novelty-bound, then α -PSRO will α -fully converge to at least one SSCC of the game.

Original: Proof.
Revised: Proof.

Original: Suppose that α -PSRO has converged, and consider a meta-SSCC.
Revised: Suppose that α -PSRO has converged, and consider a meta-SSCC.

Original: Since α -PSRO has converged, it follows that each strategy proﬁle of the meta-SSCC is an element of an SSCC of the underlying game.
Revised: Since α -PSRO has converged, it follows that each strategy proﬁle of the meta-SSCC is an element of an SSCC of the underlying game.

Original: Any strategy proﬁle in this SSCC which is not in the meta-SSCC will obtain a positive value for the PBR objective, and since α -PSRO has converged, there can be no such strategy proﬁle.
Revised: Any strategy proﬁle in this SSCC which is not in the meta-SSCC will obtain a positive value for the PBR objective, and since α -PSRO has converged, there can be no such strategy proﬁle.

Original: Thus, the meta-SSCC contains every strategy proﬁle contained within the corresponding
Revised: Thus, the meta-SSCC contains every strategy proﬁle contained within the corresponding

Original: SSCC of the underlying game, and therefore conclude that α -PSRO α -fully converges to an SSCC of the underlying game.
Revised: SSCC of the underlying game, and therefore conclude that α -PSRO α -fully converges to an SSCC of the underlying game.

Original: B. P ROOF OF P ROPOSITION
Revised: B. P ROOF OF P ROPOSITION

Original: Proposition 3.
Revised: Proposition 3.

Original: (Single-population)
Revised: (Single-population)

Original: α -PSRO converges α -partially to the unique SSCC.
Revised: α -PSRO converges α -partially to the unique SSCC.

Original: Proof.
Revised: Proof.

Original: The uniqueness of the SSCC follows from the fact that in the single-population case, the response graph is fully-connected.
Revised: The uniqueness of the SSCC follows from the fact that in the single-population case, the response graph is fully-connected.

Original: Suppose at termination of α -PSRO, the α -PSRO population contains no strategy within the SSCC, and let s be a strategy in the SSCC.
Revised: Suppose at termination of α -PSRO, the α -PSRO population contains no strategy within the SSCC, and let s be a strategy in the SSCC.

Original: We claim that s attains a higher value for the objective deﬁning the PBR oracle than any strategy in the α -PSRO population, which contradicts the fact that α -PSRO has terminated.
Revised: We claim that s attains a higher value for the objective deﬁning the PBR oracle than any strategy in the α -PSRO population, which contradicts the fact that α -PSRO has terminated.

Original: To complete this argument, we note that by virtue of s being in the SSCC, we have M 1 ( s, s (cid:48) )
Revised: To complete this argument, we note that by virtue of s being in the SSCC, we have M 1 ( s, s (cid:48) )

Original: >
Revised: >

Original: M 1 ( s (cid:48) , s ) for all s (cid:48) outside the SSCC, and in particular for all s (cid:48) ∈ S , thus the PBR objective for s is 1 .
Revised: M 1 ( s (cid:48) , s ) for all s (cid:48) outside the SSCC, and in particular for all s (cid:48) ∈ S , thus the PBR objective for s is 1 .

Original: In contrast, for any s i ∈ S , the PBR objective for s i is upper-bounded by 1 − π i .
Revised: In contrast, for any s i ∈ S , the PBR objective for s i is upper-bounded by 1 − π i .

Original: If π i > 0 , then this shows s i is not selected by the oracle, since the objective value is lower than that of s .
Revised: If π i > 0 , then this shows s i is not selected by the oracle, since the objective value is lower than that of s .

Original: If π i = 0 , then the objective value for s i is 0 , and so an SSCC member will always have a maximal PBR score of 1 against a population not composed of any SSCC member, and all members of that population have < 1 PBR scores.
Revised: If π i = 0 , then the objective value for s i is 0 , and so an SSCC member will always have a maximal PBR score of 1 against a population not composed of any SSCC member, and all members of that population have < 1 PBR scores.

Original: Consequently, singlepopulation α -PSRO cannot terminate before it has encountered an SSCC member.
Revised: Consequently, singlepopulation α -PSRO cannot terminate before it has encountered an SSCC member.

Original: By Proposition 1, the proposition is therefore proven.
Revised: By Proposition 1, the proposition is therefore proven.

Original: B. P ROOF OF P ROPOSITION
Revised: B. P ROOF OF P ROPOSITION

Original: Proposition 4.
Revised: Proposition 4.

Original: (Multi-population)
Revised: (Multi-population)

Original: Without a novelty-bound oracle, there exist games for which α -PSRO does not converge α -partially to any SSCC.
Revised: Without a novelty-bound oracle, there exist games for which α -PSRO does not converge α -partially to any SSCC.

Original: Proof.
Revised: Proof.

Original: We exhibit a speciﬁc counterexample to the claim.
Revised: We exhibit a speciﬁc counterexample to the claim.

Original: Consider the three-player, three-strategy game with response graph illustrated in Fig.
Revised: Consider the three-player, three-strategy game with response graph illustrated in Fig.

Original: B.7a; note that we do not enumerate all strategy proﬁles not appearing in the SSCC for space and clarity reasons.
Revised: B.7a; note that we do not enumerate all strategy proﬁles not appearing in the SSCC for space and clarity reasons.

Original: The sequence of updates undertaken by α -PSRO in this game is illustrated in Figs.
Revised: The sequence of updates undertaken by α -PSRO in this game is illustrated in Figs.

Original: B.7b to B.7f; whilst the singleton strategy proﬁle (3 , 2 , 3) forms the unique SSCC for this game, α -PSRO terminates before reaching it, which concludes the proof.
Revised: B.7b to B.7f; whilst the singleton strategy proﬁle (3 , 2 , 3) forms the unique SSCC for this game, α -PSRO terminates before reaching it, which concludes the proof.

Original: The steps taken by the algorithm are described below; again, we do not enumerate all strategy proﬁles not appearing in the SSCC for space and clarity reasons.
Revised: The steps taken by the algorithm are described below; again, we do not enumerate all strategy proﬁles not appearing in the SSCC for space and clarity reasons.

Original: Begin with strategies [[2] , [1] , [1]] in the α -PSRO population (Player 1 only has access to strategy 2, Players 2 and 3 only have access to strategy 1) 2.
Revised: Begin with strategies [[2] , [1] , [1]] in the α -PSRO population (Player 1 only has access to strategy 2, Players 2 and 3 only have access to strategy 1) 2.

Original: The PBR to (2,1,1) for player 2 is 2, and no other player has a PBR on this round.
Revised: The PBR to (2,1,1) for player 2 is 2, and no other player has a PBR on this round.

Original: We add 2 to the strategy space of player 2, which changes the space of available joint strategies to [(2 , 1 , 1) , (2 , 2 , 1)] .
Revised: We add 2 to the strategy space of player 2, which changes the space of available joint strategies to [(2 , 1 , 1) , (2 , 2 , 1)] .

Original: α -Rank puts all its mass on (2,2,1).
Revised: α -Rank puts all its mass on (2,2,1).

Original: The PBR to (2,2,1) for player 3 is 2, and no other player has a PBR on this round.
Revised: The PBR to (2,2,1) for player 3 is 2, and no other player has a PBR on this round.

Original: We add strategy 2 to player 3’s strategy space, which changes the space of available joint strategies to [(2 , 1 , 1) , (2 , 2 , 1) , (2 , 2 , 2)] .
Revised: We add strategy 2 to player 3’s strategy space, which changes the space of available joint strategies to [(2 , 1 , 1) , (2 , 2 , 1) , (2 , 2 , 2)] .

Original: α -Rank puts all its mass on (2,2,2).
Revised: α -Rank puts all its mass on (2,2,2).

Original: The PBR to (2,2,2) for player 1 is 1, and no other player has a PBR on this round.
Revised: The PBR to (2,2,2) for player 1 is 1, and no other player has a PBR on this round.

Original: We add strategy 1 to player 1’s strategy space, which changes the space of available joint strategies to [(1 , 1 , 1) , (1 , 1 , 2) , (1 , 2 , 1) , (1 , 2 , 2) , (2 , 1 , 1) , (2 , 2 , 1) , (2 , 2 , 2)] .
Revised: We add strategy 1 to player 1’s strategy space, which changes the space of available joint strategies to [(1 , 1 , 1) , (1 , 1 , 2) , (1 , 2 , 1) , (1 , 2 , 2) , (2 , 1 , 1) , (2 , 2 , 1) , (2 , 2 , 2)] .

Original: Deﬁne σ as the α -Rank probabilities of the meta-game.
Revised: Deﬁne σ as the α -Rank probabilities of the meta-game.

Original: Player 1 playing strategy 2 has a PBR score of σ ((1 , 1 , 1))
Revised: Player 1 playing strategy 2 has a PBR score of σ ((1 , 1 , 1))

Original: + σ ((1 , 2 , 1)) , and the same player playing strategy 3 has a PBR score of σ ((1 , 2 , 1)) , which is lower than the PBR Score of playing strategy 2.
Revised: + σ ((1 , 2 , 1)) , and the same player playing strategy 3 has a PBR score of σ ((1 , 2 , 1)) , which is lower than the PBR Score of playing strategy 2.

Original: No other player has a valid PBR for this
Revised: No other player has a valid PBR for this

Original: round, and therefore, α -PSRO terminates.
Revised: round, and therefore, α -PSRO terminates.

Original: In the above example, pictured in Fig.
Revised: In the above example, pictured in Fig.

Original: B.7, a relatively weak joint strategy (Strategy (3,2,1)) bars agents from ﬁnding the optimal joint strategy of the game (Strategy (3,2,3)) : getting to this joint strategy requires coordinated changes between agents, and is therefore closely related to the common problem of Action/Equilibrium Shadowing mentioned in (Matignon et al., 2012).
Revised: B.7, a relatively weak joint strategy (Strategy (3,2,1)) bars agents from ﬁnding the optimal joint strategy of the game (Strategy (3,2,3)) : getting to this joint strategy requires coordinated changes between agents, and is therefore closely related to the common problem of Action/Equilibrium Shadowing mentioned in (Matignon et al., 2012).

Original: B. P ROOF OF P ROPOSITION
Revised: B. P ROOF OF P ROPOSITION

Original: Proposition 5.
Revised: Proposition 5.

Original: A constant-sum game is denoted as win-loss if M k ( s ) ∈ { 0 , 1 } for all k ∈ [ K ] and s ∈ S .
Revised: A constant-sum game is denoted as win-loss if M k ( s ) ∈ { 0 , 1 } for all k ∈ [ K ] and s ∈ S .

Original: BR is compatible with PBR in win-loss games in the two-player single-population case.
Revised: BR is compatible with PBR in win-loss games in the two-player single-population case.

Original: Proof.
Revised: Proof.

Original: We manipulate the best-response objective as follows:
Revised: We manipulate the best-response objective as follows:

Original: [Equation]
Revised: [Equation]

Original: Noting that the ﬁnal line is the single-population PBR objective, we are done.
Revised: Noting that the ﬁnal line is the single-population PBR objective, we are done.

Original: B. P ROOF OF P ROPOSITION
Revised: B. P ROOF OF P ROPOSITION

Original: Proposition 6.
Revised: Proposition 6.

Original: A symmetric two-player game is denoted monotonic if there exists a function f :
Revised: A symmetric two-player game is denoted monotonic if there exists a function f :

Original: S → R and a non-decreasing function σ : R → R such that M 1 ( s, ν ) =
Revised: S → R and a non-decreasing function σ : R → R such that M 1 ( s, ν ) =

Original: σ ( f ( s ) − f ( ν )) .
Revised: σ ( f ( s ) − f ( ν )) .

Original: BR is compatible with PBR in monotonic games in the single-population case.
Revised: BR is compatible with PBR in monotonic games in the single-population case.

Original: Proof.
Revised: Proof.

Original: Rewriting the objectives given that the game is monotonic, we have that the value-based objective becomes
Revised: Rewriting the objectives given that the game is monotonic, we have that the value-based objective becomes

Original: [Equation]
Revised: [Equation]

Original: Given the fact that the only condition we have on σ is its non-decreasing character, this objective does not reduce to maximizing f ( s ) in the general case.
Revised: Given the fact that the only condition we have on σ is its non-decreasing character, this objective does not reduce to maximizing f ( s ) in the general case.

Original: The objective for PBR is
Revised: The objective for PBR is

Original: [Equation]
Revised: [Equation]

Original: 1,2, 1,1,1 3,2, 3,2, 2,1, 1,1, 2,2, 2,2, 1,2, 20,20, 4,3, 2,2, 3,3, 1,5, 3,3, 10,10, 1,1, 5,1, (a) Full response graph.
Revised: 1,2, 1,1,1 3,2, 3,2, 2,1, 1,1, 2,2, 2,2, 1,2, 20,20, 4,3, 2,2, 3,3, 1,5, 3,3, 10,10, 1,1, 5,1, (a) Full response graph.

Original: 2,1, 3,3, (b) α -PSRO
Revised: 2,1, 3,3, (b) α -PSRO

Original: Step 1.
Revised: Step 1.

Original: [Figure]
Revised: [Figure]

Original: 2,1, 2,2, 2,2, 3,3, 10,10, 1,1, (d) α -PSRO Step 3.
Revised: 2,1, 2,2, 2,2, 3,3, 10,10, 1,1, (d) α -PSRO Step 3.

Original: 2,1, 2,2, 2,2, 1,2,2 3,3, 10,10, 1,1, 5,1, (e) α -PSRO
Revised: 2,1, 2,2, 2,2, 1,2,2 3,3, 10,10, 1,1, 5,1, (e) α -PSRO

Original: Step 4.
Revised: Step 4.

Original: 1,2, 1,1, 1,1, 1,2, 2,2, 3,3, 1,5, 3,3, 10,10, 1,1, 5,1, 2,2, 2,2, 2,1, (f) α -PSRO
Revised: 1,2, 1,1, 1,1, 1,2, 2,2, 3,3, 1,5, 3,3, 10,10, 1,1, 5,1, 2,2, 2,2, 2,1, (f) α -PSRO

Original: Step 5.
Revised: Step 5.

Original: Since σ is non-decreasing,
Revised: Since σ is non-decreasing,

Original: [Equation]
Revised: [Equation]

Original: and conversely,
Revised: and conversely,

Original: [Equation]
Revised: [Equation]

Original: Without loss of generality, we reorder the strategies such that if i < k , f ( s i ) ≤ f ( s k ) .
Revised: Without loss of generality, we reorder the strategies such that if i < k , f ( s i ) ≤ f ( s k ) .

Original: Let s v maximize the value objective.
Revised: Let s v maximize the value objective.

Original: Therefore, by monotonicity, s v maximizes σ ( f ( s ) − f
Revised: Therefore, by monotonicity, s v maximizes σ ( f ( s ) − f

Original: ( s K )) .
Revised: ( s K )) .

Original: Three possibilities then ensue.
Revised: Three possibilities then ensue.

Original: If there exists s such that
Revised: If there exists s such that

Original: [Equation]
Revised: [Equation]

Original: then
Revised: then

Original: [Equation]
Revised: [Equation]

Original: since s v maximizes σ ( f ( s ) − f
Revised: since s v maximizes σ ( f ( s ) − f

Original: ( s K )) and σ is non-decreasing.
Revised: ( s K )) and σ is non-decreasing.

Original: Consequently s v maximizes the PBR objective.
Revised: Consequently s v maximizes the PBR objective.

Original: Indeed, let us remark that for all k ≤ K , we have that
Revised: Indeed, let us remark that for all k ≤ K , we have that

Original: [Equation]
Revised: [Equation]

Original: since 
Revised: since σ

Original: [Equation]
Revised: [Equation]

Original: Else, if there does not exist any policy s such that σ ( f ( s ) − f ( s K ))
Revised: Else, if there does not exist any policy s such that σ ( f ( s ) − f ( s K ))

Original: [Equation]
Revised: [Equation]

Original: σ ( f ( s K ) − f ( s )) , that is, for all s ,
Revised: σ ( f ( s K ) − f ( s )) , that is, for all s ,

Original: [Equation]
Revised: [Equation]

Original: Since s K is a possible solution to the value objective,
Revised: Since s K is a possible solution to the value objective,

Original: [Equation]
Revised: [Equation]

Original: Let n be the integer such that
Revised: Let n be the integer such that

Original: [Equation]
Revised: [Equation]

Original: f
Revised: f

Original: [Equation]
Revised: [Equation]

Original: s
Revised: s

Original: [Equation]
Revised: [Equation]

Original: s k
Revised: s k

Original: [Equation]
Revised: [Equation]

Original: Population |
Revised: Population |

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: s k 
Revised: s k )

Original: [Equation]
Revised: [Equation]

Original: >
Revised: >

Original: [Equation]
Revised: [Equation]

Original:  f [Equation] s k )
Revised: ( f ( s k )

Original: [Equation]
Revised: [Equation]

Original:  ( s )
Revised: f ( s )

Original: [Equation]
Revised: [Equation]

Original: If s n exists , then we have that for all s i such that f ( s i ) >
Revised: If s n exists , then we have that for all s i such that f ( s i ) >

Original: f ( s n ) ,
Revised: f ( s n ) ,

Original: [Equation]
Revised: [Equation]

Original: The PBR objective is
Revised: The PBR objective is

Original: [Equation]
Revised: [Equation]

Original: which, according to our assumptions, is equivalent to
Revised: which, according to our assumptions, is equivalent to

Original: [Equation]
Revised: [Equation]

Original: We know that for all i ≤ n , σ ( f ( s v ) − f
Revised: We know that for all i ≤ n , σ ( f ( s v ) − f

Original: ( s i ))
Revised: ( s i ))

Original: >
Revised: >

Original: σ ( f ( s i ) − f
Revised: σ ( f ( s i ) − f

Original: ( s v )) , and therefore, s v maximizes the
Revised: ( s v )) , and therefore, s v maximizes the

Original: PBR objective.
Revised: PBR objective.

Original: Finally, if s n doesn’t exist, then any policy is solution to the PBR objective, and therefore s v is.
Revised: Finally, if s n doesn’t exist, then any policy is solution to the PBR objective, and therefore s v is.

Original: A toy example showing the compatibility between Best Response and Preference-based Best Response is shown in Fig.
Revised: A toy example showing the compatibility between Best Response and Preference-based Best Response is shown in Fig.

Original: B.8.
Revised: B.8.

Original: The setting is that of a monotonic game where every strategy is assigned a number.
Revised: The setting is that of a monotonic game where every strategy is assigned a number.

Original: Strategies are then dominated by all strategies with higher number than theirs.
Revised: Strategies are then dominated by all strategies with higher number than theirs.

Original: We compute
Revised: We compute

Original: BR and PBR on an initial population composed of one strategy that we choose to be dominated by every other strategy.
Revised: BR and PBR on an initial population composed of one strategy that we choose to be dominated by every other strategy.

Original: Any strategy dominating the current population is a valid solution for PBR, as represented in Fig.
Revised: Any strategy dominating the current population is a valid solution for PBR, as represented in Fig.

Original: B.8c; whereas, if we consider that the game is monotonic with σ a strictly increasing function, only one strategy maximizes Best Response, strategy N – and it is thus the only solution of BR, as shown in Fig.
Revised: B.8c; whereas, if we consider that the game is monotonic with σ a strictly increasing function, only one strategy maximizes Best Response, strategy N – and it is thus the only solution of BR, as shown in Fig.

Original: B.8d.
Revised: B.8d.

Original: As we can see, the solution of BR is part of the possible solutions of PBR, demonstrating the result of
Revised: As we can see, the solution of BR is part of the possible solutions of PBR, demonstrating the result of

Original: Proposition 6: BR is compatible with PBR in monotonic games.
Revised: Proposition 6: BR is compatible with PBR in monotonic games.

Original: NN ...
Revised: NN ...

Original: [Figure]
Revised: [Figure]

Original: NN ...
Revised: NN ...

Original: [Figure]
Revised: [Figure]

Original: NN ...
Revised: NN ...

Original: [Figure]
Revised: [Figure]

Original: NN ...
Revised: NN ...

Original: [Figure]
Revised: [Figure]

Original: B. P ROOF OF P ROPOSITION
Revised: B. P ROOF OF P ROPOSITION

Original: Proposition 7.
Revised: Proposition 7.

Original: Consider symmetric win-loss games where outcomes between deterministic strategies are deterministic.
Revised: Consider symmetric win-loss games where outcomes between deterministic strategies are deterministic.

Original: A preference-based RL agent (i.e., an agent aiming to maximize its probability of winning against a distribution π of strategies { s 1 , . . .
Revised: A preference-based RL agent (i.e., an agent aiming to maximize its probability of winning against a distribution π of strategies { s 1 , . . .

Original: , s N } ) optimizes exactly the PBR objective (1) .
Revised: , s N } ) optimizes exactly the PBR objective (1) .

Original: Proof.
Revised: Proof.

Original: Commencing with the above preference-based RL objective, we calculate as follows,
Revised: Commencing with the above preference-based RL objective, we calculate as follows,

Original: [Equation]
Revised: [Equation]

Original: 1 [ σ receives a positive expected payoff against s i ] with the ﬁnal equality whenever game outcomes between two deterministic strategies are deterministic.
Revised: 1 [ σ receives a positive expected payoff against s i ] with the ﬁnal equality whenever game outcomes between two deterministic strategies are deterministic.

Original: Note that this is precisely the PBR objective (1).
Revised: Note that this is precisely the PBR objective (1).

Original: B. P ROOF OF P ROPOSITION
Revised: B. P ROOF OF P ROPOSITION

Original: Proposition 8.
Revised: Proposition 8.

Original: For symmetric two-player zero-sum games where off-diagonal payoffs have equal magnitude, all NE have support contained within that of the single-population α -Rank distribution.
Revised: For symmetric two-player zero-sum games where off-diagonal payoffs have equal magnitude, all NE have support contained within that of the single-population α -Rank distribution.

Original: Proof.
Revised: Proof.

Original: In the single-population case, the support of the α -Rank distribution is simply the (unique) sink strongly-connected component of the response graph (uniqueness follows from the fact that the response graph, viewed as an undirected graph, is fully-connected).
Revised: In the single-population case, the support of the α -Rank distribution is simply the (unique) sink strongly-connected component of the response graph (uniqueness follows from the fact that the response graph, viewed as an undirected graph, is fully-connected).

Original: We will now argue that for a strategy s in the sink strongly-connected component and a strategy z outside the sink stronglyconnected component, we have
Revised: We will now argue that for a strategy s in the sink strongly-connected component and a strategy z outside the sink stronglyconnected component, we have

Original: [Equation]
Revised: [Equation]

Original: This inequality states that when an opponent plays according to π , the expected payoff to the row player is greater if they defect to s whenever they would have played z .
Revised: This inequality states that when an opponent plays according to π , the expected payoff to the row player is greater if they defect to s whenever they would have played z .

Original: This implies that if a supposed symmetric Nash equilibrium contains a strategy z outside the sink strongly-connected component in its support, then it could receive higher reward by playing s instead, which contradicts the fact that it is an NE.
Revised: This implies that if a supposed symmetric Nash equilibrium contains a strategy z outside the sink strongly-connected component in its support, then it could receive higher reward by playing s instead, which contradicts the fact that it is an NE.

Original: We show (3) by proving a stronger result — namely, that s dominates z as strategies.
Revised: We show (3) by proving a stronger result — namely, that s dominates z as strategies.

Original: Firstly, since s is the sink strongly-connected component and z is not, s beats z , and so M 1 ( s, z ) >
Revised: Firstly, since s is the sink strongly-connected component and z is not, s beats z , and so M 1 ( s, z ) >

Original: M 1 ( s, s )
Revised: M 1 ( s, s )

Original: =
Revised: =

Original: M 1 ( z, z ) > M 1 ( z, s ) .
Revised: M 1 ( z, z ) > M 1 ( z, s ) .

Original: Next, if a (cid:54)∈ { s, z } is in the sink strongly-connected component, then a beats z , and so M 1 ( s, a ) > M 1 ( z, a ) if s beats a , and M 1 ( s, a ) =
Revised: Next, if a (cid:54)∈ { s, z } is in the sink strongly-connected component, then a beats z , and so M 1 ( s, a ) > M 1 ( z, a ) if s beats a , and M 1 ( s, a ) =

Original: M 1 ( z, a ) otherwise.
Revised: M 1 ( z, a ) otherwise.

Original: Finally, if a (cid:54) = s, z is not in the sink strongly-connected component, then M 1 ( s, a ) =
Revised: Finally, if a (cid:54) = s, z is not in the sink strongly-connected component, then M 1 ( s, a ) =

Original: M 1 ( z, a ) is z beats a , and M 1 ( s, a ) >
Revised: M 1 ( z, a ) is z beats a , and M 1 ( s, a ) >

Original: M 1 ( z, a ) otherwise.
Revised: M 1 ( z, a ) otherwise.

Original: Thus, (3) is proven, and the result follows.
Revised: Thus, (3) is proven, and the result follows.

Original: B. P ROOF OF P ROPOSITION
Revised: B. P ROOF OF P ROPOSITION

Original: Proposition 9.
Revised: Proposition 9.

Original: In a symmetric two-player zero-sum game, there exists an NE with support contained within that of the α -Rank distribution.
Revised: In a symmetric two-player zero-sum game, there exists an NE with support contained within that of the α -Rank distribution.

Original: Proof.
Revised: Proof.

Original: Consider the restriction of the game to the strategies contained in the sink strongly-connected component of the original game.
Revised: Consider the restriction of the game to the strategies contained in the sink strongly-connected component of the original game.

Original: Let π be an NE for this restricted game, and consider this as a distribution over all strategies in the original game (putting 0 mass on strategies outside the sink component).
Revised: Let π be an NE for this restricted game, and consider this as a distribution over all strategies in the original game (putting 0 mass on strategies outside the sink component).

Original: We argue that this is an NE for the full game, and the statement follows.
Revised: We argue that this is an NE for the full game, and the statement follows.

Original: To see this, note that since any strategy outside the sink strongly-connected component receives a non-positive payoff when playing against a strategy in the sink strongly-connected component, and that for at least one strategy in the sink strongly-connected component, this payoff is negative.
Revised: To see this, note that since any strategy outside the sink strongly-connected component receives a non-positive payoff when playing against a strategy in the sink strongly-connected component, and that for at least one strategy in the sink strongly-connected component, this payoff is negative.

Original: Considering the payoffs available to the row player when the column player plays according to π , we observe that the expected payoff for any strategy outside the sink strongly-connected component is negative, since every strategy in the sink
Revised: Considering the payoffs available to the row player when the column player plays according to π , we observe that the expected payoff for any strategy outside the sink strongly-connected component is negative, since every strategy in the sink

Original: strongly-connected component beats the strategy outside the component.
Revised: strongly-connected component beats the strategy outside the component.

Original: The payoff when defecting to a strategy in the sink strongly-connected component must be non-positive, since π is an
Revised: The payoff when defecting to a strategy in the sink strongly-connected component must be non-positive, since π is an

Original: NE for the restricted game.
Revised: NE for the restricted game.

Original: C A DDITIONAL D ETAILS ON E XPERIMENTS C. E XPERIMENTAL P ROCEDURES
Revised: C A DDITIONAL D ETAILS ON E XPERIMENTS C. E XPERIMENTAL P ROCEDURES

Original: The code backend for the Poker experiments used OpenSpiel (Lanctot et al., 2019).
Revised: The code backend for the Poker experiments used OpenSpiel (Lanctot et al., 2019).

Original: Specifically, we used OpenSpiel’s Kuhn and Leduc poker implementations, and exact best responses were computed by traversing the game tree (see implementation details in https://github.com/deepmind/open_spiel/blob/master/open_spiel/ python/algorithms/best_response.py ).
Revised: Specifically, we used OpenSpiel’s Kuhn and Leduc poker implementations, and exact best responses were computed by traversing the game tree (see implementation details in https://github.com/deepmind/open_spiel/blob/master/open_spiel/ python/algorithms/best_response.py ).

Original: 100 game simulations were used to estimate the payoff matrix for each possible strategy pair.
Revised: 100 game simulations were used to estimate the payoff matrix for each possible strategy pair.

Original: Although the underlying Kuhn and Leduc poker games are stochastic (due to random initial card deals), the associated meta-games are essentially deterministic (as, given enough game simulations, the mean payoffs are ﬁxed).
Revised: Although the underlying Kuhn and Leduc poker games are stochastic (due to random initial card deals), the associated meta-games are essentially deterministic (as, given enough game simulations, the mean payoffs are ﬁxed).

Original: The subsequent PSRO updates are, thus, also deterministic.
Revised: The subsequent PSRO updates are, thus, also deterministic.

Original: Despite this, we report averages over 2 runs per PSRO M , primarily to capture stochasticity due to differences in machine-speciﬁc rounding errors that occur due to the distributed computational platforms we run these experiments on.
Revised: Despite this, we report averages over 2 runs per PSRO M , primarily to capture stochasticity due to differences in machine-speciﬁc rounding errors that occur due to the distributed computational platforms we run these experiments on.

Original: For experiments involving α -Rank, we conduct a full sweep over the ranking-intensity parameter, α , following each iteration of α -PSRO.
Revised: For experiments involving α -Rank, we conduct a full sweep over the ranking-intensity parameter, α , following each iteration of α -PSRO.

Original: We implemented a version of α -Rank (building on the OpenSpiel implementation https://github.com/deepmind/open_spiel/blob/master/
Revised: We implemented a version of α -Rank (building on the OpenSpiel implementation https://github.com/deepmind/open_spiel/blob/master/

Original: open_spiel/python/egt/alpharank.py ) that used a sparse representation for the underlying transition matrix, enabling scaling-up to the large-scale NFG results presented in the experiments.
Revised: open_spiel/python/egt/alpharank.py ) that used a sparse representation for the underlying transition matrix, enabling scaling-up to the large-scale NFG results presented in the experiments.

Original: For experiments involving the projected replicator dynamics (PRD), we used uniformly-initialized meta-distributions, running PRD for 5 e 4 iterations, using a step-size of dt = 1 e − 3 , and exploration parameter γ = 1 e − 10 .
Revised: For experiments involving the projected replicator dynamics (PRD), we used uniformly-initialized meta-distributions, running PRD for 5 e 4 iterations, using a step-size of dt = 1 e − 3 , and exploration parameter γ = 1 e − 10 .

Original: Time-averaged distributions were computed over the entire trajectory.
Revised: Time-averaged distributions were computed over the entire trajectory.

Original: C. D OMAIN D ESCRIPTION AND G ENERATION C.2. N ORMAL F ORM G AMES G ENERATION
Revised: C. D OMAIN D ESCRIPTION AND G ENERATION C.2. N ORMAL F ORM G AMES G ENERATION

Original: Algorithms 2 to 4 provide an overview of the procedure we use to randomly-generate normal-form games for the oracle comparisons visualized in Fig.
Revised: Algorithms 2 to 4 provide an overview of the procedure we use to randomly-generate normal-form games for the oracle comparisons visualized in Fig.

Original: (Actions, Players, mean value =

Original: [0 . 0 , 1 . 0] , mean probability =

Original: T [ k ] = f [Equation] − [Equation]
Revised: C.

Original: | Players |

Original: [Equation]

Original: (cid:80) i (cid:54)

Original: =

Original: k C.2. K UHN AND L EDUC P OKER
Revised: K UHN AND L EDUC P OKER

Original: K -player Kuhn poker is played with a deck of K + 1 cards.
Revised: K -player Kuhn poker is played with a deck of K + 1 cards.

Original: Each player starts with 2 chips andface-down card, and antes 1 chip to play.
Revised: Each player starts with 2 chips andface-down card, and antes 1 chip to play.

Original: Players either bet (raise/call) or fold iteratively, until each player is either in (has contributed equally to the pot) or has folded.
Revised: Players either bet (raise/call) or fold iteratively, until each player is either in (has contributed equally to the pot) or has folded.

Original: Amongst the remaining players, the one with the highest-ranked card wins the pot.
Revised: Amongst the remaining players, the one with the highest-ranked card wins the pot.

Original: Leduc Poker, in comparison, has a signiﬁcantly larger state-space.
Revised: Leduc Poker, in comparison, has a signiﬁcantly larger state-space.

Original: Players in Leduc have unlimited chips, receive 1 face-down card, ante 1 chip to play, with subsequent bets limited to 2 and 4 chips in rounds 1 and 2.
Revised: Players in Leduc have unlimited chips, receive 1 face-down card, ante 1 chip to play, with subsequent bets limited to 2 and 4 chips in rounds 1 and 2.

Original: A maximum of two raises are allowed in each round, and a public card is revealed before the second round.
Revised: A maximum of two raises are allowed in each round, and a public card is revealed before the second round.

Original: C. PBR COMPUTATION IN N ORMAL F ORM G AMES
Revised: C. PBR COMPUTATION IN N ORMAL F ORM G AMES

Original: The algorithms used to compute PBR and PBR-S CORE in the games generated by the algorithm described in Section C.2.1 is shown in Algorithms 5 and 6.
Revised: The algorithms used to compute PBR and PBR-S CORE in the games generated by the algorithm described in Section C.2.1 is shown in Algorithms 5 and 6.

Original: Note that they compute the multipopulation version of PBR.
Revised: Note that they compute the multipopulation version of PBR.

Original: PCS-S CORE is computed by pre-computing the full game’s SSCC, and computing the proportion of currently selected strategies in the empirical game that also belongs to the full game’s SSCC.
Revised: PCS-S CORE is computed by pre-computing the full game’s SSCC, and computing the proportion of currently selected strategies in the empirical game that also belongs to the full game’s SSCC.

Original: Note that the PBR-S CORE and PCS-S CORE are useful measures for assessing the quality of convergence in our examples, in a manner analogous to N ASH C ONV .
Revised: Note that the PBR-S CORE and PCS-S CORE are useful measures for assessing the quality of convergence in our examples, in a manner analogous to N ASH C ONV .

Original: The computation of these scores is, however, not tractable in general games.
Revised: The computation of these scores is, however, not tractable in general games.

Original: Notably, this is also the case for N ASH C ONV (as it requires computation of player-wise best responses, which can be problematic even in moderatelysized games).
Revised: Notably, this is also the case for N ASH C ONV (as it requires computation of player-wise best responses, which can be problematic even in moderatelysized games).

Original: Despite this, these scores remain a useful way to empirically verify the convergence characteristics in small games where they can be tractably computed.
Revised: Despite this, these scores remain a useful way to empirically verify the convergence characteristics in small games where they can be tractably computed.

Original: Algorithm 5 PBR Score(Strategy S, Payoff Tensor, Current Player Id, Joint Strategies, Joint Strategy
Revised: Algorithm 5 PBR Score(Strategy S, Payoff Tensor, Current Player Id, Joint Strategies, Joint Strategy

Original: Probability) 1: New strategy score =2: for Joint strategy J, Joint probability P in Joint Strategies, Joint Strategy Probability do 3:

Original: New strategy =

Original: J 4: New strategy Current Player Id] = S 5: New strategy payoff =
Revised: Probability) 1: New strategy score =2: for Joint strategy J, probability P

Original: Payoff Tensor Strategy] 6: Old strategy payoff = Payoff Tensor
Revised: Probability do 3:

Original: J] 7:
Revised: = J 4:

Original: + P *
Revised: S 5:

Original: 8: ReturnNew strategy score
Revised: 8: Return

Original: Algorithm 6 PBR(Payoff Tensor list LM, Joint Strategies per player PJ, Alpharank Probability per
Revised: Algorithm 6 PBR(Payoff Tensor list LM, Joint Strategies per player PJ, Alpharank Probability per

Original: Joint Strategy PA, Current Player) 1: max PBR =2: max strat =
Revised: Joint Strategy PA, Current Player) 1: max PBR =2: max strat =

Original: None 3: for Strategy S available to Current Player among all possible strategies do 4: score =
Revised: None 3: for Strategy S available to Current Player among all possible strategies do 4: score =

Original: PBR Score(S, LM[Current Player Id], Current Player Id, PJ, PA) 5: ifNew Strategy Score > max PBR then 6: max PBR =
Revised: PBR Score(S, LM[Current Player Id], Current Player Id, PJ, PA) 5: if Score > max PBR then 6: max PBR =

Original: New Strategy Score 7: max strat = S 8: Return max PBR , max strat
Revised: New Strategy Score 7: max strat = S 8: Return max PBR , max strat

Original: C. A DDITIONAL O RACLE C OMPARISON R ESULTS
Revised: C. A DDITIONAL O RACLE C OMPARISON R ESULTS

Original: We present additional oracle comparisons in Fig.
Revised: We present additional oracle comparisons in Fig.

Original: C.9, all of these in the multi-population case.
Revised: C.9, all of these in the multi-population case.

Original: [Figure]
Revised: [Figure]

Revised: C. N OTES ON R ECTIFIED N ASH PERFORMANCE

Revised: This section provides additional insights into the Rectiﬁed Nash results detailed in Section 5.

Revised: We begin with an important disclaimer that Rectiﬁed Nash was developed solely with symmetric games in mind.

Revised: As Kuhn Poker and Leduc Poker are not symmetric games, they lie beyond the theoretical scope of Rectiﬁed Nash.

Revised: Nevertheless, comparing the performance of rectiﬁed and non-rectiﬁed approaches from an empirical perspective yields insights, which may be useful for future investigations that seek to potentially extend and apply rectiﬁed training approaches to more general games.

Revised: As noted in the main paper, the poor performance of PSRO using Rectiﬁed Nash (in Fig. 3) is initially surprising as it indicates premature convergence to a high-N ASH C ONV distribution over the players’ policy pools.

Revised: Investigating this further led to a counterintuitive result for the domains evaluated:

Revised: Rectiﬁed Nash was, roughly speaking, not increasing the overall diversity of behavioral policies added to each player’s population pool.

Revised: In certain regards, it even prevented diversity from emerging.

Revised: To more concretely pinpoint the issues, we detail below the ﬁrst 3 iterations of PSRO(Rectiﬁed Nash,

Revised: BR) in Kuhn Poker.

Revised: Payoff matrices at each PSRO iteration are included in Tables 6a to 6c.

Revised: For clarity, we also include the 5 best responses trained by Rectiﬁed Nash and the policies they were trained against, in their order of discovery: 2 policies for Player 1 (in Fig.

Revised: C.11) and 3 policies for

Revised: Player 2 (in Fig.

Revised: C.12).

Revised: Iteration 0: both players start with uniform random policies.

Revised: • Player 1 trains a best response against Player 2’s uniform random policy; its policy set is now the original uniform policy, and the newly-computed best response.

Revised: • Player 2 trains a best response against Player 1’s uniform random policy; its policy set is now the original uniform policy, and the newly-computed best response.

Revised: • Player 2’s best response beats both of Player 1’s policies.

Revised: • Payoff and Nash values are represented in Table 6a.

Revised: [Figure]

Revised: [Figure]

Revised: • By Rectiﬁed Nash rules, Player 1 only trains policies against policies it beats; i

Revised: e., only against Player 2’s random policy, and thus it adds the same policy as in iteration 1 to its pool.

Revised: • Player 2 trains a best response against the Nash mixture of Player 1’s

Revised: ﬁrst best response and random policy.

Revised: This policy also beats all policies of player 1.

Revised: • Payoff and Nash values are represented in Table 6b.

Revised: Iteration 3: • Player 1 only trains best responses against Player 2’s random policy.

Revised: • Player 2 only trains best responses against the Nash of Player 1’s two unique policies.

Revised: This yields the same policies for player 2 as those previously added to its pool (i.e., a loop occurs).

Revised: • Payoff and Nash values are represented in Table 6c 5.

Revised: Rectiﬁed Nash has looped.

Revised: As noted above, Rectiﬁed Nash loops at iteration 3, producing already-existing best responses against

Revised: Player 1’s policies.

Revised: Player 1 is, therefore, constrained to never being able to train best responses against any other policy than Player 2’s random policy.

Revised: In turn, this prevents Player 2 from training additional novel policies, and makes the game deadlocked.

Revised: Noise in the payoff matrices may lead to different best responses against the Nash Mixture of policies, effectively increasing diversity.

Revised: However, this effect did not seem to manifest in our experiments.

Revised: To more clearly illustrate this, we introduce a means of evaluating the policy pool diversity, counting the number of unique policies in the pool.

Revised: Speciﬁcally, given that Kuhn poker is a ﬁnite state game, comparing policies is straightforward, and only amounts to comparing each policy’s output on all states of the games.

Revised: If two policies have exactly the same output on all the game’s states, they are equal; otherwise, they are distinct.

Revised: We plot in Fig.

Revised: C.10 the policy diversity of each meta-solver, where we observe that both Rectiﬁed Nash and Rectiﬁed PRD discover a total of 5 different policies.

Revised: We have nevertheless noticed that in a few rare seeds, when using low number of simulations per payoff entry (Around 10), Rectiﬁed Nash was able to converge to low exploitability scores, suggesting a relationship between payoff noise, uncertainty and covergence of Rectiﬁed Nash whose investigation we leave for future work.

Revised: We also leave the investigation of the relationship between Policy Diversity and Exploitability for future work, though note that there appears to be a clear correlation between both.

Revised: Overall, these results demonstrate that the Rectiﬁed Nash solver fails to discover as many unique policies as the other solvers, thereby plateauing at a low N ASH C ONV .

Revised: Finally, regarding Rectiﬁed PRD, which performs better in terms of N ASH C ONV when compared to

Revised: Rectiﬁed Nash, we suspect that payoff noise in combination with the intrinsic noise of PRD, plays a key factor - but those two are not enough to deterministically make Rectiﬁed PRD converge to 0 exploitability, since in the seed that generated Fig.

Revised: C.10, it actually doesn’t (Though it indeed converges in Fig. 3).

Revised: We conjecture this noisier behavior may enable Rectiﬁed PRD to free itself from deadlocks more easily, and thus discover more policies on average.

Revised: A more detailed analysis of

Revised: Rectiﬁed PRD is left as future work.

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Table]

Original: D α -R ANK IN D ETAIL
Revised: D α -R ANK IN D ETAIL

Original: In this section we give further details of α -Rank; for a full description, see Omidshaﬁei et al.
Revised: In this section we give further details of α -Rank; for a full description, see Omidshaﬁei et al.

Original: Essentially α -Rank deﬁnes a directed response graph over the pure strategy proﬁles of the game under study, by indicating when a player has an incentive to make a unilateral deviation from their current strategy.
Revised: Essentially α -Rank deﬁnes a directed response graph over the pure strategy proﬁles of the game under study, by indicating when a player has an incentive to make a unilateral deviation from their current strategy.

Original: An irreducible (noisy) random walk over this graph is then deﬁned, and the strategy proﬁle rankings are obtained by ordering the masses of this Markov chain’s unique invariant distribution π .
Revised: An irreducible (noisy) random walk over this graph is then deﬁned, and the strategy proﬁle rankings are obtained by ordering the masses of this Markov chain’s unique invariant distribution π .

Original: The Markov transition matrix C that speciﬁes this random walk is deﬁned as follows for the multipopulation case; see Omidshaﬁei et al.
Revised: The Markov transition matrix C that speciﬁes this random walk is deﬁned as follows for the multipopulation case; see Omidshaﬁei et al.

Original: (2019) for the single-population case.
Revised: (2019) for the single-population case.

Original: Consider a pure strategy proﬁle s ∈ S , and let σ = ( σ k , s − k ) be the pure strategy proﬁle which is equal to s , except for player k , which uses strategy σ k ∈ S k instead of s k .
Revised: Consider a pure strategy proﬁle s ∈ S , and let σ = ( σ k , s − k ) be the pure strategy proﬁle which is equal to s , except for player k , which uses strategy σ k ∈ S k instead of s k .

Original: Let C s,σ denote the transition probability from s to σ , and C s,s the self-transition probability of s , with each deﬁned as: 
Revised: Let C s,σ denote the transition probability from s to σ , and C s,s the self-transition probability of s , with each deﬁned as: [Equation] if M k ( σ [Equation]

Revised: (cid:54) =

Revised: M k ( s )

Original: [Equation]
Revised: [Equation]

Revised: otherwise ,

Revised: [Equation]

Original: where η = ( (cid:80) l ( | S l | − 1)) − 1 .
Revised: where η = ( (cid:80) l ( | S l | − 1)) − 1 .

Original: If two strategy proﬁles s and s (cid:48) differ in more than one player’s strategy, then C s,s (cid:48)
Revised: If two strategy proﬁles s and s (cid:48) differ in more than one player’s strategy, then C s,s (cid:48)

Original: = 0 .
Revised: = 0 .

Original: Here α ≥ 0 and m ∈ N are parameters to be speciﬁed; the form of this transition probability is described by evolutionary dynamics models from evolutionary game theory and is explained in more detail in Omidshaﬁei et al.
Revised: Here α ≥ 0 and m ∈ N are parameters to be speciﬁed; the form of this transition probability is described by evolutionary dynamics models from evolutionary game theory and is explained in more detail in Omidshaﬁei et al.

Original: Large values of α correspond to higher selection pressure in the evolutionary model under consideration; the version of α -Rank used throughout this paper corresponds to the limiting invariant distribution as α → ∞ , under which only strategy proﬁles appearing in the sink strongly-connected components of the response graph can have positive mass.
Revised: Large values of α correspond to higher selection pressure in the evolutionary model under consideration; the version of α -Rank used throughout this paper corresponds to the limiting invariant distribution as α → ∞ , under which only strategy proﬁles appearing in the sink strongly-connected components of the response graph can have positive mass.

Original: E T OWARDS T HEORETICAL G UARANTEES FOR THE P ROJECTED R EPLICATOR D YNAMICS
Revised: E T OWARDS T HEORETICAL G UARANTEES FOR THE P ROJECTED R EPLICATOR D YNAMICS

Original: Computing Nash equilibria is intractable for general games and can suffer from a selection problem (Daskalakis et al., 2009); therefore, it quickly becomes computationally intractable to employ an exact Nash meta-solver in the inner loop of a PSRO algorithm.
Revised: Computing Nash equilibria is intractable for general games and can suffer from a selection problem (Daskalakis et al., 2009); therefore, it quickly becomes computationally intractable to employ an exact Nash meta-solver in the inner loop of a PSRO algorithm.

Original: To get around this, Lanctot et al.
Revised: To get around this, Lanctot et al.

Original: (2017) use regret minimization algorithms to attain an approximate correlated equilibrium (which is guaranteed to be an approximate Nash equilibrium under certain conditions on the underlying game, such as two-player zero-sum).
Revised: (2017) use regret minimization algorithms to attain an approximate correlated equilibrium (which is guaranteed to be an approximate Nash equilibrium under certain conditions on the underlying game, such as two-player zero-sum).

Original: A dynamical system from evolutionary game theory that also converges to equilibria under certain conditions is the replicator dynamics (Taylor and Jonker, 1978;
Revised: A dynamical system from evolutionary game theory that also converges to equilibria under certain conditions is the replicator dynamics (Taylor and Jonker, 1978;

Original: Schuster and Sigmund, 1983; Cressman and Tao, 2014; Bloembergen et al., 2015), which deﬁnes a dynamical system over distributions of strategies ( π ks ( t ) | k ∈
Revised: Schuster and Sigmund, 1983; Cressman and Tao, 2014; Bloembergen et al., 2015), which deﬁnes a dynamical system over distributions of strategies ( π ks ( t ) | k ∈

Original: [ K ] , s ∈ S k ) , given by
Revised: [ K ] , s ∈ S k ) , given by

Original: [Equation]
Revised: [Equation]

Original: for all k ∈ [ K ] , s ∈ S k , (4) with an arbitrary initial condition.
Revised: for all k ∈ [ K ] , s ∈ S k , (4) with an arbitrary initial condition.

Original: Lanctot et al.
Revised: Lanctot et al.

Original: (2017) introduced a variant of replicator dynamics, termed projected replicator dynamics (PRD), which projects the ﬂow of the system so that each distribution π k ( t ) lies in the set ∆ γS k = { π ∈ ∆ S k | π s ≥ γ/ ( | S k | + 1) , ∀ s ∈ S k } ; see, e.g.,
Revised: (2017) introduced a variant of replicator dynamics, termed projected replicator dynamics (PRD), which projects the ﬂow of the system so that each distribution π k ( t ) lies in the set ∆ γS k = { π ∈ ∆ S k | π s ≥ γ/ ( | S k | + 1) , ∀ s ∈ S k } ; see, e.g.,

Original: Nagurney and Zhang (2012) for properties of such projected dynamical systems.
Revised: Nagurney and Zhang (2012) for properties of such projected dynamical systems.

Original: This heuristically enforces additional “exploration” relative to standard replicator dynamics, and was observed to provide strong empirical results when used as a meta-solver within PSRO.
Revised: This heuristically enforces additional “exploration” relative to standard replicator dynamics, and was observed to provide strong empirical results when used as a meta-solver within PSRO.

Original: However, the introduction of projection potentially severs the connection between replicator dynamics and Nash equilibria, and the theoretical game-theoretic properties of PRD were left open in Lanctot et al.
Revised: However, the introduction of projection potentially severs the connection between replicator dynamics and Nash equilibria, and the theoretical game-theoretic properties of PRD were left open in Lanctot et al.

Original: Here, we take a ﬁrst step towards investigating theoretical guarantees for PRD.
Revised: Here, we take a ﬁrst step towards investigating theoretical guarantees for PRD.

Original: Speciﬁcally, we highlight a possible connection between α -Rank, the calculation of which requires no simulation, and a constrained variant of PRD, which we denote the ‘single-mutation PRD’ (or s-PRD), leaving formal investigation of this connection for future work.
Revised: Speciﬁcally, we highlight a possible connection between α -Rank, the calculation of which requires no simulation, and a constrained variant of PRD, which we denote the ‘single-mutation PRD’ (or s-PRD), leaving formal investigation of this connection for future work.

Original: Speciﬁcally, s-PRD is a dynamical system over distributions ( π ks ( t ) | k ∈
Revised: Speciﬁcally, s-PRD is a dynamical system over distributions ( π ks ( t ) | k ∈

Original: [ K ] , s ∈ S k ) that follows the replicator dynamics (equation 4), with initial condition restricted so that each π k 0 lies on the 1 -skeleton ∆ (1) S k
Revised: [ K ] , s ∈ S k ) that follows the replicator dynamics (equation 4), with initial condition restricted so that each π k 0 lies on the 1 -skeleton ∆ (1) S k

Original: = { π ∈ ∆ S k | (cid:80) s ∈ S
Revised: = { π ∈ ∆ S k | (cid:80) s ∈ S

Original: k
Revised: k

Original: π s (cid:54) =0 ≤ 2 } .
Revised: π s (cid:54) =0 ≤ 2 } .

Original: Further, whenever a strategy distribution π kt enters a δ -corner of the simplex, deﬁned by ∆
Revised: Further, whenever a strategy distribution π kt enters a δ -corner of the simplex, deﬁned by ∆

Original: [ δ ] S k
Revised: [ δ ] S k

Original: = { π ∈ ∆ (1) S k | ∃ s ∈ S k s.t.
Revised: = { π ∈ ∆ (1) S k | ∃ s ∈ S k s.t.

Original: π s ≥ 1 − δ } , the non-zero element of π k ( t ) with mass at most δ is replaced with a uniformly randomly chosen strategy after a random time distributed according to Exp ( µ ) , for some small µ > 0 .
Revised: π s ≥ 1 − δ } , the non-zero element of π k ( t ) with mass at most δ is replaced with a uniformly randomly chosen strategy after a random time distributed according to Exp ( µ ) , for some small µ > 0 .

Original: This concludes the description of s-PRD.
Revised: This concludes the description of s-PRD.

Original: We note at this stage that s-PRD deﬁnes, essentially, a dynamical system on the 1-skeleton (or edges) of the simplex, with random mutations towards a uniformly-sampled randomly strategy proﬁle s at the simplex vertices.
Revised: We note at this stage that s-PRD deﬁnes, essentially, a dynamical system on the 1-skeleton (or edges) of the simplex, with random mutations towards a uniformly-sampled randomly strategy proﬁle s at the simplex vertices.

Original: At a high-level, this bears a close resemblance to the ﬁnite-population α -Rank dynamics deﬁned in Omidshaﬁei et al.
Revised: At a high-level, this bears a close resemblance to the ﬁnite-population α -Rank dynamics deﬁned in Omidshaﬁei et al.

Original: ; moreover, we note that the connection between s-PRD and true α -Rank dynamics becomes even more evident when taking into account the correspondence between the standard replicator dynamics and α -Rank that is noted in Omidshaﬁei et al.
Revised: ; moreover, we note that the connection between s-PRD and true α -Rank dynamics becomes even more evident when taking into account the correspondence between the standard replicator dynamics and α -Rank that is noted in Omidshaﬁei et al.

Original: (2019, Theorem 2.1.4).
Revised: (2019, Theorem 2.1.4).

Original: We conclude by noting a major limitation of both s-PRD and PRD, which can limit their practical applicability even assuming a game-theoretic grounding can be proven for either.
Revised: We conclude by noting a major limitation of both s-PRD and PRD, which can limit their practical applicability even assuming a game-theoretic grounding can be proven for either.

Original: Speciﬁcally, with all such solvers, simulation of a dynamical system is required to obtain an approximate equilibrium, which may be costly in itself.
Revised: Speciﬁcally, with all such solvers, simulation of a dynamical system is required to obtain an approximate equilibrium, which may be costly in itself.

Original: Moreover, their dynamics can be chaotic even for simple instances of two-player two-strategy games (Palaiopanos et al., 2017).
Revised: Moreover, their dynamics can be chaotic even for simple instances of two-player two-strategy games (Palaiopanos et al., 2017).

Original: In practice, the combination of these two limitations may completely shatter the convergence properties of these algorithms in practice, in the sense that the question of how long to wait until convergence becomes increasingly difﬁcult (and computationally expensive) to answer.
Revised: In practice, the combination of these two limitations may completely shatter the convergence properties of these algorithms in practice, in the sense that the question of how long to wait until convergence becomes increasingly difﬁcult (and computationally expensive) to answer.

Original: By contrast, α -Rank does not rely on such simulations, thereby avoiding these empirical issues.
Revised: By contrast, α -Rank does not rely on such simulations, thereby avoiding these empirical issues.

Original: We conclude by remarking again that, albeit informal, these results indicate a much stronger theoretical connection between α -Rank and standard PRD that may warrant future investigation.
Revised: We conclude by remarking again that, albeit informal, these results indicate a much stronger theoretical connection between α -Rank and standard PRD that may warrant future investigation.

Original: F M U J O C O S OCCER E XPERIMENT
Revised: F M U J O C O S OCCER E XPERIMENT

Original: While the key objective of this paper is to take a ﬁrst step in establishing a theoretically-grounded framework for PSRO-based training of agents in many-player settings, an exciting question concerns the behaviors of the proposed α -Rank-based PSRO algorithm in complex domains where functionapproximation-based policies need to be relied upon for generalizable task execution.
Revised: While the key objective of this paper is to take a ﬁrst step in establishing a theoretically-grounded framework for PSRO-based training of agents in many-player settings, an exciting question concerns the behaviors of the proposed α -Rank-based PSRO algorithm in complex domains where functionapproximation-based policies need to be relied upon for generalizable task execution.

Original: In this section, we take a preliminary step towards conducting this investigation, focusing in particular on the MuJoCo soccer domain introduced in Liu et al.
Revised: In this section, we take a preliminary step towards conducting this investigation, focusing in particular on the MuJoCo soccer domain introduced in Liu et al.

Original: (2019) (refer to https://github.com/deepmind/ dm_control/tree/master/dm_control/locomotion/soccer for the corresponding domain code).
Revised: (2019) (refer to https://github.com/deepmind/ dm_control/tree/master/dm_control/locomotion/soccer for the corresponding domain code).

Original: In particular, we conduct initial experiments comparing the performance of adaptations to Deep
Revised: In particular, we conduct two sets of initial experiments.

Original: Reinforcement Learning settings of PSRO( α -Rank, RL) and PSRO(Uniform, RL), where we use an RL-agent as an oracle that approximates the best response. The modiﬁcations added to standard PSRO to help with the inherently more difﬁcult nature of Deep Reinforcement Learning training are the following:
Revised: The ﬁrst set of experiments compares the performance of PSRO( α -Rank, RL) and PSRO(Uniform, RL) in games of 3 vs. 3 MuJoCo soccer, and the second set compares PSRO( α -Rank, RL) against a population-based training pipeline in 2 vs. 2 games.

Original: • We train several agents per player (32 in our case) at the same time as in Liu et al.

Original: • We only add the top 10% of our 32 agents to the pool.

Original: Performance is measured by the α -Rank-average for PSRO( α -Rank, RL) and Nash-average for PSRO(Uniform, RL) of agents in the (Agents, Pool) vs (Agents, Pool) game. • Each oracle step in PSRO is composed of 1 billion learning steps of the agents. After each step, the top 10% of agents (the 3 best agents) are added to the pool, and training of theagents continues; training does not start from random again. • We use a constant 50% probability of training using self-play (The other 50% being training against the distribution of the pool of agents).
Revised: F. T RAINING P ROCEDURE

Original: For each of thetwo PSRO variants considered, we compose a hierarchical training procedure composed of two levels.
Revised: For each of the PSRO variants considered, we compose a hierarchical training procedure composed of two levels.

Original: At the low-level, which focuses on simulations of the underlying MuJoCo soccer game itself, we consider a collection of 32 reinforcement learners (which we call agents).
Revised: At the low-level, which focuses on simulations of the underlying MuJoCo soccer game itself, we consider a collection of 32 reinforcement learners (which we call agents) that are all trained at the same time, as in Liu et al.

Original: We compose teams corresponding to 3 clones ofeach agent (i.e., 32 homogeneous teams in total) and evaluatethe all pairwise team match-ups.
Revised: We compose teams corresponding to multiple clones of agent per team (yielding homogeneous teams, in contrast to (Liu et al., 2019), which evaluates teams of heterogeneous agents) and evaluate all pairwise team match-ups.

Original: Note that this yields a 2-“player” meta-game (where each “player” is actually a team, i.e., a team-vs.-team setting), with payoffs corresponding to the average win-rates of each team when pitted against another.
Revised: Note that this yields a 2-“player” meta-game (where each “player” is actually a team, i.e., a team-vs.-team setting), with payoffs corresponding to the average win-rates of each team when pitted against each other.

Original: The payoff matrix is estimated by simulating matches between different teams.
Revised: The payoff matrix is estimated by simulating matches between different teams.

Original: The number of simulations per entry was adaptive based on the empirical uncertainty observed on the pair-wise match outcomes.
Revised: The number of simulations per entry is adaptive based on the empirical uncertainty observed on the pair-wise match outcomes.

Original: In practice, we observed an average of 10 to 100 simulations per entry, with fewer simulations used for meta-payoffs with higher certainty.
Revised: In practice, we observed an average of 10 to 100 simulations per entry, with fewer simulations used for meta-payoffs with higher certainty.

Original: For the ﬁnal evaluation matrix reported in Appendix F (Fig.
Revised: For the ﬁnal evaluation matrix reported in Appendix F (Fig.

Original: F.10), which was computed after the conclusion of PSRO-based training,simulations were used per entry.
Revised: F.10), which was computed after the conclusion of PSRO-based training,simulations were used per entry.

Original: With a few changes motivated by scalability of the method, we use the standard PSRO procedures as outlined in this paper; i.e., we compute the meta-distribution, and expand the meta-population by 3 policies (corresponding to the 10% best RL agents). 
Revised: Additionally, instead of adding one policy per PSRO iteration per player we add three (which corresponds to the 10% best RL agents). Several additional modiﬁcations were made to standard PSRO to help with the inherently more difﬁcult nature of Deep Reinforcement Learning training:

Original: We defer further ablation studies of these changes to future work.
Revised: • Agent performance, used to choose which agents out of the 32 to add to the pool, is measured by the α -Rank-average for PSRO( α -Rank, RL) and Nash-average for PSRO(Uniform, RL) of agents in the (Agents, Pool) vs (Agents, Pool) game. • Each oracle step in PSRO is composed of 1 billion learning steps of the agents. After each step, the top 10% of agents (the 3 best agents) are added to the pool, and training of theagents continues; • We use a 50% probability of training using self-play (the other 50% training against the distribution of the pool of agents). F. R ESULTS

Original: Note that we train the PSRO( α -Rank, RL) and PSRO(Uniform, RL) agents independently (i.e., the two populations never interact with one another).
Revised: In the ﬁrst set of experiments, we train the PSRO( α -Rank, RL) and PSRO(Uniform, RL) agents independently (i.e., the two populations never interact with one another).

Original: Following training,however, we compare the effective performance of these two PSRO variants by pitting their 8 best trained agents against one another, and recording the average win rates.
Revised: Following training, we compare the effective performance of these two PSRO variants by pitting their 8 best trained agents against one another, and recording the average win rates.

Original: These results are reported in Fig.
Revised: These results are reported in Fig.

Original: F.10.
Revised: F.13 for games involving teams of 3 vs. 3.

Original: The key takeaway resulting from these experiments is as follows: upon completion of training, when computing a new play distribution based on a pool of agents trained via the α -Rank-based training approach vs. a uniform approach, the former attains essentially all of the ‘play probability’.
Revised: It is evident from these results that PSRO( α -Rank, RL) signiﬁcantly outperforms PSRO(Uniform, RL).

Original: This is evident in the colorbar on the far right of Fig.
Revised: This is clear from the colorbar on the far right of Fig.

Original: F.10, which visualizes the post-training meta-distribution over  both training pipelines.
Revised: F.13, which visualizes the post-training alpharank distribution over the payoff matrix of the metagame composed of both training pipelines.

Revised: In the second set of experiments, we compare α -PSRO-based training to self-play-based training (i.e., sampling opponents uniformly at random from the training agent population).

Revised: This provides a means of gauging the performance improvement solely due to PSRO; these results are reported in Fig.

Original:  We conclude by remarking that these results, although interesting, primarily are intended to lay the foundation for use of α -Rank as a meta-solver in complex many agent domains where RL agents serve as useful oracles; additionally, more extensive research and analysis is necessary to make these results conclusive in domains such as MuJoCo soccer.
Revised: F. for games involving teams of 2 vs. 2. We conclude by remarking that these results, although interesting, primarily are intended to lay the foundation for use of α -Rank as a meta-solver in complex many agent domains where RL agents serve as useful oracles; additionally, more extensive research and analysis is necessary to make these results conclusive in domains such as MuJoCo soccer.

Original: We plan to carry out several experiments along these lines in future work. 
Revised: We plan to carry out several experiments along these lines, including extensive ablation studies, in future work. [Figure]