Original: I TERATIVE E MPIRICAL G AME S OLVING VIA S INGLE P OLICY B EST R ESPONSE
Revised: I TERATIVE E MPIRICAL G AME S OLVING VIA S INGLE P OLICY B EST R ESPONSE

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Policy-Space Response Oracles (PSRO) is a general algorithmic framework for learning policies in multiagent systems by interleaving empirical game analysis with deep reinforcement learning (Deep RL).
Revised: Policy-Space Response Oracles (PSRO) is a general algorithmic framework for learning policies in multiagent systems by interleaving empirical game analysis with deep reinforcement learning (Deep RL).

Original: At each iteration, Deep RL is invoked to train a best response to a mixture of opponent policies.
Revised: At each iteration, Deep RL is invoked to train a best response to a mixture of opponent policies.

Original: The repeated application of Deep
Revised: The repeated application of Deep

Original: RL poses an expensive computational burden as we look to apply this algorithm to more complex domains.
Revised: RL poses an expensive computational burden as we look to apply this algorithm to more complex domains.

Original: We introduce two variations of PSRO designed to reduce the amount of simulation required during Deep RL training.
Revised: We introduce two variations of PSRO designed to reduce the amount of simulation required during Deep RL training.

Original: Both algorithms modify how
Revised: Both algorithms modify how

Original: PSRO adds new policies to the empirical game, based on learned responses to a single opponent policy.
Revised: PSRO adds new policies to the empirical game, based on learned responses to a single opponent policy.

Original: The first, Mixed-Oracles, transfers knowledge from previous iterations of Deep RL, requiring training only against the opponent’s newest policy.
Revised: The first, Mixed-Oracles, transfers knowledge from previous iterations of Deep RL, requiring training only against the opponent’s newest policy.

Original: The second,
Revised: The second,

Original: Mixed-Opponents, constructs a pure-strategy opponent by mixing existing strategy’s action-value estimates, instead of their policies.
Revised: Mixed-Opponents, constructs a pure-strategy opponent by mixing existing strategy’s action-value estimates, instead of their policies.

Original: Learning against a single policy mitigates variance in state outcomes that is induced by an unobserved distribution of opponents.
Revised: Learning against a single policy mitigates variance in state outcomes that is induced by an unobserved distribution of opponents.

Original: We empirically demonstrate that these algorithms substantially reduce the amount of simulation during training required by PSRO, while producing equivalent or better solutions to the game.
Revised: We empirically demonstrate that these algorithms substantially reduce the amount of simulation during training required by PSRO, while producing equivalent or better solutions to the game.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In (single-agent) reinforcement learning (RL), an agent repeatedly interacts with an environment until it achieves mastery, that is, can find no further way to improve performance (measured by reward).
Revised: In (single-agent) reinforcement learning (RL), an agent repeatedly interacts with an environment until it achieves mastery, that is, can find no further way to improve performance (measured by reward).

Original: In a multiagent system, the outcome of any learned policy may depend pivotally on the behavior of other agents.
Revised: In a multiagent system, the outcome of any learned policy may depend pivotally on the behavior of other agents.

Original: Direct search over a complex joint policy space for game solutions is daunting, and so more common in approaches combining RL and game theory is to interleave learning and game analysis iteratively.
Revised: Direct search over a complex joint policy space for game solutions is daunting, and so more common in approaches combining RL and game theory is to interleave learning and game analysis iteratively.

Original: In empirical game-theoretic analysis (EGTA), game reasoning is applied to approximate game models defined over restricted strategy sets.
Revised: In empirical game-theoretic analysis (EGTA), game reasoning is applied to approximate game models defined over restricted strategy sets.

Original: The strategies are selected instances drawn from a large space of possible strategies, for example, defined in terms of a fundamental policy space : mappings from observations to actions.
Revised: The strategies are selected instances drawn from a large space of possible strategies, for example, defined in terms of a fundamental policy space : mappings from observations to actions.

Original: For a given set of strategies (policies), 1 we estimate an empirical game via simulation over combinations of these strategies.
Revised: For a given set of strategies (policies), 1 we estimate an empirical game via simulation over combinations of these strategies.

Original: In iterative approaches to EGTA, analysis of the empirical game informs decisions about the sampling of strategy profiles or additions of strategies to the restricted sets.
Revised: In iterative approaches to EGTA, analysis of the empirical game informs decisions about the sampling of strategy profiles or additions of strategies to the restricted sets.

Original: Schvartzman & Wellman (2009b) first combined RL with EGTA, adding new strategies by learning a policy for one agent (using tabular Q-learning), fixing other agent policies at a Nash equilibrium of the current empirical game.
Revised: Schvartzman & Wellman (2009b) first combined RL with EGTA, adding new strategies by learning a policy for one agent (using tabular Q-learning), fixing other agent policies at a Nash equilibrium of the current empirical game.

Original: Lanctot et al.
Revised: Lanctot et al.

Original: (2017) introduced a general framework, Policy-Space Response Oracles (PSRO), interleaving empirical game analysis with deep RL.
Revised: (2017) introduced a general framework, Policy-Space Response Oracles (PSRO), interleaving empirical game analysis with deep RL.

Original: PSRO generalizes the identification of learning targets, employing an abstract method termed meta-strategy solver (MSS) that extracts a strategy profile from an empirical game.
Revised: PSRO generalizes the identification of learning targets, employing an abstract method termed meta-strategy solver (MSS) that extracts a strategy profile from an empirical game.

Original: At each epoch of PSRO, a player uses RL to derive a new policy by training against the opponentstrategies in the MSS-derived profile.
Revised: At each epoch of PSRO, a player uses RL to derive a new policy by training against the opponentstrategies in the MSS-derived profile.

Original: A particular challenge for the RL step in PSRO is that the learner must derive a response under uncertainty about opponent policies.
Revised: A particular challenge for the RL step in PSRO is that the learner must derive a response under uncertainty about opponent policies.

Original: The profile returned by the MSS is generally a mixed-strategy profile, as in strategically complex environments randomization is often a necessary ingredient for equilibrium.
Revised: The profile returned by the MSS is generally a mixed-strategy profile, as in strategically complex environments randomization is often a necessary ingredient for equilibrium.

Original: The opponent draws from this mixture are unobserved, adding uncertainty to the multiagent environment.
Revised: The opponent draws from this mixture are unobserved, adding uncertainty to the multiagent environment.

Original: We address this challenge through variants of PSRO in which all RL is applied to environments where opponents play pure strategies.
Revised: We address this challenge through variants of PSRO in which all RL is applied to environments where opponents play pure strategies.

Original: We propose and evaluate two such methods, which work in qualitatively different ways: Mixed Oracles learns separate best-responses (BR) to each pure strategy in a mixture and combines the results from learning to approximate a BR to the mixture.
Revised: We propose and evaluate two such methods, which work in qualitatively different ways: Mixed Oracles learns separate best-responses (BR) to each pure strategy in a mixture and combines the results from learning to approximate a BR to the mixture.

Original: Mixed Opponents constructs a single pure opponent policy that represents an aggregate of the mixed strategy and learns a BR to this policy.
Revised: Mixed Opponents constructs a single pure opponent policy that represents an aggregate of the mixed strategy and learns a BR to this policy.

Original: Both of our methods employ the machinery of Q-Mixing (Smith et al., 2020), which constructs policies based on an aggregation of Q-functions corresponding to component policies of a mixture.
Revised: Both of our methods employ the machinery of Q-Mixing (Smith et al., 2020), which constructs policies based on an aggregation of Q-functions corresponding to component policies of a mixture.

Original: Our methods promise advantages beyond those of learning in a less stochastic environment.
Revised: Our methods promise advantages beyond those of learning in a less stochastic environment.

Original: Mixed Oracles transfers learning across epochs, exploiting the Q-functions learned against a particular opponent policy in constructing policies for any other epoch where that opponent policy is encountered.
Revised: Mixed Oracles transfers learning across epochs, exploiting the Q-functions learned against a particular opponent policy in constructing policies for any other epoch where that opponent policy is encountered.

Original: Mixed Opponents applies directly over the joint opponent space, and so has the potential to scale beyond two-player games.
Revised: Mixed Opponents applies directly over the joint opponent space, and so has the potential to scale beyond two-player games.

Original: We evaluate our methods in a series of experimental games, and find that both Mixed-Oracles and
Revised: We evaluate our methods in a series of experimental games, and find that both Mixed-Oracles and

Original: Mixed-Opponents are able to find solutions at least as good as an unmodified PSRO, and often better, while providing a substantial reduction in the required amount of training simulation.
Revised: Mixed-Opponents are able to find solutions at least as good as an unmodified PSRO, and often better, while providing a substantial reduction in the required amount of training simulation.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: An RL agent at time t ∈ T receives the state of the environment s t ∈ S , or a partial state called an observation o t ∈O .
Revised: An RL agent at time t ∈ T receives the state of the environment s t ∈ S , or a partial state called an observation o t ∈O .

Original: The agent then chooses an action a t according to its policy , π : O→ ∆( A ) , effecting the environment and producing a reward signal r t ∈ R .
Revised: The agent then chooses an action a t according to its policy , π : O→ ∆( A ) , effecting the environment and producing a reward signal r t ∈ R .

Revised: An experience is a ( s t ,a t ,r t +1 ,s t +1 ) -tuple, and a sequence of experiences ending in a terminal state is an episode τ .

Original: How the environment changes as a result of the action is dictated by the environment’s transition dynamics p : S ×A→S .
Revised: How the environment changes as a result of the action is dictated by the environment’s transition dynamics p : S ×A → S .

Original: The agent is said to act optimally when it maximizes return G t = (cid:80) ∞
Revised: The agent is said to act optimally when it maximizes return G t = (cid:80) ∞

Original: l γ l r t + l ( γ is a discount factor).
Revised: l γ l r t + l ( γ is a discount factor).

Original: In value-based RL, agents use return as an estimate of the quality of a state V ( o t ) =
Revised: In value-based RL, agents use return as an estimate of the quality of a state V ( o t )=

Original: E π (cid:2)(cid:80) ∞ l =

Revised: E π (cid:2)(cid:80) ∞ l =

Original: γ l r ( o t + l ,a t + l ) (cid:3) , and/or taking an action in a state Q ( o t ,a t )=
Revised: γ l r ( o t + l ,a t + l ) (cid:3) , and/or taking an action in a state Q ( o t ,a t )=

Original: r ( o t ,a t )+ γ E o t +1 ∈O (cid:2) V ( o t +1 ) (cid:3) .
Revised: r ( o t ,a t )+ γ E o t +1 ∈O (cid:2) V ( o t +1 ) (cid:3) .

Original: When there is more than one agent, we denote agent-specific components with subscripts (e.g., π i ).

Original: Negated subscripts represent the joint elements of all other agents (e.g., π − i ).

Original: A normal form games (NFG) Γ =
Revised: A normal form games (NFG) Γ =

Original: (Π ,U,n ) describes a one-shot strategic interaction among n players.
Revised: (Π ,U,n ) describes a one-shot strategic interaction among n players.

Revised: When there is more than one agent, we denote agent-specific components with subscripts (e.g., π i ).

Revised: Negated subscripts represent the joint elements of all other agents (e.g., π − i ).

Original: Each of the players has a set of policies Π i = (cid:8) π 0 i ,...,π ki (cid:9) from which it may choose.
Revised: Each of the players has a set of policies Π i = (cid:8) π 0 i ,...,π ki (cid:9) from which it may choose.

Original: Player i may select a pure strategy π i ∈
Revised: Player i may select a pure strategy π i ∈

Original: Π i , or may randomize play by sampling from a mixed strategy σ i ∈ ∆(Π i ) .
Revised: Π i , or may randomize play by sampling from a mixed strategy σ i ∈ ∆(Π i ) .

Original: At the end of the interaction, each player receives a payoff U :Π → R n .
Revised: At the end of the interaction, each player receives a payoff U :Π → R n .

Original: An empirical NFG (ENFG) ˜Γ=(˜Π , ˜ U,n ) is an NFG induced by simulation of strategy profiles.
Revised: An empirical NFG (ENFG) ˜Γ=(˜Π , ˜ U,n ) is an NFG induced by simulation of strategy profiles.

Original: An ENFG approximates an underlying game Γ , estimating the payoffs through simulation.
Revised: An ENFG approximates an underlying game Γ , estimating the payoffs through simulation.

Original: ENFGs are often employed when the strategy set is too large for exhaustive representation, for example when the strategies are instances from a complex space of policies.
Revised: ENFGs are often employed when the strategy set is too large for exhaustive representation, for example when the strategies are instances from a complex space of policies.

Original: The quality of a strategy profile in an ENFG can be evaluated using regret : how much players could gain by deviating from assigned policies.
Revised: The quality of a strategy profile in an ENFG can be evaluated using regret : how much players could gain by deviating from assigned policies.

Original: Regret is measured with respect to a set of available deviation policies.
Revised: Regret is measured with respect to a set of available deviation policies.

Original: The regret of solution σ to player i when they are able to deviate to policies in Π i ⊆
Revised: The regret of solution σ to player i when they are able to deviate to policies in Π i ⊆

Original: Π i is:
Revised: Π i is:

Original: Regret i ( σ, Π i )=max π i ∈ Π
Revised: Regret i ( σ, Π i )=max π i ∈ Π

Original: i U i ( π i ,σ − i ) − U
Revised: i U i ( π i ,σ − i ) − U

Original: i ( σ i ,σ − i ) .
Revised: i ( σ i ,σ − i ) .

Original: Example deviation sets Π employed in this paper are Π PSRO , the strategies accumulated through a run of PSRO, and Π EVAL , denoting a static set of held-out evaluation policies.
Revised: Example deviation sets Π employed in this paper are Π PSRO , the strategies accumulated through a run of PSRO, and Π EVAL , denoting a static set of held-out evaluation policies.

Original: The sum of regrets across all players SumRegret ( σ, Π) =
Revised: The sum of regrets across all players SumRegret ( σ, Π) =

Original: (cid:80) i ∈ n Regret
Revised: (cid:80) i ∈ n Regret

Original: i ( σ, Π i ) , sometimes called the Nash convergence, is a measure of how stable a solution is.
Revised: i ( σ, Π i ) , sometimes called the Nash convergence, is a measure of how stable a solution is.

Original:  M ETHOD
Revised:  M ETHOD

Original: At each epoch e of the PSRO algorithm a new policy is constructed for each player by best-responding to an opponent profile σ ∗ ,e − 1 − i from the currently constructed ENFG: π ei ∈ BR ( σ ∗ ,e − 1 − i ) .
Revised: At each epoch e of the PSRO algorithm a new policy is constructed for each player by best-responding to an opponent profile σ ∗ ,e − 1 − i from the currently constructed ENFG: π ei ∈ BR ( σ ∗ ,e − 1 − i ) .

Original: These policies are then added to each player’s strategy set, ˜Π ei ← ˜Π e − 1 i ∪{ π ei } , and the new profiles are simulated to expand the ENFG.
Revised: These policies are then added to each player’s strategy set, ˜Π ei ← ˜Π e − 1 i ∪{ π ei } , and the new profiles are simulated to expand the ENFG.

Original: Algorithm 1 presents the full PSRO algorithm as defined by Lanctot et al.
Revised: Algorithm 1 presents the full PSRO algorithm as defined by Lanctot et al.

Original: One of the key design choices in iterative empirical game-solving is choosing which policies to add to the
Revised: One of the key design choices in iterative empirical game-solving is choosing which policies to add to the

Original: ENFG.
Revised: ENFG.

Original: This was first studied by Schvartzman & Wellman (2009a), who termed it the strategy exploration problem .
Revised: This was first studied by Schvartzman & Wellman (2009a), who termed it the strategy exploration problem .

Original: We want to add policies that both bring the solution to the ENFG closer to the solution of the full game and that can be calculated efficiently.
Revised: We want to add policies that both bring the solution to the ENFG closer to the solution of the full game and that can be calculated efficiently.

Original: In PSRO, the strategy exploration problem is decomposed into two steps: solution and BR via RL.
Revised: In PSRO, the strategy exploration problem is decomposed into two steps: solution and BR via RL.

Original: In the solution step, PSRO derives a profile σ ∗ ,e from the current empirical game.
Revised: In the solution step, PSRO derives a profile σ ∗ ,e from the current empirical game.

Original: The method for choosing this profile is termed the meta-strategy-solver (MSS), formally a function from empirical games to solution profiles MSS : ˜ Γ e → σ ∗ ,e .
Revised: The method for choosing this profile is termed the meta-strategy-solver (MSS), formally a function from empirical games to solution profiles MSS :˜Γ e → σ ∗ ,e .

Original: For example, the MSS might compute a Nash equilibrium of ˜Γ e . 

Revised: For example, the MSS might compute a Nash equilibrium of ˜Γ e .

Original: In the BR step, PSRO generates a new policy via RL (the response oracle ), training against the target opponent profile computed by the MSS.
Revised: In the BR step, PSRO generates a new policy via RL (the response oracle ), training against the target opponent profile computed by the MSS.

Original: The choice of MSS and response oracle algorithm constitutes a strategy exploration approach and determines the convergence speed of PSRO.
Revised: The choice of MSS and response oracle algorithm constitutes a strategy exploration approach and determines the convergence speed of PSRO.

Original: We observe and address two main problems with the standard version of PSRO (Algorithm 1).
Revised: We observe and address two main problems with the standard version of PSRO (Algorithm 1).

Original: First, note that the only thing that changes from one epoch to the next is the other-agent strategy profile σ − i ; the transition dynamics remain the same, and the mixed profile σ − i itself likely contains other-agent policies encountered in previous epochs.
Revised: First, note that the only thing that changes from one epoch to the next is the other-agent strategy profile σ − i ; the transition dynamics remain the same, and the mixed profile σ − i itself likely contains other-agent policies encountered in previous epochs.

Original: This suggests a significant opportunity to transfer learning across epochs; however, the BR calculation step works by training anew.
Revised: This suggests a significant opportunity to transfer learning across epochs; however, the BR calculation step works by training anew.

Original: Furthermore, because it is responding to some of the same strategies, the new policy learnt may be similar to the strategies added in previous epochs, and so not the most useful addition to the ENFG.
Revised: Furthermore, because it is responding to some of the same strategies, the new policy learnt may be similar to the strategies added in previous epochs, and so not the most useful addition to the ENFG.

Original: Secondly, the opponent profiles are mixtures (i.e., distribution over opponent policies).
Revised: Secondly, the opponent profiles are mixtures (i.e., distribution over opponent policies).

Original: This makes the environment dynamics more stochastic from the perspective of the RL agent, making learning more difficult.
Revised: This makes the environment dynamics more stochastic from the perspective of the RL agent, making learning more difficult.

Original: In Section 3.1 we propose an algorithm that transfers knowledge between iterations, and only trains against the single new opponent.
Revised: In Section 3.1 we propose an algorithm that transfers knowledge between iterations, and only trains against the single new opponent.

Original: In Section 3.2 we present a second algorithm that avoids responding to similar opponent strategies on subsequent iterations, while also addressing the opponent variance issue and providing scalability to multiple other agents.
Revised: In Section 3.2 we present a second algorithm that avoids responding to similar opponent strategies on subsequent iterations, while also addressing the opponent variance issue and providing scalability to multiple other agents.

Original: 3. M IXED -O RACLES
Revised: 3. M IXED -O RACLES

Original: The first problem we address is that during BR calculation there is a missed opportunity for transferring previously learned information.
Revised: The first problem we address is that during BR calculation there is a missed opportunity for transferring previously learned information.

Original: In each epoch, each player learns a BR to a mixed profile of opponent policies.
Revised: In each epoch, each player learns a BR to a mixed profile of opponent policies.

Original: This mixture typically involves the newly added strategies (one per player) for this epoch, but may also include many strategies from previous epochs.
Revised: This mixture typically involves the newly added strategies (one per player) for this epoch, but may also include many strategies from previous epochs.

Original: Training in previous epochs already captured experience against those strategies, so including them in further training may be redundant.
Revised: Training in previous epochs already captured experience against those strategies, so including them in further training may be redundant.

Original: The Mixed-Oracles algorithm is a variant of PSRO for two-player games, with a modified BR oracle designed to transfer learning across epochs.
Revised: The Mixed-Oracles algorithm is a variant of PSRO for two-player games, with a modified BR oracle designed to transfer learning across epochs.

Original: This method works by learning and maintaining a collection of BRs to each opponent policy Λ [Equation] , where λ ei is the BR to π e − 1 − i .
Revised: This method works by learning and maintaining a collection of BRs to each opponent policy Λ ei = (cid:8) λ 1 i ,...,λ ei (cid:9) , where λ ei is the BR to π e − 1 − i .

Original: During each epoch of Mixed-Oracles, a BR is learned for the single new opponent policy, rather than for the mixed opponent-profile generated by the MSS.
Revised: During each epoch of Mixed-Oracles, a BR is learned for the single new opponent policy, rather than for the mixed opponent-profile generated by the MSS.

Original: A BR to the MSS-generated target mixture is then constructed from the collection of BR results for constituent policies in the mixture.
Revised: A BR to the MSS-generated target mixture is then constructed from the collection of BR results for constituent policies in the mixture.

Original: Constructing the new policy is done through a general Transfer-Oracle function that maps a set of policies and a distribution over the policies into a single policy.
Revised: Constructing the new policy is done through a general Transfer-Oracle function that maps a set of policies and a distribution over the policies into a single policy.

Original: The resulting policy should approximately aggregate the behavior of the policies.
Revised: The resulting policy should approximately aggregate the behavior of the policies.

Original: By reusing learned behaviors from previous epochs, Mixed-Oracles allows us to focus training exclusively on new opponent policies.
Revised: By reusing learned behaviors from previous epochs, Mixed-Oracles allows us to focus training exclusively on new opponent policies.

Original: The key design choice is how to combine knowledge from the BRs to individual policies into a BR to any distribution of said policies.
Revised: The key design choice is how to combine knowledge from the BRs to individual policies into a BR to any distribution of said policies.

Original: We provide a general description of Mixed-Oracles, where the Transfer-Oracle method is abstract, as Algorithm 2.
Revised: We provide a general description of Mixed-Oracles, where the Transfer-Oracle method is abstract, as Algorithm 2.

Original: Smith et al.
Revised: Smith et al.

Original: (2020) proposes Q-Mixing as an approach for constructing policies against any mixture of opponent strategies.
Revised: (2020) proposes Q-Mixing as an approach for constructing policies against any mixture of opponent strategies.

Original: The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here.
Revised: The method employs Q-values learned against each individual opponent strategy, and thus can support the transfer sought here.

Original: Specifically, Q-Mixing averages the Q-values learned against each opponent policy π − i , weighted by their likelihood in the opponent mixture σ − i :
Revised: Specifically, Q-Mixing averages the Q-values learned against each opponent policy π − i , weighted by their likelihood in the opponent mixture σ − i :

Original: [Equation]
Revised: [Equation]

Original: where ψ determines the relative likelihood of playing an opponent ψ i : O i → ∆(Π − i ) .
Revised: where ψ determines the relative likelihood of playing an opponent ψ i : O i → ∆(Π − i ) .

Original: In this study, we use
Revised: In this study, we use

Original: Q-Mixing as our Transfer-Oracle, where ψ is the prior over the opponent distribution as given by an MSS.
Revised: Q-Mixing as our Transfer-Oracle, where ψ is the prior over the opponent distribution as given by an MSS.

Original: 3.  M IXED -O PPONENTS
Revised:  M IXED -O PPONENTS

Original: We next examine PSRO’s objective defined by BR to the opponent profile generated by an MSS.
Revised: We next examine PSRO’s objective defined by BR to the opponent profile generated by an MSS.

Original: Consider the Rock-Paper-Scissors (R-P-S) game in figure 1.
Revised: Consider the Rock-Paper-Scissors (R-P-S) game in figure 1.

Original: Player 1’s initial strategy π 11 mostly plays S, which induces a best response of R. Player 1’s second strategy
Revised: Player 1’s initial strategy π 1 1 mostly plays S, which induces a best response of R. Player 1’s second strategy

Original: π 21 is an approximate best response to R, scoring80% against it.
Revised: π 21 is an approximate best response to R, scoring

Original: The best response to π 2 1 is P. This leads to an empirical game similar to matching pennies, with player 2 playing an even mixture of Rock and Paper; in particular, Player 2’s meta-strategy never plays

Original: (Lanctot et al., 2017)
Revised: (Lanctot et al., 2017)

Original: Input: Initial policy sets for all players Π
Revised: Input: Initial policy sets for all players Π

Original: Simulate utilities ˜
Revised: Simulate utilities ˜

Original: U Π 0 for each joint π 0 ∈ Π
Revised: U Π 0 for each joint π 0 ∈ Π

Original: Initialize solution σ ∗ , 0 i =
Revised: Initialize solution σ ∗ , 0 i =

Original: Uniform (Π∗ , 0 i ) while epoch e in { 1 , 2 ,... } do for player i ∈
Revised: Uniform (Π 0 i ) while epoch e in { 1 , 2 ,... } do for player i ∈

Original: [[ n ]] do for many episodes do π − i ∼ σ  − 
Revised: [[ n ]] do for many episodes do π − i ∼ σ ∗ ,e − 1 − i

Original: i Train π ei over τ ∼ ( π ei ,π − i ) Π ei =
Revised: Train π ei over τ ∼ ( π ei ,π − i ) Π ei =

Original: Π
Revised: Π

Original: e − 1 i ∪{ π ei }
Revised: e − 1 i ∪{ π ei }

Original: Simulate missing entries in ˜ U
Revised: Simulate missing entries in ˜ U

Original: Π e from Π e
Revised: Π e from Π e

Original: Compute a solution σ ∗ ,e from ˜Γ e
Revised: Compute a solution σ ∗ ,e from ˜Γ e

Original: Output: Current solution σ ∗ ,ei for player
Revised: Output: Current solution σ ∗ ,ei for player

Original: i
Revised: i

Original: Algorithm

Original: Input: Initial policy sets for all players Π
Revised: Input: Initial policy sets for all players Π

Original: Simulate utilities ˜
Revised: Simulate utilities ˜

Original: U Π 0 for each joint π ∈ Π
Revised: U Π 0 for each joint π ∈ Π

Original: Initialize solutions σ ∗ , 

Revised: Initialize solutions σ ∗ ,

Original: i =
Revised: i =

Original: Uniform (Π∗ , 0 i )
Revised: Uniform (Π 0 i )

Original: Initialize pure-strategy BRs Λ 0 i = ∅ while epoch e in { 1 , 2 ,... } do # Best respond to each new opponent.
Revised: Initialize pure-strategy BRs Λ 0 i = ∅ while epoch e in { 1 , 2 ,... } do # Best respond to each new opponent.

Original: for player i ∈ [[ n ]] do for many episodes do
Revised: for player i ∈ [[ n ]] do for many episodes do

Original: Train λ ei over τ ∼ ( λ ei ,π e − 1 − i ) Λ
Revised: Train λ ei over τ ∼ ( λ ei ,π e − 1 − i ) Λ

Original: ei =Λ e − 1 i ∪{ λ ei } # Generate new policies.
Revised: ei =Λ e − 1 i ∪{ λ ei } # Generate new policies.

Original: for player i ∈ [[ n ]] do π ei ← TransferOracle
Revised: for player i ∈ [[ n ]] do π ei ← TransferOracle

Original: (Λ ei ,σ  − i )
Revised: (Λ ei ,σ ∗ ,e − 1 − i )

Original: Π ei =
Revised: Π ei =

Original: Π
Revised: Π

Original: e − 1 i ∪{ π ei }
Revised: e − 1 i ∪{ π ei }

Original: Simulate missing entries in ˜ U
Revised: Simulate missing entries in ˜ U

Original: Π e from Π e
Revised: Π e from Π e

Original: Compute a solution σ ∗ ,e from ˜Γ e
Revised: Compute a solution σ ∗ ,e from ˜Γ e

Original: Output: Current solution σ ∗ ,ei for player i .
Revised: Output: Current solution σ ∗ ,ei for player i .

Original:  S, despite it being good against both player 1 strategies so far.
Revised: 80% against it. The best response to π 21 is P. This leads to an empirical game similar to matching pennies, with player 2 playing an even mixture of Rock and Paper; in particular, player 2’s meta-strategy never plays S, despite it being good against both player 1 strategies so far.

Original: If, as in PSRO, player 1 then adds their BR to this meta-strategy, which is P, then their strategy set still does not contain the Nash equilibrium of the game.
Revised: If, as in PSRO, player 1 then adds their BR to this meta-strategy, which is P, then their strategy set still does not contain the Nash equilibrium of the game.

Original: The problem is that there are already strategies in the empirical game that can defeat player 2’s strategies, so the new strategy is not useful.
Revised: The problem is that there are already strategies in the empirical game that can defeat player 2’s strategies, so the new strategy is not useful.

Original: This can be detected by inspecting player 2’s Q-functions: Qhas a very low valuation of P, and Q 22 has a very low valuation of R. Responding to S, which is moderately highly- but not maximally - valued by both opponent Q functions, is more relevant.
Revised: This can be detected by inspecting player 2’s Q-functions: Q 12 has a very low valuation of P, and Q 22 has a very low valuation of R. Responding to S, which is moderately highly valued by both opponent Q functions, is more relevant.

Original: To achieve this, we can mix the opponent strategies through their action-value estimates, rather than their action distributions (Figure 1).
Revised: To achieve this, we can mix the opponent strategies through their action-value estimates, rather than their action distributions (Figure 1).

Original: The BR to this is R, which more usefully extends player 1’s strategy set.
Revised: The BR to this is R, which more usefully extends player 1’s strategy set.

Original: This example motivates our second algorithm:
Revised: This example motivates our second algorithm:

Original: Mixed-Opponents.
Revised: Mixed-Opponents.

Original: This method also employs a combination method, but instead of combining results from training against previously encountered opponents, it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training.
Revised: This method also employs a combination method, but instead of combining results from training against previously encountered opponents, it combines the strategies of the opponent mixture themselves to construct a single new opponent policy as a target for training.

Original: We refer to the method for generating a new opponent policy from a mixture of opponents the Opponent-Oracle, and it has the same functional form as the Transfer-Oracle.
Revised: We refer to the method for generating a new opponent policy from a mixture of opponents the Opponent-Oracle, and it has the same functional form as the Transfer-Oracle.

Original: The generalized Mixed-Opponent algorithm is shown in Algorithm 3.
Revised: The generalized Mixed-Opponent algorithm is shown in Algorithm 3.

Original: We employ Q-Mixing (Equation 1) as our Opponent-Oracle.
Revised: We employ Q-Mixing (Equation 1) as our Opponent-Oracle.

Original: In contrast to Mixed-Oracles, which uses Q-Mixing to transfer Q-values across epochs, here we apply it to average Q-values to define a variant training objective.
Revised: In contrast to Mixed-Oracles, which uses Q-Mixing to transfer Q-values across epochs, here we apply it to average Q-values to define a variant training objective.

Original: 3. E NVIRONMENTS
Revised: 3. E NVIRONMENTS

Original: We evaluate our algorithms on the Gathering (Perolat et al., 2017) and Leduc Poker (Southey et al., 2005) games, both of which are commonly used in the multiagent reinforcement learning field.
Revised: We evaluate our algorithms on the Gathering (Perolat et al., 2017) and Leduc Poker (Southey et al., 2005) games, both of which are commonly used in the multiagent reinforcement learning field.

Original: Gathering is a gridworld tragedy-of-the-commons game that is partially observable and general-sum.
Revised: Gathering is a gridworld tragedy-of-the-commons game that is partially observable and general-sum.

Original: Agents compete to harvest apples that regrow at a rate proportional to the number of nearby apples.
Revised: Agents compete to harvest apples that regrow at a rate proportional to the number of nearby apples.

Original: The individual’s interest in harvesting conflicts with the group’s interest in regrowth creating the dilemma.
Revised: The individual’s interest in harvesting conflicts with the group’s interest in regrowth creating the dilemma.

Original: We investigate two versions of this game that differ in the configuration of the map (apple locations, spawn points, etc.): (1) Gathering-Small has all apples in a dense grove central in the map, and (2) Gathering-Open is a larger map with many spread-out apple groves.
Revised: We investigate two versions of this game that differ in the configuration of the map (apple locations, spawn points, etc.): (1) Gathering-Small has all apples in a dense grove central in the map, and (2) Gathering-Open is a larger map with many spread-out apple groves.

Original: The former environment will be used to force two agents to interact, while the latter will allow the study of interactions between morethan two players.
Revised: The former environment will be used to force two agents to interact, while the latter will allow the study of interactions between more

Original: Further details and images of the environments are included in Section C.1.

Original: Leduc Poker is a reduced version of Poker, designed to capture the important characteristics of a full game of Poker.

Original: The game proceeds similarly to Poker, where each player chooses to raise/call/fold through two rounds of betting.

Original: In this version, we examine the 2-player versions of Leduc Poker.

Original: More detailed information about this environment is included in Section C.

Original: Input: Initial policies for all players Π
Revised: Input: Initial policies for all players Π

Original: Simulate ˜ U Π 0 for each joint π ∈ Π
Revised: Simulate ˜ U Π 0 for each joint π ∈ Π

Original: Initialize solutions σ ∗ , 

Revised: Initialize solutions σ ∗ ,

Original: i =
Revised: i =

Original: Uniform (Π∗ , 0 i ) while epoch e in { 1 , 2 ,... } do for player i ∈
Revised: Uniform (Π 0 i ) while epoch e in { 1 , 2 ,... } do for player i ∈

Original: [[ n ]] dofor many episodes do π − i ←
Revised: [[ n ]] do π − i ←

Original: OpponentOracle (Π e − 1 − i ,σ e − 1 − i ) 
Revised: OpponentOracle (Π e − 1 − i ,σ ∗ ,e − 1 − i ) for many episodes do

Original: Train π ei over τ ∼ ( π ei ,π − i ) Π ei =
Revised: Train π ei over τ ∼ ( π ei ,π − i ) Π ei =

Original: Π
Revised: Π

Original: e − 1 i ∪{ π ei }
Revised: e − 1 i ∪{ π ei }

Original: Simulate missing entries in ˜ U
Revised: Simulate missing entries in ˜ U

Original: Π e
Revised: Π e

Original: Compute a solution σ ∗ ,e from ˜Γ e
Revised: Compute a solution σ ∗ ,e from ˜Γ e

Original: Output: Solution σ ∗ ,ei for player i .
Revised: Output: Solution σ ∗ ,ei for player i .

Revised: Leduc Poker is a reduced version of Poker, designed to capture the important characteristics of a full game of Poker.

Revised: The game proceeds similarly to Poker, where each player chooses to raise/call/fold through two rounds of betting.

Revised: In this version, we examine the 2-player versions of Leduc Poker.

Revised: More detailed information about this environment is included in Section D.2.

Original: 3. H YPERPARAMETERS
Revised: 3. H YPERPARAMETERS

Original: The objective of this study is to minimize the number of timesteps spent training a RL agent across all epochs of PSRO.
Revised: The objective of this study is to minimize the number of timesteps spent training a RL agent across all epochs of PSRO.

Original: To this end, how timesteps are counted is critical to the comparison of methods.
Revised: To this end, how timesteps are counted is critical to the comparison of methods.

Original: The number of timesteps used for training is typically a hyperparameter for the RL agent.
Revised: The number of timesteps used for training is typically a hyperparameter for the RL agent.

Original: This presents a challenging dilemma, because (1) we must select this before running PSRO, and (2) any particular setting may not work against all opponent-profiles chosen by the MSS.
Revised: This presents a challenging dilemma, because (1) we must select this before running PSRO, and (2) any particular setting may not work against all opponent-profiles chosen by the MSS.

Original: To address these issues we take a two-step approach (detailed in Section A).
Revised: To address these issues we take a two-step approach (detailed in Section B).

Original: First, we select preliminary hyperparameters by searching for their setting that performs best against a random opponent policy.
Revised: First, we select preliminary hyperparameters by searching for their setting that performs best against a random opponent policy.

Original: Then we use these hyperparameters to generate a set of opponent-policies (via PSRO) to use as an objective for selecting final hyperparameters.
Revised: Then we use these hyperparameters to generate a set of opponent-policies (via PSRO) to use as an objective for selecting final hyperparameters.

Original: A subset of the generated policies are used as an opponent test-bed to collect two sets of hyperparameters: pure-hparams performed best on-average against each individual test-opponent, and mix-hparams performed best against a uniform mixture of the test-opponents.
Revised: A subset of the generated policies are used as an opponent test-bed to collect two sets of hyperparameters: pure-hparams performed best on-average against each individual test-opponent, and mix-hparams performed best against a uniform mixture of the test-opponents.

Original: These settings are meant to represent the high- and low-variance situations encountered when training mixed and pure strategies.
Revised: These settings are meant to represent the high- and low-variance situations encountered when training mixed and pure strategies.

Original: In this work, our Deep RL algorithm is Double Q-Learning (Hasslet et al., 2016), and our policies have two hidden layers with ReLU activations.
Revised: In this work, our Deep RL algorithm is Double Q-Learning (Hasslet et al., 2016), and our policies have two hidden layers with ReLU activations.

Original: 300 hyperparameter settings are sampled in each environment.
Revised: 300 hyperparameter settings are sampled in each environment.

Original: Complete details are provided in Section C.
Revised: Complete details are provided in Section D.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Regret as a measure depends on the policies that are available for deviation; therefore, it may be misleading to compare across PSRO runs directly.
Revised: Regret as a measure depends on the policies that are available for deviation; therefore, it may be misleading to compare across PSRO runs directly.

Original: As a more equitable comparison, we instead measure regret when deviating to both the run’s discovered policies Π PSRO and a held-out set of evaluation policies Π EVAL .
Revised: As a more equitable comparison, we instead measure regret when deviating to both the run’s discovered policies Π PSRO and a held-out set of evaluation policies Π EVAL .

Original: The evaluation policies’ provide a standard lower-bound on regret across all runs.
Revised: The evaluation policies’ provide a standard lower-bound on regret across all runs.

Original: These policies are constructed by independent PSRO runs, not included in the results here, by sampling from the solution’s support.
Revised: These policies are constructed by independent PSRO runs, not included in the results here, by sampling from the solution’s support.

Original: There are six evaluation policies for each environment. Unless otherwise noted, all profiles are simulated thirty times to estimate the payoff in the ENFG.
Revised: Unless otherwise noted, there are six evaluation policies and ENFG payoffs are estimated with thirty episodes.

Original: 4. M IXED -O RACLES
Revised: 4. M IXED -O RACLES

Original: The first research question we address is: does Mixed-Oracles result in a similar quality solution compared to PSRO while utilizing fewer simulation timesteps?
Revised: The first research question we address is: does Mixed-Oracles result in a similar quality solution compared to PSRO while utilizing fewer simulation timesteps?

Original: We begin by running both algorithms on the GatheringSmall and Leduc-Poker games.
Revised: We begin by running both algorithms on the GatheringSmall and Leduc-Poker games.

Original: We record regrets over PSRO epochs and cumulative training timesteps.
Revised: We record regrets over PSRO epochs and cumulative training timesteps.

Original: Figure 2 compares the convergence speed of both algorithms on the Gathering-Small environment.
Revised: Figure 2 compares the convergence speed of both algorithms on the Gathering-Small environment.

Original: We found that Mixed-Oracles converged in roughly half the number of epochs while utilizing a quarter of the number of simulations.
Revised: We found that Mixed-Oracles converged in roughly half the number of epochs while utilizing a quarter of the number of simulations.

Original: Moreover, the solution discovered by Mixed-Oracles had approximately 30% less regret.
Revised: Moreover, the solution discovered by Mixed-Oracles had approximately 30% less regret.

Original: Figure 3 shows the result on Leduc-Poker, where we observe Mixed-Oracles finding a comparable solution in dramatically fewer cumulative timesteps.
Revised: Figure 3 shows the result on Leduc-Poker, where we observe Mixed-Oracles finding a comparable solution in dramatically fewer cumulative timesteps.

Original: These results confirm our hypothesis that Mixed-Oracles will find comparable solutions to PSRO in fewer training timesteps.
Revised: These results confirm our hypothesis that Mixed-Oracles will find comparable solutions to PSRO in fewer training timesteps

Original: 4. M IXED -O PPONENTS
Revised: 4. M IXED -O PPONENTS

Original: Next we address the question: does Mixed-Opponents result in a better quality solution compared to PSRO while utilizing fewer simulation timesteps?
Revised: Next we address the question: does Mixed-Opponents result in a better quality solution compared to PSRO while utilizing fewer simulation timesteps?

Original: The methodology of this experiment follows the previous experiment: measuring SumRegret on both the Gathering-Small and Leduc-Poker games.
Revised: The methodology of this experiment follows the previous experiment: measuring SumRegret on both the Gathering-Small and Leduc-Poker games.

Original: Figure 4 compares the convergence speed of both algorithms on the Gathering-Open game.
Revised: Figure 4 compares the convergence speed of both algorithms on the Gathering-Open game.

Original: Both algorithms appear to converge initially after 25 epochs and at this point, Mixed-Opponents had discovered a solution with 50% less regret.
Revised: Both algorithms appear to converge initially after 25 epochs, and at this point, Mixed-Opponents had discovered a solution with 50% less regret.

Original: However, while PSRO remained converged as time went on, Mixed-Opponents continues to improve and nearly solves the game.
Revised: However, while PSRO remained converged as time went on, Mixed-Opponents continues to improve and nearly solves the game.

Original: Figure 5 shows the results on Leduc-Poker.
Revised: Figure 5 shows the results on Leduc-Poker.

Original: Similar to Mixed-Oracles, we also find that Mixed-Opponents finds a similar solution to PSRO in dramatically fewer timesteps.
Revised: Similar to Mixed-Oracles, we also find that Mixed-Opponents finds a similar solution to PSRO in dramatically fewer timesteps.

Original: Together these results confirm our second hypothesis that Mixed-Opponents may find a comparable solution to PSRO in less training simulations.
Revised: Together these results confirm our second hypothesis that Mixed-Opponents may find a comparable solution to PSRO in less training simulations.

Original: 4.3 > 2 P LAYER G AMES
Revised: 4.3 > 2 P LAYER G AMES

Original: An advantage of Mixed-Opponents over Mixed-Oracles is that it can be applied to games with more than two players.
Revised: An advantage of Mixed-Opponents over Mixed-Oracles is that it can be applied to games with more than two players.

Original: Does Mixed-Opponents reduce training-simulation when compared to PSRO in games where there are more than two players?
Revised: Does Mixed-Opponents reduce training-simulation when compared to PSRO in games where there are more than two players?

Original: We run both algorithms on the Gathering-Open game with three players.
Revised: We run both algorithms on the Gathering-Open game with three players.

Original: In this experiment, the ENFGis restricted toonly three simulations per profile due to  the combinatorial explosion of policies.
Revised: We limit each profile in the ENFG to three simulations, to handle the combinatorial explosion of profiles.

Original: Figure 6a shows our results where Mixed-Opponents finds a similar quality solution to PSRO in half of the training timesteps.
Revised: Figure 6a shows our results where Mixed-Opponents finds a similar quality solution to PSRO in half of the time.

Original: 4. S HARED H YPERPARAMETERS
Revised: 4. S HARED H YPERPARAMETERS

Original: An important choice in these experiments has been how to measure timesteps elapsed.
Revised: An important choice in these experiments has been how to measure timesteps elapsed.

Original: We utilized two sets of hyperparameters: specialized for low- and high-variance outcomes of state, which are a result of facing pure- and mixed-strategy opponents respectively.
Revised: We utilized two sets of hyperparameters: specialized for low- and high-variance outcomes of state, which are a result of facing pure- and mixed-strategy opponents respectively.

Original: This was motivated by the assumption that the low variance would require less training.
Revised: This was motivated by the assumption that the low variance would require less training.

Original: In this section, we question that assumption and ask the question: do MixedOracles and Mixed-Opponents perform at least as well as PSRO when given the same RL hyperparameters?
Revised: In this section, we question that assumption and ask the question: do MixedOracles and Mixed-Opponents perform at least as well as PSRO when given the same RL hyperparameters?

Original: To answer this question we run all three algorithms with the same set of hyperparameters, forcing all algorithms to utilize the same simulation budget.
Revised: To answer this question we run all three algorithms with the same set of hyperparameters, forcing all algorithms to utilize the same simulation budget.

Original: This will remove any confounding effects on our results brought about by our method for selecting hyperparameters.
Revised: This will remove any confounding effects on our results brought about by our method for selecting hyperparameters.

Original: We report our results on the Gathering-Small game in Figure 6b.
Revised: We report our results on the Gathering-Small game in Figure 6b.

Original: The trends observed in the previous experiment reoccur – Mixed-Oracles and
Revised: The trends observed in the previous experiment reoccur – Mixed-Oracles and

Original: Mixed-Opponents finding solutions at least as good as PSRO in fewer timesteps, confirming our hypothesis.
Revised: Mixed-Opponents finding solutions at least as good as PSRO in fewer timesteps, confirming our hypothesis.

Original: In particular, under the pure-hparams, where simulations are more limited, Mixed-Opponents find a better solution than PSRO in half of the simulation timesteps.
Revised: In particular, under the pure-hparams, where simulations are more limited, Mixed-Opponents find a better solution than PSRO in half of the simulation timesteps.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: The first published paper to explicitly estimate an ENFG from an agent-based simulation was a study by
Revised: The first published paper to explicitly estimate an ENFG from an agent-based simulation was a study by

Original: Walsh et al.
Revised: Walsh et al.

Original: on pricing games and auctions.
Revised: on pricing games and auctions.

Original: They analyzed symmetric games with up to twenty players over a small number of hand-crafted heuristic strategies.
Revised: They analyzed symmetric games with up to twenty players over a small number of hand-crafted heuristic strategies.

Original: A few additional works in succeeding years explored the approach, which was developed into an algorithmic framework under the EGTA label by Wellman (2006).
Revised: A few additional works in succeeding years explored the approach, which was developed into an algorithmic framework under the EGTA label by Wellman (2006).

Original: The strategy exploration problem within EGTA was first identified and studied by Schvartzman & Wellman (2009a).
Revised: The strategy exploration problem within EGTA was first identified and studied by Schvartzman & Wellman (2009a).

Original: In this work, they explored three heuristic solutions and found that BR was the most effective method.
Revised: In this work, they explored three heuristic solutions and found that BR was the most effective method.

Original: They also illustrated situations where equilibrium response had poor performance.
Revised: They also illustrated situations where equilibrium response had poor performance.

Original: [Figure]
Revised: [Figure]

Original: Before this formalization, Phelps et al.
Revised: Before this formalization, Phelps et al.

Original: (2006) were the first to utilize automated strategy generation.
Revised: (2006) were the first to utilize automated strategy generation.

Original: Their approach is still seen in PSRO today, where iterative BRs to equilibrium are used to build an empirical game.
Revised: Their approach is still seen in PSRO today, where iterative BRs to equilibrium are used to build an empirical game.

Original: They utilized a small parametric policy space and used genetic search as their approximate oracle method.
Revised: They utilized a small parametric policy space and used genetic search as their approximate oracle method.

Original: Schvartzman & Wellman (2009b) followed this up by introducing reinforcement learning as the approximate oracle method.
Revised: Schvartzman & Wellman (2009b) followed this up by introducing reinforcement learning as the approximate oracle method.

Original: These algorithms, alongside a trend of deep reinforcement learning being used to solve difficult games, set the stage for the PSRO framework.
Revised: These algorithms, alongside a trend of deep reinforcement learning being used to solve difficult games, set the stage for the PSRO framework.

Original: PSRO highlighted a subclass of
Revised: PSRO highlighted a subclass of

Original: EGTA algorithms where an iterative construction of an empirical game is built through iterative BR to an opponent-profile.
Revised: EGTA algorithms where an iterative construction of an empirical game is built through iterative BR to an opponent-profile.

Original: At each epoch, the opponent profile is determined by a chosen meta-strategy solver (MSS).
Revised: At each epoch, the opponent profile is determined by a chosen meta-strategy solver (MSS).

Original: The PSRO algorithm builds heavily upon the Double-Oracle (DO) algorithm, which iteratively solves for the Nash Equilibrium in two-player zero-sum sub-games (McMahan et al., 2003).
Revised: The PSRO algorithm builds heavily upon the Double-Oracle (DO) algorithm, which iteratively solves for the Nash Equilibrium in two-player zero-sum sub-games (McMahan et al., 2003).

Original: This has culminated in an interest in discovering good strategy-exploration methods within PSRO.
Revised: This has culminated in an interest in discovering good strategy-exploration methods within PSRO.

Original: PSRO’s strategy exploration happens in two phases: first, a MSS produces an opponent-profile as a learning objective, then reinforcement learning is used as an approximate oracle to best-respond to the objective.
Revised: PSRO’s strategy exploration happens in two phases: first, a MSS produces an opponent-profile as a learning objective, then reinforcement learning is used as an approximate oracle to best-respond to the objective.

Original: Many popular algorithms for computing strategies in game theory have been adapted as MSS: linearprogramming, replicator-dynamics Taylor & Jonker (1978), regret-minimization (Blum & Mansour, 2007), and regret-matching (Hart & Mas-Colell, 2000).
Revised: Many popular algorithms for computing strategies in game theory have been adapted as MSS: linearprogramming, replicator-dynamics Taylor & Jonker (1978), regret-minimization (Blum & Mansour, 2007), and regret-matching (Hart & Mas-Colell, 2000).

Original: Moreover, popular game-solving algorithms can also be viewed through the lens of a MSS.
Revised: Moreover, popular game-solving algorithms can also be viewed through the lens of a MSS.

Original: Fictitious play (Brown, 1951) is a MSS where the resulting profile is a uniform-strategy.
Revised: Fictitious play (Brown, 1951) is a MSS where the resulting profile is a uniform-strategy.

Original: Similarly, self-play’s resulting profile is a pure strategy containing only the newest policy.
Revised: Similarly, self-play’s resulting profile is a pure strategy containing only the newest policy.

Original: Prompting recent work to leverage the additional information gleaned through the empirical game to construct new MSSs.
Revised: Prompting recent work to leverage the additional information gleaned through the empirical game to construct new MSSs.

Original: Wright et al.
Revised: Wright et al.

Original: (2019) extends DO’s MSS to return a decay-weighted linear combination of the current and previous solutions.
Revised: (2019) extends DO’s MSS to return a decay-weighted linear combination of the current and previous solutions.

Original: Omidshafiei et al.
Revised: Omidshafiei et al.

Original: (2019) proposes Markov-Conley
Revised: (2019) proposes Markov-Conley

Original: Chains (MCC) as a solution concept that relates to their new proposed quality measure of policy called α -Rank, and Muller et al.
Revised: Chains (MCC) as a solution concept that relates to their new proposed quality measure of policy called α -Rank, and Muller et al.

Original: (2020) investigated its use as a MSS.
Revised: (2020) investigated its use as a MSS.

Original: McAleer et al.
Revised: McAleer et al.

Original: (2020) also tackles the problem of the compute requirements of PSRO by addressing its runtime.
Revised: (2020) also tackles the problem of the compute requirements of PSRO by addressing its runtime.

Original: Their Pipeline-PSRO algorithm enables the RL process across epochs to be trained in parallel.
Revised: Their Pipeline-PSRO algorithm enables the RL process across epochs to be trained in parallel.

Original: The parallel technology could be leveraged to reduce runtime of both Mixed-Oracles and Mixed-Opponents.
Revised: The parallel technology could be leveraged to reduce runtime of both Mixed-Oracles and Mixed-Opponents.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this study, we investigate PSRO’s approach to strategy exploration and extensions that focus on reducing the time spent on training new policies.
Revised: In this study, we investigate PSRO’s approach to strategy exploration and extensions that focus on reducing the time spent on training new policies.

Original: PSRO’s strategy exploration occurs in two steps: solution and BR via RL.
Revised: PSRO’s strategy exploration occurs in two steps: solution and BR via RL.

Original: We first propose Mixed-Oracles which approximates the solution step by training separate BRs to each opponent policy and then combining the results into an approximate BR to the solution.
Revised: We first propose Mixed-Oracles which approximates the solution step by training separate BRs to each opponent policy and then combining the results into an approximate BR to the solution.

Original: This affords our algorithm the ability to transfer learning about previously encountered opponents.
Revised: This affords our algorithm the ability to transfer learning about previously encountered opponents.

Original: Next, we introduce
Revised: Next, we introduce

Original: Mixed-Opponents which modifies the BR training by constructing a single opponent policy that represents an aggregate of the mixed strategy and learns a response to this new opponent.
Revised: Mixed-Opponents which modifies the BR training by constructing a single opponent policy that represents an aggregate of the mixed strategy and learns a response to this new opponent.

Original: Mixed-Opponents also trivially scales to games with more than two players.
Revised: Mixed-Opponents also trivially scales to games with more than two players.

Original: Both of these methods further mitigate the variance in state outcomes, by removing an unobserved distribution of opponents, easing learning.
Revised: Both of these methods further mitigate the variance in state outcomes, by removing an unobserved distribution of opponents, easing learning.