N EU PL: N EURAL P OPULATION L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Learning in strategy games (e.g. StarCraft, poker) requires the discovery of diverse policies.
This is often achieved by iteratively training new policies against existing ones, growing a policy population that is robust to exploit.
This iterative approach suffers from two issues in real-world games: a) under ﬁnite budget, approximate best-response operators at each iteration needs truncating, resulting in under-trained good-responses populating the population; b) repeated learning of basic skills at each iteration is wasteful and becomes intractable in the presence of increasingly strong opponents.
In this work, we propose Neural Population Learning (NeuPL) as a solution to both issues.
NeuPL offers convergence guarantees to a population of best -responses under mild assumptions.
By representing a population of policies within a single conditional model, NeuPL enables transfer learning across policies.
Empirically, we show the generality, improved performance and efﬁciency of
NeuPL across several test domains 1 .
Most interestingly, we show that novel strategies become more accessible, not less, as the neural population expands.
The need for learning not one, but a population of strategies is rooted in classical game theory.
Consider the purely cyclical game of rock-paper-scissors , the performance of individual strategies is meaningless as improving against one entails losing to another.
By contrast, performance can be meaningfully examined between populations.
A population consisting of pure strategies { rock , paper } does well against a singleton population of { scissors } because in the meta-game where both populations are revealed, a player picking strategies from the former can always beat a player choosing from the latter 2 .
This observation underpins the unifying population learning framework of
Policy Space Response Oracle (PSRO) where a new policy is trained to best-respond to a mixture over previous policies at each iteration, following a meta-strategy solver (Lanctot et al., 2017).
Most impressively, Vinyals et al.
(2019) explored the strategy game of StarCraft with a league of policies, using a practical variation of PSRO.
The league counted close to a thousand sophisticated deep RL agents as the population collectively became robust to exploits.
Unfortunately, such empirical successes often come at considerable costs.
Population learning algorithms with theoretical guarantees are traditionally studied in normal-form games (Brown, 1951;
McMahan et al., 2003) where best-responses can be solved exactly.
This is in stark contrast to real-world Game-of-Skills (Czarnecki et al., 2020) — such games are often temporal in nature, where best-responses can only be approximated with computationally intensive methods (e.g. deep RL).
This has two implications.
First, for a given opponent, one cannot efﬁciently tell apart good -responses that temporarily plateaued at local optima from globally optimal best-responses.
As a result, approximate best-response operators are often truncated prematurely, according to hand-crafted schedules (Lanctot et al., 2017; Mcaleer et al., 2020).
Second, real-world games often afford strategy-agnostic transitive skills that are pre-requisite to strategic reasoning.
Learning such skills from scratch at each iteration in the presence of evermore skillful opponents quickly becomes intractable beyond a few iterations.
This iterative and isolated approach is fundamentally at odds with human learning.
For humans, mastering diverse strategies often facilitates incremental strategic innovation and learning about new strategies does not stop us from revisiting and improving upon known ones (Caruana, 1997;
Krakauer et al., 2006).
In this work, we make progress towards endowing artiﬁcial agents with similar capability by extending population learning to real-world games.
Speciﬁcally, we propose
NeuPL, an efﬁcient and general framework that learns and represents diverse policies in symmetric zero-sum games within a single conditional network, using the computational infrastructure of simple
1 ⁄ 3 1 ⁄ 3 1 ⁄1 ⁄ 3 1 ⁄ 3 1 ⁄1 ⁄ 3 1 ⁄ 3 1 ⁄   
Population Self-Play
10 01 0
  
Strategic Cycle         1 ⁄ 2 1 ⁄ 2
  
Fictitious Play   1 − p p      p 1 − p
  
PSRO-N ASH and self-play ( Section 1.2 ).
Theoretically, we show that NeuPL converges to a sequence of iterative best-responses under certain conditions ( Section 1.3 ).
Empirically, we illustrate the generality of
NeuPL by replicating known results of population learning algorithms on the classical domain of rockpaper-scissors as well as its partially-observed, spatiotemporal counterpart running-with-scissors (Vezhnevets et al., 2020)
( Section 2.1 ).
Most interestingly, we show that NeuPL enables transfer learning across policies, discovering exploiters to strong opponents that would have been inaccessible to comparable baselines ( Section 2.2 ).
Finally, we show the appeal of NeuPL in the challenge domain of MuJoCo Football (Liu et al., 2019) where players must continuously reﬁne their movement skills in order to coordinate as a team.
In this highly transitive game, NeuPL naturally represents a short sequence of best-responses without the need for a carefully chosen truncation criteria ( Section 2.4 ).
M ETHODS
Our method is designed with two desiderata in mind.
First, at convergence, the resulting population of policies should represent a sequence of iterative best-responses under reasonable conditions.
Second, transfer learning can occur across policies throughout training.
In this section, we deﬁne the problem setting of interests as well as necessary terminologies.
We then describe NeuPL, our main conceptual algorithm as well as its theoretical properties.
To make it concrete, we further consider deep RL speciﬁcally and offer two practical implementations of NeuPL for real-world games.
1. P RELIMINARIES
Approximate Best-Response (ABR) in Stochastic Games
We consider a symmetric zero-sum
Stochastic Game (Shapley, 1953) deﬁned by ( S , O , X , A , P , R , p 0 ) with S the state space, O the observation space and X : S
→ O × O the observation function deﬁning the (partial) views of the state for both players.
Given joint actions ( a t , a (cid:48) t ) ∈
A × A , the state follows the transition distribution P : S ×
A × A → Pr( S ) .
The reward function R : S → R × R deﬁnes the rewards for both players in state s t , denoted R ( s t ) =
( r t , − r t ) .
The initial state of the environment follows the distribution p 0 .
In a given state s t , players act according to policies ( π ( ·| o ≤ t ) , π (cid:48) ( ·| o (cid:48)≤ t )) .
Player π achieves an expected return of J ( π, π (cid:48) )
=
E π,π (cid:48)
[ (cid:80) t r t ] against π (cid:48) .
Policy π ∗ is a best response to π (cid:48) if ∀ π, J ( π ∗ , π (cid:48) )
≥ J ( π, π (cid:48) ) .
We deﬁne ˆ π ← ABR ( π, π (cid:48) ) with J (ˆ π, π (cid:48) )
≥ J ( π, π (cid:48) ) .
In other words, an ABR operator yields a policy ˆ π that does no worse than π , in the presence of an opponent π (cid:48) .
Meta-game Strategies in Population Learning Given a symmetric zero-sum game and a set of N policies Π :=
{ π i } Ni =1 , we deﬁne a normal-form meta-game where players’ i -th action corresponds to executing policy
π i for one episode.
A meta-game strategy σ thus deﬁnes a probability assignment, or an action proﬁle, over Π .
Within Π , we deﬁne U ∈ R N × N ← EVAL (Π) to be the expected payoffs between pure strategies of this meta-game or equivalently, U ij :=
J ( π i , π j ) in the underlying game.
We further extend the ABR operator of the underlying game to mixture policies represented by σ , such that ˆ π ← ABR ( π, σ, Π) with E π (cid:48) ∼ P ( σ )
[ J (ˆ π, π (cid:48) )] ≥ E π (cid:48) ∼ P ( σ )
[ J ( π, π (cid:48) )] .
Finally, we deﬁne f : R | Π |×| Π | → R | Π | to be a meta-strategy solver (MSS) with σ ← f ( U ) and F : R N × N → R N × N a meta-graph solver (MGS) with Σ ← F ( U ) .
The former formulation is designed for iterative optimization of approximate best-responses as in Lanctot et al.
(2017) whereas the latter is motivated by concurrent optimization over a set of population-level objectives as in Garnelo et al.
In particular, Σ ∈ R N × N := { σ i } Ni =1 deﬁnes N population-level objectives, with π i optimized against the mixture policy represented by σ i and Π .
As such, Σ ∈ R N × N corresponds to the adjacency matrix of an interaction graph.
Figure 1 illustrates several commonly used population learning algorithms deﬁned by Σ or equivalently, their interaction graphs.
1. N EURAL P OPULATION L EARNING
We now present NeuPL and contrast it with Policy-Space Response Oracles (PSRO, Lanctot et al.
(2017)) which similarly focuses on population learning with approximate best-responses by RL.
[Table] [Table]
1: Π := { π 0 } (cid:46)
Initial policy population.
2: σ ← U NIF (Π) (cid:46)
Initial meta-game strategy.
3: f : R | Π |×| Π | → R | Π | (cid:46) Meta-strategy solver.
4:5: for i ∈
[[ N ]] do (cid:46) N-step ABR.
Initialize π θ i .
7: π θ i ← ABR ( π θ i , σ, Π) 8: Π ← Π ∪ { π θ i } 9:
U ← E VAL (Π) (cid:46) Empirical payoffs.
10: σ ← f ( U ) 11: return Π
NeuPL deviates from PSRO in two important ways.
First, NeuPL suggests concurrent and continued training of all unique policies such that no good-response features in the population prematurely due to early truncation.
Second, NeuPL represents an entire population of policies via a shared conditional network Π θ ( ·| s, σ ) with each policy Π θ ( ·| s, σ i ) conditioned on and optimised against a meta-game mixture strategy σ i , enabling transfer learning across policies.
This representation also makes NeuPL general: it delegates the choice of effective population sizes | U NIQUE (Σ) | ≤ | Σ |
=
N to the meta-graph solver F as σ i =
σ j implies Π θ ( ·| s, σ i )
≡ Π θ ( ·| s, σ j ) (cf.
Section 2.1).
Finally,
NeuPL allows for cyclic interaction graphs, beyond the scope of PSRO.
We discuss the generality of
NeuPL in the context of prior works in further details in Appendix D.
N-step Best-Responses via Lower-Triangular Graphs A popular class of population learning algorithms seeks to converge to a sequence of N iterative best-responses where each policy π i is a best-response to an opponent meta-game strategy σ i with support over a subset of the policy population Π <i
= { π j } j<i .
In NeuPL, this class of algorithms are implemented with meta-graph solvers that return lower-triangular adjacency matrices Σ with Σ i ≤ j = 0 .
Under this constraint, σbecomes a zero vector, implying that Π θ ( ·| s, σ 0 ) does not seek to best-respond to any policies.
Similar to the role of initial policies { π 0 } in PSRO (Algorithm 2), Π θ ( ·| s, σ 0 ) serves as a starting point for the sequence of N-step best-responses and any ﬁxed policy can be used.
We note that this property further allows for incorporating pre-trained policies in NeuPL, as we discuss in Appendix D.1.
One prominent example is PSRO-N ASH , where π i is optimized to best-respond to the Nash mixture policy over Π <i .
This particular meta-graph solver is shown in Algorithm 3.
1. C ONVERGENCE TO N- STEP B EST -R ESPONSES VIA N EU PL
Under certain assumptions on the best-response operator, interaction graph, and meta-graph solver (MGS) we can construct proofs that NeuPL converges to an N -step best-response.
We introduce the term grounded (Section C) to refer to interaction graphs and MGS that have a structure that imposes convergence to a unique set of policies.
Certain interaction graphs are grounded, in particular, lowertriangular graphs are one such class which describe an N -step best response.
In addition, certain
MGSs are grounded, in particular, ones that operate on the sub-payoff and output a lower-triangular interaction graph, F : U <
i,<i → Σ i,<i .
The lower-triangular maximum entropy Nash equilibrium (MENE) is one such grounded MGS.
Therefore with sufﬁciently large N , NeuPL will converge to a normal-form Nash equilibrium.
See Section C for the full deﬁnitions, theorems and proofs.
1. N EURAL P OPULATION L EARNING BY RL
We now deﬁne the discounted return maximized by Π θ ( ·| o ≤ t , σ i ) in Equation 1.
We denote P ( σ i ) as the probability distribution over policy
i ’s opponent identities σ j ∈ { σ 1 , . . .
, σ N } .
Intuitively, each policy is maximizing its expected returns in the underlying game under a double expectation: the ﬁrst is taken over its opponent distribution, with σ j ∼ P ( σ i ) and the second taken under the game dynamics partly deﬁned by the pair of policies (Π θ ( ·| o ≤ t , σ i ) , Π θ ( ·| o (cid:48)≤ t , σ j )) .
[Equation]
To optimize Π Σ θ by RL, we jointly train an opponent-conditioned action-value 3 function, approximating the expected return of choosing an action a t given an observation history o ≤ t , following Π θ ( ·| o ≤ t , σ i ) thereafter in the presence of Π θ ( ·| o (cid:48)≤ t , σ j ) , denoted by Q ( o ≤ t , a t , σ i , σ j ) =
E Π θ ( ·| o,σ i ) , Π θ ( ·| o (cid:48) ,σ j ) [ (cid:80) t
+ T τ = t
γ τ − t r τ | o ≤ t , a t ] with γ the discount factor.
In the case of ABR by deep RL, we could additionally approximate the expected payoffs matrix U by learning a payoff estimator φ ω ( σ i , σ j ) minimizing the loss
[Equation]
where the expectation is taken over the state visitation distribution D deﬁned by the pair of policies and the environment dynamics P .
In other words, φ ω ( σ i , σ j ) approximates the expected return of Π θ ( ·| o ≤ , σ i ) playing against Π θ ( ·| o (cid:48)≤ , σ j ) .
By connecting payoff matrix U to the learned Q function, we can evaluate U efﬁciently, without explicitly evaluating all policies at each iteration.
Finally, we propose Algorithm 4 in the setting where the meta-graph solver Σ const ← F ( U ) is a constant function and extends it to Algorithm 5 where the meta-graph solver is adaptive in U .
For instance, population learning algorithms such as Fictitious Play (Brown, 1951) implement static interaction graphs while algorithms such as PSRO (Lanctot et al., 2017) rely on adaptive MGS.
E XPERIMENTS
In this section, we validate different contributions of NeuPL across several domains.
First, we verify the generality of NeuPL from two aspects: a) NeuPL recovers expected results of existing population learning algorithms (Brown, 1951; Heinrich et al., 2015; Lanctot et al., 2017) on the classical game of rock-paper-scissors where we can visualize the learned policy population over time and; b) NeuPL generalises to the spatiotemporal, partially observed strategy game of runningwith-scissors (Vezhnevets et al., 2020), where players must infer opponent behaviours through tactical interactions.
Second, we show that NeuPL induces skill transfer across policies, enabling the discovery of exploiters to strong opponents that would have been out-of-reach otherwise.
This property translates to improved efﬁciency and performance compared to PSRO-N ASH baselines, even under favorable conditions.
Lastly, we show that NeuPL scales to the large-scale Game-of-Skills of
MuJoCo Football (Liu et al., 2019) where a concise sequence of best-responses are learned, reﬂecting the prominent transitive skill dimension of the game.
In all experiments, we use Maximum A Posterior Optimization (MPO, Abdolmaleki et al.
as the underlying RL algorithm, though any alternative can be used instead.
Similarly, any conditional
Fictitious Play Strategic Cycle Self-Play
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S R P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S
R
P S 1 2 31 2 1 2 31 2 epoch
R
P S 1 2 31 2 1 2 31 2 epoch
R
P S 1 2 31 2 1 2 31 2 epoch
R
P S 1 2 31 2 1 2 31 2 epoch
R
P S 1 2 31 2 1 2 31 2 epoch
R
P S 1 2 31 2 1 2 31 2 epoch
R
P S 1 2 31 2 1 2 31 2 epoch
R
P S 0 1 strategy − 1 payoff 1 2 31 2 0.
1.0 weight 1 2 31 2 epoch architecture can be used to implement Π Σ θ .
Our speciﬁc proposal reﬂects the spinning-top geometry (Czarnecki et al., 2020) so as to encourage positive transfers across polices.
Further discussions on the network design is available in Appendix B.2.
2. I S N EU PL G ENERAL ?
Static Interaction Graphs Figure 2 illustrates the effect of N EU PL-RL-S TATIC implementing popular population learning algorithms in the purely cyclical game of rock-paper-scissors .
Figure(Left) shows that learning by self-play leads to the policy cycling through the strategy space indefinitely, as expected in such games (Balduzzi et al., 2019).
By contrast, Figure 2 (Middle) shows the effect of a specialized graph that encourages the discovery of a strategic cycle as well as a ﬁnal strategy that trains against the others equally.
As a result, we obtain a population that implements the pure strategies of the game as well as an arbitrary strategy.
This ﬁnal strategy needs not be the Nash of the game as any strategy would achieve a return of zero.
Finally, Figure 2 (Right) recovers the effect of Fictitious Play (Brown, 1951; Heinrich et al., 2015) where players at each iteration optimize against the “average” previous players.
We initialize the initial sink strategy to be exploitable, heavily biased towards playing rock 4 .
The resulting population, represented by { Π θ ( a | o ≤ , σ i ) } 6 i =1 , learned to execute 6 strategies, starting with “pure-rock” which is followed by its best-response “pure-paper”, with subsequent strategies gravitating towards the Nash equilibrium (NE) of this game.
Adaptive Interaction Graphs Figure 3 illustrates that N EU PL-RL-A DAPTIVE with F PSRO-N recovers the expected result of PSRO-N ASH in rock-paper-scissors .
Speciﬁcally, the ﬁrst three strategies gradually form a strategic cycle and converge to the pure strategies of the game.
As the cycle starts to form, the ﬁnal strategy, best-responding to the NE over previous ones, shifted to optimize against a mixture of pure strategies.
These results further highlight an attractive property of
NeuPL — the number of distinct strategies represented by the neural population grows dynamically in accordance with the meta-graph solver (comparing Σ at epoch 0 and 71).
In particular, a distinct objective for strategy i + 1 is introduced if and only if Π θ ( ·| o ≤ , σ i ) gains support in the NE over Π Σ ≤ i θ .
Unlike prior works (Lanctot et al., 2017; Mcaleer et al., 2020), the effective population size is driven by the meta-graph solver, rather than handcrafted truncation criteria.
This is particularly appealing in real-world games, where we cannot efﬁciently determine if a policy has converged to a best-response or temporarily plateaued at local optima.
In NeuPL, one needs to specify the maximum number of policies N to be represented in the neural population, yet the population need not optimize
N distinct objectives at the start, nor is it required to converge to N distinct policies at convergence.
The number of distinct polices represented in the neural population is a function of the strategic complexity of the game, the capacity of the neural network, the effectiveness of the ABR operator and the nature of the meta-graph solver.
We recall this property in running-with-scissors and in MuJoCo
Football, which afford varying degrees of strategic complexities.
Stochastic Games running-with-scissors (Vezhnevets et al., 2020) extends rock-paper-scissors to the spatiotemporal and partially-observed setting.
Using ﬁrst-person observations of the game (a 4x grid in front of the agent), each player collects resources representing “rock”, “paper” and “scissors” so as to counter its opponent’s hidden inventory.
At the end of an episode or when players confront each other through “tagging”, players compare their inventories and receive rewards accordingly.
To do well, one must infer opponent behaviours from its partial observation history o ≤ t — if “rock”s went missing, then the opponent may be collecting them; if the opponent ran past “scissors”, then it may not be interested in it.
We describe the environment in details in Appendix B.1.
Figure 4 shows that NeuPL with F PSRO-N leads to a population of sophisticated policies.
As before, we set the initial sink policy to be exploitable and biased towards picking up “rock”s.
Early in training, we note that the ﬁrst three policies of the neural population implement the pure-resource policies of “rock”, “paper” and “scissors” respectively, as evidenced by their relative payoffs.
In contrast to rock-paper-scissors , the mixture of pure-resource policies is exploitable in the sequential setting, where the player can observe its opponent before implementing a counter strategy.
Indeed, policy Π θ ( ·| o ≤ , σ 4 ) observes and exploits, beating the mixture policies at epoch 680.
Following F PSRO-N , Π θ ( ·| o ≤ , σ 5 ) updates its objective to focus solely on this newfound NE over Π Σ < 5 θ , developing a deceptive counter strategy.
epoch    epoch    epoch    epoch    epoch
[Figure]
[Figure]
1.0 graph  epoch
Figure 5 (Left) quantitatively verify that NeuPL implementing F PSRO-N indeed induces a policy population that becomes robust to adversarial exploits as the population expands.
To this end, we compare independent policy populations by their Relative Population Performance (RPP 5 , Appendix A.1) across 4 independent NeuPL experiments with different maximum population sizes.
As expected, we observe that neural populations representing more best-response iterations are less exploitable.
Additionally, a population size greater than 8 has limited impact on learning, both in terms of marginal exploitability beneﬁts and rate of improvement against smaller populations (shown in blue and orange).
This further mitigate the concern of using a larger maximum population size than necessary.
In fact
[Figure]
Figure 5 (Right) shows that the effective population sizes | U NIQUE (Σ) | plateau at 12 across maximum neural population sizes.
We hypothesise that this is due to the increased difﬁculty in discovering reliable exploiter beyond 12 iterations in this domain — F PSRO-N forms a curriculum where new objectives are introduced if and only if the induced meta-game contains novel meta-game strategies worth exploiting, regardless of the maximum population size speciﬁed.
We emphasize that the only variable across these independent runs is their respective maximum neural population size
[Figure] [Figure]
P OLICIES
[Figure]
In contrast to prior works that train new policies iteratively from scratch
[Figure]
NeuPL represents diverse policies with explicit parameter sharing.
Figure 6 compares the two approaches and illustrates the signiﬁcance of transfer learning across iterations.
Speciﬁcally, we verify that the shared representation learned by training against fewer, weaker policies early in training, facilitates the learning of exploiters to stronger, previously unseen opponents.
To this end, a set of randomly initialized MPO agents with partially transferred parameters from different epochs of the experiment shown in Figure 4 are trained against ﬁxed mixture policies deﬁned by Π Σ ∗ θ ∗ , with θ ∗ and Σ ∗ =
F PSRO-N ( U ∗ ) obtained at epoch 1,200 of the same experiment.
In other words, the objective for each agent is to beat a ﬁxed NE over pre-trained policies { Π θ ∗ ( a | o ≤ , σ ∗ k ) } nk =1 , with a speciﬁc n .
Figure 6 shows the learning progressions of the agents for n ∈ { 2 , 4 , 7 } , with transferred parameters taken at epoch 0 (red) upto 1,000 (green).
Against an easily exploitable opponent mixture (NE over the ﬁrst two pure-resource policies), an agent with randomly initialized parameters (red) remains capable of learning an effective best response, albeit at a slower pace.
This difference becomes much more apparent against competent mixture policies that execute sophisticated strategies (NE over the ﬁrst 4 or 7 policies) — the randomly initialized agent failed to counter its opponent despite prolonged training while agents with
[Figure]
Policy a t g k
Encoder
Memory s ts t o t
Policy a t initialize initialize p a r a m s .
e p o c h
Against
[Figure]
Against
[Figure]
Against Nash with n
[Figure]
partially transferred parameters successfully identiﬁed exploits, leveraging effective representation of the environment dynamics and of diverse opponent strategies.
By transferring skills that support sophisticated strategic decisions across iterations, NeuPL enables the discovery of novel policies that are inaccessible to a randomly initialized agent.
In other words, learning incremental best responses becomes easier, not harder, as the population expands.
This is particularly attractive in games where strategy-agnostic skill learning is challenging in itself (Vinyals et al., 2019; Liu et al., 2021).
2. D OES N EU PL O UTPERFORM C OMPARABLE B ASELINES ?
[Figure]
We compare a NeuPL population implementing F PSRO-N to 4 comparable baselines implementing variants of PSRO.
Since PSRO does not prescribe a truncation criteria at each iteration, we investigate
PSRO baselines with 100k and 200k gradient steps per iteration respectively.
Further, we consider the effect of continued training across iterations, by initializing new policies with the policy obtained at the end of the preceding iteration instead of random initialization.
We refer to this continued variant as PSRO-C. All PSRO populations are initialized with the same initial policy used for the
NeuPL population.
Figure 7 illustrates the quantitative beneﬁts of NeuPL, measuring RPP between a
NeuPL population of maximum population size of 8 against the ﬁnal population of 8 policies obtained via PSRO after 7 iterations.
The vertical dashed lines indicate that both the NeuPL population and the PSRO population have cumulatively undergone identical amount of training to allow for fair comparison.
In purple, we show the effective population size of the NeuPL population which has been shown previously in Figure 5.
We make the following observations: i) with a population size of 8, the NeuPL population successfully exploits PSRO baselines representing an equal number of policies, even if the latter performed twice as many gradient updates; ii) the increase in RPP coincides with an increase in the effective population size, from 5 to 8, reaching the maximum number of distinct policies that this NeuPL population can represent; iii) the amount of training each PSRO generation received has limited impact on the robustness of policy populations at convergence.
This corroborates our observations in Figure 6, where the agent (in red) failed to exploit strong opponents despite continued training.
Interestingly, PSRO-C proves equally exploitable.
We hypothesize that the learned policies failed to develop reusable representations that can support diverse strategic decisions.
Details of the PSRO baselines across 3 seeds are available in Appendix B.4, demonstrating the strategic complexities captured by the PSRO baseline populations.
2. D OES N EU PL S CALE TO H IGHLY T RANSITIVE G AME - OF -S KILLS ?
If a game is purely transitive, all policies share the same best-response policy.
In this case, self-play offers a natural curriculum that efﬁciently converges to this best-response (Balduzzi et al., 2019).
Nevertheless, this approach is infeasible in real-world games as one cannot rule out strategic cycles in the game without exhaustive policy search.
The MuJoCo Football domain (Liu et al., 2019) is one such example: it challenges agents to continuously reﬁne their motor control skills while coordinated team-play intuitively suggests the presence of strategic cycles.
In such games, PSRO is a challenging proposal as it requires carefully designed truncation criteria.
If an iteration terminates prematurely due to temporary plateaus in performance, good- responses are introduced and convergence is slowed unnecessarily; if iterations rarely terminate, then the population may unduly delay the representation of strategic cycles.
In such games, NeuPL offers an attractive proposal that retains the ability to capture strategic cycles, but also falls back to self-play if the game appears transitive.
Figure 8 shows the learning progression of NeuPL implementing F PSRO-N in this domain, starting with a sink policy that exerts zero torque on all actuators.
We observe that Π θ ( ·| o ≤ , σ 2 ) exploits the sink policy by making rapid, long-range shots which is in turn countered by Π θ ( ·| o ≤ , σ 3 ) , that intercepts shots and coordinates as a team to score.
Impressively, the off-ball blue player learned to blocked off defenders, creating scoring opportunities for its teammate.
With the interaction graph focused on the lower diagonal elements, this training regimes closely matches that of self-play.
2.0e+      1.6e+      3.1e+      4.7e+ training epoch p
[Figure]
s g
r a ph s
R ELATED W ORK
Prior works attempted at making population learning scalable, motivated by similar concerns as ours.
Mcaleer et al.
(2020) recognized the sequential nature of PSRO as a limiting factor to scaling and proposed pipeline PSRO (P2SRO) which learns iterative best-responses concurrently in a staggered, hierarchical fashion.
P2SRO offers a principled way to make use of additional computation resources while retaining the convergence guarantee of PSRO.
Nevertheless, it does not induce more efﬁcient learning per unit of computation cost, with basic skills re-learned at each iteration albeit asynchronously.
In contrast, Smith et al.
(2020a) focused on the lack of transfer learning across best-response iterations and proposed “Mixed-Oracles” where knowledge acquired over previous iterations is accumulated via an ensemble of policies.
In this setting, each policy is trained to best-respond to a meta-game pure strategy, rather than a mixture strategy as suggested by the meta-strategy solver.
To approximately re-construct a best-response to the desired mixture strategy, a specialized RL algorithm ( Q-mixing , Smith et al.
(2020b))
that re-weights expert policies at each step is used.
Compared to NeuPL, where Bayes-optimal RL objectives are retained via a form of meta-learning (Ortega et al., 2019), “Mixed-Oracles” proposes a mixture-of-expert system that encourages the learning of specialised exploiter strategies to each opponent. While the implication of this discrepancy in the context of population learning remains to be further explored, it’s worth noting that “Mixed-Oracles” can be suitably expressed through NeuPL, too.
This can be achieved with a lower-diagonal interaction graph where Q-mixing is used to construct suitable best-responses to meta-game mixture strategies, offering a computationally efﬁcient implementation of “MixedOracles”.
Finally, we note that although we primarily focused on grounded interaction graphs for which we can construct convergence proofs, NeuPL additionally enables efﬁcient population learning with “un-grounded” interaction graphs, beyond the scope of prior works.
C ONCLUSION AND F UTURE W ORK
We proposed an efﬁcient, general and principled framework that learns and represents strategically diverse policies in real-world games within a single conditional-model, making progress towards scalable policy space exploration.
In addition to exploring suitable technique from the multi-task, continual learning literature, going beyond the symmetric zero-sum setting remain interesting future works, too, as discussed in Appendix D.
A M ETHODS A. P OPULATION L EARNING A LGORITHMS AS I NTERACTION G RAPHS
Figure 1 shows several population learning algorithms represented as directed interaction graphs (Garnelo et al., 2021).
In particular, Policy Space Response Oracle with a Nash meta-game solver (PSRO-N ASH ) (Lanctot et al., 2017) proposes to learn a best-response π i to the Nash equilibrium over policies
Π <i at each iteration, resulting in an adaptive interaction graph.
Speciﬁcally, the out-edges of node i + 1 are weighted according to the Nash equilibrium over the ﬁrst i policies (e.g. shown as (1 − p, p ) for node 3 in Figure 1).
A. R ELATIVE P OPULATION P ERFORMANCE
Relative population performance was ﬁrst introduced in Balduzzi et al.
(2019) as the performance of individual agents is meaningless in purely cyclical games.
Intuitively, a relative population performance measure of v ( B , D ) implies that there exists a mixture policy in the population B that achieves a payoff at least v ( B , D ) against all mixture policies in the opponent population D .
This measure is deﬁned formally in Deﬁnition A.1.
Deﬁnition A.1 (Relative Population Performance) .
Given two populations of policies B , D and let ( p , q ) be a Nash equilibrium over the zero-sum game on U B , D ∈ R M × N , Relative Population
Performance measures their relative performance: v ( B , D ) :
= p T · U B , D · q .
A. N EU PL WITH S TATIC I NTERACTION G RAPH AND ABR BY RL
In this section, we describe a specialised implementation of NeuPL given static meta-graph solvers
F with an approximate best-response operator implemented via deep Reinforcement Learning.
The algorithm is described in Algorithm 4.
1: function R UN -E PISODES ( Σ , Π θ , M ) (cid:46) Visualized in Figure 10 (Right).
D Π ← {} ; D Q ← {} (cid:46) Initialize actor, critic trajectory buffers.
3: for Episode ∈ { 1 , . . .
, M } do
Generate trajectories for M episodes.
Sample
NIQUE uniformly
Sample trajectories T σ i , T j from interactions between (Π σ i
, Π θ
D
← D Π ∪ T
i ; D Q ← D Q ∪ T σ i ∪ T σ j .
7: return D Π , D Q 8:9: function N EU PL-RL-S TATIC ( Σ , T ) (cid:46) Σ ∈ R N × N the static interaction graph.
Initialize Π θ ( a | o, σ i ) , the task-conditioned policy.
Initialize Q
θ ( o, a, σ i , σ j ) , the action-value function.
12: for Iteration t ∈ { 1 , . .
, T } do (cid:46) T the number of iterations to run for.
D Π , D Q ← R UN -E PISODES (Σ , Π θ , M ) 14:
Optimize policy Π θ from D Π and Q θ from D Q by RL.
A. N EU PL WITH A DAPTIVE M ETA -G RAPH S OLVERS AND ABR BY RL
Building on Algorithm 4, we now extend to the case of adaptive meta-graph solvers F that are functions of empirical payoff matrices U .
Speciﬁcally, the set of meta-game strategies the set of policies seek to best-respond to are given by Σ ← F ( U ) .
This algorithm is described in Algorithm 5.
B E XPERIMENTS B. R UNNING - WITH -S CISSORS ENVIRONMENT
Figure 9 shows an example view of the running-with-scissors environment at the start of an episode.
The dashed squares shows the type of resources initialized in the enclosed area with some of them consistently initialized with one type of resources while others randomly initialized with one of the
1: function N EU PL-RL-A DAPTIVE ( F , K, T, M, (cid:15) )
(cid:46) F : R N × N → R N × N .
Initialize Π θ ( a | o ≤ , σ i ) , the neural population network.
Initialize Q θ ( o ≤ , a, σ i , σ j ) , the action-value function.
Initialize φ ω ( σ i , σ j ) , the empirical payoff estimator.
Let Σ c ∈ R N × N ← 1 / N .
(cid:46) All-to-all interactions.
6: for Epoch ∈ { 1 , . . .
, K } do (cid:46) K epochs to run.
7: ∀ i, j : U ij ← φ ω ( σ i , σ j ) the payoff matrix over N policies.
(cid:46) Re-compute payoff estimates.
8: Σ ∈ R N × N ← F ( U ) the interaction graph over N policies.
(cid:46) Update the interaction graph.
9: for Iteration t ∈ { 1 , . . .
, T } do (cid:46) T iterations per epoch.
D Π , D Q ← R UN -E PISODES (Σ , π θ , M × (1 − (cid:15) ))
(cid:46) M episodes per iteration.
11: _ , D (cid:15)Q ← R UN -E PISODES (Σ c , π θ , M × (cid:15) )
(cid:46) (cid:15) the prop.
of evaluation episodes.
Optimize policy π θ from D Π and Q θ from D Q ∪ D (cid:15)Q by RL. 13:
Optimize φ ω from D Q ∪ D (cid:15)Q by minimizing L ij .
[Figure]
the right, we visualize the 4x4 pixel observations of the two players.
The visual observation is oriented along each player’s forward direction.
In addition to the visual observations, each player observes their current inventory of the three types of resources, expressed in terms of their normalized ratios.
Each player is initialized with an equal weight inventory at the start of an episode.
To move around, a player can turn left, turn right, strafe left, strafe right, move forward or move backwards.
Finally, each player can proactively seek out its opponent and “tag” it to terminate the game.
A player is considered tagged if it falls into the tagging area of its opponent.
Bottom right view illustrates the shape of the tagging area in front of a player.
If neither player tags its opponent, the game ends after a ﬁxed number of 500 steps.
On the terminal step, the game resolves by comparing the inventory of both players and the rewards are assigned according to the classical anti-symmetric payoff matrix of rock-paper-scissors .
On all other steps, both players receive zero rewards.
B. C ONDITIONAL N ETWORK A RCHITECTURE FOR N EU PL
Figure 10 (Left) illustrates the general network architecture of a NeuPL population for a typical
Q-learning based RL agent.
In particular, the encoder, memory and policy head network modules are shared across all policies within the neural population with the conditioning variable σ introduced at the ﬁnal policy head layer via concatenation.
This reﬂects the hypothetical Game-of-Skills geometry proposed in Czarnecki et al.
(2020), where each policy can be understood as a point within a “spinning-top” volume.
Each policy is interpreted as a combination of strategy-agnostic transitive skills (e.g. movement skills for embodied agents; representational capability of past observations in partially-observed games) and a cyclic strategic element that decides which mixture policy to best-respond to, leveraging its transitive skills.
At a high-level, the goal of NeuPL is to represent a compact set of policies that corresponds to the top layer of this “spinning-top” geometry, or the NE of the game.
As such, our proposed conditional network architecture facilitates the sharing of strategy-agnostic transitive skills across policies, while isolating strategy-speciﬁc decisions that may call for different actions in the same state within the
NeuPL
Encoder
Memory h
th t
Q Π θ ( a |o ≤ , σ ) + + q t σ i σ j o t a t
Interaction graph Σ σ 1 … σ k … σ N
Self-Play Episode Π
θ (a|o, σ i )
Π
θ (a ’ |o ’ , σ j )
Env.
( σ i , σ j ) (a, a ’ )
(o, o ’ )
[Figure]
at
[Figure]
ﬁnal opponent-conditioned policy head network, mitigating the concern of negative transfer.
We note that while we limited ourselves to the simplest conditioning architecture in this work, investigating conditional network architecture with suitable inductive biases could be an interesting future direction.
B. H YPER -P ARAMETERS & T RAINING I NFRASTRUCTURE
For experiments on the normal form game of rock-paper-scissors , the “Encoder” and “Memory” components are omitted and the policy head network π θ alone is parameterized and learned, with its only input g .
The policy and action-value networks are both parameterized by 4-layer MLPs, withneurons at each layer.
We use a small entropy cost of 0 .
01 , learning rates of 0 .
001 and 0 .
01 for the main networks and the MPO dual variables (Abdolmaleki et al., 2018) respectively.
As is typical in an MPO agent, the online network parameters are copied to target networks periodically at every 10 gradient steps.
For the spatiotemporal, partially observable game of running-with-scissors , we use a small convolutional network to encode the agent’s partial observation of the environment (a 4x4 grid surrounding itself).
The encoding is further concatenated with the agent’s own inventory information and encoded through a 2-layer MLP with 256 neurons each, with relu activation.
The memory module corresponds to a recurrent LSTM network, with a hidden size of 256.
The policy, action-value networks are parameterized as 4-layer MLP networks.
The learned payoff estimator φ ω ( σ i , σ j ) is parameterized as a 3-layer MLP network.
The learning rate of the agent networks is set to 0 .
0001 while the MPO dual variables are optimized with a learning rate of 0 .
The online network parameters are copied to target networks every 100 gradient steps.
For the MuJoCo Football environment, we used a domain speciﬁc encoder network that encodes egocentric observations of each player individually.
The state representation is then implemented as a weighted sum of per-player embedding, using a learned attention mask.
Instead of outputting a discrete categorical action proﬁle in each state, the policy head network is trained to output a Gaussian distribution with learned mean and variance distribution parameters.
As is common in continuous control literature, we used elu as the activation function between network components.
The rest of the network architecture and hyper-parameters remain consistent with that of running-with-scissors .
The same network architecture and hyper-parameters are used across all experiments for the same environment.
Figure 10 (Right) illustrates how self-play experience data is generated for a NeuPL population at training time.
At the start of an episode, a pair of conditioning vectors is sampled ( σ i , σ j ) and used to condition the pair of policies interacting in the game.
In running-with-scissors , each NeuPL experiment uses 128 actor workers running the policy environment interaction loops and a single
TPU-v2 chip running gradient updates to the agent networks.
The same computational resources are
[Figure]
PSRO
[Figure]
[Figure] [Figure]
used across different maximum population size for NeuPL as well as the PSRO baseline experiments.
For MuJoCo Football, 256 CPU actors are used per learner.
For the game of rock-paper-scissors , a single CPU worker is used instead.
In running-with-scissors , all experiments converge within 2 days wall-clock time.
For all experiments using NeuPL, an evaluation split (cid:15) = 0 .
3 is used.
B. B ASELINE PSRO EXPERIMENTS
Figure 11 visualizes the empirical payoff matrices obtained in 12 independent experiments.
In each experiment, we develop a discrete population of policies via 8 iterations of PSRO training.
In particular, we show the payoff matrices between trained policies where each PSRO iteration lasts 100k or 200k gradient steps.
PSRO-C further shows the effect of initializing each iteration by the policy obtained at the end of the preceding iteration instead of random initialization as prescribed in (Lanctot et al., 2017).
We note that across all experiments, policy populations successfully recover the rock-paper-scissors dynamics among the ﬁrst three policies.
In most but not all trails, PSRO also manages to learn a reliable exploiter of the mixture of the three pure strategies.
Finally, we note that continued training appears to facilitate the discovery of richer strategic cycles, and so does allowing each PSRO iteration to perform more gradient updates.
B. S ENSITIVITY TO THE CHOICE OF H YPER -P ARAMETERS
Figure 12 investigates the sensitivity of NeuPL to the choice of hyper-parameters.
In particular,
NeuPL introduces two additional hyper-parameters: the proportion of simulation episodes used for learning pairwise empirical payoffs ( σ in Algorithm 5) and the interval between interaction graph update ( T in Algorithm 5).
Intuitively, if (cid:15) is too low, the empirical payoff estimator risk being under-trained when its estimations are used by the meta-graph solver while a value too high would delay the policy learning due to insufﬁcient simulation data being generated for policy learning.
When it comes to the meta-graph update interval, a value too high may slow down strategic exploration if the approximate best-response operators have already converged to best-responses while a value too low may lead to noisy gradients in the optimization process, as the set of learning objectives may change too frequently.
[Figure]
In practice
[Figure]
we observe that NeuPL is reasonably robust to these hyper-parameters across a wide range of choices.
Across all experiments, we evaluate the NeuPL population’s relative population performance against a ﬁxed, held-out population obtained via PSRO-C@200k, seed = 1 .
B. T EST - TIME E XECUTION OF L EARNED P OLICIES
A distinctive property of population learning is that we obtain a population of policies that can be executed individually at test time.
In NeuPL, the population of policies is jointly deﬁned by two elements: a conditional network Π θ ( ·| o ≤ t , σ ) and a set of meta-game strategies Σ = { σ i } Ni =1 , derived from the empirical payoffs between policies.
To execute a mixture policy deﬁned over the population, it sufﬁce to execute Π θ ( ·| o ≤ t , ˆ σ ) with ˆ σ ∼ Pr (Σ) .
Speciﬁc mixture policies have well-understood properties.
For instance, playing the NE mixture policy implies that an opponent who has access to the same set of policies is indifferent to playing any mixtures.
This could be a principled option in the absence of prior knowledge about one’s opponent.
C C ONVERGENCE P ROOFS
In this section we detail the conditions upon which NeuPL will converge to a unique sets of policies.
First, we discuss the theory when the interaction graph is static and grounded (Section C.1).
Then we discuss the theory when the interaction graph is a function of the payoff matrix, a so-called meta-graph solver (Section C.2).
Finally we discuss a speciﬁc function, Nash Equilibrium meta-graph solver, that is popular in the literature, and prove NeuPL’s convergence to a normal-form Nash Equilibrium under certain conditions (Section C.3).
C. G ROUNDED I NTERACTION G RAPHS
Assumption C.1 (Unique, Exact, Finite Best Response) .
Assume that we have access to a best response (BR) operator that responds to a policy, π n , (or equivalently a mixture over policies) exactly converges to a unique best response, π n +1 =
BR ( π n ) , in ﬁnite time.
When using RL as the BR operator, uniqueness can be approximated with a small entropy term on policy being optimized.
Therefore, in the situation where there is more than one best response, the
BR operator will opt for the one with maximum entropy.
This is a common additional loss that is often added to RL agents, and is also believed to aid exploration (Haarnoja et al., 2018).
The second assumption, that the BR will be exact, is idealised.
With sufﬁcient model capacity, enough time, and appropriate hyper-parameter annealing schedules, RL can get close to an exact BR.
The ﬁnal assumption, that the BR operator converges in ﬁnite time, is assumed because proofs follow easily from this assumption.
It may be possible to relax this assumption to converge as t → ∞ , but we leave this for future work.
Deﬁnition C.1 (Grounded Interaction Graph) .
An interaction graph’s edges deﬁne how policies should best respond to each other.
We call an interaction graph grounded if its structure imposes convergence to a unique set of policies, where each policy is a unique, exact, ﬁnite BR over opponent policies deﬁned by the interaction graph.
Theorem 1 (Grounded Lower Triangular) .
All lower-triangular interaction graphs ( Σ i ≥ j = 0 ) are grounded.
Proof.
The ﬁrst row of a lower triangular matrix is all zeros meaning that the ﬁrst policy does not respond to any other policies.
The ﬁrst policy is therefore static (and arbitrary) and does not change over time.
The second row only responds to the ﬁrst policy.
Since the ﬁrst policy is static and we are using a unique, exact and ﬁnite BR, the second policy will converge exactly to a unique policy.
The n th row will respond to the n − 1 previous policies.
Since over time all previous policies will converge to uniquely, so will the n th policy.
It is easy to imagine other grounded interaction graphs with interesting structure, however we will only focus on lower-triangular graphs for the purpose of this section.
Research into other grounded interaction graphs may be interesting future work.
Deﬁnition C.2 (Static Interaction Graph) .
We call an interaction graph static if it does not change over time: Σ t = Σ .
We now prove our ﬁrst NeuPL result, that it can ﬁnd an exact N -step best response under certain assumptions.
Theorem 2 (NeuPL Static Lower-Triangular Exact N -Step Best Response) .
NeuPL with a static, lower triangular interaction graph, an arbitrary ﬁxed initial policy, π 0 , and N more policies, will converge to an exact N -step best response, assuming that the BR is unique, exact, and converges in ﬁnite time.
Proof.
The bulk of this proof can recycle the arguments in Theorem 1.
Note that NeuPL trains its conditional policies in parallel according to an interaction graph.
If that graph is lower-triangular, it is grounded, and NeuPL will ﬁnd the conditioned policies corresponding to the N -step best response.
C. G ROUNDED M ETA -G RAPH S OLVER
It is possible to extend this result on any deterministic function acting on a sub-payoff, producing a lower triangular interaction graph.
Deﬁnition C.3 (Grounded Meta-Graph Solver) .
A meta-graph solver is a function that takes the payoff matrix as an argument and outputs an interaction graph as an output.
We call a meta-graph solver, F , grounded if, when using unique, exact, ﬁnite BRs, it converges to a grounded interaction graph in ﬁnite time.
Theorem 3 (Any Deterministic Lower-Triangular Meta-Graph Solver is Grounded) .
Any deterministic function, F , that maps a sub-payoff , U <i,<i , to a row, Σ i,<i , in a lower-triangular interaction graph is a grounded meta-graph solver.
[Equation]
Proof.
The difﬁculty here is that in general, as the policies change, so will the payoff, and hence so will interaction graph, etc.
Similar to the arguments in Theorem 1 the ﬁrst policy is static and does not change.
The second policy can only respond to the ﬁrst policy (under the lower-triangular constraint), so the meta-graph solver has no ﬂexibility to do otherwise, and therefore the second policy will also converge to a unique result.
Note that as these policies converge, so will their sub-payoffs, U < 3 ,< 3 .
Therefore, for row n any deterministic mapping F : U <i,<i → Σ i,<i will result in a unique set of policies.
We can now make a further claim about NeuPL: that it will converge to a unique set of conditioned policies under certain grounded meta-graph solvers.
Theorem 4 (NeuPL Deterministic Lower-Triangular Exact N -Step Best Response) .
Assuming any deterministic lower-triangular meta-graph solver, an arbitrary ﬁxed initial policy, π 0 , and N more policies, NeuPL will converge to an exact N -step best response, assuming a unique, exact and ﬁnite
BR.
Proof.
Similar in structure to Theorem 2 and Theorem 3.
Of course, the more general result, that NeuPL will also converge for any grounded meta-graph solver is also true.
C. N ASH E QUILIBRIUM M ETA -G RAPH S OLVER
Note that the maximum entropy Nash Equilibrium (MENE) is such as grounded meta-graph solver.
This will result in an algorithm similar to PSRO-N ASH (Lanctot et al., 2017) or Double Oracle (McMahan et al., 2003).
Theorem 5 (NeuPL Nash Exact N -Step Best Response) .
Using an interaction-graph function that maps payoff to a lower triangular NE distribution:
[Equation]
With a such a function, an arbitrary ﬁxed initial policy, and sufﬁciently large N policies, NeuPL will converge to an exact normal-form Nash Equilibrium (NE), assuming a unique, exact, ﬁnite BR.
Proof.
Starting from the proof in Theorem 4, we follow additional arguments from DO (McMahan et al., 2003) to prove that for sufﬁcient N we will converge to a normal form NE.
Of course, many other algorithms can be recovered using speciﬁc meta-graph solvers (Muller et al., 2020; Marris et al., 2021).
D G ENERALITY OF N EU PL AND D ISCUSSION ON F UTURE W ORKS
For simplicity of presentation, we focused on the simple setting of learning from scratch in symmetric zero-sum games.
In this section, we discuss elements needed towards applying NeuPL more generally from several aspects.
D. I NCORPORATING P RIOR K NOWLEDGE IN N EU PL
As we alluded to in Section 1.2, the formulation of NeuPL offers a principled way to incorporate prior knowledge in the form of pre-trained policies.
In short, pre-trained policies can be incorporated in the same way as the sink policy, with the requirement that it can only gain in-edges in the interaction graph.
Figure 13 offers an illustration of such an example, where the population includes 2 pre-trained policies ( Π θ ( ·| o ≤ t , σ 1 ) and Π θ ( ·| o ≤ t , σ 2 ) ) while Π θ ( ·| o ≤ t , σ 3 ) is optimized to best-respond to a mixture over its predecessors according to a suitable meta-graph solver.
As an implementation details due to the use of neural networks, σ 1 and σ 2 need to be unique vectors and they need to be treated speciﬁcally in the corresponding MGS as they no longer represent valid probability distribution and should be excluded from policy training, similar to the treatment of the sink policy.
Finally, we note that our proposed conditional network architecture is in fact synergistic with the use of pre-trained policies, beyond naively including pre-trained expert policies as opponents in
1 − p p -    -  p 1 − p    the population.
This is because the shared action-value function Q θ ( o ≤ t , σ i , σ j ) would learn to predict expected returns between ﬁxed pre-trained expert policies, kick-starting the learning of encoder and memory components in the underlying game.
As proposed in Figure 10, this kick-started representation learning process should in principle transfer to the learning of other policies within the neural population as well, rendering it an attractive proposal.
D. G ENERALIZATION TO N- PLAYER G ENERAL - SUM G AMES
NeuPL can in principle extend to n-player general-sum games.
We offer elements of the solution in this section while deferring thorough investigation in this direction to dedicated future work.
Consider an n -player general-sum game where the i -th player can play one of M i policies, the induced meta-game corresponds to a normal-form game between n players, each selecting a policy to execute for an episode for one player.
This yields n empirical payoff tensors
U
=
{U i } ni =1 with
U i ∈
R M 1 × M 2 ···× M n the payoff tensor for player i , given all players’ policy selections.
Coarse Correlated Equilibrium (CCE): in this setting, solving for NE becomes intractable but solution concepts such as CCE could be readily used instead (Marris et al., 2021).
This leads to a solver that takes on the form P ← F ( U ) with P ∈ R M 1 × M 2 ···× M n a joint-distribution over the cartesian product of policy choices across n players.
This solver can thus be used in place of the meta-strategy solver, similarly and repeatedly invoked by the meta-graph solver.
Note that instead of obtaining a marginal distribution for a given player as in NE, CCE offers the joint distribution which can be marginalized for each player.
Heterogeneous Neural Populations: in the n-player setting, different players may take on different roles in the game and admit entirely different observation, action spaces.
This implies that heterogeneous neural populations are needed.
Speciﬁcally, each player’s policies can be represented by its own neural population of maximum capacity M i and its own n -dimensional marginal interaction tensor .
The i -th neural population is deﬁned by a conditional model Π iθ ( ·| o ≤ t , σ i ) and a marginalized interaction tensor Σ i ∈ R M 1 × M 2 ···× M n = { σ ik } M
i
k
=1 , derived from incremental CCE joint distributions.