Original: Turbocharging Solution Concepts: Solving NEs, CEs and CCEs with Neural Equilibrium Solvers
Revised: Turbocharging Solution Concepts: Solving NEs, CEs and CCEs with Neural Equilibrium Solvers

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Solution concepts such as Nash Equilibria, Correlated Equilibria, and Coarse
Revised: Solution concepts such as Nash Equilibria, Correlated Equilibria, and Coarse

Original: Correlated Equilibria are useful components for many multiagent machine learningalgorithms.
Revised: Correlated Equilibria are useful components for many multiagent machine learningalgorithms.

Original: Unfortunately, solving a normal-form game could take prohibitiveor non-deterministic time to converge, and could fail.
Revised: Unfortunately, solving a normal-form game could take prohibitiveor non-deterministic time to converge, and could fail.

Original: We introduce the Neural
Revised: We introduce the Neural

Original: Equilibrium Solver which utilizes a special equivariant neural network architectureto approximately solvethe space of all games of ﬁxed shape, buying speed anddeterminism.
Revised: Equilibrium Solver which utilizes a special equivariant neural network architectureto approximately solve all games of ﬁxed shape, buying speed and determinism.

Original: We deﬁne a ﬂexible equilibrium selection framework, that is capableof uniquely selecting an equilibrium that minimizes relative entropy, or maximizeswelfare.
Revised: We deﬁne a ﬂexible equilibrium selection framework, that is capable of uniquelyselecting an equilibrium that minimizes relative entropy, or maximizes welfare.

Original: The network is trained without needing to generate any supervised trainingdata.
Revised: The network is trained without needing to generate any supervised training data.

Original: We show remarkable zero-shot generalization to larger games.

Original: We argue thatsuch a network is a powerful module for many possible multiagent algorithms.
Revised: We argue that such a network is a powerful module for many possible multiagentalgorithms.

Original:  Introduction
Revised:  Introduction

Original: Solution concepts such as Nash Equilibrium (NE) [41], Correlated Equilibrium (CE) [3], and Coarse
Revised: Solution concepts such as Nash Equilibrium (NE) [28], Correlated Equilibrium (CE) [3], and Coarse

Original: Correlated Equilibrium (CCE) [40] are useful components for many machine learning algorithms [19].
Revised: Correlated Equilibrium (CCE) [27] are useful components for many machine learning algorithms [13].

Original: They are used in Double Oracle [39], PSRO [33], and JPSRO [36] to determine what distributions ofco-players agents should train with.
Revised: They are used in Double Oracle [26], PSRO [21], and JPSRO [23] to determine what distributions ofco-players agents should train with.

Original: Furthermore equilibria have been critical in driving agents tosuperhuman performance in Go [49], Chess [50], and StarCraft [54].
Revised: Furthermore equilibria have been critical in driving agents tosuperhuman performance in Go [36], Chess [37], and StarCraft [41].

Original: Unfortunately, solving for anequilibrium of a game can be computationally complex [9, 8], and often requires deploying iterativesolvers, which can scale but may (i) take an unpredictable amount of time to converge, (ii) take aprohibitive amount of time to do so, and (iii) may fail unpredictably on ill-conditioned problems.
Revised: Unfortunately, solving for anequilibrium of a game can be computationally complex [7, 6], and often requires deploying iterativesolvers, which can scale but may (i) take a non-deterministic amount of time to converge, (ii) takea prohibitive amount of time to do so, and (iii) may fail unpredictably on ill-conditioned problems.

Original: Furthermore, classical methods [16, 34, 38] (i) do not scale, and (ii) are non-differentiable.
Revised: Furthermore, classical methods [12, 22, 25] (i) do not scale, and (ii) are non-differentiable.

Original: Thislimits the applicability of equilibrium solution concepts in multiagent machine learning algorithms.
Revised: Thislimits the applicability of equilibrium solution concepts in multiagent machine learning algorithms.

Original: We propose the Neural Equilibirum Solver (NES), a neural network that approximately solves forsuch solutions, allowing equilibria to be found quickly and in a deterministic amount of time.
Revised: We propose the Neural Equilibirum Solver (NES), a neural network that approximately solves forsuch solutions, allowing equilibria to be found quickly and in a deterministic amount of time.

Original: Thisnetwork is trained to optimize a composite objective function that weights accuracy of the returnedequilibrium against auxiliary objectives that a user may desire such as maximum entropy, maximumwelfare, or minimum distance to some target distribution.
Revised: Thisnetwork is trained to optimize a composite objective function that weights accuracy of the returnedequilibrium against auxiliary objectives that a user may desire such as maximum entropy, maximumwelfare, or minimum distance to some target distribution.

Original: We introduce several innovations into thedesign and training of NES so that it is efﬁcient and accurate.
Revised: We introduce several innovations into thedesign and training of NES so that it is efﬁcient and accurate.

Original: Unlike most supervised deep learningmodels, NES avoids the need for explicitly constructing a labeled dataset of (game, equilibrium)pairs.
Revised: Unlike most supervised deep learningmodels, NES avoids the need for explicitly constructing a labeled dataset of (game, equilibrium)pairs.

Original: Instead we derive a loss function that can be minimized in an unsupervised fashion from onlygame inputs.
Revised: Instead we derive a loss function that can be minimized in an unsupervised fashion from onlygame inputs.

Original: We also exploit the duality of the equilibrium problem.
Revised: We also exploit the duality of the equilibrium problem.

Original: Instead of solving for equilibriain the primal space, we train NES to solve for them in the dual space, which has a much lowerdimensional representation.
Revised: Instead of solving for equilibriain the primal space, we train NES to solve for them in the dual space, which has a much lowerdimensional representation.

Original: We utilize a training distribution that efﬁciently represents the spaceof all normal-form games of a desired shape and use an invariant preprocessing step to map gamesat test time to this space.
Revised: We utilize a training distribution that efﬁciently represents the spaceof all normal-form games of a desired shape and use an invariant preprocessing step to map gamesat test time to this space.

Original: In terms of the network architecture, we design a series of layers that areequivariant to symmetries in games such as permutations of players and strategies, which reduces the number of training steps and improves generalization performance.The network architecture isindependent of the number of strategies in the game and we show interesting zero-shot generalizationto larger games.
Revised: In terms of the network architecture, we design a series of layers that areequivariant to symmetries in games such as permutations of players and strategies, which reduces the number of training steps and improves generalization performance.

Original: This network can either be pretrained before being deployed, trained online alongsideanother machine learning algorithm, or a mixture of both.
Revised: This network can either bepretrained before being deployed, trained online alongside another machine learning algorithm, or amixture of both.

Original:  Preliminaries
Revised:  Preliminaries

Original: Game Theory Game theory is the study of the interactive behaviour of rational payoff maximizingagents in the presence of other agents.
Revised: Game Theory Game theory is the study of the interactive behaviour of rational payoff maximizingagents in the presence of other agents.

Original: The environment that the agents operate in is called a game.
Revised: The environment that the agents operate in is called a game.

Original: We focus on a particular type of single-shot, simultaneous move game called a normal-form game.
Revised: We focus on a particular type of single-shot, simultaneous move game called a normal-form game.

Original: A normal-form game consists of N players, a set of strategies available to each player, a p 2 A p ,and a payoff for each player under a particular joint action, G p ( a ) , where a = ( a 1 , ..., a N ) =( a p , a  p ) 2 A
Revised: A normal-form game consists of N players, a set of strategies available to each player, a p 2 A p ,and a payoff for each player under a particular joint action, G p ( a ) , where a = ( a 1 , ..., a N ) =( a p , a  p ) 2 A

Original: = ⌦ p
Revised: = ⌦ p

Original: A p .
Revised: A p .

Original: The subscript notation  p is used to mean “all players apart from player  p ”.
Revised: The subscript notation  p is used to mean “all players apart from player 46 p ”.

Original: 47 Games are sometimes referred to by their shape, for example: | A 1 | ⇥ ... ⇥ | A N | .

Original: The distribution48 of play is described by a joint  ( a ) .
Revised: The distribution of play is described by a joint  ( a ) .

Original: The goal of each player is to maximize their expected payoff,
Revised: The goal of each player is to maximizetheir expected payoff,

Original: P a 2 A  ( a ) G p ( a ) .
Revised: P a 2 A  ( a ) G p ( a ) .

Original: Players could play independently by selecting a strategy according to theirmarginal  ( a p ) over joint strategies, such that  ( a ) =
Revised: Players could play independently by selecting a strategyaccording to their marginal  ( a p ) over joint strategies, such that  ( a ) =

Original: ⌦ p  ( a p ) .
Revised: ⌦ p  ( a p ) .

Original: However this is limiting becauseit does not allow players to coordinate.
Revised: However thisis limiting because it does not allow players to coordinate.

Original: A mediator called a correlation device could be employedto allow players to execute arbitrary joint strategies  ( a ) that do not necessarily factorize into theirmarginals.
Revised: A mediator called a correlation devicecould be employed to allow players to execute arbitrary joint strategies  ( a ) that do not necessarilyfactorize into their marginals.

Original: Such a mediator would sample from a publicly known joint  ( a ) and secretly communicateto each player their recommended strategy.
Revised: Such a mediator would sample from a publicly known joint  ( a ) andsecretly communicate to each player their recommended strategy.

Original: Game theory is most developed in a subset of games:those with two players and a restriction on the payoffs, G 1 ( a 1 , a 2 )
Revised: Game theory is most developed in asubset of games: those with two players and a restriction on the payoffs, G 1 ( a 1 , a 2 )

Original: =
Revised: =

Original:  G 2 ( a 1 , a 2 ) , known as zerosum.
Revised:  G 2 ( a 1 , a 2 ) ,known as zero-sum.

Original: Particularly in N-player, general-sum games, it is difﬁcult to deﬁne a single criterion to ﬁndsolutions to games.
Revised: Particularly in N-player, general-sum games, it is difﬁcult to deﬁne a singlecriterion to ﬁnd solutions to games.

Original: One approach is instead to consider joints that are in equilibrium: distributionssuch that no player has incentive to unilaterally deviate from a recommendation.
Revised: One approach is instead to consider joints that are in equilibrium:distributions such that no player has incentive to unilaterally deviate from a recommendation.

Original: Equilibrium Solution Concepts Correlated Equilibria (CEs) [3] can be deﬁned in terms of linearinequality constraints.
Revised: Equilibrium Solution Concepts Correlated Equilibria (CEs) [3] can be deﬁned in terms of linearinequality constraints.

Original: The deviation gain of a player is the change in payoff the player achieveswhen deviating to action a 0 p from a recommended action a 00 p , when the other players play a  p .
Revised: The deviation gain of a player is the change in payoff the player achieveswhen deviating to action a 0 p from a recommended action a 00 p , when the other players play a  p .

Original: [Equation]
Revised: [Equation]

Original: otherwise
Revised: otherwise

Original: [Equation]
Revised: [Equation]

Original: [Equation] A distribution,  ( a ) , is in ✏ -CE if the deviation gain is no more than some constant ✏ p  ✏ for everypair of recommendation, a 00 p , and deviation strategies, a 0 p , for every player, p .
Revised: A distribution,  ( a ) , is in ✏ -CE if the deviation gain is no more than some constant ✏ p  ✏ for everypair of recommendation, a 00 p , and deviation strategies, a 0 p , for every player, p .

Original: These linear constraintsgeometrically form a convex polytope of feasible solutions.
Revised: These linear constraintsgeometrically form a convex polytope of feasible solutions.

Original: [Equation]
Revised: [Equation]

Original: Coarse Correlated Equilibria (CCEs) [40] are similar to CEs but a player may only consider deviatingbefore receiving a recommended strategy.
Revised: Coarse Correlated Equilibria (CCEs) [27] are similar to CEs but a player may only consider deviatingbefore receiving a recommended strategy.

Original: Therefore the deviation gain for CCEs can be derived fromthe CE deﬁnition by summing over all possible recommended strategies a 00 p .
Revised: Therefore the deviation gain for CCEs can be derived fromthe CE deﬁnition by summing over all possible recommended strategies a 00 p .

Original: [Equation]
Revised: [Equation]

Original: A distribution,  ( a ) , is in ✏ -CCE if the deviation gain is no more than some constant ✏ p  ✏ for everydeviation strategy, a 0 p , and for every player, p .
Revised: A distribution,  ( a ) , is in ✏ -CCE if the deviation gain is no more than some constant ✏ p  ✏ for everydeviation strategy, a 0 p , and for every player, p .

Original: [Equation]
Revised: [Equation]

Original: NEs [41] have similar deﬁnitions to CCEs but have an extra constraint: the joint distribution factorizes⌦ p  ( a p ) =
Revised: NEs [28] have similar deﬁnitions to CCEs but have an extra constraint: the joint distribution factorizes⌦ p  ( a p ) =

Original:  ( a ) , resulting in nonlinear constraints 1 .
Revised:  ( a ) , resulting in nonlinear constraints 1 .

Original: [Equation]
Revised: [Equation]

Original: Note that the deﬁnition of the NE uses the same deviation gain as the CCE deﬁnition.
Revised: Note that the deﬁnition of the NE uses the same deviation gain as the CCE deﬁnition.

Original: Anotherremarkable fact is that the marginals of any joint CCE in two-player constant-sum games,  ( a p ) =
Revised: Anotherremarkable fact is that the marginals of any joint CCE in two-player constant-sum games,  ( a p ) =

Original: P a  p  ( a ) , is also an NE, when ✏ p = 0 .
Revised: P a  p  ( a ) , is also an NE, when ✏ p = 0 .

Original: Therefore we can use CCE machinery to solve for NEs insuch classes of games.
Revised: Therefore we can use CCE machinery to solve for NEs insuch classes of games.

Original: When a distribution is in equilibrium, no player has incentive to unilaterally deviate from it to achievea better payoff.
Revised: When a distribution is in equilibrium, no player has incentive to unilaterally deviate from it to achievea better payoff.

Original: There can however be many equilibria in a game, choosing amongst these is known asthe equilibrium selection problem [20].
Revised: There can however be many equilibria in a game, choosing amongst these is known asthe equilibrium selection problem [14].

Original: For (C)CEs, the valid solution space is convex (Figure 1), soany strictly convex function will sufﬁce (in particular, Maximum Entropy (ME) [43] and Maximum
Revised: For (C)CEs, the valid solution space is convex (Figure 1), soany strictly convex function will sufﬁce (in particular, Maximum Entropy (ME) [30] and Maximum

Original: Gini (MG) [36] have been proposed).
Revised: Gini (MG) [23] have been proposed).

Original: For NEs, the solution space is convex for only certain classes ofgames such as two-player constant-sum games.
Revised: For NEs, the solution space is convex for only certain classes ofgames such as two-player constant-sum games.

Original: Indeed, NEs are considered fundamental in this classwhere they have powerful properties, such as being unexploitable, interchangeable, and tractable tocompute.
Revised: Indeed, NEs are considered fundamental in this classwhere they have powerful properties, such as being unexploitable, interchangeable, and tractable tocompute.

Original: However, for N-player general-sum games (C)CEs may be more suitable as they remaintractable and permit coordination between players which results in higher-welfare equilibria.
Revised: However, for N-player general-sum games (C)CEs may be more suitable as they remaintractable and permit coordination between players which results in higher-welfare equilibria.

Original: If✏ p  0 , there must exist at least one NE, CE, and CCE for any ﬁnite game, because a NE alwaysexists and NE ✓ CE ✓ CCE.
Revised: If✏ p  0 , there must exist at least one NE, CE, and CCE for any ﬁnite game, because a NE alwaysexists and NE ✓ CE ✓ CCE.

Original: Learning NEs [39] and CCEs [21] on a single game is well studied. Neural Network Solvers Approximating NEs using neural networks is known to be PAC learnable[14].
Revised: Neural Network Solvers Approximating NEs using neural networks is known to be PAC learnable[10]. Learning NEs [26] and CCEs [15] on a single game is well studied.

Original: There is also work learning (C)CEs [4, 30] and training neural networks to approximate NEs[14, 22] on subclasses of games.Learned NE meta-solvers have been deployed in PSRO [17]. Differentiable neural networks have been developed to learn QREs [35]. NEs for contextual gameshave been learned using ﬁxed point (deep equilibrium) networks [24].
Revised: There is also work learning(C)CEs [4, 19] and training neural networks to approximate NEs [10, 16] on subclasses of games.

Original: A related ﬁeld, L2O [7],aims to learn an iterative optimizer more suited to a particular distribution of inputs, while this workfocuses on learning a direct mapping.

Original: To the best of our knowledge, no work exists training a generalapproximate mapping from the full space of games to (C)CEs with ﬂexible selection criteria.
Revised: But, to the best of our knowledge, no work exists training a general approximate solver for the fullspace of games for (C)CEs with ﬂexible selection criteria.

Original:  Maximum Welfare Minimum Relative Entropy (C)CEs
Revised:  Maximum Welfare Minimum Relative Entropy (C)CEs

Original: Previous work has argued that having a unique objective to solve for equilibrium selection is important.
Revised: Previous work has argued that having a unique objective to solve for equilibrium selection is important.

Revised: However justifying why a particular solution is better than another is thorny.

Original: The principle of maximum entropy [28] has been used to justify entropy maximizing equilibria [43].
Revised: Appeals to Occam’srazor and the principle of maximum entropy have been used to justify entropy maximizing equilibria.

Original: In maximum entropy selection, payoffs are ignored and selection is based on minimizing the distanceto the uniform distribution.
Revised: In maximum entropy selection, payoffs are ignored and selection is based on minimizing the distanceto the uniform distribution.

Original: This has two interesting properties: (i) it makes deﬁning unique solutionseasy (ii) the solution is invariant to offset and positive scaling of the payoff tensor.
Revised: This has two interesting properties: (i) it makes deﬁning unique solutionseasy (ii) the solution is invariant to offset and positive scaling of the payoff tensor.

Original: While thesesolutions are unique, they both result in weak and low payoff equilibria because they ﬁnd solutionson the boundary of the polytope.
Revised: While thesesolutions are unique, they both result in weak and low payoff equilibria because they ﬁnd solutionson the boundary of the polytope.

Original: Meanwhile, the literature tends to favour Maximum Welfare (MW)because it results in large payoffs for the agents and is a linear objective, however it is not unique ingeneral.
Revised: Meanwhile, the literature tends to favour Maximum Welfare (MW)because it results in large payoffs for the agents and is a linear objective, however it is not unique ingeneral.

Original: We consider a composite objective function composed of (i) Minimum Relative Entropy(MRE, also known as Kullback-Leibler divergence) between a target joint, ˆ  ( a ) , and the equilibriumjoint,  ( a ) , (ii) distance between a target approximation, ˆ ✏ p , and the equilibrium approximation,✏ p , (iii) maximum of a linear objective, W ( a ) .
Revised: We consider a composite objective function composed of (i) Minimum Relative Entropy(MRE) between a target joint, ˆ  ( a ) , and the equilibrium joint,  ( a ) , (ii) distance between a targetapproximation, ˆ ✏ p , and the equilibrium approximation, ✏ p , (iii) maximum of a linear objective, W ( a ) .

Original: The objective is constrained by the (i) distributionconstraints (
Revised: The objective is constrained by the (i) distribution constraints (

Original: P a  ( a ) = 

Revised: P a  ( a ) =

Original: and  ( a )  0 ) and, (ii) either CCE constraints (Equation (4)) or CEconstraints (Equation (2)).
Revised: and  ( a )  0 ) and, (ii)either CCE constraints (Equation (4)) or CE constraints (Equation (2)).

Original: [Equation]
Revised: [Equation]

Original: The approximation weight, ⇢ , and welfare weight, µ , are hyperparameters that control the balanceof the optimization.
Revised: The approximation weight, ⇢ , and welfare weight, µ , are hyperparemeters that control the balanceof the optimization.

Original: The maximum approximation parameter, ✏ + p , is another constant that is usuallychosen to be equal to the payoff scale (Section 4.1).
Revised: The maximum approximation parameter, ✏ + p , is another constant that is usuallychosen to be equal to the payoff scale (Section 4.1).

Original: The approximation term is designed to havea similar form to the relative entropy and is maximum when ˆ ✏ p = ✏ p .
Revised: The approximation term is designed to havea similar form to the relative entropy and is maximum when ˆ ✏ p = ✏ p .

Original: We refer to this equilibriumselection framework as Target Approximate Maximum Welfare Minimum Relative Entropy ( ˆ ✏ -
Revised: We refer to this equilibriumselection framework as Target Approximate Maximum Welfare Minimum Relative Entropy ( ˆ ✏ -

Original: MWMRE).
Revised: MWMRE).

Original: 3.1 Dual of ✏ -MWMRE (C)CEs
Revised: 3.1 Dual of ✏ -MWMRE (C)CEs

Original: Rather than performing a constrained optimization, it is easier to solve the dual problem,arg min ↵ p L (C)CE (derived in Section A), where ↵ CE p ( a 0 p , a 00 p )  0 are the dual deviation gains [Figure] corresponding to the CE constraints, and ↵ CCE p ( a 0 p )  0 are the dual deviation gains corresponding tothe CCE constraints.
Revised: Rather than performing a constrained optimization, it is easier to optimize the dual Lagrangian,arg min ↵ p L (C)CE (derived in Section A), where ↵ CE p ( a 0 p , a 00 p )  0 are the dual deviation gainscorresponding to the CE constraints, and ↵ CCE p ( a 0 p )  0 are the dual deviation gains corresponding tothe CCE constraints.

Original: Note that we do not need to optimize over the primal joint,  ( a ) .
Revised: Note that we do not need to optimize over the primal joint,  ( a ) .

Original: Choosing oneof the elements in the curly brackets, the Lagrangian is deﬁned:
Revised: Choosing one [Figure] of the elements in the curly brackets, the Lagrangian is deﬁned:

Original: [Equation]
Revised: [Equation]

Original: The logits l (C)CE ( a ) are deﬁned:
Revised: The logits l (C)CE ( a ) are deﬁned:

Original: [Equation]
Revised: [Equation]

Original: The primal joint and primal approximation parameters are deﬁned:
Revised: The primal joint and primal approximation parameters are deﬁned:

Original: [Equation]
Revised: [Equation]

Revised: ↵ p ( a 0 p , a 00 p ) ,

Revised: X a 0 p

Revised: CCE ↵ p ( a 0 p )= ;

Revised: A + ✏ + p (10)

Original:  Neural Network Training
Revised:  Neural Network Training

Original: The network maps the payoffs of a game, G p ( a ) , and the targets ( ˆ  ( a ) , ˆ ✏ p , W ( a ) ) to the dualdeviation gains, ↵ (C)CE p , that deﬁne the equilibrium.
Revised: The network maps the payoffs of a game, G p ( a ) , and the targets ( ˆ  ( a ) , ˆ ✏ p , W ( a ) ) to the dualdeviation gains, ↵ (C)CE p , that deﬁne the equilibrium.

Original: The duals are a signiﬁcantly more space efﬁcientobjective target (
Revised: The duals are a much more space efﬁcientobjective target (

Original: P p |A
Revised: P p |A

Original: p | 2 for CEs and
Revised: p | 2 for CEs and

Original: P p |A
Revised: P p |A

Original: p | for CCEs) than the full joint (
Revised: p | for CCEs) than the full joint (

Original: Q p |A p | ), particularly 129 when scaling the number of strategies and players.
Revised: Q p |A p | ), particularly 124 when scaling the number of strategies and players.

Original: The joint,  ( a ) , and approximation, ✏ p , can becomputed analytically from the dual deviation gains and the inputs using Equations (9) and (10).
Revised: The joint,  ( a ) , and approximation, ✏ p , can becomputed analytically from the dual deviation gains and the inputs using Equations (9) and (10).

Original: Thenetwork is trained by minimizing the loss, L (C)CE (Equation (7)).

Original: We call the resulting architecture a Neural Equilibrium Solver (NES).
Revised: Wecall the resulting architecture a Neural Equilibrium Solver (NES).

Original: 4.1 Input and Preprocessing
Revised: 4.1 Input and Preprocessing

Original: The MRE objective, and the (C)CEs constraints are invariant to payoff offset and scaling.
Revised: The MRE objective, and the (C)CEs constraints are invariant to payoff offset and scaling.

Original: Thereforewe can assume that the payoffs have been standardized without loss of generality.
Revised: Thereforewe can assume that the payoffs have been standardized without loss of generality.

Original: Each player’spayoff should have zero-mean.
Revised: Each player’spayoff should have zero-mean.

Original: Furthermore, it should be scaled such that the L m = ||
Revised: Furthermore, it should be scaled such that the L m = ||

Original: · || m norm of 137 the payoff tensor equals Z m , where Z m is a scale hyperparameter chosen such that the elements of the inputs have unit variance (a property that ensures neural networks train quickly with standardparameter initializations [18]).
Revised: · || m norm of 131 the payoff tensor equals Z m .

Original: We will ensure both these properties by including a preprocessinglayer in NES.
Revised: We will ensure both these properties by including a preprocessing layerin NES.

Original: Thepreprocessed inputs ( G  p ( a ) , ˆ   ( a ) , ˆ ✏  p , W  ( a ) ) are then broadcast and concatenatedtogether so that they result in an input of shape [ C, N, |A 1 | , ..., |A
Revised: The inputs ( G L m p ( a ) , ˆ  L 1 ( a ) , ˆ ✏ L m p , W L m ( a ) ) are then broadcast and concatenated togetherso that they result in an input of shape [ C, N, |A 1 | , ..., |A

Original: N | ] , where the channel dimension142 C
Revised: N | ] , where the channel dimension C

Original: = 4 , if  all inputs are required.
Revised: = 4 , if 134 all inputs are required.

Original: 4.2 Training Distribution
Revised: 4.2 Training Distribution

Original: Recall that offset and positive scale transformations to the payoff are invariant.

Original: Zero-mean andconstant norm scale geometrically results, in the edge of a disc centered on the origin (Section D.4).
Revised: Zero-mean and constant norm scale geometrically results in the edge of a disc.

Original: It makes sense to uniformly sample over this space so as to not bias the training data to a particularoffset or scaling.
Revised: It makes sense touniformly sample over this space so as to not bias the training data to a particular offset or scaling.

Original: The literature tends to favour the unnormalized uniform distribution to generaterandom games, which would unnecessarily require a network to learnoffset and scale invariance.
Revised: The literature tends to favour the unnormalized uniform distribution to generate random games, whichwould unnecessarily require a network to learn scale invariance.

Original: When normalized to a constant scale and zero-mean the uniform results in the L 1 disc.
Revised: When normalized to a constant scaleand zero-mean the uniform results in the L 1 disc.

Original: We proposeusing disc distributions for training, and use the L 2 disc in the experiments in this paper.
Revised: We propose using disc distributions for training,and use the L 2 disc in the experiments in this paper.

Original: We choosethe norm scaling constant, Z m, such that the elements of the payoffs maintain unit variance (forexample, Z 2 = p |A| ).
Revised: We choose Z m such that the elements of thepayoffs maintain unit variance (for example, Z 2 = p |A| ).

Original: 4.3 Gradient Calculation
Revised: 4.3 Gradient Calculation

Original: The gradient update is found by taking the derivativeof the loss (Equation (7)) with respect to thedual variables, ↵ .
Revised: The gradient update, is found by taking the derivative with respect to the dual deviation gains(Equation (7)).

Original: Note that computing a gradient does not require knowing the optimal joint  ⇤ ( a ) ,so the network can be trained in an unsupervised fashion, from randomly generated inputs, G p ( a ) ,ˆ  ( a ) , ˆ ✏ p , and W ( a ) .[Equation]
Revised: Note that computing a gradient does not require knowing the optimal joint  ⇤ ( a ) ,so the network can be trained in an unsupervised fashion, from randomly generated inputs, G p ( a ) ,ˆ  ( a ) , ˆ ✏ p , and W ( a ) .

Original: [Equation] The dual variables, { ↵ CE p ( a 0 p , a 00 p ) , ↵ CCE p ( a 0 p ) } , are outputs of the neural network, with learned parameters ✓ . Gradients for these parameters can be derived using the chain rule:
Revised: Furthermore, many powerful neural network optimizers [39, 20, 11] and MLframeworks [1, 5, 31] can be leveraged to update network parameters.

Original: [Equation]
Revised: [Equation]

Original: Backprop efﬁciently calculates these gradients, and many powerful neural network optimizers[52, 32, 15] and ML frameworks [1, 5, 44] can be leveraged to update the network parameters.

Original: 4.4 Equivariant Architectures
Revised: 4.4 Equivariant Architectures

Original: The ordering of strategies and players in a normal-form game is unimportant, therefore the output ofthe network should be equivariant under two types of permutation; (i) strategy permutation, and (ii)player permutation.
Revised: The ordering of strategies and players in a normal-form game is unimportant, therefore the output ofthe network should be equivariant under two types of permutation; (i) strategy permutation, and (ii)player permutation.

Original: Speciﬁcally, for some strategy permutation ⌧ p (1) , ..., ⌧ p ( |A p | ) applied to eachelement of a player’s inputs (payoffs, target joint, and welfare), the outputs must also have permuteddimensions: ( ↵ p ( a p ) =
Revised: Speciﬁcally, for some strategy permutation ⌧ p (1) , ..., ⌧ p ( |A p | ) applied to eachelement of a player’s inputs (payoffs, target joint, and welfare), the outputs must also have permuteddimensions: ( ↵ p ( a p ) =

Original: ↵ p ( ⌧ p ( a p )) and  ⌧ ( a 1 , ..., a N ) =  ( ⌧ 1 ( a 1 ) , ..., ⌧ N ( a N )) ).
Revised: ↵ p ( ⌧ p ( a p )) and  ⌧ ( a 1 , ..., a N ) =  ( ⌧ 1 ( a 1 ) , ..., ⌧ N ( a N )) ).

Original: Likewise, forsome player permutation ⌧ (1) , ..., ⌧ ( N ) , the outputs must be transposed: ( ↵ ⌧ p ( a p ) =
Revised: Likewise, forsome player permutation ⌧ (1) , ..., ⌧ ( N ) , the outputs must be transposed: ( ↵ ⌧ p ( a p ) =

Original: ↵ ⌧ ( p ) ( a ⌧ ( p ) )and  ⌧ ( a 1 , ..., a N ) =
Revised: ↵ ⌧ ( p ) ( a ⌧ ( p ) )and  ⌧ ( a 1 , ..., a N ) =

Original:  ( a ⌧ (1) , ..., a ⌧ ( N ) ) ).
Revised:  ( a ⌧ (1) , ..., a ⌧ ( N ) ) ).

Original: The latter equivariance can only be exploited by anetwork if all players have the same number of strategies (“cubic games”).
Revised: The latter equivariance can only be exploited by anetwork if all players have the same number of strategies (“cubic games”).

Original: There are |A p | !
Revised: There are |A p | !

Original: possiblestrategy permutations for each player and N ! player permutations, resulting in N !
Revised: possiblestrategy permutations for each player and N ! player permutations, resulting in N !

Original: ( |A p | !)
Revised: ( |A p | !)

Original: N possibleequivariant permutations of each sampled payoff.
Revised: N possibleequivariant permutations of each sampled payoff.

Original: Note that this is much greater than the numberof joint strategies in a game, N !
Revised: Note that this is much greater than the numberof joint strategies in a game, N !

Original: ( |A p | !)
Revised: ( |A p | !)

Original: N  |A p | N , which is an encouraging observation when 174 considering how this approach will scale to large games.
Revised: N  |A p | N , which is an encouraging observation when 161 considering how this approach will scale to large games.

Original: Utilizing an equivariant [47, 55] architectureis therefore crucial to scale to large games because each sample represents many possible inputs.
Revised: Utilizing an equivariant [34, 42] architectureis therefore crucial to scale to large games because each sample represents many possible inputs.

Original: Equivariant architectures have been used before for two-player games [22].
Revised: Equivariant architectures have been used before for two-player games [16].

Original: Payoffs
Revised: Payoffs

Original: Transformations The main layers of the architecture consist of activations with shape[ C, N, |A 1 | , ..., |A
Revised: Transformations The main layers of the architecture consist of activations with shape[ C, N, |A 1 | , ..., |A

Original: N | ] , which is the same shape as a payoff tensor (with a channel dimension).
Revised: N | ] , which is the same shape as a payoff tensor (with a channel dimension).

Original: We  refer to layers with this shape as “payoff” layers.
Revised: We 166 refer to layers with this shape as “payoff” layers.

Original: We consider transformations of the form:
Revised: We consider transformations of the form:

Original: [Equation]
Revised: [Equation]

Original:  where f is any equivariant nonlinearity 2 ,w are learned network weights, b are learned networkbiases, and h i is one of many possible equivariant pooling functions (Section C) and Con isthe concatenate function along the channel dimension.
Revised: [Equation] A ( [Equation] where f is any equivariant nonlinearity 2 , and h i is one of many possible equivariant pooling functions(Section C).

Original: For example, consider one such function, h i =
Revised: For example, consider one such function, h i =

Original: P a 1 g ( p, a 1 , ..., a N ) , which is invariant across any permutation of a 1 (similar tosum-pooling in CNNs), and equivariant over permutations of p, a 2 , ..., a N .
Revised: P a 1 g ( p, a 1 , ..., a N ) , which is invariantacross any permutation of a 1 (similar to sum-pooling in CNNs), and equivariant over permutationsof p, a 2 , ..., a N .

Original: In general we canuse  ✓ { p,a 1 ,...a N } g ( p, a 1 , ..., a N ) , where  is any equivariant  continuous pooling function (mean- 186 pooling and max-pooling are good choices).
Revised: In general we can use  ✓ { p,a 1 ,...a N } g ( p, a 1 , ..., a N ) , where  is any equivariant 171 continuous pooling function (mean-pooling and max-pooling are good choices).

Original: If all players have an equal number of strategies, forsome functions, weights can be shared over all p 2 [1 , N ] because of symmetry [48].
Revised: If all players havean equal number of strategies, for some functions, weights can be shared over all p 2 [1 , N ] becauseof symmetry [35].

Original: Note thatnumber of trainable parameters scales with the number of input and output channels, and not withthesize of the game, therefore it is possible for the network to generalize to games with different numbersof strategies.
Revised: Note that number of trainable parameters scales with the number of input andoutput channels, and not with the size of the game.

Original: The basic layer, g l +1 , therefore comprises of a linear transform of a concatenated,broadcasted set of pooling functions.
Revised: The basic layer, g l +1 , therefore comprises of alinear transform of a stacked, broadcasted set of pooling functions.

Original: Payoffs to CCE Duals Transformations Payoffs can be transformed to CCE duals, ↵ CCE p ( c l +1 , a 0 p ) ,by using a combination of a subset of the equivariant functions
Revised: Payoffs to CCE Duals Transformations Payoffs can be transformed to CCE duals, ↵ CCE p ( c l +1 , a 0 p ) ,by using a combination of a subset of the equivariant functions

Original: h i discussed above that sum over atleast  p .
Revised: h i discussed above that sum over atleast  p .

Original: If the number of strategies are equal for each player, the transformation weights can beshared and the duals can be stacked into a single object for more efﬁcient computation in later layers:
Revised: If the number of strategies are equal for each player, the transformation weights can beshared and the duals can be stacked into a single object for more efﬁcient computation in later layers:↵ CCE ( c l +1 , p, a 0 p ) =

Original: [Equation]
Revised: Stack p  ↵ CCE p ( c l +1 , a 0 p )  .

Original: Payoffs to CE Duals Transformations The transformation to produce the CE duals is morecomplex.
Revised: The transformation to produce the CE duals is morecomplex.

Original: CE duals, ↵ CCE p ( c l +1 , a 00 p , a 0 p ) , need to be symmetrically equivariant .
Revised: CE duals, ↵ CCE p ( c l +1 , a 00 p , a 0 p ) , need to be symmetrically equivariant .

Original: This property can beobtained by (i) independently generating two CCE duals and, (ii) taking outer operations,  (forexample, sum or product), over them. 
Revised: This property can beobtained by (i) independently generating two CCE duals and, (ii) taking outer operations,  (forexample, sum or product), over them. [Equation]

Original: [Equation]
Revised: [Equation]

Original: Where ˆ f is any equivariant nonlinearity with zero diagonal 3 .
Revised: Where ˆ f is any equivariant nonlinearity with zero diagonal 3 .

Original: We know that the diagonal is zerobecause it represents the dual of the deviation gain when deviating from a strategy to itself, which iszero, and therefore cannot be violated.
Revised: We know that the diagonal is zerobecause it represents the dual of the deviation gain when deviating from a strategy to itself, which iszero, and therefore cannot be violated.

Original: This is a useful property which will be exploited in later duallayers.
Revised: This is a useful property which will be exploited in later duallayers.

Original: These can also be stacked if players have equal number of strategies: ↵ CCE ( c l +1 , p, a 0 p , a 00 p ) =
Revised: These can also be stacked if players have equal number of strategies: ↵ CCE ( c l +1 , p, a 0 p , a 00 p ) =

Original: Stack p
Revised: Stack p

Original: [Equation]
Revised: [Equation]

Original:  CCE Duals Transformations Because the payoff activations are high-dimensional, it is worthwhileto operate on them in dual space.
Revised:  . CCE Duals Transformations Because the payoff activations are high-dimensional, it is worthwhileto operate on them in dual space.

Original: When transforming CCE duals we consider a mapping:
Revised: When transforming CCE duals we consider a mapping:

Original: [Equation]
Revised: [Equation]

Original: where f is any equivariant nonlinearity, and h i is from a set of only two possible equivarianttransformation functions (and two more if the game is cubic).
Revised: where f is any equivairant nonlinearity, and h i is from a set of only two possible equivarianttransformation functions (and two more if the game is cubic).

Original: For the ﬁnal layer, we use a SoftPlusnonlinearity to ensure the output is nonnegative and has gradient everywhere.
Revised: For the ﬁnal layer, we use a SoftPlusnonlinearity to ensure the output is nonnegative and has gradient everywhere.

Original: CE Duals Transformations When transforming CE duals we consider functions of the form:
Revised: CE Duals Transformations When transforming CE duals we consider functions of the form:

Original: [Equation]
Revised: [Equation]

Original: For the CE case, the equivariant linear transformations are more complex: it is symmetric over therecommended and deviation strategies.
Revised: For the CE case, the equivariant linear transformations are more complex: it is symmetric over therecommended and deviation strategies.

Original: Fortunately this is a well studied equivariance class [51],which can be fully covered by combining seven transforms (Section C) which comprise of differentsums and transpositions of the input.Common nonlinearities such as ReLU tanh sigmoid
Revised: Fortunately this is a well studied equivariance class [38],which can be fully covered by combining seven transforms (Section C) which comprise of differentsums and transpositions of the input.

Original: SoftMax

Original: Masking is sufﬁcient: e.g. ˆ f ( a 0 p , a 00 p ) =

Original: (1  I ( a 0 p , a 00 p )) f ( a 0 p , a 00 p ) , where I is the identity matrix. [Figure] Activation Variance Because the equivariant network possibly involves summing over dimensionsof the inputs, activations are no longer independent of one another, so extra care needs to be takenwhen initializing the network to avoid variance explosion.
Revised: Activation Variance Because the equivariant network possibly involves summing over dimensionsof the inputs, activations are no longer independent of one another, so extra care needs to be takenwhen initializing the network to avoid variance explosion.

Original: To combat this we use three techniques:(i) inputs are scaled to unit variance as described previously (Section 4.1), (ii) the network israndomly initialized with variance scaling to ensure the variance at every layer is one, and (iii) weuse BatchNorm [27] between every layer.
Revised: To combat this we use three techniques:(i) inputs are scaled to unit variance as described previously (Section 4.1), (ii) the network israndomly initialized with variance scaling to ensure the variance at every layer is one, and (iii) weuse BatchNorm [18] between every layer.

Original: We also used weight decay to regularize the network.
Revised: We also used weight decay to regularize the network.

Original: Advanced Architectures More advanced architectures such as ResNet [23] or Transformers [53]are possible.
Revised: Advanced Architectures More advanced architectures such as ResNet [17] or Transformers [40]are possible.

Original: An example of the ﬁnal architecture is summarized in Figure 2.
Revised: An example of the ﬁnal architecture is summarized in Figure 2.

Original: 4.5 Parameterizations
Revised: 4.5 Parameterizations

Original: The composite objective framework allows us to deﬁne a number of combinations of auxiliaryobjectives.
Revised: The composite objective framework allows us to deﬁne a number of combinations of auxiliaryobjectives.

Original: We highlight several interesting speciﬁcations (Appendix Table 3).
Revised: We highlight several interesting speciﬁcations (Table 1).

Original: The most basic is 
Revised: The most basic is Maximum

Original: Maximum Entropy (ME) which simply ﬁnds the unique equilibrium closest to the uniform distributionaccording to the relative entropy distance.
Revised: Entropy (ME) which simply ﬁnds the unique equilibrium closest to the uniform distribution accordingto the relative entropy distance.

Original: This distribution need not be uniform, it could be anytarget distribution.
Revised: This distribution need not be uniform, it could be any target [Figure] distribution.

Original: We could instead favour a welfare objective parameterized on the payoffs to ﬁnd a
Revised: We could instead favour a welfare objective parameterized on the payoffs to ﬁnd a

Original: Maximum Welfare (MW) solution.
Revised: Maximum Utilitarian (MU) solution.

Original: The two previous solutions can be generalized to solve for anywelfare (Maximum Welfare Maximum Entropy (MWME)) or any target (Minimum Relative Entropy(MRE)).
Revised: The two previous solutions can be generalized to solve for anywelfare (Maximum Welfare Maximum Entropy (MWME)) or any target (Minimum Relative Entropy(MRE)).

Original: Furthermore we need not limit ourselves to approximation parameters equal to zero, forexample by ﬁnding the minimum possible approximation parameter we have the Maximum Strength(MS) solution.
Revised: Furthermore we need not limit ourselves to approximation parameters equal to zero, forexample by ﬁnding the minimum possible approximation parameter we have the Maximum Strength(MS) solution.

Original: Finally, we can ﬁnd solutions for any approximation parameter for the solutionsdiscussed so far ( ˆ ✏ -MWME and ˆ ✏ -MRE).
Revised: Finally, we can ﬁnd solutions for any approximation parameter for the solutionsdiscussed so far ( ˆ ✏ -MWME and ˆ ✏ -MRE).

Original:  Performance Experiments
Revised:  Performance Experiments

Original: Traditionally performance of NE, and (C)CE solvers has focused on evaluating time to converge toa solution within some tolerance.
Revised: Traditionally performance of NE, and (C)CE solvers has focused on evaluating time to converge toa solution within some tolerance.

Original: Feedforward neural networks can produce batches of solutionsquickly 4 and deterministically.
Revised: Feedforward neural networks can produce batches of solutionsquickly 4 and deterministically.

Original: For non-trivial games this is much faster than what an iterative solvercould hope to achieve.
Revised: For non-trivial games this is much faster than what an iterative solvercould hope to achieve.

Original: We therefore focus our evaluation on the trade-offs of the neural networksolver, namely (i) how long it takes to train, and (ii) how accurate the solutions are.
Revised: We therefore focus our evaluation on the trade-offs of the neural networksolver, namely (i) how long it takes to train, and (ii) how accurate the solutions are.

Original: For the latter weuse two metrics:
Revised: For the latter weuse two metrics:

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst (Solver Gap) measures the distance to the exact solution found by an iterative solver 5 ,  ⇤ ( a ) ,and is bounded between 0 and 1 .
Revised: The ﬁrst (Solver Gap) measures the distance to the exact solution found by an iterative solver 5 ,  ⇤ ( a ) ,and is bounded between 0 and 1 .

Original: The second ((C)CE Gap) measures the distance to the equilibriumsolution polytope, and is zero if it is within the polytope.
Revised: The second ((C)CE Gap) measures the distance to the equilibriumsolution polytope, and is zero if it is within the polytope.

Original: Parameterization Sweeps We show performance across a number of parameterizations, including(i) different equilibrium selection criteria (Figure 3a), (ii) different shapes of games (Figure 3b), and(iii) different solution concepts (Figure 3c).[Figure]
Revised: Parameterization Sweeps We show performance across a number of parameterizations, including(i) different equilibrium selection criteria (Figure 3a), (ii) different shapes of games (Figure 3b), and(iii) different solution concepts (Figure 3c).

Original: Classes of Games It is known that some distributions of game payoffs are harder for some methodsto solve than others [46].
Revised: Classes of Games It is known that some distributions of game payoffs are harder for some methodsto solve than others [33].

Original: We compare performance across a number of classes of transfer games(Appendix Table 4) for a single MECCE and a single MECE[43] Neural Equilibrium Solver trainedon 8 ⇥ 8 games.
Revised: We compare performance across a number of classes of games (Table 2) fora single MECCE and a single MECE Neural Equilibrium Solver trained on 8 ⇥ 8 games.

Original: Figure 4 shows the worst, mean, and best performance over 512 samples from eachclass in terms of (i) distance to any equilibrium, and (ii) distance to the target equilibrium found byan iterative solver.
Revised: Figure 4shows the worst, mean, and best performance over 512 samples from each class in terms of (i)distance to any equilibrium, and (ii) distance to the target equilibrium found by an iterative solver.

Original: We also plot the performance of a uniform joint as a naive baseline, as the gapcan be artiﬁcially reduced by scaling the payoffs.
Revised: We also plot the performance of a uniform joint as a naive baseline, as the gap can be artiﬁciallyreduced by scaling the payoffs.

Original: In regards to equilibrium violation, ME is trickybecause it lies on the boundary of the polytope, so some violation is expected in an approximatesetting.
Revised: In regards to equilibrium violation, ME is tricky because it lies on theboundary of the polytope, so some violation is expected in an approximate setting.

Original: The plots showing the failure rate and run time of the iterative solver are to intuit difﬁcultclasses.
Revised: The plots showingthe failure rate and run time of the iterative solver are to intuit difﬁcult classes.

Original: The baseline iterative solver take about 0 .

Original: 05 s to solve a single game, the network can solve abatch of 4096 games in 0 .

Original: 0025 s. We see that for most classes the NES is very accurate with a solvergap of around 10  2 .
Revised: We see that for mostclasses the NES is very accurate with a solver gap of around 10  2 .

Original: Some classes of games are indeed more difﬁcult and these align with games thatiterative equilibrium solvers struggle with.
Revised: Some classes of games are indeedmore difﬁcult and these align with games that iterative equilibrium solvers struggle with.

Original: This hints that difﬁcult games are ill-conditioned.
Revised: This hintsthat difﬁcult games are ill-conditioned.

Original: [Table]
Revised: [Figure]

Original: [Table] Ablations We show the performance (Figure 3d) of the proposed method compared with (i) a linearnetwork [Table] and (ii) no invariant pre-processing with naive payoff sampling (each element sampledusing a uniform distribution).
Revised: Ablations We show the performance (Figure 3d) of the proposed method compared with (i) a linearnetwork, and (ii) no invariant pre-processing with naive payoff sampling.

Original: Both result in signiﬁcant reduction in performance.
Revised: Both result in signiﬁcantreduction in performance.

Original: Scaling Due to the size of the representation of the payoffs, G p ( a ) , the inputs and thereforethe activations of the network grow signiﬁcantly with the number of joint strategies in the game.

Original: Therefore without further work on sparser payoff representation, NES is limited by size of payoffinputs.

Original: For further discussion see Section 7.

Original: Nevertheless, Table 1 shows good performance whenscaling to moderately sized games.

Original: Note that the “solver gap” metric is incomplete on larger gamesbecause the ECOS evaluation solver fails to converge.

Original: Generalization An interesting property of the NES architecture is that its parameters do not dependon the number of strategies in the game.

Original: Therefore we can test the generalization ability of the networkzero-shot on games with different numbers of strategies (Table 2).

Original: There are two observations: (i)

Original: NES only weakly generalizes to other game sizes under the solver gap metric, and (ii) NES stronglygeneralizes to larger games under the CCE gap, remarkably achieving zero violation.

Original: Therefore thenetwork retains the ability to reliably ﬁnd CCEs in larger games, but does struggle to accurately selectthe target MWMRE equilibrium.

Original: This could be mitigated by training the network on a mixture ofgame sizes, which we leave to future work.

Original:  Applications
Revised:  Applications

Original: The key beneﬁt of this work is that it allows fast and general solving of games at the expenseof guaranteed correctness.
Revised: The key beneﬁt of this work is that it allows fast and general solving of games at the expense ofguaranteed correctness.

Original: This allows the application of solution concepts into areas that wouldotherwise be too time-expensive and are not as sensitive to approximations.
Revised: This allows the application of solution concepts into areas that would beotherwise be too time-expensive and are not as sensitive to approximations.

Original: There are several possibleapplications for this approach, most of which we leave to future work, but mention here.
Revised: There are several possibleapplications for this approach, most of which we leave to future work, but mention here.

Original: Warm-Start Iterative Solvers
Revised: Warm-Start Iterative Solvers

Original: Many iterative solvers start with a guess of the parameters andreﬁne them over time to ﬁnd an accurate solution.
Revised: Many iterative solvers start with a guess of the parameters andreﬁne them over time to ﬁnd an accurate solution.

Original: It is possible to use NES to warm-start iterativesolver algorithms, potentially signiﬁcantly improving convergence.
Revised: It is possible to use NES to warm-start iterativesolver algorithms, potentially signiﬁcantly improving convergence.

Original: Polytope Approximation
Revised: Polytope Approximation

Original: The framework can be used to approximate the full space of solutions byﬁnding extreme points of the convex polytope.
Revised: The framework can be used to approximate the full space of solutions byﬁnding extreme points of the convex polytope.

Original: Because of convexity, any convex mixture of theseextreme points is also a valid solution.
Revised: Because of convexity, any convex mixture of theseextreme points is also a valid solution.

Original: Two approaches could be used to ﬁnd extreme points (i) usingdifferent welfare objectives (ii) or using different target joint objectives.
Revised: Two approaches could be used to ﬁnd extreme points (i) usingdifferent welfare objectives (ii) or using different target joint objectives.

Original: For example, using pure joint targets:
Revised: For example, using pure joint targets:

Original: [Equation]
Revised: [Equation]

Original: These could be computed in a single batch, and would cover a reasonably large subset of full polytopeof solutions (the latter approach is demonstrated in Figure 1).
Revised: These could be computed in a single batch, and would cover a reasonably large subset of full polytopeof solutions (the latter approach is demonstrated in Figure 1).

Original: It would be easy to develop an algorithmthat reﬁnes the targets at each step to gradually ﬁnd all vertices of the polytope, if desired.
Revised: It would be easy to develop an algorithmthat reﬁnes the targets at each step to gradually ﬁnd all vertices of the polytope, if desired.

Original: Differentiable Model and Mechanism Design Mechanism design (MD) is a sub-ﬁeld of economicsoften described as “inverse game theory”, where instead of studying the behavior of rational payoffmaximizing agents on a given game, we are tasked with designing a game so that rational payoffmaximizing participants will exhibit behaviours at equilibrium that we deem desirable.
Revised: Differentiable Model and Mechanism Design Mechanism design (MD) is a sub-ﬁeld of economicsoften described as “inverse game theory”, where instead of studying the behavior of rational payoffmaximizing agents on a given game, we are tasked with designing a game so that rational payoffmaximizing participants will exhibit behaviours at equilibrium that we deem desirable.

Original: The ﬁeld hasa long history to which it is near impossible to do justice; see [37] for a review.
Revised: The ﬁeld hasa long history to which it is near impossible to do justice; see [24] for a review.

Original: The work presentedhere could impact MD in two ways.
Revised: The work presentedhere could impact MD in two ways.

Original: First, by making it easy to compute equilibrium strategies, NEScould widen the class of acceptable output games, relaxing the restrictive requirements (e.g. strategicdominance) often imposed of the output games out of concern more permissive solution conceptscould be hard for participants to compute.
Revised: First, by making it easy to compute equilibrium strategies, NEScould widen the class of acceptable output games, relaxing the restrictive requirements (e.g. strategicdominance) often imposed of the output games out of concern more permissive solution conceptscould be hard for participants to compute.

Original: Second, NES maps payoffs to joint strategies and it isdifferentiable, one could imagine turning a mechanism design task to a optimization problem thatcould be solved using standard gradient descent (e.g. design a general-sum game where strategies atequilibrium maximize some non-linear function of welfare and entropy, with payoff lying in a usefulconvex and closed subset).A related idea is to ﬁnd a game that produces a certain equilibrium [35]. Given an equilibrium, a payoff could be trained through the differentiable model that results in thedesired speciﬁc behaviour.
Revised: Second, NES maps payoffs to joint strategies and it isdifferentiable, one could imagine turning a mechanism design task to a optimization problem thatcould be solved using standard gradient descent (e.g. design a general-sum game where strategies atequilibrium maximize some non-linear function of welfare and entropy, with payoff lying in a usefulconvex and closed subset).

Original:  Discussion
Revised:  Discussion

Original: With Neural Equilibrium Solvers (NES) it is possible to quickly ﬁnd approximate equilibriumsolutions using a variety of selection criteria.
Revised: With Neural Equilibrium Solvers (NES) it is possible to quickly ﬁnd approximate equilibriumsolutions using a variety of selection criteria.

Original: There are many exciting applications of such a networktoinner-loops or subroutines of MARL algorithmswhere speed is critical, and the size of games ismodest but numerous.
Revised: There are many exciting applications of such a networkto MARL algorithms and mechanism design.

Original: Thedifferentiable property means that this could also be applied to mechanismdesign, or other inverse game theory problems. The main limitation of the approach is that the activation space of the network is large, particularlywith a large number of players and strategies which limits the size of games that can be tackled.
Revised: The main limitation of the approach is that, while the network is extremely efﬁcient in terms ofparameters, the activation space of the network is large, particularly with a large number of playersand strategies which limits the size of games that can be tackled.

Original: Futurework could look at restricted classes of games, such as polymatrix games [10, 11], or graphical games[29], which consider only local payoff structure and have much smaller payoff representations.

Original: Thisis a promising direction because NES otherwise has good scaling properties: (i) the dual variables arespace efﬁcient, (ii) there are relatively few parameters, (iii) the number of parameters is independentof the number of strategies in the game, (iv) equivariance means each training sample is equivalent totraining under all payoff permutations, and (v) there are promising zero-shot generalization results tolarger games.

Original: Solving for equilibria has the potential to promote increased cooperation in general-sum games,which could increase the welfare of all players.
Revised: Solving for equilibria has the potential to promote increased cooperation in general-sum games,which could increase the welfare of all players.

Original: However, if a powerful and unethical actor hadinﬂuence on the game being played, welfare gains of some equilibria could unfairly come at theexpense of other players.
Revised: However, if a powerful and unethical actor hadinﬂuence on the game being played, welfare gains of some equilibria could unfairly come at theexpense of other players.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] Main limitation is that the activation space of the network is very large.
Revised: [Yes] Main limitation is that theactivation space of the network is very large.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] We brieﬂy
Revised: [Yes] We brieﬂy

Original: discuss the abuse of equilibria could cause harm if used by a malicious actor, as well
Revised: discuss the abuse of equilibria could cause harm if used by a malicious actor, as well

Original: as the obvious beneﬁts.
Revised: as the obvious beneﬁts.

Original: We think most of game theory can be abused this way.
Revised: We think most of game theory can be abused this way.

Original: Users
Revised: Users

Original: must make sure they apply any work in an ethical way.
Revised: must make sure they apply any work in an ethical way.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes] We have no ethics concerns.
Revised: [Yes] We have no ethics concerns.

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] Proofs are in appen- dices.
Revised: [Yes] Proofs are in appendices.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [No] We have not
Revised: [No] We have not

Original: included code, although we are working on open-sourcing a library.
Revised: included code, although we are working on open-sourcing a library.

Original: We have described
Revised: We hope this will

Original:  the approach in enough detail that a competent practitioner should be able to reproduce our results in less than a week.
Revised: be available by rebuttal time. We have described the approach in enough detail that a competent practitioner should be able to reproduce our results in less than a week.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you report error bars (e.g ., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] We did not run with multiple random seeds dueto computation constraints.
Revised: [Yes] We did not run with multiple random seeds dueto computation constraints.

Original: We do, however, report the worst, mean, and best caseperformance over many examples.
Revised: We do, however, report the worst, mean, and best case performance over many examples.(d)

Original: (d) Did you include the total amount of compute and the type of resources used (e g., type of GPUs, internal cluster, or cloud provider)?
Revised: Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] In the appendix.
Revised: [Yes] In the appendix.

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets.
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] We use third party [Table] code which we cite (e.g. GAMUT).[Table]
Revised: [Yes] We use third partycode which we cite (e.g. GAMUT).

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: [Table] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: [Table] (d) Did you discuss whether and how consent was obtained from people whose data you’re[Table]
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A

Original: [Table]

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Original: [Table]
Revised: [Table]

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: [Table]
Revised: [Table]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: [Table]
Revised: [Table]

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: [Table]
Revised: [Table]

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: [Table]
Revised: [Table]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: [Table]
Revised: [Table]

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: [Table]
Revised: [Table]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: [Table]
Revised: [Table]

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A] [Table]
Revised: [N/A] [Table]