Exploitability Minimization in Games and Beyond
Abstract
Pseudo-games are a natural and well-known generalization of normal-form games, in which the actions taken by each player a ￿ ect not only the other players’ payo ￿ s, as in games, but also the other players’ strategy sets.
The solution concept par excellence for pseudo-games is the generalized Nash equilibrium (GNE), i.e., a strategy pro ￿ le at which each player’s strategy is feasible and no player can improve their payo ￿ s by unilaterally deviating to another strategy in the strategy set determined by the other players’ strategies.
The computation of GNE in pseudo-games has long been a problem of interest, due to applications in a wide variety of ￿ elds, from environmental protection to logistics to telecommunications.
Although computing GNE is PPAD-hard in general, it is still of interest to try to compute them in restricted classes of pseudo-games.
One approach is to search for a strategy pro ￿ le that minimizes exploitability, i.e., the sum of the regrets across all players. As exploitability is nondi ￿ erentiable in general, developing e ￿ cient ￿ rst-order methods that minimize it might not seem possible at ￿ rst glance. We observe, however, that the exploitability-minimization problem can be recast as a min-max optimization problem, and thereby obtain polynomial-time ￿ rstorder methods to compute a re ￿ nement of GNE, namely the variational equilibria (VE), in convex-concave cumulative regret pseudo-games with jointly convex constraints.
More generally, we also show that our methods ￿ nd the stationary points of the exploitability in polynomial time in Lipschitz-smooth pseudo-games with jointly convex constraints. Finally, we demonstrate in experiments that our methods not only outperform known algorithms, but that even in pseudo-games where they are not guaranteed to converge to a GNE, they may do so nonetheless, with proper initialization.
Introduction
Nash equilibrium [1] is the canonical solution concept used to predict the outcome of models of rational agents known as games.
A game comprises a set of players, each of whom simultaneously chooses a strategy (or action) from a strategy set, and then receives a payo ￿ based on all the players’ choices.
A Nash equilibrium (NE) is a strategy pro ￿ le, i.e., a collection of strategies, one per player, from which no player can improve their payo ￿ via a unilateral deviation.
Recently, the literature on NE has expanded beyond classic economic applications to the question of its computation, a problem known as the Nash equilibrium problem (NEP) [2, 3].
Unfortunately, it is unlikely that there exists an e ￿ cient algorithm to compute NE, as NEP is PPAD-complete, i.e.,
NEP is equivalent to the problem of computing solutions to a class of ￿ xed-point problems known as PPAD [4], which are believed to be computationally intractable [5, 6, 7].
Nonetheless, there has been a great deal of recent progress developing e ￿ cient algorithms to compute NE in special classes of games, such as two-player zero-sum [8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 36th Conference Neural Information Processing Systems ( 26, 27, 28, 29, 30, 31], potential [32, 33, 34], and monotone (or, more generally, variationally stable) games [35, 36, 37, 38, 39, 40, 41].
Interest in two-player zero-sum games, for example, stems in part from emerging applications in machine learning, e.g., the training of GANs [42, 43].
A natural generalization of games are pseudo-games or abstract economies , ￿ rst introduced by
Arrow and Debreu [44] in the context of markets, in which the actions taken by each player a ￿ ect not only the other players’ payo ￿ s, as in games, but also the other players’ strategy sets.
In other words, players (simultaneously) take actions that can restrict the other players’ strategy sets.
Such a model is not technically a game, giving rise to the pseudo-game terminology.
Pseudo-games generalize games, and hence are even more widely applicable.
Some recently studied applications include adversarial classi ￿ cation [45, 46], energy resource allocation [47, 48], environmental protection [49, 50], cloud computing [51, 52], adversarial classi ￿ cation, ride sharing services [53], transportation [54], wireless and network communication [55, 56].
The solution concept par excellence for pseudo-games is the generalized Nash equilibrium (GNE) [44, 57, 58], i.e., a strategy pro ￿ le at which each player’s strategy is feasible and no player can improve their payo ￿ s by unilaterally deviating to another strategy in the strategy set determined by the other players’ strategies.
The aforementioned applications speak to the importance of developing algorithms that can approximate GNEs e ￿ ciently in the largest class of pseudo-games possible.
Work in this direction, on the generalized Nash equilibrium problem (GNEP) , is progressing; see, for example, [59, 57, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77].
Nonetheless, there are still few, if any [78], GNE- ￿ nding algorithms with computational guarantees, even for restricted classes of pseudo-games
Exploitability , or the Nikoida-Isoda potential [79], is de ￿ ned as the sum of the players’ payo ￿ - maximizing unilateral deviations w.r.t.
a given strategy pro ￿ le.
Minimizing exploitability is a logical approach to computing GNE, especially in cases where exploitability is convex, such as in the pseudo-games that result from replacing each player’s payo ￿ function in a monotone pseudo-game by a second-order Taylor approximation [80].
When the players’ payo ￿ s are also Lipschitz-smooth and strongly concave (in their own action), this approximation can be quite good, because the error in the approximation will be bounded by the Lipschitz-smoothness and strong concavity constant.
Minimizing exploitability can thus be an e ￿ ective way of ￿ nding GNEs in pseudo-games; however, to our knowledge, no convergence rates for this approach are known, and few methods have been proposed for pseudo-games whose exploitability is non-convex.
In this paper, we develop e ￿ cient exploitability-minimization algorithms that converge to variational equilibrium (VE), a re ￿ nement of GNE, exactly in a large class of pseudo-games with jointly convex constraints (which includes, but is not limited to, two-player zero-sum, n -player pairwise zero-sum, and a large class of monotone and bilinear pseudo-games, as well as Cournot oligopoly games); and approximately, in Lipschitz-smooth pseudo-games with jointly convex constraints, where exact computation is intractable.
Our algorithms apply not only to games with discrete action spaces but also to games with continuous action spaces; previous exploitability-minimization algorithms concern only ￿ nite games [81, 82].
The cumulative regret of two strategy pro ￿ les is de ￿ ned as the sum, across all players, of the change in utility due to unilateral deviations from one strategy pro ￿ le to the other.
Our algorithms and their analysis rely on the following key insights: assuming jointly convex constraints (i.e., symmetric and convex-valued joint constraint correspondences)
the exploitability-minimization problem is equivalent to solving a cumulative regret min-max optimization problem, which although non-convex-concave in general can nonetheless be solved by gradient descent ascent (GDA) methods 1 [25, 85] using a parametric variant of Moreau’s envelope theorem that we introduce (Theorem 4.1).
We apply this logic to develop two algorithms for Lipschitz-smooth pseudo-games with jointly convex constraints.
To the best of our knowledge, this makes ours the ￿ rst exploitabilityminimization methods with convergence rate guarantees in this class of pseudo-games.
The ￿ rst (EDA; Algorithm 1) is an extragradient method that converges in average iterates to an " -VE, and hence an " -GNE, in O ( 1 / " ) iterations in pseudo-games with convex-concave cumulative regret, where " is the exploitability of the game (Theorem 3.4).
This class of pseudo-games includes zero-sum, potential, Cournot, a large class of monotone pseudo-games, and a class of bilinear pseudo-games, namely those with convex second-order approximations.
Our second algorithm (ADA; Algorithm 2) is a nested GDA algorithm, whose best iterate converges more generally to a stationary point of (regularized) exploitability in all Lipschitz-smooth pseudo-games with jointly convex constraints at the slightly slower rate of O ( log( 1 / " ) / " )
(Theorem 4.2).
When the (regularized) exploitability is additionally invex, our method also converges to a GNE in best iterates at the same rate.
This convergence can be strengthened to convergence in last iterates in O (log( 1 / " ) 2 ) iterations in strongly-convex exploitability pseudo-games and in a certain class of pseudo-games with non-convex exploitability (Theorem 4.3).
Pseudo-games with jointly convex constraints are of interest in decentralized resource allocation problems with feasibility (e.g., supply) constraints.
Feasibility constraints are almost always joint, and often also jointly convex, since we generally require the sum of the allocations to be less than or equal to the supply.
One of the many examples of such pseudo-games is wireless and network communication, where agents communicate packets across a shared network, with limited capacities [55].
The experiments we run in this paper suggest that our algorithms have the potential to yield improved solutions to such pseudo-games as compared to other known approaches, and thus open up avenues for practical applications of our work in the future.
Related Work Following Arrow and Debreu’s introduction of GNE, Rosen [86] initiated the study of the mathematical and computational properties of GNE in pseudo-games with jointly convex constraints, proposing a projected gradient method to compute GNE.
Thirty years later,
Uryas’ev and Rubinstein [87] developed the ￿ rst relaxation methods for ￿ nding GNEs, which were improved upon in subsequent works [88, 89].
Two other types of algorithms were also introduced to the literature: Newton-style methods [59, 64, 68, 69, 70, 90] and interior-point potential methods [90].
Many of these approaches are based on minimizing the exploitability of the pseudo-game, but others use variational inequality[91, 92] and Lemke methods [93].
More recently, novel methods that transform GNEP to NEP were analyzed.
These models take the form of either exact penalization methods, which lift the constraints into the objective function via a penalty term [72, 73, 76, 77, 94], or augmented Lagrangian methods [71, 74, 76, 95], which do the same, augmented by dual Lagrangian variables.
Using these methods, Jordan, Lin, and Zampetakis [78] provide the ￿ rst convergence rates to a " -GNE in monotone (resp.
strongly monotone) pseudogames with jointly a ￿ ne constraints in ˜ O
( ˜ O ( 1 / p " ) ) iterations.
These algorithms, despite being highly e ￿ cient in theory, are numerically unstable in practice [78].
Nearly all of the aforementioned approaches concerned pseudo-games with jointly convex constraints.
Exploitability minimization has also been a valuable tool in multi-agent reinforcement learning; algorithms in this literature that aim to minimize exploitability are known as exploitability descent algorithms [81].
Lockhart et al.
[81] analyzed exploitability descent in two-player, zero-sum, extensive-form games with ￿ nite action spaces.
Variants of exploitability-descent have also been combined with entropic regularization and homotopy methods to solve for NE in large games [82].
Preliminaries
Notation We use caligraphic uppercase letters to denote sets (e.g., X ); bold lowercase letters to denote vectors (e.g., p , ⇡ ); bold uppercase letters to denote matrices (e.g., X ) and lowercase letters to denote scalar quantities (e.g., x,  ).
We denote the i th row vector of a matrix (e.g., X ) by the corresponding bold lowercase letter with subscript
i (e.g., x i ) .
Similarly, we denote the j th entry of a vector (e.g., p or x i ) by the corresponding Roman lowercase letter with subscript j (e.g., p j or x ij ).
We denote functions by a letter determined by the value of the function, e.g., f if the mapping is scalar-valued, f if the mapping is vector-valued, and F if the mapping is set-valued.
We denote the set of integers { 1 , . . . , n } by [ n ] , the set of natural numbers by N , the set of real numbers by R , and the positive and strictly positive elements of a set by a + and ++ subscript, respectively, e.g., R + and R ++ .
We de ￿ ne the gradient r x : C 1 ( X ) !
C
( X ) as the operator which takes as input a functional f : X ! R , and outputs a vector-valued function consisting of the partial derivatives of f w.r.t.
x .
We denote the orthogonal projection onto a set C by ⇧ C , i.e., ⇧ C ( x ) =
arg min y 2 C k x
y k 22 .
Finally, we write X ( a ) to denote the indicator function of the set X , i.e., X ( x ) {+= 
+}
if x 2 X and X ( x ) {+= 
+}
if x / 2 X .
A pseudo-game [44] G .
=
( n, A , X , h , u ) comprises n 2 N + players, each i 2 [ n ] of whom chooses an a i from an action space
A i .
We denote the players’ joint action space by A = ⇥
i 2 [ n ] A i .
Each player i aims to maximize their continuous utility
u i :
A ! R , which is concave in a i , by choosing a feasible action from a set of actions
X i ( a  i ) ✓ A i determined by the actions a  i
A  i ⇢ R m of the other players, where X i : A  i ◆ A i is a non-empty, continuous, compact- and convex-valued action correspondence, which for convenience we represent as
X i ( a  i ) =
{ a i 2 A i | h ik ( a i , a  i )  0 , for all k [-
-]2 [ d ] } , where for all i 2 [ n ] , and k 2 [ d ] , h ik is a continuous and concave function in a i , which de ￿ nes the constraints.
We denote the product of the feasible action correspondences of all players by X ( a ) = ⇥
i 2 [ n ] X i ( a  i ) , which we note is guaranteed to be non-empty, continuous, and compact-valued, but not necessarily convexvalued.
Additionally, overloading notation, we de ￿ ne the set of jointly feasible strategy pro ￿ les
X = { a 2 A | h ik ( a )  0 , for all i 2 [ n ] , k 2 [ d ] } .
A two player pseudo-game is called zero-sum if for all a 2 A , u 1 ( a ) =
u 2 ( a ) .
A pseudo-game is called monotone if for all a , b 2 A
P i 2 [ n ]  r a i u i ( a )  r a i u i ( b )
T ( a  b )  0 .
A pseudogame is said to have joint constraints if there exists a function g : A !
R d s.t.
g = h
= . . .
=
h d .
If additionally, for all k 2 [ d ] , g k ( a ) is concave in a , then we say that the pseudo-game has jointly convex constraints , as this assumption implies that X is a convex set.
A game [1] is a pseudogame where, for all players i 2 [ n ] , X i is a constant correspondence, i.e., for all players i 2 [ n ] , X i ( a  i ) =
X i ( b  i ) , for all a , b 2 A . Moreover, while X ( a ) is convex, for all action pro ￿ les a 2 A , X is not guaranteed to be convex unless one assumes joint convexity.
Given a pseudo-game G , an " - generalized Nash equilibrium (GNE) is strategy pro ￿ le a ⇤ 2 X ( a ⇤ ) s.t.
for all i 2 [ n ] and a i 2 X i ( a ⇤ i ) , u i ( a ⇤ )
u i ( a i , a ⇤ i )  " .
An " - variational equilibrium (VE) (or normalized GNE ) of a pseudo-game is a strategy pro ￿ le a ⇤ 2 X ( a ⇤ ) s.t.
for all i 2 [ n ] and a 2 X , u i ( a ⇤ )
u i ( a i , a ⇤ i )  " .
We note that in the above de ￿ nitions, one could just as well write a ⇤ 2 X ( a ⇤ ) as a ⇤ 2 X , as any ￿ xed point of the joint action correspondence is also a jointly feasible action pro ￿ le, and vice versa.
A GNE (VE) is an " -GNE (VE) with " = 0 .
Under our assumptions, while GNE are guaranteed to exist in all pseudo-games by Arrow and Debreu’s lemma on abstract economies [44], VE are only guaranteed to exist in pseudo-games with jointly convex constraints [65].
Note that the set of " -VE of a pseudo-game is a subset of the set of the " -GNE, as X ( a ⇤ ) ✓ X , for all a ⇤ which are GNE of G .
The converse, however, is not true, unless A ✓ X .
Further, when G is a game, GNE and VE coincide; we refer to this set simply as NE.
Mathematical Preliminaries Finally, we de ￿ ne several mathematical concepts that are relevant to our convergence proofs.
Given A ⇢ R n , the function f : A !
R is said to be ` f - Lipschitzcontinuous i ￿ 8 x 1 , x 2 2 X , k f ( x 1 )  f ( x 2 )
k  ` f k x 1  x 2 k .
If the gradient of f is ` r f - Lipschitz-continuous, we then refer to f as ` r f - Lipschitz-smooth .
A function f : X !
R is said to be invex w.r.t.
a function h :
X ⇥ X !
X if f ( x )  f ( y )
h ( x , y ) ·
r f ( y ) , for all x , y 2 X .
A function f :
X !
R is convex if it is invex w.r.t.
h ( x , y ) =
x  y and concave if  f is convex.
A function f : X !
R is µ - strongly convex (SC) if f ( x 1 )  f ( x 2 )
+ hr x f ( x 2 )
, x 1  x 2 i + µ / 2 k x 1  x 1 k 2 , and µ - strongly concave if  f is µ -strongly convex.
From Exploitability Minimization To Min-Max Optimization
In this section, we reformulate the exploitability minimization problem as a min-max optimization problem.
This reformulation is a consequence of the following observation, which is key to our analysis:
In pseudo-games for which a VE exists, e.g., pseudo-games with jointly convex constraints, the quasi-optimization problem of minimizing exploitability, whose solutions characterize the set of
GNEs, reduces to the reduces to a standard min-max optimization problem (with independent constraint sets) , which characterizes the set of VEs, a subset of the GNEs.
This new perspective allows us to e ￿ ciently solve GNEP by computing VEs in a large class of pseudo-games as an immediate consequence of known results on the computation of saddle points in convex-concave min-max optimization problems.
Given a pseudo-game G , we de ￿ ne the regret felt by any player i 2 [ n ] for an action a i as compared to another action b i , given the action pro ￿ le a  i of other players, as follows:
Regret i ( a i , b i ; a  i ) = u i ( b i , a  i )
u i ( a i , a  i ) .
Additionally, the cumulative regret , or the
Nikaida-Isoda function , : A ⇥ A !
R between two action pro ￿ les a 2 X and b 2 X across all players in a pseudo-game is given by ( a , b ) =
P i 2 [ n ] Regret i ( a i , b i ; a  i ) .
Further, the exploitability , or Nikaido-Isoda potential function , ' : A !
R of an action pro ￿ le a is de ￿ ned as ' ( a ) =
P i 2 [ n ] max b i
X i ( a  i ) Regret
i ( a i , b i ; a  i ) [57].
Notice that the max is taken over X i ( a  i ) , since a player can only deviate within the set of available strategies.
A well-known [57] 2 result is that any unexploitable strategy pro ￿ le in a pseudo-game is a GNE.
Lemma 3.1.
Given a pseudo-game G , for all a 2 A , ' ( a )  0 .
Additionally, a strategy pro ￿ le a ⇤ 2 X ( a ⇤ ) is a GNE i
￿ it achieves the lowerbound, i.e., ' ( a ⇤ )
= 0 .
This lemma tells us that we can reformulate GNEP as the quasi-optimization problem of minimizing exploitability, i.e., min a 2 X ( a ) ' ( a ) .
Here, “quasi” refers to the fact that a solution to this problem is both a minimizer of ' and a ￿ xed point a ⇤ s.t.
a ⇤ 2 X ( a ⇤ ) .
Despite this reformulation of
GNEP in terms of exploitability, no exploitability-minimization algorithms with convergence rate guarantees are known.
The unexploitability (!) of exploitability may be due to the fact that it is not
Lipschitz-smooth.
The key insight that allows us to obtain convergence guarantees is that we treat the GNE problem not as a quasi-minimization problem, but rather as a quasi-min-max optimization problem, namely min a 2 X ( a ) ' ( a ) =
min a 2 X ( a ) max b 2 X ( a ) ( a , b ) .
Observation 3.2.
Given a pseudo-game G , for all a 2 A , ' ( a ) = max b 2 X ( a ) ( a , b ) .
Proof.
The per-player maximum operators can be pulled out of the sum, as the i th player’s best action in hindsight is independent of the other players’ best actions in hindsight, since the action pro ￿ le a is ￿ xed:
[Equation]
That is, the optimization problem in the i th summand is independent of the optimization problem in the other summands, e.g., for any y [Equation] [Equation] If we restrict our attention to VEs, a subset of GNEs, VE exploitability— hereafter exploitability for short —is conveniently expressed as ' ( a ) = max b 2 X ( a , b ) .
When a VE exists, e.g., in pseudo-games with jointly convex constraints, this exploitability is guaranteed to achieve the lower bound of 0 for some a 2 X .
In such cases, formulation of exploitability minimization as a quasi-min-max optimization problem reduces to a standard min-max optimization problem, namely min a 2 X max b 2 X ( a , b ) , which characterizes VEs.
This problem is well understood when is a convex-concave objective function [21, 83, 84, 96].
Furthermore, the cumulative regret is indeed convex-concave, i.e., convex in a and concave in b , in many pseudo-games of interest: e.g., two-player zero-sum, n -player pairwise zero-sum, and a large class of monotone and bilinear pseudo-games, as well as Cournot oligopoly games.
Going forward, we restrict our attention to pseudo-games satisfying the following assumptions.
Lipschitz-smoothness is a standard assumption in the convex optimization literature [97], while joint convexity is a standard assumption in the GNE literature [57].
Furthermore, these are weaker assumptions than ones made to obtain the few known results on convergence rates to GNEs [78].
Assumption 3.3.
The pseudo-game G has joint constraints g : A !
R d , and additionally, 1.
(Lipschitz smoothness and concavity) for any player
i 2 [ n ] , their utility function u i is ` r u -Lipchitz smooth; 2.
(Joint Convexity) g is component-wise concave.
Using the simple observation that every VE of a pseudo-game is the solution to a min-max optimization problem we introduce our ￿ rst algorithm (EDA; Algorithm 1), an extragradient method [83].
The algorithm works by interleaving extragradient ascent and descent steps: at iteration t , given a ( t ) , it ascends on ( a ( t ) , · ) , thereby generating a better response b ( t +1) , and then descends on ( · , b ( t +1) ) , thereby decreasing exploitability. We combine several known results about the
This problem could also be formulated as minimizing over the feasible e
Inputs: n, u , X , ⌘ , T, a (0) , b (0)
Outputs: ( a ( t ) , b ( t ) )
Tt = 1: for t = 0 , . .
, T  1 do 2: a ( t +1 / 2) =
⇧ X ⇥ a ( t )  ⌘ r a ( a ( t ) , b ( t ) )
⇤ 3: b ( t +1 / 2) =
⇧ X ⇥ b ( t ) + ⌘ r b ( a ( t ) , b ( t ) )
⇤ 4: a ( t +1) =
⇧ X h a ( t )  ⌘ r a ( a ( t +1 / 2) , b ( t +1 / 2) )
i 5: b ( t +1) =
⇧ X h b ( t ) + ⌘ r b ( a ( t +1 / 2) , b ( t +1 / 2) )
i 6: return ( a ( t ) , b ( t ) )
Tt = convergence of extragradient descent methods in min-max optimization problems to obtain the following convergence guarantees for EDA in pseudo-games.
Theorem 3.4 (Convergence rate of EDA) .
Consider a pseudo-game G with convex-concave cumulative regret that satis ￿ es Assumption 3.3.
Suppose that EDA (Algorithm 1) is run with ⌘ < 1 / ` r , and that doing so generates the sequence of iterates ( a ( t ) , b ( t ) )
Tt =0 .
Let a ( T ) = 1 / T
P
Tt =1 a ( t ) and b ( T ) = 1 / T
P
Tt =1 b ( t ) .
Then the following convergence rate to a VE, i.e., to zero exploitability holds: max b 2 X ( a ( T ) , b )  max a 2 X ( a , b ( T ) )
 1 / T ( d ` r ) , where d = max ( a , b ) 2 X ⇥ X  ( a , b )  ( a (0) , b (0) )
2 .
If, additionally, is µ -strongly-convex- µ -stronglyconcave, and the learning rate ⌘ = 1 / 4 ` , then the following convergence bound also holds: max b 2 X
[Equation]
a [Equation] [Equation]
/ 4 ` [Equation] T . Remark 3.5.
If the players’ utility functions have Lipschitz-continuous third-order derivatives, then convergence in last iterates can be obtained in O ( 1 / " 2 ) iterations [98]. Since the complexity of two-player zero-sum convex-concave games, a special case of our model, is ⌦ ( 1 / " ) , the iteration complexity we derive is optimal. The exploitability minimization via EDA is thus an optimal approach to GNEP in convex-concave cumulative regret pseudo-games. Additionally, since pseudo-games generalize games, our results show that the complexity of this class of pseudogames is the same as that of the corresponding games.
More generally, the exploitability minimization problem is equivalent to a non-convex-concave min-max optimization problem, i.e., cumulative regret is non-convex-concave. In such cases, a minimax theorem does not hold, which precludes the existence of a saddle point, making the problem much harder. Solving non-convex-concave min-max optimization problems is NP-hard in general [99], so we instead set as our goal ￿ nding a ( ￿ rst-order) stationary point a ⇤ 2 X of the exploitability ' , i.e., a point at which ￿ rst-order deviations cannot decrease the exploitability, which we show can be found in polynomial-time.
Note that any stationary point a ⇤ of the exploitability is a ' ( a ⇤ ) -GNE of the associated pseudo-game.
In recent years, a variety of methods have been developed that could be applied to ￿ nding stationary points of the exploitability [19, 25, 26, 27, 31].
Using the most e ￿ cient of these methods, one could obtain convergence to an " -stationary point of the exploitability in O ( 1 / " 6 ) iterations [19].
However, as we will see, this rate would be very slow, and the convergence metric, not very strong, e.g., convergence in expected iterates.
Instead, we demonstrate how to leverage the structure of the exploitability-minimization problem to obtain much faster convergence rates.
Unfortunately, the exploitability associated with the min-max characterization of pseudo-games with joint constraints is non-di ￿ erentiable in general.
This fact poses an obstacle when trying to design e ￿ cient algorithms that ￿ nd its stationary points.
However, by exploiting the structure of cumulative regret, we can regularize it to obtain a smooth objective.
Observe the following: if a ⇤ 2 arg min a 2 X max b 2 X ( a , b ) , then a ⇤ 2 arg max b 2 X ( a ⇤ , b ) . In other words, b ⇤ =
a ⇤ is a solution to the inner maximization problem.
As a result, we can penalize exploitability in proportion to the distance between a and b , while still ensuring that this penalized exploitability is minimized at a VE.
We thus optimize the ↵ - regularized cumulative regret ↵ : A ⇥ A !
R , de ￿ ned as ↵ ( a , b ) .
= ( a , b )  ↵ 2 k a  b k 22 , whose associated ↵ -regularized exploitability ' ↵ : A !
R is given by ' ↵ ( a ) = max b 2 X ↵ ( a , b ) .
Von Heusinger and Kanzow show that a strategy pro ￿ le a ⇤ has no ↵ -regularized-exploitability, i.e., ' ↵ ( a ⇤ )
= 0 , i ￿ a ⇤ is a VE for all ↵ >(Theorem 3.3 [65]).
To better understand the smoothness properties of the regularized exploitability, we present the parametric Moreau envelope , a Moreau envelope [100] in which the objective function is a function of the point w.r.t.
which we regularize the objective function, which in our case is cumulative regret.
Our next theorem is a generalization of the Moreau envelope theorem [100], which shows that ↵ -regularized exploitability is Lipschitz-smooth, by allowing us to see ' ↵ as a parametric Moreau envelope for , an observation which to our knowledge has not been made in previous work.
This Lipschitz-smoothness of ' ↵ gives rise to the possibility of deriving algorithms that converge to a stationary point of ' ↵ , the key idea underlying our second algorithm (ADA; Algorithm 2).
Theorem 4.1 (Parameteric Moreau Envelope Theorem) .
Given ↵ > 0 , consider the parametric
Moreau envelope ' ↵ ( a ) .
= max b 2 X n ( a , b )  ↵ 2 k a  b ko and the associated proximal operator { b ⇤ ( a ) } . = arg max b 2 X n
[Equation] a [Equation] o , where is ` r -Lipschitz smooth.
Then ' ↵ ( a ) is ⇣
[Equation]
-Lipschitz-smooth, with gradients r
a [Equation] ↵ ( [Equation]
Next, we formalize the de ￿ nition of a stationary point.
As the exploitability ' ↵ is in general non-convex, and the optimization problem min a 2 X ' ↵ ( a ) is constrained, the ￿ rst-order condition r ' ↵ ( a ) = 0 is not su ￿ cient for stationarity, since a solution on the boundary of the constraint set could also be ￿ rst-order stationary.
As a result, we use proximal mappings in our de ￿ nition.
In particular, we reformulate the constrained exploitability minimization problem as the unconstrained minimization problem min a 2 R nm ' ↵ ( a ) + X ( a )[-is the indicator function of the set X ,i.e., X ( x ) = 
 if x 2 X and X ( x ) = 
 if x / 2 X-] .
In this optimization problem, ' ↵ is continuous and Lipschitz smooth, while X is continuous and convex.
Additionally, the set of solutions is non-empty.
As a result, one can solve this problem using the proximal gradient method, which, in this case, is equivalent to the projected gradient method [97, 101].
This view of the problem as proximal minimization allows us to provide a clear de ￿ nition of a ￿ rst-order stationary point.
De ￿ ne the projected gradient operator of a function ' : A ! R for some step size ⌘ a > 0 as follows: G
[Equation] i .
The projected gradient operator is a special case of the gradient mapping operator, and a generalization of the gradient that is projected back onto the feasible set X [101].
The zeros of the projected gradient operator capture stationary points both at the boundary of the set X and in its interior.
Indeed, a 2 A is a stationary point of ' if G ' ⌘ a ( a ) = 0 .
Note that, for an action pro ￿ le a ⇤ 2 A to be a VE, it is necessary that it is stationary with respect to the VE exploitability (Proposition 3, [80]).
Our goal will thus be to achieve convergence to a strategy pro ￿ le a 2 X s.t.
G ' ↵ ⌘ a ( a ) = 0 .
ADA (Algorithm 2) is a nested GDA algorithm, similar to those of Nouiehed et al. [25] and Goktas and Greenwald [85], with the objective function augmented by a smart regularizer.
The algorithm is reminiscent of EDA, interleaving gradient ascent and descent steps.
The key di ￿ erence is that whereas EDA runs only a single step of extragradient ascent (to ￿ nd a better response), ADA runs multiple steps of gradient ascent to approximate a best response.
Indeed, as Theorem 4.2 shows, ADA’s accuracy depends on the accuracy of the best-response found.
To derive convergence rates for ADA, we rely on two technical lemmas. The ￿ rst tells us that the inner loop can approximate the gradient of the exploitability to any desired precision. More speci ￿ cally, we bound the error [Equation]
 " by providing a lower bound on the number of inner loop iterations T b necessary to achieve a desired precision " , for all outer loop iterations t 2 [ T a ] (Lemma B.3). The second lemma bounds the algorithm’s progress at each iteration of the outer loop by relating " and G ' ↵ ⌘ a ( a t ) (Lemma B.4). Finally, by telescoping the inequalities in the progress lemma, we obtain our main theorem: The best iterate found by
Inputs: n, u , g , ↵ , ⌘ a , ⌘ b , T a , T b , a (0) , b (0)
Outputs: ( a ( t ) , b ( t ) )
T a t = 1: for t = 0 , . .
, T a  1 do 2: a ( t +1) =
⇧ X h a ( t )  ⌘ a  r a ( a ( t ) , b ( t ) )
↵ ( a ( t )  b ( t ) )
i 3: b =4: for s = 0 , . .
, T b  1 do 5: b = ⇧ X ⇥ b + ⌘ b  r b ( a ( t +1) , b )  ↵ ( a ( t +1)  b ) ⇤ 6: b ( t +1)
=
b 7: return ( a ( t ) , b ( t ) )
T a t =
ADA, as measured by the norm of the projected gradient operator, converges to an approximate stationary point of the exploitability in all Lipschitz-smooth pseudo-games with jointly convex constraints.
We note that if the gradient of the regularized exploitability can be computed exactly, then convergence to an exact stationary point can be obtained.
Theorem 4.2 (Convergence to Stationary Point of Exploitability) .
Suppose that ADA is run on a pseudo-game G which satis
￿ es Assumption 3.3 with learning rates ⌘ a > ` r ↵ + ` 2 r ↵ ↵ and ⌘ b = 1 ` ↵ , for any number of outer loop iterations T a 2 N ++ and for T b  2log ⇣ " ` r ↵ p 2 ↵ c ⌘ log ⇣ ↵ ` r ↵ ⌘ total inner loop iterations where " > 0 .
Then the outputs ( a ( t ) , b ( t ) )
Tt =
satisfy min t =0 ,...,T a
G ' ↵ ⌘ a ( a ( t ) )  2  1 ⌘ a  ` r ↵ + ` 2 r ↵ ↵' ↵ ( a (0) ) T a + " ⇣ ` r ↵ + `
[Equation]
Note that the stationary points to which ADA converges are global minima of ' ↵ i ￿ ' ↵ is invex, in which case ADA converges to a VE, and hence a GNE [102].
We can also obtain faster and last iterate convergence to VE in a more restricted class of games.
A pseudo-game is said to be a µ - PL-pseudo-game if the regularized exploitability ' ↵ satis ￿ es the µ - projection-PolyakLojasiewicz (PL) condition, i.e., the projected gradient operator associated with ' ↵ satis ￿ es
[Equation]
for all a 2 X .
5 A similar PL-condition for games was recently used by Raghunathan, Cherian, and Jha [104], which they argued is natural.
Theorem 4.3 (PL Exploitability Convergence) .
Suppose ADA is run on a PL-pseudo-game G which satis ￿ es Assumption 3.3 with learning rates ⌘ ah
[Equation]
i and ⌘ b = 1 / ` r ↵ , for any number of outer loop iterations T a
[Equation]
+ and for T b
[Equation]
total inner loop iterations.
Then the outputs ( a ( t ) , b ( t ) )
Tt
[Equation]
satisfy
[Equation]
We note that this result also implies convergence to a VE in strongly-convex exploitability games in linear time as well, since strong convexity implies the PL-condition.
Experiments
In this section, we report on experiments that demonstrate the e ￿ ectiveness of our algorithms as compared to others that were also designed to compute GNE.
We present numerical results on benchmark psuedo-games that were studied empirically in previous work [104]: 1) monotone pseudo-games with a ￿ ne constraints and 2) bilinear pseudo-games with jointly convex constraints.
We compare our algorithms to the accelerated mirror-prox quadratic penalty method (AMPQP), the only GNE- ￿ nding algorithm with theoretical convergence guarantees and rates [78].
For the AMPQP algorithm, we use the hyperparameter settings derived in theory when available, and otherwise conduct a grid search for the best ones.
We compare the convergence rates of our algorithms to that of AMPQP, which converges at the same rate as EDA, but slower than ADA, in settings in which it is guaranteed to converge, i.e., in monotone pseudo-games.
In the monotone pseudo-games, for EDA we use a constant learning rate of ⌘ = 0 .
02 , while for
ADA we use the constant learning rates of ⌘ a = 0 .
02 and ⌘ a = 0 . 05 .
These rates approximate the Lipschitz-smoothness parameter of the pseudo-games. In the bilinear pseudo-games, we use the theoretically-grounded (Theorem 3.4) learning rate of ⌘ = 1 / k Q k for EDA, and ⌘ a = k Q k + k Q k 2 ↵ and ⌘ b = 1 / k Q k for ADA.
In both settings, we use a regularization of ↵ = 0 .
1 for ADA, which we found by grid search.
Likewise, in our implementation of AMPQP, grid search led us to initialize  0 = 0 .
01 ,  = 1 . 05 , ↵ = 0 .
2 in both settings.
Additionally, because the iterates generated by AMPQP are not necessarily feasible, we projected them onto the feasible set of actions, so as to avoid in ￿ nite exploitability.
This heuristic improved the convergence rate of AMPQP signi ￿ cantly.
The ￿ rst iterate for all methods was common, and initialized at random.
Monotone Games with Jointly A ￿ ne Constraints We begin by experimenting with monotone games with jointly a ￿ ne constraints, the largest class of pseudo-games for which convergence rates exist for AMPQP.
In particular, we consider norm-minimization pseudo-games with payo ￿ functions u
i ( a ) =
P i 2 [ n ] a i  s i  , for all players i 2 [ n ] , and for some randomly initialized shifting factors s i , as well as jointly a ￿ ne constraints g ( a ) = 1
P j 2 [ m ] a 1 j
P i 2 [ m ] a 2 j , with action spaces A i =
[  10 , 10] m , for all players i 2 [ n ] .
Although norm-minimization games are trivial, in the sense that they can be solved in closed form, this is not the case for pseudo-games; on the contrary, they form a bedrock example for monotone pseudo-games ([57], Example 1).
In Figure 1a, we observe that all three algorithms converge to a GNE.
Interestingly, EDA and ADA ￿ nd distinct GNEs from AMPQP, which is not entirely surprising, as AMPQP is not an exploitability minimization algorithm.
Although all the algorithms eventually ￿ nd a GNE, exploitability does not decrease monotonically (Figure 1b); on the contrary, it increases before eventually decreasing to zero.
Convergence to a GNE is expected for AMPQP, as per the theory.
EDA, however, is not guaranteed to converge to a GNE in pseudo-games beyond convex exploitability; yet, we still observe convergence, albeit at a slower rate than AMPQP or ADA.
Likewise, ADA converges to a GNE, although it is again not guaranteed, much faster than AMPQP.
Bilinear General-Sum Games with Jointly Convex Constraints
Second, we consider the class of two-player general-sum games with jointly convex constraints with payo ￿ functions u 1 ( a ) =
a 1 T Q 1 a 2 , u 2 ( a ) = a 1 T Q 2 a 2 , g ( a ) =
k a k 22 , A
=
A 2 =
[  10 , 10] m .
Such games are not monotone, and hence AMPQP is not guaranteed to converge in theory.
In Figure 1c, we see that convergence is not guaranteed empirically either; shown is a sample run in which ADA and EDA converge, while AMPQP exhibits erratic behavior.
Non-convergence for AMPQP was [Figure] observed in 93 of the 100 experiments run.
In fact, even though AMPQP gets very close a GNE (Figure 1d), i.e., zero exploitability, it does not stabilize there.
We likewise found that ADA and EDA do not converge to a GNE from all random initializations.
In such cases, we restarted the algorithms with a new random initialization; this process lead to on average 11 restarts for EDA and 10 restarts for ADA, meaning both algorithms converged to a GNE within this window.
Conclusion
In this paper, we provide an a ￿ rmative answer to an open question ￿ rst posed by Flam and
Ruszczynski [80], who suggested that projected gradient methods could be developed to minimize exploitability in pseudo-games.
Although some existing methods [57] make use of the exploitability concept to analyze algorithms that take alternative optimization approaches, we know of none that are exploitability-minimization algorithms per se, as exploitability itself is not explicitly minimized.
Our main contribution is the observation that the exploitability minimization problem can be recast as a simple and seemingly trivial min-max optimization problem (Observation 3.2). Our analysis e ￿ ectively extends the class of games which can be solved as potential games, since the exploitability can be seen as a potential for any game. We provide a de ￿ nition of stationarity for exploitability via the gradient mapping operator from proximal theory, and we use this de ￿ nition to characterize the limit points of our algorithms.
Our characterization relies on novel proof techniques inspired by recent analyses of algorithms for min-max optimization problems [25], as well as a parameterized version of Moreau’s envelope theorem, which may be of independent interest. This approach, which takes advantage of the structural properties of exploitability, allows us to rigorously analyze an algorithm that converges to stationary points of the VE exploitability in all Lipschitz-smooth pseudo-games with jointly convex constraints, and obtain algorithms which are orders of magnitude faster than known minmax optimization algorithms for similar types of problems. To summarize, our results 1. extend known polynomial-time convergence guarantees to VEs, and thus GNEs, to a class of pseudo-games beyond monotone, and 2.
provide a general approach to approximate, with convergence guarantees, solutions to Lipschitz-smooth pseudo-games with jointly convex constraints.
The design of e ￿ cient algorithms for constrained non-convex-concave min-max optimization problems is an open problem.
The literature thus far has focused on problems in which only the variable to be maximized is constrained [25, 27, 29].
Our techniques may provide a path to solving non-convex-concave min-max optimization problems in entirely constrained domains, since our algorithms solve a non-convex-concave optimization problem where both variables are constrained. Acknowledgments and Disclosure of Funding
The ideas in this paper stemmed from conversations and seminars held during the Learning in Games workshop at the Simons Institute for the Theory of Computing.
This work was also supported by
NSF Grant CMMI-1761546.