F INDING AND ONLY FINDING LOCAL N ASH EQUILIBRIA BY BOTH PRETENDING TO BE A FOLLOWER
A BSTRACT
Finding local Nash equilibria in two-player differentiable games is a classical problem in game theory with important relevance in machine learning.
We propose double Follow-the-Ridge (double-FTR), an algorithm that locally converges to and only to local Nash equilibria in general-sum two-player differentiable games.
To our knowledge, double-FTR is the first algorithm with such guarantees for general-sum games.
Furthermore, we show that by varying its preconditioner, double-FTR leads to a broader family of algorithms with the same properties.
Double-FTR avoids oscillation near equilibria due to the real-eigenvalues of its Jacobian at critical points.
Finally, we empirically verify the effectiveness of double-FTR in finding local Nash equilibria in two simple examples.
I NTRODUCTION
Much of the recent success in deep learning can be attributed to the effectiveness of gradient-based optimization.
It is well-known that for a minimization problem, with appropriate choice of learning rates, gradient descent has convergence guarantee to local minima (Lee et al., 2016; 2019).
Based on this foundational result, an array of accelerated and higher-order methods have since been proposed and widely applied in training neural networks (Duchi et al., 2011; Kingma & Ba, 2014; Reddi et al., 2018; Zhang et al., 2019).
However once we leave the realm of minimization problems and consider the multi-agent setting the optimization landscape becomes much more complicated.
Multi-agent optimization problems arise in diverse fields such as robotics, economics and machine learning (Foerster et al., 2016;
Von Neumann & Morgenstern, 2007; Goodfellow et al., 2014; Ben-Tal & Nemirovski, 2002; Gemp et al., 2020; Anil et al., 2021).
A classical abstraction that is especially relevant for machine learning is two-player differentiable games, where the objective is to find local Nash equilibria.
The equivalent of gradient descent in such a game would be that each agent applies gradient descent to minimize their own objective function.
However, in stark contrast with minimization problems, this gradient-descent-style algorithm may converge to critical points that are not local Nash equilibria, and in the general-sum game case, local
Nash equilibria might not even be stable critical points for this algorithm (Mazumdar et al., 2020)!
These negative results have driven a surge of recent interest in developing other gradient-based algorithms for finding Nash equilibria in differentiable games.
Among them is Mazumdar et al.
(2019), who proposed an update algorithm whose attracting critical points are only local Nash equilibria in the special case of zero-sum games.
However, to the best of our knowledge, such guarantees have not been extended to general-sum games.
We propose double Follow-the-Ridge (double-FTR), a gradient-based algorithm for general-sum differentiable games whose attracting critical points are equivalent to differential Nash equilibria.
Double-FTR is closely related to the Follow-the-Ridge (FTR) algorithm for Stackelberg games (Wang et al., 2019), which converge to and only to local Stackelberg equilibria (Fiez et al., 2019).
Double-FTR can be viewed as its counterpart for simultaneous games, where each player adopts the “follower” strategy in FTR.
The rest of this paper is organized as follows.
In Section 2, we give background on two-player differentiable games and equilibrium concepts.
We also explain the issues with using gradient- descent-style algorithm on such games.
In Section 3, we present the double-FTR algorithm and prove the equivalence of its attracting critical points and differential Nash equilibria.
We also identify a more general class of algorithms that share the properties.
We discuss recent work that is closely-connected to double-FTR in Section 4 and other related work in Section 5.
In Section 6, we show empirical evidence of double-FTR’s convergence to and only to local Nash equilibria.
B ACKGROUND 2. T WO - PLAYER DIFFERENTIABLE GAMES AND EQUILIBRIUM CONCEPTS
In a general-sum two-player differentiable game, player 1 aims to minimize f : R n + m → R with respect to x ∈ R n , whereas player 2 aims to maximize g : R n + m → R with respect to y ∈ R m .
Following the notation in Mazumdar et al.
(2019), we denote such the game as { ( f, − g ) , R n + m } .
We also assume that f and g are twice-differentiable.
For two rational, non-cooperative players, their optimal outcome is to achieve a local Nash equilibrium.
A point ( x ∗ , y ∗ ) is a local Nash equilibrium of { ( f, − g ) , R n + m } if there exists open sets S x ⊂ R n , S y ⊂ R m such that x ∗ ∈ S x , y ∗ ∈ S y , and
[Equation]
As shown in Ratliff et al.
when f and g are twice-differentiable, local Nash equilibria generally satisfy stronger conditions, and are differential Nash equilibria (DNE).
Definition 2.1 (Differential Nash equilibrium) .
( x ∗ , y ∗ ) is a differential Nash equilibrium of { ( f, − g ) , R n + m } if:
• ∇ x f ( x ∗ , y ∗ ) = 0 and ∇ y g ( x ∗ , y ∗ ) = 0 .
• ∇ 2 xx f ( x ∗ , y ∗ ) ≻ 0 and ∇ 2 yy g ( x ∗ , y ∗ ) ≺ 0 .
2. I SSUES WITH GRADIENT - BASED ALGORITHMS
A natural strategy for agents to search for local Nash equilibria in a differentiable game is to use gradient-based algorithms.
The simplest gradient-based algorithm is the gradient descent-ascent (GDA)
(Ryu & Boyd, 2016; Zhang et al., 2021b)
(Algorithm 1) or its variants (Zhang et al., 2021a;
Korpelevich, 1976; Mokhtari et al., 2020).
Require: Number of iterations T , learning rates γ 1 , . . .
, γ T 1: for t = 1 , . . .
, T do 2:
[Equation]
[Equation]
4: end for
To help analyzing the dynamics, we study GDA in a continuous-time limit.
Let z
=
( x , y ) and ω GDA ( z ) =
[ ∇ x f ( x , y ) −∇ y g ( x , y )]
⊤ , the continuous-time flow of GDA is:
[Equation]
[Equation]
where the Jacobian matrix ( J := ∂ ω ( z ) ∂ z ) is:
[Equation]
Using the Jacobian matrix, we then introduce some characterizations on the critical points of the update (as in Mazumdar et al. [Figure]
Definition 2.2 ((Hyperbolic) critical point) .
z ∗ ∈ R n + m is a critical point of ω if ω ( z ∗ ) = 0 .
It is a hyperbolic critical point if Re ( λ ) ̸ = 0 for ∀ λ ∈ spec( J ( z ∗ )) .
Definition 2.3 (Locally asymptotically stable equilibrium (LASE)) .
z ∗ is a locally asymptotically stable equilibrium of the continuous-time dynamics ˙ z = − ω
( z ) if ω ( z ∗ ) = 0 and Re( λ ) > 0 for ∀ λ ∈ spec( J ( z ∗ )) .
LASE points are important for deriving local convergence results, as they are isolated and exponentially attracting under ˙
z = − ω
( z ) (Sastry, 2013).
This means that with proper discretization (i.e. with appropriate choice of learning rate η ), when initialized within the neighbourhood of the LASE point, the discretized system z t +1 = z t − η ω ( z t ) has linear convergence.
We focus on games where all critical points ω GDA are hyperbolic.
This is a reasonable assumption, as hyperbolic critical points are generic for smooth dynamical systems (Sastry, 2013).
Unfortunately, GDA is not guaranteed to converge to DNE, nor are DNE necessarily LASE of the GDA dynamics.
Even in the special case of zero-sum games ( g = f ), GDA dynamics can still have
LASE that are not DNE (Mazumdar et al., 2020).
The relationship is shown in the Venn diagrams in Figure 1.
In Figure 2, we provide simple 2-D examples that demonstrate the failure modes of GDA in zerosum games .
In 2a, GDA converges to a spurious LASE which is not DNE.
In 2b, GDA converges to a limit cycle instead of the unique DNE (Hsieh et al., 2020).
[Figure]
D OUBLE F OLLOW - THE -R IDGE
We propose double Follow-the-Ridge (double-FTR), an update rule for general-sum two-player continuous games whose attracting critical points are equivalent to differential Nash equilibria.
The double-FTR update is shown in Algorithm 2 (the arguments x t , y t of f and g are dropped to avoid notational clutter).
Algorithm
Require:
Learning rate η x and η y ; number of iterations T .
1: for t = 1 , . . .
, T do 2: x t [Equation] ← x t [Equation]
∇
[Equation]
xx f
[Equation]
y g 3: y t +1 ← y t + η y ∇ y g + η x ( ∇ 2 yy g ) − [Equation] ∇ 2 yx f ∇ x f 4: end for
In the limit of infinitesimal learning rates, double-FTR behaves as a continuous flow (let c = η y η x ):
[Equation]
To characterize the stationary points of equation 2, we study its Jacobian in the neighbourhood of an equilibrium. Let z
= ( x , y ) . Applying the approximation
[Equation]
g
[Equation]
≈ J GDA z in the neighbourhood of an equilibrium, we get the following factorized flow for double-FTR: ˙ z = ω FTR ( z ) , where ω FTR ( z ) = − (cid:20) ( ∇ 2 xx f ) −− ( ∇ 2 yy g ) −(cid:21) J ⊤ GDA (cid:20)
[Equation]
3. LASE OF DOUBLE -FTR ⇐⇒ DIFFERENTIAL N ASH EQUILIBRIA
Our main theoretical result is stated below.
Theorem 1.
For a two-player game { ( f, − g ) , R n + m } where all critical points of the GDA flow (equation 1) are hyperbolic, the continuous-time dynamical system in equation 3 satisfies:
• z ∗ is a locally asymptotically stable equilibrium (LASE) of equation 3 ⇐⇒ z ∗ is a differential Nash equilibrium of the game { ( f, − g ) , R n + m } .
• The Jacobian J FTR := ∂ ω FTR ∂
z
at any critical point of equation 3 has real eigenvalues.
In practice, we need to apply a discretization of the continuous flow in equation 3 with sufficiently small learning rates.
We defer the proof of Theorem 1 to Appendix A.
Combining Theorem 1 with the local convergence property of LASE points, we conclude that equation 3 locally converges to and only to differential Nash equilibria.
To the best of our knowledge, the double FTR is the first algorithm with such local convergence result for general-sum games.
3. G ENERAL PRECONDITIONERS FOR DOUBLE -FTR
In the following remark, we show that double-FTR can be generalized to include family of algorithms.
Remark 1.
Theorem 1 applies to a more general version of the double FTR algorithm.
In particular, we can generalize equation 3 to allow a broader class of “preconditioners”:
[Equation]
where P x , P y are functions of x , y respectively, and satisfy P x ≻
⇐⇒ ∇ 2 xx f ≻ 0 , and P y ≺ 0 ⇐⇒ ∇ 2 yy g ≺ 0 .
Remark 1 also provides intuition on the convergence of double-FTR. Without the preconditioner P x and P y , double-FTR reduces to Hamiltonian gradient descent (Mescheder et al., 2017; Balduzzi et al., 2018; Abernethy et al., 2021), which is not guaranteed to only converge to local Nash equilibria.
It is the introduction of the preconditioner that enables LASE points to satisfy the second-order condition of DNE.
C ONNECTION WITH OTHER ALGORITHMS
Mazumdar et al.
(2019) proposed local symplectic surgery (LSS) – a gradient-based algorithm whose LASE are only local Nash equilibria in zero-sum games.
LSS avoids oscillatory behaviour at local Nash equilibria (similar to double-FTR).
Compared to LSS, double-FTR appears to have a simpler form and enables a broader family of algorithms with such LASE characterizations in general-sum games.
The Follow-the-Ridge (FTR) algorithm (Wang et al., 2019) is closely related to our proposed doubleFTR.
FTR was proposed for two-player sequential games and is guaranteed to converge to and only to local minimax.
FTR applies a gradient correction term on the follower in a sequential game, so that the agents approximately follow a ridge in the landscape of the objective function.
The doubleFTR can be viewed as a counterpart of FTR for simultaneous games.
The update rule of double-FTR resembles that of FTR, with the gradient modification term applied on both players.
Another related algorithm is the Hamiltonian gradient descent (HGD)
(Abernethy et al., 2021;
Loizou et al., 2020).
HGD performs gradient-descent on the Hamiltonian, or the squared norm of the gradient.
HGD is guaranteed to converge, as it is essentially a minimization problem.
However, in general it is not guaranteed to converge only to local Nash equilibria.
Interestingly, our double-FTR can be viewed as a preconditioned HGD.
R ELATED WORK
Mazumdar et al.
(2020) introduced a general framework for competitive gradient-based learning.
They characterized local Nash equilibria in terms of the critical points of the gradient algorithms.
They showed the lack of convergence of the gradient algorithm in games, which motivated the development of the double-FTR algorithm.
Much work has focused on improving the dynamics in finding stable fixed points, which is crucial in applications such as GANs, where oscillation caused by eigenvalues with zero real parts of large imaginary parts in the gradient Jacobian can lead to training instability.
Mescheder et al.
(2017) proposes Consensus Optimization, which encourages agreement between the two players by introducing a regularization term in the objectives of both players.
The regularization term results in a more negative real-part for the eigenvalues of the gradient Jacobian, therefore reduces oscillation and allows larger learning rates.
Balduzzi et al.
; Gemp & Mahadevan (2018) proposes Symplectic Gradient Adjustment (SGA), which decomposes the gradient Jacobian into symmetric (potential) and asymmetric (Hamiltonian) parts and adds a gradient adjustment term for rapid convergence to stable fixed points.
Sch¨afer & Anandkumar (2019) proposes Competitive Gradient Descent (CGD), whose update is given by the Nash equilibrium of a regularized bilinear approximation of the original game.
Compared to other methods, CGD has the advantage of not needing to adapt step size when the interaction strength changes between players.
Many other methods have been proposed with different strategies for predicting other agents’ moves, such as Learning with Opponent Learning
Awareness (LOLA) (Foerster et al., 2016) and optimistic gradient descent-ascent (OGDA) (Popov, 1980; Rakhlin & Sridharan, 2013; Daskalakis et al., 2018; Mertikopoulos et al., 2018).
However, none of these existing methods address the problem of spurious (i.e. non-Nash) stable fixed points.
E XPERIMENTS
We conduct simple experiments to demonstrate the implications of our theoretical results.
In Section 6.1, we show that the double-FTR algorithm empirically converges to and only to differential
Nash equilibria, as predicted by Theorem 1.
In Section 6.2, we demonstrate the practical implications of another property of double-FTR — eigenvalues of J FTR at critical points are real. [Figure]
6. 2-D TOY EXAMPLE
We consider the zero-sum game { f, − f } , R 2 with the following 2-D function (same as in Mazumdar et al.
[Equation]
This function has several LASE points for the GDA dynamics, among which some are DNE and some are not.
As shown in Figure 3, while GDA may converge to critical points that are not local
Nash equilibria, double-FTR avoids such spurious critical points.
Also, in the neighbourhood of local Nash equilibria, GDA exhibits oscillatory behaviour due to complex eigenvalues of the Jacobian matrix.
In contrast, the double-FTR does not have oscillatory behaviour near local Nash equilibria.
For reference, we also show the trajectories of the Local Symplectic Surgery (LSS).
In this experiment, LSS has similar convergence properties – it avoids spurious critical points and does not have oscillatory behaviour near local Nash equilibria.
6. P ARAMETERIZED BILINEAR GAME
The fact that the Jacobian of the double-FTR at critical points have real eigenvalues is also beneficial for stochastic games, in which the GDA might not converge due to its oscillatory behaviour.
Consider the following stochastic parameterized bilinear game:
[Equation]
where x ∼ N ( µ x , σ 2 x ) , y ∼ N ( µ y , σ 2 y ) , r ( x, y )
= xy.
The unique Nash equilibrium with respect to ( x, y ) is (0 , 0) .
However, the learnable parameters are the mean and the standard deviation of the distributions where x and y are drawn from.
We denote the learnable parameters for x and y as θ and ϕ respectively.
At each time step, we obtain an unbiased estimate of the gradient using REINFORCE, over a mini-batch of size B :
[Equation]
[Figure] The second-order derivatives are estimated as below ( ˜ ∇ 2 ϕϕ r ( θ , ϕ ) and ˜ ∇ 2 ϕθ r ( θ , ϕ ) are computed similarly):
[Equation]
As is often the case, GDA has oscillatory behaviour due to the complex eigenvalues of its Jacobian at critical points.
In this stochastic setting, the oscillation prevents convergence for GDA (Figure 4).
In contrast, the double-FTR algorithm does not have rotational behaviour at critical points, and converges to the unique Nash equilibrium ( x, y )
= (0 , 0) .
C ONCLUSION
We propose double Follow-the-Ridge (double-FTR), an gradient-based algorithm for finding local
Nash equilibria in differentiable games.
We prove that when only hyperbolic critical points are present, double-FTR converges to and only to local Nash equilibria in the general-sum setting, and avoids oscillation in the neighbourhood of critical points.
Furthermore, we remark that by varying the preconditioner, double-FTR leads to a broader family of algorithms that share the same convergence guarantee.
Finally, we empirically verify the effectiveness of double-FTR in finding and only finding local Nash equilibria through two simple examples
A D EFERRED PROOFS
Theorem 1.
For a two-player game { ( f, − g ) , R n + m } where all critical points of the GDA flow (equation 1) are hyperbolic, the continuous-time dynamical system in equation 3 satisfies:
• z ∗ is a locally asymptotically stable equilibrium (LASE) of equation 3 ⇐⇒ z ∗ is a differential Nash equilibrium of the game { ( f, − g ) , R n + m } .
• The Jacobian J FTR := ∂ ω FTR ∂
z
at any critical point of equation 3 has real eigenvalues.
Proof.
To avoid notational clutter, we drop the subscripts for the Jacobian matrix of the GDA flow in this proof (i.e. we use J to refer to J GDA ).
Define A := (cid:20) ( ∇ 2 xx f ) −− ( ∇ 2 yy g ) −(cid:21) and
˜ J := (cid:20) I √ c
[Equation]
J , we re-write equation 2 as:
[Equation]
Since we restrict the games to only have hyperbolic critical points, we have Re ( λ ) ̸ = 0 for all λ ∈ spec( J ) , hence J ,
˜ J have full rank, and ( ˜ J ⊤ ˜ J ) 12 exists.
Then, notice that A ˜ J ⊤ ˜ J is similar to ( ˜ J ⊤ ˜ J )
12 A ( ˜ J ⊤ ˜ J ) 12 , we conclude that all eigenvalues of A ˜ J ⊤ ˜ J are real.
The rest of the proof consists of two directions: 1) we show that all locally asymptotically stable equilibria (LASE) of equation 5 are differential Nash equilibria (DNE); and 2) we show that for two-player zero games with only hyperbolic critical points, all DNE are LASE of equation 5.
[Equation]
DNE.
Let ( x ∗ , y ∗ ) be a LASE of equation 5, we have: [Equation] [Equation] and spec
[Equation]
A [Equation] + . First-order condition for DNE Since all eigenvalues of A ˜ J ⊤ ˜ J are in the strictly positive real part, we can conclude that the matrices A , J and ˜ J are all of full-rank.
Therefore, the nullspace of AJ ⊤ (cid:20)
I c I (cid:21) has only the zero vector, and
[Equation]
implies
[Equation]
The first-order condition for DNE is satisfied.
Second-order condition for DNE
Since spec( A ˜ J ⊤ ˜ J ) ⊂
C + , we have that ( ˜ J ⊤ ˜ J )
12 A
˜ J ⊤ ˜ J )
12 ≻ 0 (which means that in fact, spec( A ˜ J ⊤ ˜ J ) ⊂ R + ).
Since positive-definiteness is preserved from congruence transformation, we have A ≻ 0 .
Finally, A ≻ 0 =
⇒ ( ∇ 2 xx f ) − 1 ≻ 0 , ( ∇ 2 yy g ) − 1 ≺ 0 = ⇒ ∇ 2 xx f ≻ 0 , ∇ 2 yy g ≺ 0 .
The second-order condition for DNE is satisfied.
[Equation]
⇒ LASE.
Let ( x ∗ , y ∗ ) be a DNE of the game { ( f, − f ) , R n + m } , we have: ∇ x f ( x ∗ , y ∗ ) = 0 , ∇ y f ( x ∗ , y ∗ )
= 0 , and ∇ 2 xx f ≻ 0 , ∇ 2 yy g ≺ 0 .
First, from the update rule of simultaneous FTR, we immediately have
[Equation]
∇ y
[Equation]
y
[Equation]
) (cid:21) =
(cid:20) (cid:21) =
⇒ (cid:20) ˙
x ∗ ˙ y ∗ (cid:21) = (cid:20) (cid:21) .
Then, since we restrict the games to only have hyperbolic critical points, we have Re ( λ ) ̸ = 0 for all λ ∈ spec( J ) , hence J ,
˜ J have full rank, and ( ˜ J ⊤ ˜ J ) 12 exists.
∇ 2 xx f ≻ 0 , ∇ 2 yy g ≺ 0 = ⇒
A ≻ 0 =
⇒ ( ˜ J ⊤ ˜ J ) 12 A ( ˜ J ⊤ ˜ J ) 12 ≻ 0 .
Since A ˜ J ⊤ ˜ J is similar to ( ˜ J ⊤ ˜ J ) 12 A ( ˜ J ⊤ ˜ J ) 12 , we have that spec( A ˜ J ⊤ ˜ J )
⊂ R + .
( x ∗ , y ∗ ) is a LASE of equation 5.