Robust No-Regret Learning in Min-Max Stackelberg Games
Abstract
The behavior of no-regret learning algorithms is well under- stood in two-player min-max (i.e, zero-sum) games.
In this paper, we investigate the behavior of no-regret learning in min-max games with dependent strategy sets , where the strategy of the first player constrains the behavior of the second.
Such games are best understood as sequential, i.e., min-max
Stackelberg, games.
We consider two settings, one in which only the first player chooses their actions using a no-regret algorithm while the second player best responds, and one in which both players use no-regret algorithms.
For the former case, we show that no-regret dynamics converge to a Stack- elberg equilibrium.
For the latter case, we introduce a new type of regret, which we call Lagrangian regret, and show that if both players minimize their Lagrangian regrets, then play converges to a Stackelberg equilibrium.
We then observe that online mirror descent (OMD) dynamics in these two settings correspond respectively to a known nested (i.e., sequential) gradient descent-ascent (GDA) algorithm and a new simultaneous GDA-like algorithm, thereby establishing convergence of these algorithms to Stackelberg equilibrium.
Finally, we analyze the robustness of OMD dynamics to perturbations by investigating dynamic min-max Stackelberg games.
We prove that OMD dynamics are robust for a large class of dynamic min-max games with independent strategy sets.
In the dependent case, we demonstrate the robustness of OMD dynamics experimentally by simulating them in dynamic Fisher markets, a canonical example of a min-max Stackelberg game with dependent strategy sets.
Introduction
Min-max optimization problems (i.e., zero-sum games) have been attracting a great deal of attention recently because of their applicability to problems in fairness in machine learning (Dai et al.
; Edwards and Storkey 2016;
Madras et al.
; Sattigeri et al.
2018), generative adversarial imitation learning (Cai et al. 2019; Hamedani et al. 2018)
, reinforcement learning (Dai et al. 2018), generative adversarial learning (Sanjabi et al. 2018a), adversarial learning (Sinha et al. 2020), and statistical learning, e.g., learning parameters of exponential families (Dai et al. 2019).
These problems are often modelled as minmax games , i.e., constrained min-max optimization prob- lems of the form: min x ∈ X max y ∈ Y
f ( x , y ) , where f :
X × Y → R is continuous, and X ⊂ R n and Y ⊂ R m are non-empty and compact.
In convex-concave minmax games , where f is convex in x and concave in y , von Neumann and Morgenstern’s seminal minimax theorem holds (Neumann 1928): i.e., min x ∈ X max y ∈ Y
f
( x , y ) =
max y ∈ Y min x ∈ X f ( x , y ) , guaranteeing the existence of a saddle point, i.e., a point that is simultaneously a minimum of f in the x -direction and a maximum of f in the y -direction.
This theorem allows us to interpret the optimization problem as a simultaneous-move, zero-sum game, where y ∗ (resp.
x ∗ ) is a best-response of the outer (resp.
inner) player to the other’s action x ∗ (resp.
y ∗ ) , in which case a saddle point is also called a minimax point or a Nash equilibrium.
In this paper, we study min-max Stackelberg games (Goktas and Greenwald 2021), i.e., constrained min-max optimization problems with dependent feasible sets of the form: min x ∈ X max y ∈ Y : [-g ( x , y ) ≥ 
-][Equation] [Equation]
{+, y ) ≥ 
+} f ( x , y ) , where f : X × Y → R is continuous, X ⊂ R n and Y ⊂ R m are non-empty and compact, and g ( x , y )
= ( g 1 ( x , y ) , . .
. , g K ( x , y ))
T with g k :
X × Y → R .
Goktas and Greenwald observe that the minimax theorem does not hold in these games (2021).
As a result, such games are more appropriately viewed as sequential, i.e., Stackelberg, games for which the relevant solution concept is the Stackelberg equilibrium, 1 where the outer player chooses ˆ x ∈ X before the inner player responds with their choice of y (ˆ x )
∈ Y s.t.
g (ˆ x , y (ˆ x )) ≥ 0 .
In these games, the outer player seeks to minimize their loss, assuming the inner player chooses a feasible best response: i.e., the outer player’s objective, also known as their value function in the economics literature (Milgrom and Segal 2002), is defined as V X ( x ) =
max y ∈ Y : g ( x , y ) ≥
f ( x , y ) .
The inner player’s value function, V Y :
X → R , which they seek to maximize, is simply the objective function given the outer player’s action ˆ x : i.e., V Y ( y ; ˆ x )
= f
(ˆ x , y ) .
Goktas and Greenwald (2021) proposed a polynomialtime first-order method by which to compute Stackelberg equilibria, which they called nested gradient descent ascent (GDA) .
This method can be understood as an algorithm a third party might run to find an equilibrium, or as a game dynamic that the players might employ if their long-run goal were to reach an equilibrium.
Rather than assume that players are jointly working towards the goal of reaching an equilibrium, it is often more reasonable to assume that they play so as to not regret their decisions: i.e., that they employ a no-regret learning algorithm , which minimizes their loss in hindsight.
It is well known that when both players in a min-max game are no-regret learners, the players’ strategy profile over time converges to a Nash equilibrium in average iterates: i.e., empirical play converges to a Nash equilibrium (e.g., (Freund and Schapire 1996)).
In this paper, we investigate no-regret learning dynamics in min-max Stackelberg games.
We assume both pessimistic and optimistic settings.
In the pessimistic setting, the outer player is a no-regret learner while the inner player best responds; in the optimistic setting, both players are no-regret learners.
In the pessimistic case, we show that if the outer player uses a no-regret algorithm that achieves ε -pessimistic regret after T iterations, then the outer player’s empirical play converges to their ε -Stackelberg equilibrium strategy.
In the optimistic case, we introduce a new type of regret, which we call Lagrangian regret, which assumes access to a solution oracle for the optimal KKT multipliers of the game’s constraints.
We then show that if both players use no-regret algorithms that achieve ε -Lagrangian regret after
T iterations, the players’ empirical play converges to an εStackelberg equilibrium.
We then restrict our attention to online mirror descent (OMD) dynamics, which yield two algorithms, namely maxoracle gradient descent (Jin, Netrapalli, and Jordan 2020) and nested GDA (Goktas and Greenwald 2021) in the pessimistic setting, and a new simultaneous GDA-like algorithm (Nedic and Ozdaglar 2009) in the optimistic setting, which we call Lagrangian GDA (LGDA).
Convergence of the former two algorithms in O ( 1 / ε 2 ) iterations then follows from our previous theorems.
Additionally, the iteration complexity of O ( 1 / ε 2 ) suggests the superiority of LGDA over nested-GDA when a Lagrangian solution oracle exists, since nested-GDA converges in O ( 1 / ε 3 ) iterations (Goktas and Greenwald 2021), while LGDA converges in O ( 1 / ε 2 ) iterations, assuming the objective function is only Lipschitz continuous.
Finally,we analyze the robustness of OMD dynamics to perturbations by investigating dynamic min-max Stackelberg games.
We prove that OMD dynamics are robust, in that even when the game changes with each iteration of the algorithm, OMD dynamics track the changing equilibrium closely for a large class of dynamic min-max games with independent strategy sets.
In the dependent strategy set case, we demonstrate the robustness of OMD dynamics experimentally by simulating them in dynamic Fisher markets, a canonical example of a min-max Stackelberg game (with dependent strategy sets).
Even when the Fisher market changes with each iteration, our OMD dynamics are able to track the changing equilibria closely.
Our findings can be summarized as follows:
• In min-max Stackelberg games, when the outer player is a no-regret learner and the inner-player best-responds, the average of the outer player’s strategies converges to their Stackelberg equilibrium strategy.
• We introduce a new type of regret we call Lagrangian regret and show that in min-max Stackelberg games when both players minimize Lagrangian regret, the average of the players’ strategies converges to a Stackelberg equilibrium.
• We provide novel convergence guarantees for two known algorithms, max-oracle gradient descent and nested gradient descent ascent, to an ε -Stackelberg equilibrium in O ( 1 / ε 2 ) in average iterates.
• We introduce a new simultaneous GDA-like algorithm and prove that its average iterates converge to an εStackelberg equilibrium in O ( 1 / ε 2 ) iterations.
• We prove that max-oracle gradient descent and simultaneous GDA are robust to perturbations in a large class of min-max games (with independent strategy sets).
• We run experiments with Fisher markets which suggest that max-oracle gradient descent and simultaneous GDA are robust to perturbations in these min-max Stackelberg games (with dependent strategy sets).
We provide a review of related work in Appendix BThis paper is organized as follows.
In the next section, we present the requisite mathematical preliminaries.
In Section 3, we present no-regret learning dynamics that converge in a large class of min-max Stackelberg games.
In Section 4, we study the convergence and robustness properties of a particular noregret learning algorithm, namely online mirror descent, in min-max Stackelberg games.
Mathematical Preliminaries
Our notational conventions can be found in Appendix A.
Game Definitions A min-max Stackelberg game , ( X, Y, f, g ) , is a two-player, zero-sum game, where one player, who we call the outer, or x -, player (resp.
the inner, or y -, player), is trying to minimize their loss (resp.
maximize their gain), defined by a continuous objective function f : X × Y → R , by taking an action from their strategy set
X ⊂ R n , and (resp.
Y ⊂ R m )
s.t.
g ( x , y ) ≥ 0 where g ( x , y )
= ( g 1 ( x , y ) , . .
. , g K ( x , y ))
T with g k :
X × Y → R continuous.
A strategy profile ( x , y ) ∈
X × Y is said to be feasible iff for all k ∈ [ K ] , g k ( x , y ) ≥ 0 .
The function f maps a pair of actions taken by the players ( x , y ) ∈
X × Y to a real value (i.e., a payoff), which represents the loss (resp.
the gain) of the x -player (resp.
the y -player).
A min-max game is said to be convex-concave if the objective function f is convex-concave.
Onewaytosee this game is as a Stackelberg game , i.e., a sequential game with two players, where WLOG, we assume that the minimizing player moves first and the maximizing player moves second.
The relevant solution concept for Stackelberg games is the Stackelberg equilibrium :
A strategy profile ( x ∗ , y ∗ ) ∈
X × Y s.t.
g ( x ∗ , y ∗ ) ≥ 0 is an ( ϵ, δ ) -Stackelberg
equilibrium
if max y ∈ Y : g ( x ∗ , y ) ≥ 0 f ( x ∗ , y ) − δ ≤ f ( x ∗ , y ∗ ) ≤ min x ∈ X max y ∈ Y : g ( x , y ) ≥
f ( x , y ) + ϵ .
Intuitively, a ( ε, δ ) -Stackelberg
equilibrium is a point at which the xplayer’s (resp.
y -player’s) payoff is no more than ε (resp.
δ ) away from its optimum.
A (0 , 0) -Stackelberg equilibrium is guaranteed to exist in min-max Stackelberg games (Goktas and Greenwald 2021).
Note that when g ( x , y ) ≥ 0 , for all x ∈ X and y ∈ Y , the game reduces to a min-max game (with independent strategy sets), for which, by the min-max theorem, a Nash equilibrium is guaranteed to exist (Neumann 1928).
In a min-max
Stackelberg game, the outer player’s best-response set BR X
⊂
X , defined as
BR X = arg min x ∈ X V X ( x ) , is independent of the inner player’s strategy, while the inner player’s bestresponse correspondence BR Y : X ⇒ Y , defined as
BR Y ( x ) = arg max y ∈ Y : g ( x , y ) ≥ 0 V Y ( y ; x ) , depends on the outer player’s strategy.
A (0 , 0) -Stackelberg equilibrium ( x ∗ , y ∗ ) ∈
X × Y is then a tuple of strategies such that ( x ∗ , y ∗ )
∈
BR X × BR Y ( x ∗ ) .
A dynamic min-max
Stackelberg game , (cid:8) ( X, Y, f ( t ) , g ( t ) ) (cid:9) T t =1 , is a sequence of min-max Stackelberg games played for T time periods.
We define the players’ value functions at time t in a dynamic min-max
Stackelberg game in the obvious way.
Note that when g ( t ) ( x , y ) ≥ 0 for all x ∈ X, y ∈ Y and all time periods t ∈
[ T ] , the game reduces to a dynamic minmax game (with independent strategy sets).
Moreover, if ∀ t, t ′ ∈
[ T ] , f ( t ) =
f ( t ′ ) , and g ( t ) =
g ( t ′ ) , then the game reduces to a (static) min-max Stackelberg game, which we denote simply by ( X, Y, f, g ) .
Mathematical Preliminaries Given A ⊂ R n , the function f : A → R is said to be ℓ f - Lipschitz-continuous iff ∀ x 1 , x 2 ∈ X, ∥ f
( x 1 ) − f ( x 2 ) ∥
≤ ℓ f ∥ x 1 − x 2 ∥ .
If the gradient of f , ∇ f , is ℓ ∇ f -Lipschitz-continuous, we refer to f as ℓ ∇ fLipschitz-smooth .
We provide a review of online convex optimization in Appendix A.
No-Regret Learning Dynamics
In this section we explore no-regret learning dynamics in min-max Stackelberg games, and prove the convergence of no-regret learning dynamics in two settings: a pessimistic setting in which the outer player is a no-regret learner while the inner player best-responds, and an optimistic setting in which both players are no-regret learners.
All the results in this paper rely on the following assumptions:
Assumption 1. 1.
( Slater’s condition (Slater 1959, 2014))
∀ x ∈ X, ∃ (cid:98)
y ∈
Y s.t.
g k ( x , (cid:98) y ) > 0 , for all k = 1 , . . .
, K ; 2.
f, g 1 , . .
. , g K are continuous and convexconcave; and 3.
∇ x f, ∇ x g 1 , . .
. , ∇ x g K are well-defined for all
( x , y ) ∈
X × Y and continuous in ( x , y ) .
We note that these assumptions are in line with previous work geared towards solving min-max Stackelberg games (Goktas and Greenwald 2021).
Part 1 of Assumption 1, Slater’s condition, is a standard constraint qualification condition (Boyd, Boyd, and Vandenberghe 2004), which is needed to derive the optimality conditions for the inner player’s maximization problem; without it the problem becomes analytically intractable.
Part 2 of Assumptionis is required for the value function of the outer player to be continuous and convex ((Goktas and Greenwald 2021),
Proposition A1)
so that the problem is solvable efficiently.
Finally, we note that Part 3 of Assumption 1 can be replaced by a subgradient boundedness assumption instead; however, for simplicity, we assume this stronger condition.
Pessimistic Learning Setting
In Stackelberg games, the leader decides their strategy assuming that the inner player will best respond which leads us to first consider a repeated game setting in which the inner player always best responds to the strategy picked by the outer player.
Such a setting also makes sense as in zerosum Stackelberg games the outer player and inner player are adversaries, and in most applications of interest we are concerned by optimal strategies for the outer player; hence, assuming a strong adversary which always best-responds allows us to consider more robust strategies for the outer player.
For any x ∈ X , denote y ∗ ( x ) ∈ BR Y ( x ) , in such a setting, intuitively, the regret should be equal to the difference between the cumulative loss of the outer player w.r.t.
to their sequence of actions to which the inner player best responds, and the smallest cumulative loss that the outer player can achieve by picking a fixed strategy to which the inner player best responds, i.e., 1 T (cid:80) Tt =1 f ( t ) ( x ( t ) , y ∗ ( x ( t ) )) − (cid:80) Tt =1 1 T f ( t ) ( x , y ∗ ( x )) .
We call this regret the pessimistic regret which can be more conveniently defined as the regret incurred by an action x ∈ X of the outer player w.r.t.
a sequence of actions (cid:8) ( X, Y, f ( t ) , g ( t ) ) (cid:9) T t =1 and a dynamic min-max Stackelberg game { G t } t ∈ T w.r.t.
to the loss given by their value function { V ( t ) X } Tt =1 , i.e.:
PesRegret
[Equation]
That is, the pessimistic regret of the outer player compares the outer player’s play history to the smallest cumulative loss the outer player could achieve by picking a fixed strategy assuming that the inner player best-responds .
It is pessimistic in the sense that the outer player assumes the worst possible outcome for themself.
The main theorem in this section states the following: assuming the inner player best responds to the actions of the outer player, if the outer player employs a no-regret algorithm, then the outer player’s average strategy converges to a Stackelberg equilibrium.
Before presenting this theorem,we recall the following property of the outer player’s value function.
Proposition 2 ((Goktas and Greenwald 2021), Proposition
A.1) .
In a min-max Stackelberg game ( X, Y, f, g ) , the outer player’s value function, V ( x ) = max y ∈ Y : g ( x , y ) ≥
f ( x , y ) , is continuous and convex.
Theorem 3.
Consider a min-max Stackelberg game ( X, Y, f, g ) , and suppose the outer player plays a sequence of actions { x ( t ) }
Tt =1 ⊂ X .
If, after T iterations, the outer player’s pessimistic regret is bounded by ε for all x ∈ X , then (cid:0) ¯ x ( T ) , y ∗ (¯ x ( T ) ) (cid:1) is a ( ε, 0) -Stackelberg
equilibrium, where y ∗ (¯ x ( T ) )
∈ BR Y (¯ x ( T ) ) .
We remark that even though the definition of pessimistic regret looks similar to the standard definition of regret, its structure is very different.
In particular, without Proposition 2, it is not clear that the value (cid:80) Tt =1 1 T f ( t ) ( x , y ∗ ( x ))
= (cid:80) Tt =1 V ( t ) ( x ) is convex in x .
Optimistic Learning Setting
We now turn our attention to a learning setting in which both players are no-regret learners.
The most straightforward way to define regret is by considering the outer and inner players’ “vanilla” regrets, respectively:
Regret ( T ) X ( x ) = 1 T (cid:80) Tt =1 f ( x ( t ) , y ( t ) ) −
[Equation]
and
Regret ( T ) Y ( y ) =
1 T (cid:80) Tt =
[Equation]
x
[Equation]
− 1 T (cid:80) Tt =1 f ( x ( t ) , y ( t ) ) .
In convex-concave min-max games (with independent strategy sets), when both players minimize their vanilla regret, the players’ average strategies converge to Nash equilibrium.
In min-max Stackelberg games (with dependent strategy sets), however, convergence to a Stackelberg equilibrium in not guaranteed.
Example 4.
Consider the min-max
Stackelberg game
min
[Equation]
max y ∈
[Equation] [Equation]
1 − ( x + y ) x
2 + y + 1 .
The Stackelberg equilibrium of this game is given by x ∗ = 1 / 2 , y ∗ = 1 / 2 .
Suppose both players employ no-regret algorithms that generate strategies { x ( t ) , y ( t ) } t ∈ N + .
Then at time T ∈ N + , there exists ε > 0 , s.t.
[Equation]
Simplifying yields:
[Equation]
Since both players are no-regret learners, there exists T ∈ N + large enough s.t.
[Equation]
In other words, the average iterates converge to x = 0 , y = 1 , which is not the Stackelberg equilibrium of this game.
If the inner player minimizes their vanilla regret without regard to the game’s constraints, then their actions are not guaranteed to be feasible, and thus cannot converge to a
Stackelberg equilibrium.
To remedy this infeasibility, we introduce a new type of regret we call Lagrangian regret , and show that assuming access to a solution oracle for the optimal KKT multipliers of the game’s constraints, if both players minimize their Lagrangian regret, then no-regret learning dynamics converge to a Stackelberg equilibrium.
Define L x ( y , λ ) =
f ( x , y ) + (cid:80)
Kk =
λ k g
k
( x , y ) to be the Lagrangian associated with the outer player’s value function, or equivalently, the inner player’s maximization problem given the outer player’s strategy x ∈ X .
If the optimal KKT multipliers λ ∗ ∈ R K , which are guaranteed to exist by Slater’s condition (Slater 1959), were known for the problem min x ∈ X max y
[Equation]
x ,
[Equation]
min
[Equation]
x ( y , λ ) , then one could plug them back into the Lagrangian to obtain a convex-concave saddle point problem given by min x ∈ X
max y ∈ Y L x ( y , λ ∗ ) .
Note that a saddle point of this problem is guaranteed to exist by the minimax theorem (Neumann 1928), since L x ( y , λ ∗ ) is convex in x and concave in y .
The next lemma states that the Stackelberg equilibria of a minmax Stackelberg game correspond to the saddle points of
L x ( y , λ ∗ ) .
Lemma 5.
Any Stackelberg equilibrium ( x ∗ y ∗ ) ∈
X ×
Y of any min-max Stackelberg game ( X, Y, f, g ) corresponds to a saddle point of L x ( y , λ ∗ ) , where λ ∗ ∈ arg min λ ≥ 0 min x ∈ X max
y ∈
Y L x ( y , λ ) .
This lemma tells us that the function L x ( y , λ ∗ ) represents a new loss function that enforces the game’s constraints.
Based on this observation, we assume access to a Lagrangian solution oracle that provides us with λ ∗ ∈ arg min λ ≥
min x ∈ X max y ∈
Y L x ( y , λ ∗ ) .
Further, we define a new type of regret which we call Lagrangian regret.
Given a sequence of actions (cid:8) x
( t ) , y ( t ) (cid:9) T t = taken by the outer and inner players in a dynamic min-max
Stackelberg game (cid:8) ( X, Y, f ( t ) , g ( t ) ) (cid:9) T t =1 , we define their
Lagrangian regret, respectively, as LagrRegret ( T ) X (
[Equation]
and LagrRegret ( T
[Equation]
Y ( y
[Equation]
The saddle point residual of a point ( x ∗ , y ∗ ) ∈
X × Y with respect to a convex-concave function f : X × Y → R is given by max y ∈ Y f
( x ∗ , y ) − min x ∈ X f ( x , y ∗ ) .
When the saddle point residual is 0, the saddle point is a (0 , 0) -
Stackelberg equilibrium.
The main theorem of this section now follows: if both players play so as to minimize their Lagrangian regret, then their average strategies converge to a Stackelberg equilibrium.
The bound is given in terms of the saddle point residual of L x ( y , λ ∗ ).
Theorem 6.
Consider a min-max Stackelberg game ( X, Y, f, g ) , and suppose the outer and the players generate sequences of actions { ( x ( t ) , y ( t ) ) }
Tt =1 ⊂ X using a no-Lagrangian-regret algorithm.
If after T iterations, the
Lagrangian regret of both players is bounded by ε for all x ∈ X , the following convergence bound holds on the saddle point residual of (¯ x ( T ) , ¯ y ( T ) ) w.r.t.
the Lagrangian:
[Equation]
Having provided convergence to Stackelberg equilibrium of general no-regret learning dynamics in min-max Stackelberg games, we now proceed to investigate the convergence and robustness properties of a specific example of a no-regret learning dynamic, namely online mirror descent (OMD) dynamics.
Online Mirror Descent
In this section, we apply the results we have derived for no-regret learning dynamics to Online Mirror Descent (OMD) (Zinkevich 2003; Shalev-Shwartz et al. 2011).
We apply the theorems we derived above to OMD, and then we study the robustness properties of OMD in min-max Stackelberg games.
Convergence Analysis
When the outer player is an OMD learner minimizing its pessimistic regret and the inner player best responds, we obtain the max-oracle gradient descent algorithm (Algorithm- Appendix D) first proposed by Jin, Netrapalli, and Jordan (2020) for min-max games.
Following Jin, Netrapalli, and Jordan (2020),
Goktas and
Greenwald extend the max-oracle gradient descent algorithm to min-max Stackelberg games and prove its convergence of in best iterates.
The following corollary of Theorem 3, which concerns convergence of this algorithm in average iterates, complements their result: the max-oracle gradient descent algorithm is guaranteed to converge to an ( ε, 0)
-Stackelberg
equilibrium strategy of the outer player in average iterates after O ( 1 / ε 2 ) iterations, assuming the inner player best responds.
We note that since V X is convex, by Proposition 2, V X is subdifferentiable.
Moreover, for all (cid:98) x ∈ X , (cid:98) y ∈ BR Y ( (cid:98) x ) , ∇ x f ( (cid:98) x , (cid:98) y )
+ (cid:80)
Kk =
λ k g
k
( (cid:98) x , (cid:98) y ) is an arbitrary subgradient of the value function at (cid:98) x by Goktas and Greenwald’s subdifferential envelope theorem (2021).
We add that similar to Goktas and Greenwald, we assume that the optimal
KKT multipliers λ ∗ ( x ( t ) , (cid:98) y
( x ( t ) )) associated with a solution (cid:98) y
( x ( t ) )) can be computed in constant time.
Corollary 7.
Let c = max x ∈ X ∥ x ∥ and let ℓ f = max ( (cid:98) x , (cid:98) y ) ∈
X
× Y ∥∇ x f ( (cid:98) x , (cid:98) y ) ∥ .
If Algorithm 1 (Appendix D) is run on a minmax Stackelberg game ( X, Y, f, g ) with η t = c ℓ f √ 2 T for all iteration t ∈
[ T ] and any x (0) ∈ X , then (¯ x ( T ) , y ∗ (¯ x ( T ) )) is a ( cℓ f √ 2 / √ T , 0) -Stackelberg
equilibrium.
Furthermore, for ε ∈ (0 , 1) , if we choose T ≥ N T ( ε ) ∈ O ( 1 / ε 2 ) , then there exists an iteration T ∗ ≤ T s.t.
(¯ x ( T ) , y ∗ (¯ x ( T ) )) is an ( ε, 0) -Stackelberg equilibrium.
Note that we can relax Theorem 3 to instead work with an approximate best response of the inner player, i.e., given the strategy of the outer player (cid:98) x , instead of playing an exact best-response, the inner player computes a (cid:98) y s.t.
f ( (cid:98) x , (cid:98) y ) ≥ max y ∈ Y : g ( (cid:98) x , y ) ≥
f
( (cid:98) x ) − ε .
Combine with results on the convergence of gradient ascent on smooth functions, the average iterates computed by Goktas and Greenwald’s nested
GDA algorithm converge to an ( ε, ε ) -Stackelberg equilibrium in O ( 1 / ε 3 ) iterations.
If additionally, f is strongly convex in y , then the iteration complexity can reduced to
O ( 1 / ε 2 log( 1 / ε )) .
Similarly, we can also consider the optimistic case, in which both the outer and inner players minimize their Lagrangian regrets, as OMD learners with access to a Lagrangian solution oracle that returns λ ∗ ∈ arg min λ ≥
min x ∈ X max y ∈ Y L x ( y , λ ∗ ) .
In this case, we obtain the Lagrangian GDA (LGDA) algorithm
(Algorithm 2 - Appendix D).
The following corollary of Theorem 6 states that LGDA converges in average iterates to an approximate-Stackelberg equilibrium in O ( 1 / ε 2 ) iterations.
Corollary 8.
Let b = max x ∈ X ∥ x ∥ , c = max y ∈ Y ∥
y ∥ , and ℓ L = max ( (cid:98) x , (cid:98) y
) ∈
X × Y ∥∇ x L (cid:98) x
( (cid:98) y , λ ∗ ) ∥ .
If Algorithm 2 (Appendix D) is run on a min-max Stackelberg game ( X, Y, f, g ) with η x t = b ℓ L √ 2 T and η y t = c ℓ L √ 2 T for all iterations t ∈
[ T ] and any x (0) ∈ X , then the following convergence bound holds on the saddle point residual (¯ x ( T ) , ¯ y ( T ) )
w.r.t.
the Lagrangian:
[Equation]
We remark that in certain rare cases the Lagrangian can become degenerate in y , in that the y terms in the Lagrangian might cancel out when λ ∗ is plugged back into
Lagrangian, leading LGDA to not update the y variables, as demonstrated by the following example:
Example 9.
Consider this min-max Stackelberg game: min x ∈ [ − 1 , 1] max y ∈ [ − 1 , 1]:0 ≤ 1 − ( x + y )
x
2 + y + 1 .
When we plug the op- timal KKT multiplier λ ∗ = 1 into the Lagrangian associated with the outer player’s value function, we obtain
L x ( y, λ ) =
x 2 + y + 1 − ( x + y ) =
x 2 − x + 1 , with ∂ L ∂x = 2 x − 1 and ∂ L ∂y = 0 .
It follows that the x iterate converges to 1 / 2 , but the y iterate will never be updated, and hence unless y is initialized to its Stackelberg equilibirium value, LGDA will not converge to a Stackelberg equilibrium.
In general, this degeneracy issue occurs when ∀ x ∈
X, ∇ y f ( x , y )
= − (cid:80)
Kk =1 λ ∗ k ∇ y g k
( x , y ) .
We can sidestep the issue by restricting our attention to min-max Stackelberg games with convex- strictly -concave objective functions, which is sufficient to ensure that the Lagrangian is not degenerate in y (Boyd, Boyd, and Vandenberghe 2004).
Robustness Analysis
Although the OMD dynamics we analyzed in the previous section describe a dynamic behavior in nature, they assume that the game and its properties, i.e., the objective function and constraints, are static and thus do not change over time.
In many real-world games, however, the game itself is subject to perturbations, i.e., dynamic changes, in the sense that the agents’ objectives and constraints might be perturbed by external influences.
Analyzing and providing dynamics that are robust to ongoing changes in games is critical, since the real world is rarely static.
This makes the study of dynamic min-max Stackelberg games and their associated optimal dynamic strategies for both players an important goal.
Dynamic games bring with them a series of interesting issues; notably, even though the environment might change at each time period, in each period of time the game still exhibits a Stackelberg equilibrium.
However, one cannot sensibly expect the players to play a Stackelberg equilibrium strategy at each time period since even in the static setting, known game dynamics require multiple time steps in order for players to reach even an approximate Stackelberg equilibrium.
When players cannot directly best respond or pick the optimal strategy for themselves, they essentially become boundedly rational agents in that they can only take a step towards their optimal strategy but they cannot reach it in just one time step.
Hence, in dynamic games, equilibria also become dynamic objects, which can never be reached unless the game stops changing significantly.
Corollaries 8 and 7 tell us that OMD dynamics are effective equilibrium finding strategies in min-max Stackelberg games.
However, they do not provide any intuition about the robustness of OMD dynamics to perturbations in the game.
That is, we would like to know whether or not OMD dynamics are able to track the equilibrium even when the game changes slowly.
Robustness is a desirable property for noregret learning dynamics as many real-world applications of games involve changing environments.
In this section, we provide theoretical guarantees that show that even when the game changes at each iteration, OMD dynamics closely track the changing equilibria of the dynamic game.
Unfortunately, our theoretical results only concern min-max games (with independent strategy sets).
Nevertheless, we provide experimental evidence that suggests that the results we prove may also apply more broadly to min-max Stackelberg games (with dependent strategy sets).
We first consider the pessimistic setting in which the outer player is a no-regret learner and the inner player bestresponds.
In this setting, we show that when the outer player follows online projected gradient descent dynamics in a dynamic min-max game, i.e., a min-max game in which the objective function constantly changes, the outer player’s strategies closely track their Stackelberg equilibrium strategy.
Intuitively, the following result implies that irrespective of the initial strategy of the outer player, online projected gradient descent dynamics follow the Nash equilibrium strategy of the outer player s.t.
the strategy determined by the outer player is always within a 2 d / δ radius of the outer player’s
Nash equilibrium strategy.
Theorem 10.
Consider a dynamic min-max game (cid:8) ( X, Y, f ( t ) ) (cid:9) T t =1 .
Suppose that, for all t ∈ [ T ] , f ( t ) is µ -strongly convex in x and strictly concave in y , and f ( t ) is ℓ ∇ f -Lipschitz smooth.
Suppose that the outer player generates a sequence of actions { x ( t ) }
Tt =1 ⊂ X by using an online projected gradient descent algorithm on the loss functions { V ( t ) }
Tt =1 with learning rate η ≤ 2 µ + ℓ ∇ f and suppose that the inner player generates a sequence of bestresponses to each iterate of the outer player { y ( t ) }
Tt =1 ⊂ Y .
For all t ∈ [ T ] , let x ( t ) ∗ ∈ arg min x ∈ X V ( t ) ( x ) , ∆ ( t ) =
(cid:13)(cid:13)(cid:13) x
( t +1) ∗ − x ( t ) ∗ (cid:13)(cid:13)(cid:13) , and δ = 2 ηµℓ ∇ f ℓ ∇ f + µ , we then have:
[Equation]
−
[Equation]
[Equation]
If additionally, for all t ∈ [ T ] , ∆ ( t ) ≤ d , then:
[Equation]
We can extend a similar robustness result to the setting in which the outer and inner players are both OMD learners.
The following theorem implies that irrespective of the initial strategies of the two players, online projected gradient descent dynamics follow the Nash equilibrium of the game, always staying within a 4 d / δ radius.
Theorem 11.
Consider a dynamic min-max game { G t } Tt = = (cid:8) ( X, Y, f ( t ) ) (cid:9) T t =1 .
Suppose that, for all t ∈ [ T ] , f ( t ) is µ x -strongly convex in x and µ ystrongly concave in y , f ( t ) is ℓ ∇ f -Lipschitz smooth.
Let { ( x ( t ) , y ( t ) ) }
Tt =1 ⊂ X
× Y be the strategies generated by the outer and inner players assuming that the outer player uses a online projected gradient descent algorithm on the losses { f ( t ) ( · , y ( t ) ) }
Tt =1 with η x = 2 µ x + ℓ ∇ f and that the inner player uses a online projected gradient descent algorithm on the losses {− f ( t ) ( x ( t ) , · ) } Tt =1 with η y = 2 µ y + ℓ ∇ f .
For all t ∈ [ T ] , let x ( t ) ∗ ∈ arg min x ∈ X f ( t ) ( x , y ( t ) ) , y ( t ) ∗ ∈ arg min
y ∈
Y f
( t ) ( x ( t ) , y ) .
∆ ( t ) x
= (cid:13)(cid:13)(cid:13) x ( t +1) ∗ − x ( t ) ∗ (cid:13)(cid:13)(cid:13) , ∆ ( t )
y
=
(cid:13)(cid:13)(cid:13)
y
( t +1) ∗ − y ( t ) ∗ (cid:13)(cid:13)(cid:13) ,
δ x = 2 ηµ x ℓ ∇ f ℓ ∇ x f + µ x , and δ y = 2 ηµ y ℓ ∇ f ℓ ∇ f + µ y we then have: (cid:13) (cid:13)(cid:13)
[Equation]
( T )
[Equation]
[Equation]
(cid:13) (cid:13)(cid:13)
[Equation]
(cid:13) (cid:13)(cid:13)
[Equation]
( T )
[Equation]
( T
[Equation]
(cid:13) (cid:13)(cid:13)
[Equation]
[Equation]
(cid:13)(cid:13)(cid:13) x (0) ∗
[Equation]
x (0) (cid:13)(cid:13)(cid:13)
+ (
[Equation]
δ y ) T / 2 (cid:13)(cid:13)(cid:13) y (0) ∗ − y (0)
(cid:13)(cid:13)(cid:13)
+
[Equation]
(cid:88)
[Equation]
If additionally, ∆ ( t ) x ≤ d and ∆ ( t ) y ≤ d for all t ∈
[ T ] , and δ = min { δ y , δ x } , then: (cid:13)
(cid:13)(cid:13) x ( T ) ∗ − x ( T ) (cid:13) (cid:13)(cid:13) + (cid:13) (cid:13)(cid:13) y ( T ) ∗ −
[Equation]
The proofs of the above theorems are relegated to Appendix C.
The theorems we have proven in this section establish the robustness of OMD dynamics for min-max games in both the pessimistic and optimistic settings by showing that the dynamics closely track the Stackelberg equilibrium in a large class of min-max games.
As we are not able to extend these theoretical robustness guarantees to min-max
Stackelberg games (with dependent strategy sets), we instead ran a series of experiments on (dynamic) Fisher markets, which are canonical examples of min-max Stackelberg games (Goktas and Greenwald 2021), to investigate the empirical robustness guarantees of OMD dynamics for this class of min-max Stackelberg games.
Dynamic Fisher Markets
The Fisher market model, attributed to Irving Fisher (Brainard, Scarf et al. 2000), has received a great deal of attention in the literature, especially by computer scientists, as it has proven useful in the design of online marketplaces.
We now study OMD dynamics in dynamic Fisher markets, which are instances of min-max Stackelberg games (Goktas and Greenwald 2021).
A Fisher market consists of n buyers and m divisible goods (Brainard, Scarf et al. 2000).
Each buyer i ∈ [ n ] has a budget b i ∈ R + and a utility function u
i : R m + → R .
Each good j ∈ [ m ] has supply s j ∈ R + .
A Fisher market is thus given by a tuple ( n, m, U, b , s ) , where U = { u 1 , . . .
, u n } is a set of utility functions, one per buyer, b ∈ R n + is a vector of buyer budgets, and s ∈ R m
+ is a vector of good supplies.
We abbreviate as ( U, b , s ) when n and m are clear from context.
A dynamic Fisher market is a sequence of Fisher markets (cid:0) U ( t ) , b ( t ) , s ( t ) (cid:1) ([ T ]) t =1 .
An allocation X =
( x 1 , . . .
, x n ) T ∈ R n × m + is a map from goods to buyers, represented as a matrix s.t.
x ij ≥ 0 denotes the amount of good j ∈ [ m ] allocated to buyer i ∈ [ n ] .
Goods are assigned prices p
= ( p 1 , . .
, p m ) T ∈ R m + .
A tuple ( p ∗ , X ∗ ) is said to be a competitive (or Walrasian) equilibrium of Fisher market ( U, b , s ) if 1.
buyers are utility maximizing, constrained by their budget, i.e., ∀ i ∈ [ n ] , x ∗ i ∈ arg max x : x · p ∗ ≤ b i u i ( x ) ; and 2.
the market clears, i.e., ∀ j ∈ [ m ] , p ∗ j > 0 ⇒ (cid:80) i ∈ [ n ] x ∗ ij =
s j and p ∗ j =
⇒ (cid:80) i ∈ [ n ] x ∗ ij ≤ s j .
Goktas and Greenwald (2021) observe that any competitive equilibrium ( p ∗ , X ∗ ) of a Fisher market ( U, b ) corresponds to a Stackelberg equilibrium of the following minmax Stackelberg game:
[Equation]
Let L : R m + × R n × m → R + be the Lagrangian of the outer player’s value function in Equation (10), i.e.,
L p ( X , λ ) [Equation]
[Table]
· p ) .
One can show the existence of a Lagrangian solution oracle for the Lagrangian of Equation (10) such that λ ∗ = 1 m .
We then have: 1. by Goktas and
Greenwald’s envelope theorem, the subdifferential of the outer player’s value function is given by ∇ p V ( p ) =
s − (cid:80) i ∈ [ n ] x ∗ i ( p ) , where x ∗ i ( p ) ∈ arg
max x ∈ R m + x · p ≤ b i u i ( x ) , 2.
the gradient of the Lagrangian w.r.t.
the prices, given the Langrangian solution oracle, is ∇ p L p ( X , λ ∗ ) =
s − (cid:80) i ∈ [ n ] x i and ∇ x
i L p ( X , λ ∗ ))
=
b i u i ( x i ) ∇ x i u i ( x i ) − p , where λ ∗ = 1 m .
We first consider OMD dynamics for Fisher markets in the pessimistic setting, in which the outer player determines their strategy via online projected gradient descent and the inner player best-responds.
In this setting, we obtain a dynamic version of a natural price adjustment process known as tˆatonnement (Walras 1969), which was first studied by Cheung, Hoefer, and Nakhe (2019)
(Algorithm 3,
Appendix D).
We then consider OMD dynamics in the optimistic setting, in which case both the outer and inner players employ online projected gradient descent, which yields myopic best-response dynamics (Monderer and Shapley 1996)
(Algorithm 4, Appendix D).
In words, at each time step, the (fictional Walrasian) auctioneer takes a gradient descent step to minimize its regret, and then all the buyers take a gradient ascent step to minimize their Lagrangian regret.
These gradient descent-ascent dynamics can be seen as myopic best-response dynamics for sellers and buyers who are both boundedly rational (Camerer 1998).
Experiments In order to better understand the robustness properties of Algorithms 3 and 4 in a dynamic min-max
Stackelberg game that is subject to perturbation across time, we ran a series of experiments with dynamic Fisher Markets assuming three different classes of utility functions.
Each utility structure endows Equation (10) with different smoothness properties, which allows us to compare the efficiency of the algorithms under varying conditions.
Let v i ∈
R m be a vector of valuation parameters that describes the utility function of buyer i ∈ [ n ] .
We consider the following utility function classes: 1.
linear: u i ( x i ) =
(cid:80) j ∈ [ m ] v ij x ij ; 2.
Cobb-Douglas: u i ( x i ) = (cid:81) j ∈ [ m ] x v ij ij ; and 3.
Leontief:
u i ( x i ) =
min j ∈ [ m ] (cid:110) x ij v ij (cid:111) .
To simulate the dynamic Fisher markets, we fix a range for every market parameter and draw from that range uniformly at random during each iteration.
Our goal is to understand how closely OMD dynamics track the Stackelberg equilibria of the game as the latter varies with time.
To do so, we compare the distance between the iterates (cid:0) p ( t ) , X ( t ) (cid:1) computed by the algorithms and the equilibrium of the game at each iteration t .
This distance is measured as (cid:13)(cid:13) p ( t ) ∗ − p ( t ) (cid:13)(cid:13)
+ (cid:13)(cid:13)
X ( t ) ∗ − X ( t ) (cid:13)(cid:13) 2 , where (cid:0) p ( t ) ∗ , X ( t ) ∗ (cid:1) is the Stackelberg equilibrium of the Fisher market ( U ( t ) , b ( t ) , s ( t ) ) at time
t ∈
[ T ] .
In our experiments, we ran Algorithms 3 and 4 on 100 ran- domly initialized dynamic Fisher markets.
We depict the distance to equilibrium at each iteration for a randomly chosen experiment in Figures 1 and 2.
In these figures, we observe that our OMD dynamics are closely tracking the Stackelberg equilibria as they vary with each iteration.
A more detailed description of our experimental setup can be found in Appendix E.
We observe from Figures 1 and 2 that for both Algorithms 3 and 4, we obtain an empirical convergence rate relatively close to O ( 1 / √ T ) under Cobb-Douglas utilities, and a slightly slower empirical convergence rate under linear utilities.
Recall that O ( 1 / √ T ) is the convergence rate guarantee we obtained for both algorithms, assuming a fixed learning rate in a static Fisher market (Corollaries 7 and 8).
Dynamic Fisher markets with Leontief utilities, in which the objective function is not differentiable, are the hardest markets of the three for our algorithms to solve.
Still, we only see a slightly slower than O ( 1 / √ T ) empirical convergence rate for both Algorithms 3 and 4.
In these experiments, the convergence curve generated by Algorithm 4 has a less erratic behavior than the one generated by Algorithm 3.
Due to the non-differentiability of the objective function, the gradient ascent step in Algorithm 4 for buyers with Leontief utilities is very small, effectively dampening any potentially erratic changes it the iterates.
Our experiments suggest that even when the game changes at each iteration, OMD dynamics (Algorithmsand 4 - Appendix D) are robust enough to closely track the changing Stackelberg equilibria of dynamic Fisher markets.
We note that tˆatonnement dynamics (Algorithm 3) seem to be more robust than myopic best response dynamics (Algorithm 4), i.e., the distance to equilibrium allocations is smaller at each iteration of tˆatonnement.
This result is not surprising, as tˆatonnement computes a utility-maximizing allocation for the buyers at each time step.
Even though Theorems 10 and 11 only provide theoretical guarantees on the robustness of OMD dynamics in dynamic min-max games (with independent strategy sets), it seems like similar theoretical robustness results may be attainable in dynamic minmax Stackelberg games (with dependent strategy sets).
Conclusion
We began this paper by considering no-regret learning dynamics for min-max Stackelberg games in two settings: a pessimistic setting in which the outer player is a no-regret learner and the inner player best responds, and an optimistic setting in which both players are no-regret learners.
For both of these settings, we proved that no-regret learning dynamics converge to a Stackelberg equilibrium of the game.
We then specialized the no-regret algorithm employed by the players to online mirror descent (OMD), which yielded two known algorithms, namely max-oracle gradient descent (Jin,
Netrapalli, and Jordan 2020) and nested GDA (Goktas and
Greenwald 2021) in the pessimistic setting, and a new simultaneous GDA-like algorithm (Nedic and Ozdaglar 2009), which we call Lagrangian GDA, in the optimistic setting.
As these algorithms are no-regret learning algorithms, our previous theorems imply convergence to Stackelberg equilibria in O ( 1 / ε 2 ) iterations.
Finally, we investigated the robustness of OMD dynamics to perturbations in the parameters of a min-max Stackelberg game.
To do so, we analyzed how closely OMD dynamics track Stackelberg equilibria in dynamic min-max Stackelberg games.
We proved that in minmax games (with independent strategy sets) OMD dynamics closely track the changing Stackelberg equilibria of a game.
As we were not able to extend these theoretical robustness guarantees to min-max Stackelberg games (with dependent strategy sets), we instead ran a series of experiments on dynamic Fisher markets, which are canonical examples of min-max Stackelberg games.
Our experiments suggest that
OMD dynamics are robust for min-max Stackelberg games so that perhaps the robustness guarantees we have provided for OMD dynamics in min-max games can be extended to min-max Stackelberg games.
The theory developed in this paper opens the door to extending the myriad applications of
Stackelberg games in AI to incorporating dependent strategy sets.
A Background
Notation We use Roman uppercase letters to denote sets (e.g., X ), bold uppercase letters to denote matrices (e.g.,
X ), bold lowercase letters to denote vectors (e.g., p ), and
Roman lowercase letters to denote scalar quantities, (e.g., c ).
We denote the i th row vector of a matrix (e.g., X ) by the corresponding bold lowercase letter with subscript
i (e.g., x i ) .
Similarly, we denote the j th entry of a vector (e.g., p or x i ) by the corresponding Roman lowercase letter with subscript j (e.g., p j or x ij ).
We denote the vector of ones of size n by 1 n .
We denote the set of integers { 1 , . . . , n } by [ n ] , the set of natural numbers by N , the set of positive natural numbers by N + the set of real numbers by R , the set of non-negative real numbers by R + , and the set of strictly positive real numbers by R ++ .
We denote the orthogonal projection operator onto a convex set C by Π C , i.e., Π C ( x ) =
arg min y ∈ C
∥ x − y ∥ 2 .
Given a sequence of iterates { z ( t ) } Tt =1 ⊂
Z , we denote the average iterate ¯ z ( T ) = 1 T (cid:80)
Tt =1 z ( t ) .
Online Convex Optimization
An online convex optimization problem (OCP) is a decision problem in a dynamic environment which comprises a finite time horizon
T , a compact, convex feasible set X , and a sequence of convex differentiable loss functions { ℓ ( t ) } Tt =1 , where ℓ ( t ) :
X → R for all t ∈ [ T ] .
A solution to an OCP is a sequence { x ( t ) } Tt =1 ⊂ X .
A preferred solution is one that minimizes average regret given by Regret ( T ) ( x ) = (cid:80)
Tt =1 1 T ℓ ( t ) ( x ( t ) ) − (cid:80)
Tt =1 1 T ℓ ( t ) ( x ) , over all x ∈ X .
An algorithm A : X R → X T that takes as input a sequence of loss functions and outputs decisions such that (cid:80)
Tt =
[Equation]
t )
[Equation]
A t
[Equation] [Equation]
( t ) }
Tt =1 )
[Equation]
t ) ( x ) → 0 as T → ∞ is called a no-regret algorithm .
A first-order method that solves OCPs is Online Mirror Descent (OMD) .
For some initial iterates u (0)
=
and x ( t ) ∈ X , OMD performs the following update in the dual space X ∗ at each time step t : u ( t +1)
=
u ( t ) − η ∇ x ℓ ( t ) ( x ( t ) ) , and then projects the iterate computed in the dual space X ∗ back to the primal space
X : x ( t +1) =
arg min x ∈ X (cid:8) R ( x ) − (cid:10) u
( t +1) , x (cid:11)(cid:9) , where
R : X → R is a strongly-convex differentiable function.
When R ( x ) =
∥ x ∥ 2 2 , OMD reduces to projected online gradient descent , given by the update rule: x ( t +1) =
Π X (cid:0) x ( t ) − η ∇ x ℓ ( t ) ( x ( t ) ) (cid:1) .
The following theorem bounds the average regret of OMD (Kakade, Shalev-Shwartz, and
Tewari 2012):
Theorem 12.
Let c = max x ∈ X ∥ x ∥ , and let { ℓ ( t ) } t be a sequence of ℓ -Lipschitz loss functions s.t.
for all t ∈ N + , ℓ ( t ) : R n → R with respect to the dual norm ∥·∥ ∗ .
Then, if η = c ℓ √ 2 T , online projected gradient descent achieves bounded average regret bounded as follows:
[Equation]
[Table]
B Additional Related Work
Related Work Stackelberg games (Von Stackelberg 1934) have found important applications in the domain of security (e.g., (Nguyen et al. 2016; Sinha et al. 2018)) and environmental protection (e.g., (Fang and Nguyen 2016)).
These applications have thus far been modelled as Stackelberg games with independent strategy sets.
Yet, the increased expressiveness of Stackelberg games with dependent strategy sets may make them a better model of the real world, as they provide the leader with more power to achieve a better outcome by constraining the follower’s choices.
The study of algorithms that compute competitive equilibria in Fisher markets was initiated by Devanur et al.
(Devanur et al. 2002), who provided a polynomial-time method for solving these markets assuming linear utilities.
More recently, Cheung, Hoefer, and Nakhe (Cheung, Hoefer, and Nakhe 2019) studied two price adjustment processes, tˆatonnement and proportional response dynamics, in dynamic Fisher markets and showed that these price adjustment processes track the equilibrium of Fisher markets closely even when the market is subject to change.
Additional Related Work Much progress has been made recently in solving min-max games with independent strategy sets, both in the convex-concave case and in nonconvex-concave case.
We provide a survey of the literature as presented by Goktas and Greenwald in what follows.
For the former case, when f is µ x -strongly-convex in x and µ y -strongly-concave in y , Tseng (Tseng 1995),
Yurii Nesterov (Yurii Nesterov 2011), and Gidel et al.
(Gidel et al. 2020) proposed variational inequality methods, and
Mokhtari, Ozdaglar, and Pattathil (Mokhtari, Ozdaglar, and
Pattathil 2020), gradient-descent-ascent (GDA)-based methods, all of which compute a solution in ˜ O ( µ y + µ x ) iterations.
These upper bounds were recently complemented by the lower bound of ˜Ω( √ µ y µ x ) , shown by Ibrahim et al.
(Ibrahim et al. 2019) and Zhang, Hong, and Zhang (Zhang,
Hong, and Zhang 2020).
Subsequently, Lin, Jin, and Jordan (Lin, Jin, and Jordan 2020b) and Alkousa et al.
(Alkousa et al. 2020) analyzed algorithms that converge in ˜
O ( √ µ y µ x ) and ˜ O (min (cid:8) µ x √ µ y , µ y √ µ x (cid:9) ) iterations, respectively.
Forthespecial case where f is µ x -strongly convex in x and linear in y , Juditsky, Nemirovski et al.
(Juditsky, Nemirovski et al. 2011), Hamedani and Aybat (Hamedani and
Aybat 2018), and Zhao (Zhao 2019)
all present methods that converge to an ε -approximate solution in O ( (cid:112) µ x / ε ) iterations.
When the strong concavity or linearity assumptions of f on y are dropped, and f is assumed to be µ x -stronglyconvex in x but only concave in y , Thekumparampil et al.
(Thekumparampil et al. 2019) provide an algorithm that converges to an ε -approximate solution in ˜ O ( µ x / ε ) iterations, and Ouyang and Xu (Ouyang and Xu 2018) provide a lower bound of ˜Ω (cid:16)(cid:112) µ x / ε (cid:17) iterations on this same computation.
Lin, Jin, and Jordan then went on to develop a faster algorithm, with iteration complexity of ˜ O
(cid:16)(cid:112) µ x / ε (cid:17) , under the same conditions.
When f is simply assumed to be convex-concave, Nemirovski (Nemirovski 2004), Nesterov (Nesterov 2007), and
Tseng (Tseng 2008) describe algorithms that solve for an ε -approximate solution with ˜ O (cid:0) ε − 1 (cid:1) iteration complexity, and Ouyang and Xu (Ouyang and Xu 2018) prove a corresponding lower bound of Ω( ε − 1 ) .
When f is assumed to be non-convex- µ y -stronglyconcave, and the goal is to compute a first-order Nash, Sanjabi et al.
(Sanjabi et al. 2018b) provide an algorithm that converges to ε -an approximate solution in O ( ε − 2 ) iterations.
Jin, Netrapalli, and Jordan (Jin, Netrapalli, and Jordan 2020), Rafique et al.
(Rafique et al. 2019), Lin, Jin, and
Jordan (Lin, Jin, and Jordan 2020a), and Lu, Tsaknakis, and
Hong (Lu, Tsaknakis, and Hong 2019) provide algorithms that converge in ˜ O
(cid:0) µ 2 y ε − 2 (cid:1) iterations, while Lin, Jin, and
Jordan (Lin, Jin, and Jordan 2020b) provide an even faster algorithm, with an iteration complexity of ˜ O (cid:0) √ µ y ε − 2 (cid:1) .
When f is non-convex-non-concave and the goal to compute is an approximate first-order Nash equilibrium, Lu,
Tsaknakis, and Hong (Lu, Tsaknakis, and Hong 2019) provide an algorithm with iteration complexity ˜ O
( ε − 4 ) , while
Nouiehed et al.
(Nouiehed et al. 2019) provide an algorithm with iteration complexity ˜
O ( ε − 3 . 5 ) .
More recently, Ostrovskii, Lowy, and Razaviyayn (Ostrovskii, Lowy, and Razaviyayn 2020) and Lin, Jin, and Jordan (Lin, Jin, and Jordan 2020b) proposed an algorithm with iteration complexity ˜
O (cid:0) ε − 2 .
5 (cid:1) .
When f is non-convex-non-concave and the desired solution concept is a “local” Stackelberg equilibrium, Jin,
Netrapalli, and Jordan (Jin, Netrapalli, and Jordan 2020),
Rafique et al.
(Rafique et al. 2019), and Lin, Jin, and Jordan (Lin, Jin, and Jordan 2020a) provide algorithms with a ˜
O (cid:0) ε − 6 (cid:1) complexity.
More recently, Thekumparampil et al.
(Thekumparampil et al. 2019), Zhao (Zhao 2020), and Lin,
Jin, and Jordan (Lin, Jin, and Jordan 2020b) have proposed algorithms that converge to an ε -approximate solution in ˜
O (cid:0) ε − 3 (cid:1) iterations.
We summarize the literature pertaining to the convexconcave and the non-convex-concave settings in Tablesand 2 respectively.
[Table]
Mokhtari,
Ozdaglar, and
Pattathil
[Table]
Alkousa et
[Table]
[Table] [Table]
Lin,
[Table]
and Jordan 2020b)
[Table]
et
[Table]
and
[Table]
Juditsky,
Ne
[Table]
mirovski et al.
[Table]
µ x / ε (cid:17) (Hamedani and
Aybat 2018)
(Zhao 2019)
[Table]
(Thekumparampil et al. 2019)
˜ O ( µ x / √ ε ) (Lin, Jin, and Jordan 2020b)
˜ O
( (cid:112) µ x / ε )
(Ouyang and Xu 2018)
˜Ω (cid:16)(cid:112) µ x / ε (cid:17)
[Table]
(Nemirovski 2004)
O (cid:0) ε − 1 (cid:1) (Nesterov 2007)
(Tseng 2008)
(Lin, Jin, and Jordan 2020b)
˜ O (cid:0) ε − 1 (cid:1) (Ouyang and Xu 2018)
Ω( ε − 1 )
Note that although all these results assume that the objective function is Lipschitz-smooth, some authors make additional assumptions: e
[Table]
g., (Nouiehed et al.
2019) obtain their result for objective functions that satisfy the Lojasiwicz condition
[Table]
LocalStackelbergEquilibrium
[Table]
Proof of Theorem 3.
Since pessimistic regret is bounded by ε after T iterations, it holds that:
[Equation]
[Table]
[Equation]
[Table]
[Equation]
T
[Equation]
Since the game is static
[Equation]
and it further holds that:
[Equation]
Thus, by the convexity of V X (see Proposition 2)
[Equation]
−
[Equation]
∈
[Equation]
V X ( x ) ≤ ε .
Now replacing V X by its definition
[Equation]
and setting y ∗ (¯ x ( T ) )
∈ BR Y (¯ x ( T ) )
[Equation]
we obtain that (cid:0) ¯ x ( T ) , y ∗ (¯ x ( T ) ) (cid:1) is ( ε, 0)
-Stackelberg equilibrium:
[Equation]
Proof of Lemma 5.
We can relax the inner player’s payoff maximization problem via the problem’s Lagrangian and since by assumption 1, Slater’s condition is satisfied, strong duality holds, giving us for all x ∈ X : [Equation] = [Equation]
L
[Equation]
y [Equation] λ ) .
We can then re-express the min-max game as: min x ∈ X max y ∈ Y : g ( x , y ) ≥
[Equation]
f
[Equation]
x ,
[Equation]
min λ
[Equation]
min
[Equation]
max
[Equation] Y {+L x ( y , λ ) . Letting λ ∗ ∈ arg min λ ≥ 
 min x ∈ X max y ∈+}
Y L x ( y , λ ) , we have min x ∈ X max
[Equation]
x ,
[Equation]
max
[Equation] y , λ ∗ ) .
Note that L x ( y , λ ∗ ) is convex-concave in ( x , y ) .
Hence, any Stackelberg equilibrium ( x ∗ , y ∗ ) ∈
X × Y of ( X, Y, f, g ) is a saddle point of L x ( y , λ ∗ ) , i.e., ∀ x ∈
X, y ∈ Y, L x ∗ ( y , λ ∗ ) ≤
L x ∗ ( y ∗ , λ ∗ ) ≤
L x ( y ∗ , λ ∗ ) .
Proof of Theorem 6.
Since the Lagrangian regret is bounded for both players we have:
[Equation]
The last line follows because the min-max Stackelberg game is static.
Summing the final two inequalities yields:
[Equation]
where the second inequality was obtained by an application of Jensen’s inequality on the first and second terms.
Since L is convex in x and concave in y , we have that
[Equation]
) ( y , λ ∗ ) is convex in x and min x ∈ X L x ( y ( t ) , λ ∗ ) is convex in y , which implies that max
y ∈ Y L ¯ x ( T ) ( y , λ ∗ )
− min x ∈ X L x
(¯ y ( T ) , λ ∗ ) ≤ 2 ε .
By the max-min inequality ((Boyd, Boyd, and Vandenberghe 2004), Equation 5.46), it also holds that min x ∈ X L x (¯ y ( T ) , λ ∗ ) ≤ max
y ∈
Y L ¯ x ( T ) ( y , λ ∗ ) .
Combining these two inequality yields the desired result.
Proof of Theorem 10.
The value function of the outer player in the game (cid:8) ( X, Y, f ( t ) ) (cid:9) T t =1 at iteration t ∈
[ T ] , is given by V ( t ) ( x ) = max
y ∈
Y f
( t ) ( x , y ) .
Hence, for all t ∈ [ T ] , as f ( t ) is µ -strongly-convex, V ( t ) is also strongly concave since the maximum preserves strong-convexity.
Additionally, since for all t ∈ [ T ] , f ( t ) is strictly concave in y , by Danskin’s theorem (Danskin 1966), for all t ∈
[ T ] , V ( t ) is differentiable and its derivative is given by ∇ x V ( t ) ( x )
= ∇ x f ( x , y ∗ ( x )) where y ∗ ( x ) ∈ max
y ∈
Y f
( t ) ( x , y ) .
Thus, as ∇ x f ( x , y ∗ ( x )) is ℓ ∇ f -lipschitz continuous, so is ∇ x V ( t ) ( x ) .
The result follows from Cheung, Hoefer, and Nakhe’s bound for gradient descent on shifting strongly convex functions ((Cheung,
Hoefer, and Nakhe 2019), Proposition 12).
Proof of Theorem 11.
By the assumptions of the theorem, the loss functions of the outer player { f ( t ) ( · , y ( t ) ) }
Tt = are µ and functions.
Similarly the loss functions of the inner player {− f ( t ) ( x ( t ) , · ) } Tt =1 are µ y -strongly-convex and ℓ ∇ fLipschitz continuous functions.
Using Cheung, Hoefer, and
Nakhe’s Proposition 12 (Cheung, Hoefer, and Nakhe 2019), we then obtain the following bounds:
[Equation]
Combining the two inequalities, we obtain:
[Equation]
The second part of the theorem follows by taking the sum of the geometric series
[Equation]
Inputs: X, Y, f, g , η , T, x (0)
Output: ( x ∗ , y ∗ ) 1: for t = 1 , . .
, T do 2:
Find y ∗ ( x ( t − 1) )
∈ BR Y ( x ( t − 1) ) 3:
Set y ( t − 1) =
y ∗ ( x ( t − 1) ) 4:
Set λ ( t − 1) =
λ ∗ ( x ( t − 1) , y ( t − 1) ) 5:
Set x ( t ) =
Π X (cid:2)
x ( t − 1) − η t ∇ x L x ( t − 1) (cid:0) y
( t − 1) , λ ( t − 1) (cid:1)(cid:3) 6: end for 7: Set ¯ x ( T ) =
1 T (cid:80)
Tt =1 x ( t ) 8: Set y ∗ (¯ x ( T ) )
∈ BR Y (¯ x ( T ) ) 9: return (¯ x ( T ) , y ∗ (¯ x ( T ) ))
Algorithm
Inputs: λ ∗ , X, Y, f, g , η x , η y , T, x (0) , y (0)
Output: x ∗ , y ∗ 1: for t = 1 , . .
, T − 1 do 2:
Set x ( t +1) =
Π X (cid:0) x ( t ) − η x t ∇ x L x ( t ) ( y ( t ) , λ ∗ ) (cid:1) 3:
Set y ( t +1) =
Π Y (cid:0) y ( t ) + η y t ∇ y L x ( t ) ( y ( t ) , λ ∗ ) (cid:1) 4: end for 5: return { ( x ( t ) , y ( t ) ) }
Tt =
Inputs: T, { ( U ( t ) , b ( t ) , s ( t ) ) }
Tt =1 , η , p (0) , δ
Output: x ⋆ , y ⋆ 1: for t = 1 , . .
, T − 1 do 2:
For all i ∈ [ n ] , find x ( t ) i ∈ arg max x i ∈
R
m
+ : x i · p ( t − 1) ≤ b ( t ) i u i ( x i ) 3:
Set p ( t ) =
Π R m + (cid:16) p ( t − 1) − η t ( s ( t ) − (cid:80) i ∈ [ n ] x ( t ) i ) (cid:17) 4: end for 5: return ( p ( t ) , X ( t ) )
Tt =
Inputs: { ( U ( t ) , b ( t ) , s ( t ) ) }
Tt =1 , η p , η X , T, X (0) , p (0)
Output: x ⋆ , y ⋆ 1: for t = 1 , . .
, T − 1 do 2:
Set p ( t +1) =
Π R m + (cid:16) p ( t ) − η p t ( s ( t ) − (cid:80) i ∈
[ n ] x ( t ) i ) (cid:17) 3:
For all i ∈ [ n ] , set x ( t +1) i
=
Π R m + (cid:18) x ( t ) i + η X t (cid:18) b ( t ) i u ( t ) i (cid:16) x ( t ) i (cid:17)
∇ x i u ( t ) i (cid:16) x ( t ) i (cid:17) − p ( t ) (cid:19)(cid:19) 4: end for 5: return ( p ( t ) , X ( t ) )
T t =
E An Economic Application: Details
Our experimental goal was to understand if Algorithmand Algorithm 4 converges in terms of distance to equilibrium and if so how the rate of convergences changes under different utility structures, i.e. different smoothness and convexity properties of the value functions.
To answer these questions, we ran multiple experiments, each time recording the prices and allocations computed by
Algorithm 3, in pessimistic learning setting, and by Algorithm 4, in optimistic learning setting, during each iteration t of the loop.
Moreover, at each iteration t , we solve the competitive equilibrium ( p ( t ) ⋆ , X ( t ) ⋆ ) for the Fisher market ( U ( t ) , b ( t ) , s ( t ) ) .
Finally, for each run of the algorithm on each market, we then computed distance between the computed prices, allocations and the equilibrium prices, allocations, which we plot in Figure 1 and Figure 2.
Hyperparameters We set up 100 different linear, CobbDouglas, Leontief dynamic Fisher markets with random changing market parameters across time, each with 5 buyers and 8 goods, and we randomly pick one of these experiments to graph.
In our execution of Algorithm 3, buyer i ’s budget at iteration t , b ( t ) i , was drawn randomly from a uniform distribution ranging from 10 to 20 (i.e., U [10 , 20] ), each buyer i ’s valuation for good j at iteration t , v ( t ) ij , was drawn randomly from U [5 , 15] , while each good j ’s supply at iteration t , s ( t ) j , was drawn randomly from U [100 , 110] .
In our execution of
Algorithm 4, buyer i ’s budget at iteration t , b ( t ) i , was drawn randomly from a uniform distribution ranging from 10 to(i.e.
, U [10 , 15] ), each buyer i ’s valuation for good j at iteration t , v ( t ) ij , was drawn randomly from U [10 , 20] , while each good j ’s supply at iteration t , s ( t ) j , was drawn randomly from
U [10 , 15] .
We ran both Algorithm 3 and Algorithm 4 for 1000 iterations on linear, Cobb-Douglas, and Leontief Fisher markets.
We started the algorithm with initial prices drawn randomly from U [5 , 55] .
Our theoretical results assume fixed learning rates, but since those results apply to static games while our experiments apply to dynamic Fisher markets, we selected variable learning rates.
After manual hyper-parameter tuning, for Algorithm 3, we chose a dynamic learning rate of η t = 1 √ t , while for Algorithm 4, we chose learning rates of η x t = 5 √ t and η y t = 0 .
01 √ t , for all t ∈
[ T ] .
For these choices of learning rates, we obtain empirical convergence rates close to what the theory predicts.
Programming Languages, Packages, and Licensing We ran our experiments in Python 3.7 (Van Rossum and
Drake Jr 1995), using NumPy (Harris et al. 2020), Pandas (pandas development team 2020), and CVXPY (Diamond and Boyd 2016).
Figure 1 and Figure 2 were graphed using
Matplotlib (Hunter 2007).
Python software and documentation are licensed under the PSF License Agreement.
Numpy is distributed under a liberal BSD license.
Pandas is distributed under a new BSD license.
Matplotlib only uses BSD compatible code, and its license is based on the PSF license.
CVXPY is licensed under an APACHE license.
Implementation Details In order to project each allocation computed onto the budget set of the consumers, i.e., { X ∈ R n × m + | Xp ≤ b } , we used the alternating projection algorithm for convex sets, and alternatively projected onto the sets R n × m + and { X ∈ R n × m | Xp ≤ b } .
To compute the best-response for the inner play in Algorithm 3, we used the ECOS solver, a CVXPY’s first-order convex-program solvers, but if ever a runtime exception occurred, we ran the SCS solver.
When computing the distance from the demands X ( t ) computed by our algorithms to the equilibrium demands
X ( t ) ⋆ , we normalize both demands to satisfy ∀ j ∈ [ m ] , (cid:80) i ∈ [ n ] x ij = 1 m to reduce the noise caused by changing supplies.
Computational Resources Our experiments were run on
MacOS machine with 8GB RAM and an Apple M1 chip, and took about 2 hours to run.
Only CPU resources were used.
Code Repository
The data our experiments generated, as well as the code used to produce our visualizations, can be found in our code repository (https://anonymous.4open.science/r/Dynamic-MinmaxGames-8153/).