Zero-Sum Stochastic Stackelberg Games
Abstract
Zero-sum stochastic games have found important applications in a variety of ﬁelds, from machine learning to economics.
Work on this model has primarily focused on the computation of Nash equilibrium due to its effectiveness in solving adversarial board and video games.
Unfortunately, a Nash equilibrium is not guaranteed to exist in zero-sum stochastic games when the payoffs at each state are not convexconcave in the players’ actions.
A Stackelberg equilibrium, however, is guaranteed to exist.
Consequently, in this paper, we study zero-sum stochastic Stackelberg games.
Going beyond known existence results for (non-stationary)
Stackelberg equilibria, we prove the existence of recursive (i.e., Markov perfect) Stackelberg equilibria (recSE) in these games, provide necessary and sufﬁcient conditions for a policy proﬁle to be a recSE, and show that recSE can be computed in (weakly) polynomial time via value iteration.
Finally, we show that zero-sum stochastic
Stackelberg games can model the problem of pricing and allocating goods across agents and time.
More speciﬁcally, we propose a zero-sum stochastic Stackelberg game whose recSE correspond to the recursive competitive equilibria of a large class of stochastic Fisher markets.
We close with a series of experiments that showcase how our methodology can be used to solve the consumption-savings problem in stochastic Fisher markets.
Min-max optimization has paved the way for recent progress in a variety of ﬁelds, from machine learning to economics.
These applications require computing solutions to a constrained min-max optimization problem i.e., min x 2 X max y 2 Y f
( x , y ) , where the objective function f : X
⇥ Y !
R is continuous, and the constraint sets X ⇢ R n and Y ⇢ R m are nonempty and compact.
When f is convex-concave, and the constraint sets
X and Y are convex, the seminal minimax theorem [1, 2] holds, i.e., min x 2 X max y 2 Y
f
( x , y ) =
max y 2 Y min x 2 X f ( x , y ) , and such problems can be interpreted as computing a Nash equilibrium of a simultaneous-move min-max (or zero-sum ) game between an outer player x and an inner player y with respective payoff functions  f , f and respective action sets X , Y , where the solutions ( x ⇤ , y ⇤ )
2 X ⇥ Y are best responses to one another.
More generally, one can consider zero-sum stochastic games , played over an inﬁnite discrete time horizon N + .
The game starts at some initial state S (0)
⇠ µ (0) .
At each subsequent time-step t 2 N + , players encounter a new state s ( t ) 2 S .
After taking their respective actions ( x ( t ) , y ( t ) ) from their respective action spaces X ( s ( t ) ) ✓ R n and Y ( s ( t ) )
✓ R m , they receive payoffs r ( s ( t ) , x ( t ) , y ( t ) ) , and then either transition to a new state S ( t +1)
⇠ p ( · | s ( t ) , x ( t ) , y ( t ) ) with probability  , or the game ends with the remaining probability.
The goal of the outer (resp.
inner) player is to play a sequence 36th of actions { x ( t ) } t (resp.
{ y ( t ) } t ), that maximizes (resp.
minimizes) their expected cumulative discounted payoff (resp.
loss) E ⇥P
[Equation]
⇤ , ﬁxing their opponent’s policy.
A (stationary) policy is a mapping from states to actions.
When r ( s , x , y ) is bounded, continuous, and concave-convex in ( x , y ) , for all s 2 S , we are guaranteed the existence of a stationary policy proﬁle , i.e., a pair of policies ⇡ x :
[Equation]
for the outer and inner players, respectively, specifying the actions taken at each state, with a unique value such that both players maximize their expected payoffs, as a generalization of the minimax theorem holds [3]:
[Equation]
In other words, under the aforementioned assumptions, we are guaranteed the existence of a recursive
Nash equilibrium (sometimes called a Markov perfect Nash equilibrium [4]), a stationary policy proﬁle in which players not only best respond to one another, but they do so at every state of the game.
Additionally, when the rewards at each state are convex-concave, a recursive Nash equilibrium can be computed in polynomial time by iterative application of the min-max operator [3].
Zero-sum stochastic games generalize zero-sum games from a single state to multiple states, and have found even more applications in a variety of ﬁelds [5].
Unfortunately, when the objective function in a min-max optimization problem is not convexconcave, a minimax theorem is not guaranteed to hold, precluding the interpretation of the game as simultaneous-move, and the guaranteed existence of Nash equilibrium.
Nonetheless, the game can still be viewed as a Stackelberg game, in which the outer player moves before the inner one.
The canonical solution concept in such games is Stackelberg equilibrium (SE) .
Moreover, in Stackelberg games, the inner player’s actions can be constrained by the outer player’s choice, without impacting existence [6].
The result is a min-max Stackelberg game : i.e., min x 2 X max y 2 Y : h ( x , y )  0 f
( x , y ) where f, h :
X ⇥ Y !
R are continuous, and X , Y are non-empty and compact.
Even more problems of interest can be captured by this model [7, 8, 9, 10].
One can likewise consider zero-sum stochastic Stackelberg games , which generalize both zero-sum
Stackelberg games and zero-sum stochastic games.
Similar to zero-sum stochastic games, these games are played over an inﬁnite discrete time horizon N + , start at some state S (0)
⇠ µ (0) and consist of nonempty and compact actions spaces X ⇢ R n and Y ⇢ R m 2 , a state-dependent payoff function r ( s , x , y ) , a transition probability p ( s 0 | s , x , y ) , and a discount rate  , but are in addition augmented with a state-dependent (joint action) constraint function g ( s , x , y ) , with two players that seek to optimize their cumulative discounted payoffs, in expectation, while satisfying the constraint g ( s , x , y )  0 at each state s 2 S .
Applications of this model include autonomous driving [8, 10], reach-avoid problems in human-robot interaction [9], and robust optimization in stochastic environments [7], and, as we show, economic markets.
While in stochastic games, players announce their policies simultaneously before play commences, in stochastic Stackelberg games, the outer player, announces their (in general, non-stationary) policy: i.e., the action they will take at each time step, ﬁrst, after which the inner player announces theirs.
The canonical solution concept for such games is the Stackelberg equilibrium , which is guaranteed to exist (in non-stationary policies) under mild assumptions [11, 6].
The computational complexity of non-stationary equilibrium policies in stochastic games can be prohibitive, since even representing such policies in an inﬁnite horizon setting is intractable.
A natural question to ask then is whether stationary equilibria exist in zero-sum Stackelberg games, i.e., stationary policy proﬁles at which the outer player maximizes their expected discounted cumulative payoff while the inner player best responds.
We call such policies recursive Stackelberg equilibria (recSE) (or Markov perfect Stackelberg equilibria).
In this paper, we deﬁne and prove the existence of recSE in zero-sum stochastic Stackelberg games, provide necessary and sufﬁcient conditions for a policy proﬁle to be a recSE, and show that a recSE can be computed in (weakly) polynomial time via value iteration.
We further show that zero-sum stochastic Stackelberg games can be used to solve problems of pricing and allocating goods across agents and time.
In particular, we introduce stochastic Fisher markets , a stochastic generalization of the Fisher market [12], and a special case of Friesen’s [13] ﬁnancial market model, which itself is a stochastic generalization of the Arrow and Debreu model of a competitive economy [14].
We then prove the existence of recursive competitive equilibrium (recCE) [15] in this model, under the assumption that consumers have continuous and homogeneous utility functions, by characterizing the recCE of any stochastic Fisher market as the recSE of a corresponding zero-sum stochastic Stackelberg game.
Finally, we use value iteration to solve various stochastic Fisher markets, highlighting the issues that value iteration can encounter, depending on the smoothness properties of the utilities.
Related Work Algorithms for min-max optimization problems (i.e., zero-sum games) with independent strategy sets have been extensively studied [16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39] (for a summary see, Section G [6]).
Goktas and Greenwald studied min-max games with dependent strategy sets, proposing polynomial-time nested gradient descent ascent (GDA) [6] and simultaneous GDA algorithms for such problems [40].
The computation of Stackelberg equilibrium in two-player stochastic Stackelberg games has been studied in several interesting settings, in which the leader moves before the follower, but without the leader’s actions impacting the followers’ choice sets.
Bensoussan, Chen, and Sethi [41] study continuous-time general-sum stochastic Stackelberg games with continuous action spaces, and prove existence of a solution in this setting.
Vorobeychik and Singh [11] consider a general-sum stochastic
Stackelberg game with ﬁnite state-action spaces and an inﬁnite horizon.
These authors show that stationary SE policies do not exist in this very general setting, but nonetheless identify a subclass of games, namely team (or potential) Stackelberg games for which stationary Stackelberg equilibrium policies do exist.
Vu et al.
[42] study the empirical convergence of policy gradient methods in the same setting as Vorobeychik and Singh [11], while Ramponi and Restelli [43] study non-stationary equilibria in this same setting, assuming a ﬁnite horizon.
Chang, Erera, and White [44] and Sengupta and Kambhampati [45] consider a partially observable version of Vorobeychik and Singh’s [11] model, and provide methods to compute Stackelberg equilibria in their setting.
Some recent research concerns one leader-many followers Stackelberg games.
Vasal [46] studies a discrete-time, ﬁnite horizon one leader-many follower stochastic Stackelberg game with discrete action and state spaces, and provides algorithms to solve such games.
DeMiguel and Xu [47] consider a stochastic Stackelberg game-like market model with n leaders and m followers; they prove the existence of a SE in their model, and provide (without theoretical guarantees) algorithms that converge to such an equilibrium in experiments.
Dynamic Stackelberg games [48] have been applied to a wide range of problems, including security [46, 11], insurance provision [49, 50], advertising [51], robust agent design [52], allocating goods across time intertemporal pricing [53].
The study of algorithms that compute competitive equilibria in Fisher markets was initiated by
Devanur et al.
, who provided a polynomial-time method for solving these markets assuming linear utilities.
More recently, there have been efforts to study markets in dynamic settings [55, 56, 6], in which the goal is to either track the changing equilibrium of a changing market, or minimize some regret-like quantity for the market.
The models considered in these earlier works differ from ours as they do not have stochastic structure and do not invoke a dynamic solution concept.
Preliminaries
Notation We use caligraphic uppercase letters to denote sets (e.g., X ); bold lowercase letters to denote vectors (e.g., p , ⇡ ); bold uppercase letters to denote matrices and vector-valued random variables (e.g., X ,  )—which one should be clear from context; lowercase letters to denote scalar quantities (e.g., x,  ); and uppercase letters to denote scalar-valued random variables (e.g., X,  ).
We denote the i th row vector of a matrix (e.g., X ) by the corresponding bold lowercase letter with subscript
i (e.g., x i ) .
Similarly, we denote the j th entry of a vector (e.g., p or x i ) by the corresponding
Roman lowercase letter with subscript j (e.g., p j or x ij ).
We denote functions by a letter: e.g., f if the function is scalar valued, and f if the function is vector valued.
We denote the vector of ones of size n by 1 n .
We denote the set of integers { 1 , . . . , n } by [ n ] , the set of natural numbers by N , the set of real numbers by R .
We denote the postive and strictly positive elements of a set by a + and ++ subscript respectively, e.g., R + and R ++ .
We denote the orthogonal projection operator onto a set C by ⇧ C , i.e., ⇧ C ( x ) =
arg min y 2 C k x  y k 2 .
We denote by  n = { x 2 R n + |
P ni =1 x i = 1 } , and by  ( A ) , the set of probability measures on the set A .
A stochastic Stackelberg game ( S , X , Y , µ (0) , r x , r y , g , p,  ) is a two-player game played over an inﬁnite discrete time horizon N + .
At each time-step t 2 N + , the players, who we call the outer- (resp.
inner-) players, encounter a new state s 2 S , and choose an action to play from their continuous set of actions X ⇢ R n (resp.
Y ⇢ R m ).
Play initiates at a start state S (0) drawn from a distribution µ (0) :
S !
At each state s 2 S the action x 2 X chosen by the outer player determines the set of feasible actions { y 2 Y | g ( s , x , y )  0 } available to the inner player, where g : S ⇥ X ⇥ Y !
R d .
After the outer and inner players both make their moves, they receive payoffs r x : S ⇥ X ⇥ Y !
R and r y : S ⇥ X ⇥ Y !
R , respectively, and the game either ends with probability  , where  2 (0 , 1) is called the discount factor , or transitions to a new state s 0 2 S , according to a transition probability function p :
S ⇥ S ⇥ X ⇥ Y !
[0 , 1] s.t. p ( s 0 | s , x , y ) 2 [0 , 1] denotes the probability of transitioning to state s 0 2 S from state s 2 S
when action proﬁle ( x , y ) 2 X ⇥
Y is chosen by the players.
In this paper, we focus on zero-sum stochastic Stackelberg games G (0) .
= ( S , X , Y , µ (0) , r, g , p,  ) , in which the outer player’s loss is the inner player’s gain, i.e., r x =  r y .
A zero-sum stochastic
Stackelberg game reduces to zero-sum (simultaneous-move) stochastic game [3] in the special case where g ( s , x , y )  0 , for all state-action tuples ( s , x , y ) 2 S ⇥ X ⇥ Y .
More generally, a policy proﬁle ( ⇡ x , ⇡ y ) 2 X S ⇥ Y S is said to be feasible if g ( s , ⇡ x ( s ) , ⇡ y ( s ))
0 , for all states s 2 S .
To simplify notation, we introduce a function G : X S ⇥ Y S !
R |S| ⇥ d such that G ( ⇡ x , ⇡ y ) =
( g ( s , ⇡ x ( s ) , ⇡ y ( s ))) s 2 S , and deﬁne feasible policy proﬁles as those ( ⇡ x , ⇡ y ) 2 X S ⇥
Y S s.t.
G ( ⇡ x , ⇡ y )  0 .
From now on, we assume:
Assumption 1.1. 1.
For all states s 2 S , the functions r ( s , · , · ) , g ( s , · , · ) are continuous in ( x , y )
X ⇥ Y , and payoffs are bounded, i.e., k r k 1  r max < 1 , for some r max 2 R + , 2.
X , Y are non-empty and compact, and for all s 2 S and x 2 X there exists y 2 Y s.t.
g ( s , x , y )  0 .
Given a zero-sum stochastic Stackelberg game G (0) , the state-value function , v : S ⇥ X S ⇥ Y S !
R , and the action-value function , q : S ⇥ X ⇥ Y ⇥ X
S ⇥ Y S !
R , respectively, are deﬁned as:
[Equation] [Equation]
, ⇡
[Equation]
=
[Equation]
y
[Equation]
Again, to simplify notation, we write expectations conditional on X ( t ) =
⇡ x ( S ( t ) ) and
Y ( t ) =
⇡ y ( S ( t ) ) as E ⇡ x , ⇡ y , and denote the state- and action-value functions by v ⇡ x ⇡ y ( s ) , and q ⇡ x ⇡ y ( s , x , y ) , respectively.
Additionally, we let V =
[  r max / 1   , r max / 1   ] S be the space of all state-value functions of the form v : S !
[  r max / 1   , r max / 1   ] , and we let
Q =
[  r max / 1   , r max / 1   ] S ⇥ X ⇥
Y be the space of all action-value functions of the form q :
S ⇥ X ⇥ Y !
[  r max / 1   , r max / 1   ] .
Note that by Assumption 1.1 the range of the stateand action-value functions is [  r max / 1   , r max / 1   ] .
The cumulative payoff function of the game u : X S ⇥ Y S !
R is the total expected loss (resp.
gain) of the outer (resp.
inner) player, given by u ( ⇡ x , ⇡ y )
=
E s ⇠ µ (0) ( s )
[ v ⇡ x ⇡ y ( s )] .
The canonical solution concept for stochastic Stackelberg games is the Stackelberg equilibrium (SE) .
A feasible policy proﬁle ( ⇡ ⇤ x , ⇡ ⇤ y ) 2 X S ⇥ Y S is said to be a Stackelberg equilibrium (SE) of a zero-sum stochastic Stackelberg game G (0) iff
[Equation]
Note the strength of this deﬁnition, as it requires the constraints g ( s , ⇡ x , ⇡ y )  0 to be satisﬁed at all states s 2 S , not only states which are reached with strictly positive probability.
A SE is guaranteed to exist in zero-sum stochastic Stackelberg games, under Assumption 1.1, as a corollary of Goktas and Greenwald’s [6] Proposition B.2.
; however, this existence result is non-constructive.
In this paper, we study a Markov perfect reﬁnement of SE, which we call recursive Stackelberg equilibrium (recSE).
Deﬁnition 1.2 (Recursive Stackelberg Equilibrium (recSE)) .
A policy proﬁle ( ⇡ ⇤ x , ⇡ ⇤ y ) 2 S X ⇥ S Y is a recursive Stackelberg equilibrium (recSE) iff, for all s 2 S , it holds that:
[Equation]
Equivalently, a policy proﬁle ( ⇡ ⇤ x , ⇡ ⇤ y ) is a recSE if ( ⇡ ⇤ x ( s ) , ⇡ ⇤ y ( s )) is a SE with value v ⇡ ⇤ x
⇡ ⇤ y ( s ) at each state s 2 S : i.e., v ⇡ ⇤ x ⇡
[Equation]
y ( s
[Equation]
= min
[Equation]
[Equation]
q
[Equation]
y ( s
[Equation]
x
[Equation]
y
[Equation]
, for all s 2 S .
Mathematical Preliminaries
A probability measure q 1 2  ( S ) convex stochastically dominates (CSD) q 2 2  ( S ) if
R
[Equation]
s
[Equation]
ds
R S v ( s ) q 2 ( s ) ds for all continuous, bounded, and convex functions v on S .
A transition function p is termed CSD convex in x if, for all  2 (0 , 1) , y 2 Y and any ( s 0 , x 0 ) , ( s † , x † ) 2 S ⇥ X , with ( s , x ) =
( s 0 , x 0 ) + (1   )( s † , x † ) , it holds that  p ( · | s 0 , x 0 , y )
+ (1   ) p ( · | s † , x † , y )
CSD p ( · | s , x , y ) .
A transition function p is termed
CSD concave in y if, for all  2 (0 , 1) and any ( s 0 , y 0 ) , ( s † , y † ) 2 S ⇥ X ⇥ Y , with ( s , y ) =
( s 0 , y 0 )
+(1   )( s † , y † ) , it holds that p ( · | s , x , y ) CSD  p ( · | s 0 , x , y 0 )
+(1   ) p ( · | s † , x , y † ) .
A mapping L : A !
B is said to be a contraction mapping (resp.
non-expansion ) w.r.t.
norm k · k iff for all x , y 2 A , and for k 2 [0 , 1) (resp.
k
= 1 ) such that k L ( x )  L ( y ) k  k k x  y k .
The min-max operator min x 2 X max y 2 Y :
R X ⇥ Y !
R w.r.t.
to sets X , Y takes as input a function f :
X ⇥ Y !
R and outputs min x 2 X max y 2 Y f
( x , y ) .
The generalized min-max operator min x 2 X max y 2 Y : g ( x , y )  0 : R X ⇥ Y !
R w.r.t.
to sets X , Y and the function g : X ⇥ Y !
R takes as input a function f : X ⇥ Y !
R and outputs min x 2 X max y 2 Y : g ( x , y )  0 f
( x , y ) .
Properties of Recursive Stackelberg equilibrium
In this section, we show that a recSE exists in all zero-sum stochastic Stackelberg games.
5 To do so, we ﬁrst associate an operator C : V !
V with any zero-sum stochastic Stackelberg game G (0) , the ﬁxed points of which satisfy Deﬁnition 1.2, and hence correspond to the value function associated with a recSE of G (0) .
We then show that this operator is a contraction mapping, thereby establishing the existence of such a ﬁxed point.
This result generalizes Shapley’s theorem on the existence of Markov perfect Nash equilibria in zero-sum stochastic games [3].
Deﬁne C : V !
V for any zero-sum stochastic Stackelberg game G (0) as the operator ( Cv ) ( s ) =
min x 2 X
[Equation]
2 Y : g ( s , x , y )
[Equation]
E S
⇠ p ( ·| s , x , y )
[ r
[Equation]
s , x ,
[Equation]
+  v
[Equation]
S 0 )] .
We ﬁrst show that the ﬁxed points of C correspond to the recSE of the associated game.
Theorem 2.1.
( ⇡ ⇤ x , ⇡ ⇤ y ) is a recSE of G (0) of v ⇡ x ⇡ y iff it induces a value function which is a ﬁxed point of C : i.e., ( ⇡ ⇤ x , ⇡ ⇤ y )
is a Stackelberg equilbrium iff, for all s 2 S ,
[Equation]
The following technical lemma is crucial to proving that C is a contraction mapping.
It tells us that the generalized min-max operator is non-expansive; in other words, the generalized min-max operator is 1-Lipschitz w.r.t.
the sup-norm.
Lemma 2.2.
Suppose that f, h : X ⇥ Y !
[Equation]
R d are continuous functions, and X , Y are compact sets.
Then
[Equation]
With the above lemma in hand, we can now prove that C is a contraction mapping.
Theorem 2.3.
Consider the operator C associated with a stochastic Stackelberg game G (0) .
Under
Assumption 1.1, C is a contraction mapping w.r.t.
to the sup norm k .
k 1 with constant  .
Proof of Theorem 2.3.
We will show that C is a contraction mapping, which then by Banach ﬁxed point theorem establish the result.
Let v, v 0 2 V be any two state value functions and q, q 0 2 Q be the respective associated action-value functions.
We then have by Lemma 2.2:
[Equation]
Replacing the deﬁnition of the state-action value function in the above, we get that C is a contraction mapping since  2 (0 , 1) :
[Equation]
Given an initial state-value function v (0) 2 V , we deﬁne the value iteration process as v ( t +1) =
Cv ( t ) , for all t 2 N + (Algorithm 2).
One way to interpret v ( t ) is as the function that returns the value v ( t ) ( s ) of each state s 2 S in the t -stage zero-sum stochastic Stackelberg game starting at the last stage t and continuing until stage 0 , with terminal payoffs given by v (0) .
The following theorem, which is a consequence of Theorems 2.1 and 2.3, and the Banach ﬁxed point theorem [57], not only proves the existence of a recSE, but further provides us with a means of computing a recSE via value iteration.
Theorem 2.4.
Consider a zero-sum stochastic Stackelberg game G (0) .
Under Assumption 1.1, G (0) has a unique value function v ⇡ ⇤ x
⇡ ⇤ y associated with all recSE ( ⇡ ⇤ x , ⇡ ⇤ y ) , which can be computed by iteratively applying C to any initial state-value function v (0) 2 V : i.e., lim t !
1 v ( t ) = v ⇡ ⇤ x ⇡ ⇤ y .
Remark 2.5.
Unlike Shapley’s existence theorem for recursive Nash equilibria in zero-sum stochastic games, Theorem 2.4 does not require the payoff function to be convex-concave.
The only conditions needed are continuity of the payoffs and constraints, and bounded payoffs.
This makes the recSE a potentially useful solution concept, even for non-convex-non-concave stochastic games.
Since a recSE is guaranteed to exist, and is by deﬁnition independent of the initial state distribution, we can infer that the recSE of any zero-sum stochastic Stackelberg game G (0) =
( S , X , Y , µ (0) , r, g , p,  ) is independent of the initial state distribution µ (0) .
Hence, in the remainder of the paper, we denote zero-sum stochastic Stackelberg games by G .
= ( S , X , Y , r, g , p,  ) .
Theorem 2.4 tells us that value iteration converges to the value function associated with a recSE.
Additionally, under Assumption 1.1, recSE is computable in (weakly) polynomial time.
Theorem 2.6 (Convergence of Value Iteration) .
Suppose value iteration is run on input G .
Let ( ⇡ ⇤ x , ⇡ ⇤ y ) be recSE of G with value function v ⇡ ⇤ x ⇡ ⇤ y .
Under Assumption 1.1, if we initialize v (0) ( s ) = 0 , for all s 2 S , then for k  11   log r max ✏ (1   ) , it holds that v ( k ) ( s )  v ⇡ ⇤ x
⇡ ⇤ y ( s )  ✏ .
Subdifferential Envelope Theorems and Optimality Conditions for Recursive Stackelberg Equilibrium
In this section, we derive optimality conditions for recursive Stackelberg equilibria.
In particular, we provide necessary conditions for a policy proﬁle to be a recSE of any zero-sum stochastic Stackelberg game, and show that under additional convexity assumptions, these conditions are also sufﬁcient.
The Benveniste-Scheinkman theorem characterizes the derivative of the optimal value function associated with a recursive optimization problem w.r.t.
its parameters, when it is differentiable [58].
Our proofs of the necessary and sufﬁcient optimality conditions rely on a novel subdifferential generalization (Theorem C.2, Appendix C) of this theorem, which applies even when the optimal value function is not differentiable.
A consequence of our subdifferential version of the BenvenisteScheinkman theorem is that we can easily derive the ﬁrst-order necessary conditions for a policy proﬁle to be a recSE of any zero-sum stochastic Stackelberg game G satisfying Assumption 1.1, under standard regularity conditions.
Theorem 3.1.
Consider a zero-sum stochastic Stackelberg game G , where
X = { x 2 R n | q 1 ( x )  0 , . . .
, q p ( x )  0 } and Y = { y 2 R m | r 1 ( y )  0 , . .
, r l ( y )  0 } are convex.
Let
L s , x ( y ,  ) =
r ( s , x , y )
+  E S
⇠ p ( ·| s , x , y )
[ v ( S 0 , x )]
+
P dk
=
k g k
( s , x , y ) where Cv = v .
Suppose that
Assumption 1. holds, and that 1.
for all s
S , max y 2 Y : g ( s , x , y )
r ( s , x , y )
+  E S
[Equation]
p
[Equation]
·| s
[Equation]
x , y ) [ v ( S
[Equation]
, x )] is concave in x , 2.
r x r ( s , x , y ) , r x g 1 ( s , x , y ) , . . .
, r x g d ( s , x , y ) , r y r ( s , x , y ) , r y g 1 ( s , x , y ) , . . .
, r y g d ( s , x , y ) exist, for all s 2 S , x 2 X , y 2 Y , 4.
p (
s
|
s
x
y
) is continuous and differentiable in
( x , y
and
Slater’s condition holds, i.e., 8 s 2 S , x 2 X ,
[Equation]
y
[Equation]
Y s.t.
g k ( s , x , b y ) > 0 , for all k = 1 , . . .
, d and r j ( b y ) > 0 , for all j = 1 , . . .
, l , and 9 x 2 R n s.t. q k
( x ) <
for all k = 1 . . .
, p .
Then, there exists µ ⇤ : S !
R p + ,  ⇤ : S ⇥ X !
R d + , and ⌫ ⇤ :
S ⇥ X !
R l + s.t.
a policy proﬁle ( ⇡ ⇤ x , ⇡ ⇤ y ) 2 X S ⇥ Y S is a recSE of
G only if it satisﬁes the following conditions, for all s 2 S :
[Equation]
⇤
[Equation]
[Equation]
⇡
[Equation] [Equation]
⌫ ⇤ k
( s
[Equation]
⇡ ⇤ x ( s )
[Equation]
x
[Equation]
s
[Equation]
Under the conditions of Theorem 3.1, if we additionally assume that for all s 2 S and x 2 X , both r ( s , x , y ) and g 1 ( s , x , y ) , . . .
, g d ( s , x , y ) are concave in y , and p ( s| s , x , y ) is continuous, CSD concave in y , and differentiable in ( x , y ) , Equations (59) to (63) become necessary and sufﬁcient optimality conditions.
For completeness, the reader can ﬁnd the necessary and sufﬁcient optimality conditions for convex-concave stochastic Stackelberg games under standard regularity conditions in
Theorem C.3 (Appendix C).
The proof follows exactly as that of Theorem 2.1.
Recursive Market Equilibrium
We now introduce an application of zero-sum stochastic Stackelberg games, which generalizes a well known market model, the Fisher market [12], to a dynamic setting in which buyers not only participate in markets across time, but their wealth persists.
A (static) Fisher market consists of n buyers and m divisible goods [12].
Each buyer i 2 [ n ] is endowed with a budget b i 2 B i ✓ R + and a utility function u i : R m + ⇥ T i !
R , which is parameterized by a type t i 2 T i that deﬁnes a preference relation over the consumption space R m + .
Each good is characterized by a supply q j 2 Q j ⇢ R + .
An instance of a Fisher market is then a tuple M .
=
( n, m, U , T , b , q ) , where U = { u 1 , . . .
, u n } is a set of utility functions, one per buyer, b 2 R n + is the vector of buyer budgets, and q 2 R m + is the vector of supplies.
When clear from context, we simply denote M by ( T , b , q ) .
A stochastic Fisher market with savings is a dynamic market in which each state corresponds to a static Fisher market: i.e., each state s 2 S is characterized by a tuple s .
=
( T , b , q ) .
At each state, the market determines the prices p of the goods, while the buyers choose their allocations
X = ( x 1 , . . .
, x n ) T 2 R n ⇥ m + and potentially set aside some savings  i 2 [0 , b i ] to spend at some future state.
Once allocations, savings, and prices have been determined, the market terminates with probability 1   , or it transitions to a new state s 0 with probability  p ( s 0 | s ,  ) , depending on the buyers’ saving decisions.
7 We denote a stochastic Fisher market by F (0) .
= ( n, m, U , S , s (0) , p,  ) .
Given a stochastic Fisher market with savings F (0) a recursive competitive equilibrium (recCE) [15] is a tuple ( X ⇤ ,  ⇤ , p ⇤ )
2 R n ⇥ m ⇥ S + ⇥ R n ⇥ S + ⇥ R m ⇥ S
+ , which consists of an allocation, savings, and price system s.t. 1) the buyers are expected utility maximizing, constrained by their savings and spending constraints, i.e., for all buyers i 2 [ n ] , ( x ⇤ i ,  ⇤ i ) is an optimal policy that, for all states s .
=
( T , b , q ) 2 S , solves the consumption-savings problem , deﬁned by the following Bellman equations: for all s 2 S , ⌫ i ( s ) =
[Equation]
where X ⇤ i ,  ⇤ i denote the allocation and saving systems excluding buyer i ; and 2) the market clears in each state so that unallocated goods in each state are priced at 0, i.e., for all
j 2 [ m ] and s 2 S , p ⇤ j ( s ) > 0 = )
P
[Equation] [Equation]
ij
[Equation]
q j and p
⇤ j
[Equation]
s )  0 = )
P i 2 [ n ] x ⇤ ij ( s )  q j .
By analogy with subgame perfect equilibrium, we can view a recCE as a “submarket” perfect equilibrium, as a recCE corresponds to a competitive equilibrium of the market starting from any state, i.e., buyers are allocated expected discounted cumulative utility-maximizing goods starting at any state , and the aggregate demand for any good is equal to its aggregate supply at all encountered markets.
The following theorem states that any recSE of a stochastic Fisher market with savings is in fact a recCE.
Since recSE are guaranteed to exist, recCE are also guaranteed to exist.
As recSE, and hence recCE, are independent of the initial market state, we denote any stochastic Fisher market with savings by F .
=
( n, m, U , S , p,  ) .
Theorem 4.1.
A stochastic Fisher market with savings F in which U is a set of continuous and homogeneous utility functions and the transition function is continuous in  i has at least one recCE.
Additionally, the recSE that solves the following Bellman equation corresponds to a recCE of F :
[Equation]
Remark 4.2.
This result cannot be obtained by modifying the Lagrangian formulation, i.e., the simultaneous-move game form, of the Eisenberg-Gale program, because the inner maximization problem is convex-non-concave, while recursive Nash equilibria are guaranteed to exist in zero-sum stochastic games only under the assumption of convex-concave payoffs [5].
Experiments
The zero-sum stochastic Stackelberg game associated with a stochastic Fisher market, can, in theory, be solved via value iteration (Algorithm 1) assuming that one can compute the solution to the min-max optimization in line 4 of Algorithm 1.
However, the min-max optimization problem which has to be solved at each step of value iteration is convex-non-concave.
Speciﬁcally, the
P i 2 [ n ] ( b i   i ) log( u i ( x i , t i )) term renders the objective function , convex-non-concave.
This means that in general one is not guaranteed compute a globally optimal solution to the min-max optimization problem in line 4 of Algorithm 1 and instead can only converge to a local min-max solution with known (ﬁrst-order) methods, e.g. nested gradient descent ascent [6].
Unfortunately, if we are not able to compute a globally optimal solution to the generalized min-max optimization, our guarantees for the convergence of value iteration do not apply.
That said, gradient methods have been observed to escape local solutions in many non-convex optimization problems (e.g., see [60, 61]) leading us to investigate how well we can solve the generalized min-max operator in Algorithmusing nested gradient descent ascent [6], and in turn how effectively we can implement value iteration (Algorithm 1) in practice.
)
Xp
 b
P
[Equation]
2 [ m ]
[Equation]
log
[Equation]
⇤
[Equation]
To do so, we computed the recursive Stackelberg equilibria of three different classes of stochastic
Fisher markets with savings.
Speciﬁcally, we created markets with three classes of utility functions, each of which endowed the state-value function with different smoothness properties.
Let t i 2 R m be a vector of parameters, i.e., a type , that describes the utility function of buyer i 2 [ n ] .
We considered the following (standard) utility function classes: 1.
linear : u i ( x i ) =
P j 2 [ m ] t ij x ij ; 2.
Cobb-Douglas : u i ( x i ) =
Q j 2 [ m ] x t ij ij ; and 3.
Leontief :
u i ( x i ) =
min
j 2 [ m ] n x ij t ij o .
We ran two different experiments.
First, we modeled a small stochastic Fisher market with savings without interest rates.
In this setting, buyers’ budgets, which are initialized at the start of the game, persist across states, and are replenished by a constant amount with each state transition.
Thus, the buyers’ budgets from one state to the next are deterministic.
Second, we modeled a larger stochastic Fisher market with savings and probabilistic interest rates.
In this model, although buyers’ savings persist across states, they are nondeterministic, as they increase or decrease based on the random movements of an interest rate with each state transition.
More speciﬁcally, we chose ﬁve different equiprobable interest rates (0.9, 1.0, 1.1, 1.2, and 1.5) to provide buyers with more incentive to save as compared to the model without interest rates.
Since budgets are a part of the state space in stochastic Fisher markets, the state space is continuous; so we attempted to estimate the value function at each iteration of value iteration by running linear regression on a sample of state and associated min-max value pairs, ﬁnding a ﬁt via linear regression (e.g., [62]).
To compute the min-max value of each state that we sampled, i.e., the solution to the optimization problem in line 4 of Algorithm 1, we used nested gradient descent ascent [6] which runs a step of gradient descent on the prices and a loop of gradient ascent on allocations and savings repeatedly (Algorithm 3), where we computed gradients via auto-differentiation using JAX [63] which we observed achieved better numerical stability than analytically derived gradients as can often be the case with autodifferentiation [64].
In both experiments, to check whether the optimal value function was found, we measured the exploitability of the market, meaning the distance between the recCE computed and the actual recCE.
To do so, we checked two conditions: 1) whether each buyer’s expected utility was maximized at the computed allocation and savings, at the prices outputted by the algorithm, and 2) whether the market always cleared.
In both settings, we extracted the greedy policy from the value function computed by value iteration, and unrolled it across time to obtain the greedy actions ( X ( t ) ,  ( t ) , p ( t ) ) at each state s ( t ) .
We then computed the cumulative utility of these allocation and savings, i.e., for all i 2 [ n ] , ˆ u i . =
P
Tt =
t u i ( x ( t ) i ) .
We compared these values to the expected maximum utility u
⇤ i , given the prices and the other buyers’ allocations computed by our algorithm.
We report the normalized distance between these two values, ˆ u i and u ⇤ i , which we call the normalized distance to utility maximization (UM) .
For example, in the case of two buyers, the normalized distance to UM
=
||
(ˆ u 1 , ˆ u 2 )  ( u ? 1 ,u ? 2 ) |||| ( u ⇤ 1 ,u ⇤ 2 )
||.
Finally, we also measured excess demand, which we took as the distance to market clearance (MC) , i.e.,
[Equation]
In the experiment with smaller markets and without interest rates, Figure 1 depicts the average value of the value function across a sample of states as it varies with time, and Table 1 records the exploitability of the recCE found by nested GDA.
For all three class of utility functions, not only do the value functions converge, exploitability is also sufﬁciently minimized, as all the buyer utilities are maximized and the market always clears.
[Table] [Table]
In the experiment with larger markets and probabilistic interest rates (Figure 2, Table 3), in linear and
Cobb-Douglas markets, the value functions converge, and exploitability is sufﬁciently minimized.
In Leontief markets, however, although the value function converges and the markets almost clear, the buyers’ utilities are not fully maximized, since the cumulative utilities they obtained are less than half of their expected maximum utilities.
The difﬁculty in this case likely arises from the fact that the Leontief utility function is not differentiable, so the problem for Leontief markets is neither smooth nor convex-concave, which makes it difﬁcult, if not impossible, for nested GDA to ﬁnd even a stationary point of the objective of the min-max optimization problem in line 4 of Algorithm 1, since gradient ascent on a function is not guaranteed to converge to a stationary point of that function if it is non-convex-non-smooth [65].
Conclusion
In this paper, we proved the existence of recursive Stackelberg equilibria in zero-sum stochastic
Stackelberg games, provided necessary and sufﬁcient conditions for a policy proﬁle to be a recursive
Stackelberg equilibrium, and showed that a Stackelberg equilibrium can be computed in (weakly) polynomial time via value iteration.
Finally, we showed that recursive Stackelberg equilibria coincide with recursive competitive equilibria in stochastic Fisher markets, and we used value iteration together with nested GDA to solve for them.
Future work in this space could try using deep reinforcement learning methods to learn better (i.e., nonlinear) representations of the value functions.
It is also conceivable that deep reinforcement learning would be able to learn better policies, thereby resolving the difﬁculties that our methods face due to non-smoothness and non-concavity have in solving for global solutions of the min-max optimization problem in line 4 of Algorithm 1 .
Acknowledgments and Disclosure of Funding
This work was supported by NSF Grant CMMI-1761546.