Zero-Sum Stochastic Stackelberg Games
Abstract
Min-max optimization problems (i.e., zero-sum games) have been used to model problems in a variety of fields in recent years, from machine learning to economics.
The literature to date has mostly focused on static zero-sum games, assuming independent strategy sets.
In this paper, we study a form of dynamic zero-sum games, called stochastic games, with dependent strategy sets.
Just as zero-sum games with dependent strategy sets can be interpreted as zero-sum
Stackelberg games, stochastic zero-sum games with dependent strategy sets can be interpreted as zero-sum stochastic Stackelberg games.
We prove the existence of an optimal solution in zero-sum stochastic Stackelberg games (i.e., a recursive
Stackelberg equilibrium), provide necessary and sufficient conditions for a solution to be optimal, and show that a recursive Stackelberg equilibrium can be computed in polynomial time via value iteration.
Finally, we show that stochastic Stackelberg games can model the problem of pricing and allocating goods across agents and time; more specifically, we propose a stochastic Stackelberg game whose solutions correspond to a recursive competitive equilibrium in a stochastic Fisher market.
We close with a series of experiments which confirm our theoretical results and show how value iteration performs in practice.
Min-max optimization has paved the way for recent progress in a variety of fields, from machine learning to economics.
These applications require computing solutions to the min-max operator , i.e., solving a constrained min-max optimization problem , also known as zero-sum games (with independent strategy sets): i.e., min x ∈X max y ∈Y f ( x , y ) , where the objective function f :
X ×Y → R is continuous, and the strategy sets X ⊂ R n and Y ⊂
R m are nonempty and compact.
When f is convex-concave, the seminal minimax theorem (Neumann, 1928; Sion et al., 1958) holds, and such a problem can be interpreted as a simultaneous-move zero-sum game between an outer player x and an inner player y , with the solutions ( x ∗ , y ∗ )
∈
X × Y of the min-max operator corresponding to a Nash equilibrium.
More generally, one can consider zero-sum stochastic games (with independent strategy sets)
X ⊂ R n and Y ⊂ R m , nonempty and compact, and a state-dependent payoff function r ( s , x , y ) , for all s ∈ S , where the players seek to optimize their cumulative (discounted) payoffs, in expectation.
When r ( s , x , y ) is bounded, continuous, and concave-convex in ( x , y ) , for all s ∈ S , these games are guaranteed to have a unique solution (Shapley, 1953), 1 while the optimal policies , i.e., the per-state collection of solutions to the minmax operator can be interpreted as a recursive Nash equilibrium of a zero-sum stochastic game, and can be computed in polynomial time by iterative application of the min-max operator (Shapley, 1953).
Zero-sum stochastic games generalize zero-sum games from a single state to multiple states, and have found even more applications in a variety of fields (Jaśkiewicz & Nowak, 2018).
Recently, Goktas & Greenwald (2021) studied the computation of a generalized min-max operator i.e., solving a constrained min-max game with dependent strategy sets : i.e., min x ∈X max y ∈Y : h ( x , y ) ≥ 0 r ( x , y ) where, in addition to the aforementioned assumptions, h :
X × Y → R is continuous.
Goktas & Greenwald have shown that even more problems of interest can be captured by solutions to the generalized min-max operator.
However, a minimax theorem is not guaranteed to hold in this setting; as a result, these problems are best interpreted as zero-sum sequential, i.e., Stackelberg (Von Stackelberg, 1934), games, and their solutions, as
Stackelberg equilibria.
One can likewise consider zero-sum stochastic games with dependent feasible strategy sets X ⊂ R n and Y ⊂ R m , nonempty and compact and a state-dependent payoff function r ( s , x , y ) , as well as a state dependent constraint function g ( s , x , y ) , for all s ∈ S , where the players seek to optimize their cumulative (discounted) payoffs, in expectation, while satisfying the constraint g ( s , x , y ) ≥ 0 at each state s ∈ S .
Such a problem is best interpreted as a zero-sum stochastic Stackelberg game, for which the appropriate solution concept is the recursive Stackelberg equilibrium (recSE) .
Although very little is known about the properties of such problems, they generalize both min-max games with dependent strategy sets and zero-sum stochastic games (with independent strategy sets), and, as we show, have novel applications.
In this paper, we prove the existence of recSE in zero-sum stochastic Stackelberg games, provide necessary and sufficient conditions for a solution to be a recSE, and show that a recSE can be computed in (weakly) polynomial time via value iteration.
We further show that stochastic Stackelberg games can be used to solve problems of pricing and allocating goods across agents and time.
In particular, we introduce stochastic Fisher markets , a stochastic generalization of the Fisher market (Brainard et al., 2000), and a special case of Friesen’s (1979) financial market model, which itself is a stochastic generalization of the Arrow & Debreu model of a competitive economy (1954).
We then prove the existence of recursive competitive equilibrium (Mehra & Prescott, 1977) in this model, under the assumption that consumers have continuous and homogeneous utility functions, by characterizing the competitive equilibria of any stochastic Fisher market as the Stackelberg equilibria of the corresponding stochastic Stackelberg game.
Finally, we use value iteration to solve various stochastic Fisher markets, highlighting the issues that value iteration might face as a consequence of the smoothness properties of the utility functions.
Related Work Algorithms for min-max optimization problems (i.e., zero-sum games) with independent strategy sets have been extensively studied; for a summary see Goktas & Greenwald (2021), Section G. Goktas & Greenwald (2021) and (2022) studied zero-sum games with dependent strategy sets, proposing polynomial-time nested gradient descent ascent (GDA) and simultaneous
GDA algorithms for such problems.
The computation of Stackelberg equilibrium in dynamic Stackelberg games has been studied in several interesting settings, but always with independent strategy sets.
Bensoussan et al.
(2015) study continuous-time stochastic Stackelberg games with continuous action spaces, and prove existence of a solution in their setting.
Vasal (2020) and Vorobeychik & Singh (2012) study discrete-time stochastic Stackelberg games with discrete action and state spaces, and provide algorithms to solve such games.
DeMiguel & Xu (2009) consider a stochastic Stackelberg game-like market model with n leaders and m followers; they prove the existence of a Stackelberg equilibrium in their model, and provide (without theoretical guarantees) algorithms that converge to such an equilibrium in experiments.
Dynamic Stackelberg games (Li & Sethi, 2017) have been applied to a wide range of problems, including security (Vasal, 2020; Vorobeychik & Singh, 2012), insurance provision (Chen & Shen, 2018; Yuan et al., 2021), advertising (He et al., 2008), robust agent design (Rismiller et al., 2020), allocating goods across time intertemporal pricing (Oksendal et al., 2013).
The study of algorithms that compute competitive equilibria in Fisher markets was initiated by
Devanur et al.
(2002), who provided a polynomial-time method for solving these markets assuming linear utilities.
More recently, there have been efforts to study markets in dynamic settings (Cheung et al., 2019; Gao et al., 2021; Goktas & Greenwald, 2021), in which the goal is to either track the changing equilibrium of a changing market, or minimize some regret-like quantity for the market.
The models considered in these earlier works differ from ours as they do not have stochastic structure and do not invoke a dynamic solution concept.
Preliminaries
Notation We use caligraphic uppercase letters to denote sets (e.g., X ), bold uppercase letters to denote matrices (e.g., X ), bold lowercase letters to denote vectors (e.g., p ), bold uppercase letters to denote vector-valued random variables (e.g., Γ ), lowercase letters to denote scalar quantities, (e.g., x ), and uppercase letters to denote scalar-valued random variables (e.g., X ).
We denote the i th row vector of a matrix (e.g., X ) by the corresponding bold lowercase letter with subscript
i (e.g., x i ) .
Similarly, we denote the j th entry of a vector (e.g., p or x i ) by the corresponding Roman lowercase letter with subscript j (e.g., p j or x ij ).
We denote functions by a letter: e.g., f if the function is scalar valued, and f if the function is vector valued.
We denote the vector of ones of size n by 1 n .
We denote the set of integers { 1 , . . . , n } by [ n ] , the set of natural numbers by N , the set of real numbers by R .
We denote the postive and strictly positive elements of a set by a + and ++ subscript respectively, e.g., R + and R ++ .
We denote the orthogonal projection operator onto a set
C by Π C , i.e., Π C ( x ) =
arg min y ∈ C
∥ x − y ∥ 2 .
We denote by ∆ n = { x ∈ R n + | (cid:80) ni
=1 x i = 1 } , and by ∆( A ) the set of probability measures on the set A .
A stochastic Stackelberg game ( S , X , Y , µ (0) , r x , r y , g , p, γ ) is a dynamic two-player sequen- tial, i.e., Stackelberg, game where one player we call the outer-player (resp.
inner-player) moves through a set of states S picking an action to play from their continuous set of actions X ⊂ R n (resp.
Y ⊂ R m ).
Players start the game at an initial state determined by an initial state distribution µ (0) :
S → [0 , 1] s.t. for all states s ∈ S , µ (0) ( s ) ≥ 0 denotes the probability of the game being initialized at state s .
At each state s ∈ S the action x ∈ X chosen by the outer player determines the set of feasible actions { y ∈ Y | g ( s , x , y ) ≥ 0 } that the inner player can in turn play.
Once the outer and inner players make their moves, they receive payoffs r x : s × X × Y
→ R and r y : S ×
X × Y → R , respectively, and the game either ends with probability 1 − γ , where γ ∈ (0 , 1) is called the discount factor , or transitions to a new state s ′ ∈ S , according to a transition probability function p : S × S × X × Y → [0 , 1] s.t. p
( s ′ | s , x , y ) ∈ [0 , 1] denotes the probability of transitioning to state s ′ ∈ S from state s ∈ S
when a strategy profile ( x , y ) ∈
X ×Y is chosen by the players.
In this paper, we focus on stochastic
Stackelberg zero-sum games
G (0) =
( S , X , Y , µ (0) , r, g , p, γ ) , in which the outer player’s loss is the inner player’s gain, i.e., r x =
−
r
y .
We say that a zero-sum stochastic Stackelberg game is convex-concave if, for all s ∈ S , r ( s , x , y ) , g 1 ( s , x , y ) , . . .
, g d ( s , x , y ) are convex-concave in ( x , y ) .
A zero-sum stochastic Stackelberg game reduces to zero-sum stochastic game (Shapley, 1953) if for all state-action tuples ( s , x
, y ) ∈ S ×
X × Y , g ( s , x , y ) ≥ 0 .
As we will show, it suffices to focus on deterministic policies in which a policy for the outer (resp.
inner) player is a mapping from states to actions π x : S → X (resp.
π y : S → Y ).
The outcome of a zero-sum stochastic Stackelberg game G (0) is a policy profile ( π x ,
π y ) ∈
X S × Y S consisting of policies for the outer and inner players, respectively.
An outcome ( π x , π y ) ∈
X S
× Y S is said to be feasible if, for all states s ∈ S , g ( s , π x ( s ) , π y ( s )) ≥ 0 .
For simplicity, we introduce a function G : X S × Y S → R |S|× d such that
G ( π x , π y )
=
( g ( s , π x ( s ) , π y ( s ))) s ∈S , and define feasible outcomes as those ( π x ,
π y ) ∈
X S
× Y S s.t.
G ( π x , π y ) ≥ 0 .
For the rest of this paper, we assume:
Assumption 1.1. 1.
For all states s ∈ S , the functions r ( s , · , · ) , g 1 ( s , · , · ) , . . .
, g d ( s , · , · ) are continuous in ( x , y ) ∈
X × Y with payoffs r bounded, i.e., ∥ r ∥ ∞ ≤ α < ∞ , for some α ∈ R + , and 2 X , Y are non-empty and compact.
Given a zero-sum stochastic Stackelberg game G (0) , the state-value function , v : S×X S ×Y S →
R , and the action-value function , q : S ×
X × Y × X S
× Y S
→ R are respectively defined as: [Equation]
[-E π x , π y S ( t +1) ∼ p ( ·| S ( t ) , X ( t ) , Y ( t ) ) (cid:2)(cid:80) ∞ t =
 (1 − γ ) γ t r ( S ( t ) , X ( t ) , Y (-] t ) )
| S (0) =
s (cid:3) ; q ( s , x , y ; π x , π y ) =
E π x , π y S ( t +1) ∼ p ( ·| S ( t ) , X ( t ) , Y ( t ) )
(cid:2)(cid:80) ∞ t =
(1 − γ ) γ t r ( S ( t ) , X ( t ) , Y ( t ) )
| S (0) = s
X (0) =
x , Y (0) =
y (cid:3) .
For clarity, we write expectations conditional on X ( t ) =
π x ( S ( t ) ) and Y ( t ) =
π y ( S ( t ) ) as
E π x , π y , and denote the state- and action-value functions by v π x π y ( s ) , and q π x π y ( s , x , y ) , respectively.
Additionally, we let V =
[ − α, α ] S be the space of all state-value functions of the form v : S → [ − α, α ] , and we let Q =
[ − α, α ] S×X×Y be the space of all action-value func- tions of the form q : S ×
X × Y
→
[ − α, α ] .
Note that by Assumption 1.1 the range of the state- and action-value functions is [ − α, α ] .
The cumulative payoff function of the game u : X S × Y S
→ R is the total expected loss (resp.
gain) of the outer (resp.
inner) player is then given by u ( π x , π y ) =
E s ∼ µ (0) ( s )
[ v π x π y ( s )] .
Definition 1.2.
A feasible policy profile ( π ∗ x , π ∗ y ) ∈
X S × Y S is said to be a recursive
Stackelberg equilibrium (recSE) of a zero-sum stochastic
Stackelberg game
[Equation]
≤
[Equation]
Remark 1.3.
This is a strong definition of a recSE, since we require the constraints g ( s , π x , π y ) ≥to be satisfied at all states s ∈ S , not only states which are reached with strictly positive probability.
Mathematical Preliminaries
A probability measure q 1 ∈ ∆( S ) convex stochastically dominates (CSD) q 2 ∈ ∆( S ) if (cid:82) S v ( s ) q 1 ( s ) ds ≥ (cid:82) S v ( s ) q
( s ) ds for all continuous, bounded, and convex functions v on S .
A transition function p is termed CSD convex in x if for all λ ∈ (0 , 1) , y ∈ Y and any ( s ′ , x ′ ) , ( s † , x † ) ∈ S × X , with ( s , x ) = λ
( s ′ , x ′ )
+ (1 − λ )( s † , x † ) , we have λp ( · | s ′ , x ′ , y )
+ (1 − λ ) p
( · | s † , x † , y )
CSD p ( · | s , x , y ) .
A transition function p is termed
CSD concave in y if for all λ ∈ (0 , 1) and any ( s ′ , y ′ ) , ( s † , y † ) ∈ S ×
X × Y , with ( s , y ) =
λ ( s ′ , y ′ )
+ (1 − λ )( s † , y † ) , we have p ( · | s , x , y ) CSD λp ( · | s ′ , x , y ′ )
+ (1 − λ ) p
( · | s † , x , y † ) .
A mapping
L : A → B is said to be a contraction mapping (resp.
non-expansion ) w.r.t.
norm ∥·∥ iff for all x , y ∈ A , and for k ∈ [0 , 1) (resp.
k
= 1 ) such that ∥ L ( x ) − L ( y )
∥
≤ k ∥ x − y ∥ .
Properties of Recursive Stackelberg eqilibrium
We first state a necessary condition that the state-value function induced by a policy profile needs to satisfy in order for that policy profile to be a recSE.
It is not directly clear if there exists a policy for any zero-sum stochastic Stackelberg game which satisfies this necessary condition since there is no way to guarantee that such an inequality will hold at all states.
Lemma 2.1. Consider a zero-sum stochastic Stackelberg game G (0) .
A policy profile ( π ∗ x , π ∗ y ) ∈
S X × S Y is a recSE if, for all s ∈ S , we have that: max y ∈Y : g ( s , π ∗ x
( s ) , y ) ≥
q π ∗ x π ∗ y ( s , π ∗ x ( s ) , y ) ≤ q π ∗ x
π ∗ y ( s , π ∗ x ( x ) , π ∗ y ( y ))
≤ min x ∈X max y ∈Y : g ( s , x , y ) ≥
q π ∗ x π ∗ y ( s , x , y ) .
Equivalently, a policy profile ( π ∗ x , π ∗ y ) is a recSE if ( π ∗ x ( s ) , π ∗ y ( s )) is a Stackelberg equilibrium: i.e., a solution to min x ∈X max y ∈Y : g ( s , x , y ) ≥
q π ∗ x π ∗ y ( s , x , y ) at each s ∈ S .
We define an operator C : V → V associated with a zero-sum stochastic Stackelberg game
G (0) whose fixed points satisfy the condition given in Lemma 2.1, and hence correspond to the value function associated with a recSE of G (0) .
We then show that this operator is a contraction mapping, thereby establishing the existence of such a fixed point.
This result generalizes a result first shown by Shapley (1953) for zero-sum stochastic games, i.e., zero-sum stochastic Stackelberg games in which G ( π x , π y ) ≥ 0 , for all ( π x , π y ) ∈
X S × Y S .
Define C : V → V for a stochastic Stackelberg game G (0) as the operator ( Cv ) ( s ) =
min x ∈X max y ∈Y : g ( s , x , y ) ≥ 0 E S ′ ∼ p ( ·| s , x , y )
[ r ( s , x , y ) + γv ( S ′ )] .
Theorem 2.2.
( π ∗ x , π ∗ y ) is a recSE of G (0) of v π x π y iff it induces a value function which is a fixed point of C : i.e., ( π ∗ x , π ∗ y ) is a Stackelberg equilbrium iff, for all s ∈ S , (cid:16)
Cv π ∗ x π ∗ y (cid:17) ( s ) =
v π ∗ x π ∗ y
( s ) .
The following technical lemma is crucial to proving that C is a contraction mapping.
It tells us that the generalized min-max operator is non-expansive; in other words, the generalized min-max operator is 1-Lipschitz w.r.t.
the sup-norm.
Lemma 2.3.
Suppose that f, h :
X × Y →
R , g :
X × Y
(cid:12)(cid:12) min x ∈X max y ∈Y : g ( x {+, y ) ≥ 
+}
f ( x , y ) − min x ∈X max y ∈Y : g ( x , y {+) ≥ 
+}
h ( x , y ) (cid:12)(cid:12)
[-y ) ≥ 
 h [Equation] x-]≤
max ( x , y ) ∈X×Y | f ( x , y ) − h ( x , y ) | .
With the above lemma in hand, we can now prove that C is a contraction mapping.
Theorem 2.4.
Consider the operator C associated with a stochastic Stackelberg game G (0) .
If Assumption 1.1 holds, then C is a contraction mapping w.r.t.
to the sup norm ∥ .
∥ ∞ with constant γ .
Given some initial state-value function v (0) ∈ V , we define the value iteration process as v ( t +1) =
Cv ( t ) , for all t ∈ N + (Algorithm 1).
One way to interpret v ( t ) is as the function that returns the value v ( t ) ( s ) of each state s ∈ S in the t stage zero-sum Stackelberg game starting at the last stage t and continuing until stage 0 , with terminal payoffs given by v (0) .
The following theorem, which is a consequence of Theorems 2.2 and 2.4, not only states the existence of a recSE but also provides us with a means of computing a recSE via value iteration.
Theorem 2.5.
Consider a zero-sum stochastic Stackelberg game G (0) .
If Assumption 1.1 holds, then
G (0) has at least one recSE ( π ∗ x , π ∗ y ) with unique value function v π ∗ x π ∗ y .
Further, v π ∗ x π ∗ y can be computed by iteratively applying C to any initial state-value function v (0) ∈ V : lim t →∞ v ( t ) =
v π ∗ x π ∗ y .
Remark 2.6.
Unlike Shapley’s existence theorem for recursive Nash equilibria in zero-sum stochastic games, the above theorem does not require that the payoff function be convex-concave.
The only conditions needed are continuity of the payoffs and constraints, and bounded payoffs.
This makes the recSE a potentially useful solution concept, even for non-convex-non-concave stochastic games.
Note that the proof of Theorem 2.5 shows that there exists a policy profile which satisfies the conditions of Lemma 2.1.
Since this recSE definition is independent of the initial state distribution, we can infer that the recSE of any zero-sum Stackelberg game G (0) =
( S , X , Y , µ (0) , r, g , p, γ ) is independent of the initial state distribution µ (0) .
Hence, from now on, we denote a zero-sum
Stackelberg game by G .
It seems that a stronger equilibrium concept for zero-sum stochastic Stackelberg games would be one that requires the strategy profile be a recSE of each subgame, aptly called subgame perfect Stackelberg equilibrium .
A feasible policy profile ( π ∗ x , π ∗ y ) ∈
S X × S Y is a sub- game perfect Stackelberg equilibrium of a zero-sum stochastic Stackelberg game G if, for all s ∈ S , t ∈ N , µ ( t +1)
( s ) =
(cid:80) s ′ ∈S p ( s | s ′ , π ∗ x ( s ′ ) , π ∗ y ( s ′ ))
µ ( t ) ( s ′ ) , ( π ∗ x , π ∗ y ) is a recSE of ( S , X , Y , µ ( t ) , r, g , p, γ ) .
By definition, the set of subgame perfect Stackelberg equilibria of a zerosum stochastic Stackelberg game is a subset of the set of recSE of the game.
Since the set of recSE is independent of the initial state distribution µ (0) , the reverse inclusion also holds.
We thus have the following corollary:
Corollary 2.7.
The set of recSE and subgame perfect Stackelberg equilibria coincide in zero-sum stochastic Stackelberg games.
Theorem 2.5 tells us that value iteration converges to the value function associated with a recSE.
Additionally, under Assumption 1.1, recSE is computable in (weakly) polynomial time.
Theorem 2.8.
[Convergence of Value Iteration] Suppose value iteration is run on input G .
If Assumption 1.1 holds, and if we initialize v (0) ( s ) = 0 , for all s ∈ S , then for k ≥ 1 1 − γ log 2 α ϵ (1 − γ ) , we have v ( k ) ( s ) − v π ∗ x π ∗ y ( s ) ≤ ϵ .
Subdifferential
Envelope Theorems and Optimality Conditions for Recursive Stackelberg Eqilibrium
In this section, we derive optimality conditions for recursive Stackelberg equilibria.
In particular, we provide necessary conditions for a policy profile to be a recSE of any zero-sum stochastic Stackelberg game, and we provide sufficient conditions for a policy profile to be a recursive Stackelberg equilibrium of a zero-sum convex-concave stochastic Stackelberg game.
Using these results, we prove in the next section that recursive market equilibrium (Mehra & Prescott, 1977) is an instance of recSE in a large class of stochastic markets.
The Benveniste-Scheinkman theorem characterizes the derivative of the optimal value function associated with a recursive optimization problem w.r.t.
its parameters, when it is differentiable (Benveniste & Scheinkman, 1979).
Our proofs of the necessary and sufficient optimality conditions rely on a novel subdifferential generalization (Theorem C.3, Appendix C) of this theorem, which applies even when the optimal value function is not differentiable.
A consequence of our subdifferential version of the Benveniste-Scheinkman theorem is that we can easily derive the firstorder necessary conditions for a policy profile to be a recSE of any stochastic Stackelberg game G satisfying Assumption 1.1, under standard regularity conditions.
Theorem 3.1.
Consider a zero-sum stochastic Stackelberg game G , where X = { x ∈ R n | q 1 ( x ) ≤ 0 , . .
. , q p ( x ) ≤ 0 } and Y = { y ∈
R m | r 1 ( y ) ≥ 0 , . .
, r l ( y ) ≥ 0 } .
Let L s , x ( y , λ ) =
r ( s , x , y )
+ γ E S ′ ∼ p ( ·| s , x , y )
[ v ( S ′ , x )]
+ (cid:80) dk
=
λ k g
k
( s , x , y ) .
Suppose that Assumption 1.1 holds, and that 1.
for all s ∈
S , y ∈
Y , r ( s , x , y ) , g 1 ( s , x , y ) , . .
g d ( s , x , y ) are concave in x , 2.
∇ x r ( s , x , y ) , ∇ x g 1 ( s , x , y ) , . .
. , ∇ x g d ( s , x , y ) , ∇ y r ( s , x , y ) , ∇ y g
( s , x , y ) , . . .
, ∇ y g d ( s , x , y ) exist, for all s ∈ S , x ∈ X , y ∈ Y , 4.
p ( s ′ | s , x , y ) is continuous CSD convex and differentiable in ( x , y ) , and 5.
Slater’s condition holds, i.e., ∀ s ∈ S , x ∈ X , ∃ (cid:98)
y ∈
Y s.t.
g k ( s , x , (cid:98) y ) > 0 , for all k = 1 , . .
, d and r j ( (cid:98) y ) > 0 , for all j = 1 , . . .
, l , and ∃ x ∈ R n s.t.
q k ( x ) < 0 for all k = 1 . .
, p .
Then, there exists µ ∗ :
S → R p + , λ ∗ : S×X → R d + , and ν ∗ : S × X
→ R l + s.t.
a policy profile ( π ∗ x , π ∗ y ) ∈
X S
× Y S is a recSE of G only if it satisfies the following conditions, for all s ∈ S :
[Equation]
[Equation] Note, that in Theorem 3.1, when one additionally assumes concavity of max y ∈Y :
[Equation]
[Equation]
[Equation] , y ) + [Equation]
[Equation]
·| s , x , y ) [ v ( S ′ )] (cid:9) in x , Equations (74) to (78) be- come necessary and sufficient optimality conditions.
For completeness, the reader can find the necessary and sufficient optimality conditions for convex-concave stochastic Stackelberg games under standard regularity conditions in Theorem C.4 (Appendix C).
The proof follows exactly as that of Theorem 2.2.
Recursive Market Eqilibrium
We now introduce an application of zero-sum stochastic Stackelberg games, which generalizes a well known market model, the Fisher market (Brainard et al., 2000), to a dynamic setting in which buyers not only participate in markets across time, but their wealth persists.
A (static) Fisher market consists of n buyers and m divisible goods (Brainard et al., 2000).
Each buyer i ∈ [ n ] is endowed with a budget b i ∈ B i
⊂ R + and a utility function u i : R m + × T i → R , which is parameterized by a type t i ∈ T i that defines a preference relation over the consumption space R m + .
Each good is characterized by a supply q j ∈ Q j ⊂ R + .
A stochastic Fisher market is a dynamic market in which each state corresponds to a static Fisher market: i.e., each state s ∈ S is characterized by a tuple ( t , b , q ) .
In each state, the buyers choose their allocations X
= ( x 1 , . . .
, x n ) T ∈ R n × m + and the market determines prices, after which the market comes to an end with probability (1 − γ ) , or it moves into a new state with probability p ( s ′ | s , X ) .
4 A stochastic
Fisher market with savings is a stochastic Fisher market in which, at each state, each buyer i , in addition to choosing their allocation, can set aside some savings β
i ∈ R + to spend at some future state, and transitions depend on their savings choices as well: i.e., p ( s ′ | s , X , β ) .
A stochastic Fisher market with savings is denoted by ( S , U , b (0) , p, γ ) .
Given such a market, a recursive competitive equilibrium (recCE) (Mehra & Prescott, 1977) is a tuple ( X ∗ , β ∗ , p ∗ ) ∈
R n × m ×S + × R n ×S + × R m ×S + , which consists of an allocation, savings, and price system s.t. 1) the buyers are expected utility maximizing, constrained by their savings and spending constraints, i.e., for all buyers i ∈
[ n ] , ( x ∗ i , β ∗ i ) is the optimal policy that, for all states ( t , b , q ) ∈ S , solves the Bellman equation
[Equation]
where X ∗− i , β ∗− i denote the allocation and saving systems excluding buyer i , and 2) the market clears in each state so that unallocated goods in each state are priced at 0, i.e., for all j ∈ [ m ] and s ∈
S , p ∗ j ( t , b , q ) > 0 = ⇒ (cid:80) i ∈ [ n ] x ∗ ij ( t , b , q ) = 1 and p ∗ j ( t , b , q ) ≥ 0 = ⇒ (cid:80) i ∈ [ n ] x ∗ ij ( t , b , q ) ≤ 1 .
The following theorem shows that the recSE of a stochastic Fisher market with savings are in fact recursive competitive equilibria.
We note that if one ignores the savings terms, then the recSE are recursive competitive equilibria of the same market without savings.
Theorem 4.1.
A stochastic Fisher market with savings ( S , U , b (0) , p, γ ) in which U is a vector of continuous and homogeneous utility functions has at least one recCE.
Additionally, the recSE ( p ∗ , X ∗ , β ∗ ) that solves the following Bellman equation corresponds to the recCE of ( S , U , b (0) , p, γ ) :
[Equation]
Remark 4.2.
This result could not be obtained by modifying the Lagrangian formulation, i.e., the simultaneous-move game form, of the Eisenberg-Gale program, because the saving problem is a convex-non-concave problem, and existence of recursive Nash equilibrium in deterministic policies requires convex-concavity of payoffs (Jaśkiewicz & Nowak, 2018).
Experiments
In order to better understand the iteration complexity of value iteration, and to understand how it performs in a continuous state space, we computed the recursive Stackelberg equilibria of various stochastic Fisher market with savings.
We created markets with three different classes of utility functions, each of which endowed the state-value function with different smoothness properties.
Let θ i ∈
R m be a vector of parameters that describes the utility function of buyer i ∈ [ n ] .
We considered the following (standard) utility function classes: 1.
linear: u i ( x i ) =
(cid:80) j ∈ [ m ] θ ij x ij ;
Cobb-Douglas: u i ( x i ) = (cid:81) j ∈ [ m ] x θ ij ij ; and 3.
Leontief:
u i ( x i ) =
min j ∈ [ m ] (cid:110) x ij θ ij (cid:111) .
Since the state space is continuous, the value function is also continuous in stochastic Fisher markets.
As a result, we used fitted value iteration, finding a fit via linear regression (e.g., Boyan &
Moore (1994)).
To solve the generalized min-max operator at each step of value iteration, we used two methods: 1.
nested gradient descent ascent (GDA) (Goktas & Greenwald (2021); Algorithm 2), which is not guaranteed to converge to a global optimum since the zero-sum Stackelberg game for stochastic Fisher markets is convex-non-concave; and 2.
max-oracle gradient descent (Goktas & Greenwald (2021); Algorithm 3), where we used simulated annealing (Bertsimas & Tsitsiklis, 1993), a metaheuristic which aims to find a global optimum, as the max-oracle.
Although simulated annealing is not guaranteed to converge to a global optimum, we observed that it outperformed nested GDA, more often finding a global maximum for the inner player.
To check whether the value function computed was optimal, we measured the exploitability of the market, meaning the distance between the recursive competitive equilibrium computed and the actual competitive equilibrium.
To do so required that we check two conditions: 1) if each buyer’s expected utility is maximized at the computed allocation and saving system at the price system output by the algorithm, and 2) if the market always clears.
For both settings, given the value the function computed by value iteration, we extracted the greedy policy and unrolled it across time to obtain the greedy actions ( X ( t ) , β ( t ) , p ( t ) ) at each state s ( t ) .
We then computed the cumulative utility of the allocation and saving systems computed by the algorithms, i.e., for all i ∈ [ n ] , (cid:80) Tt =
γ t u i ( x ( t ) i ) , and compared this value to the expected maximum utility u ∗ i , obtained by solving the consumption/saving problem for individual buyers given the price systems computed by our algorithms.
We report the normalized distance between these two values: e.g., in the case of two buyers, we report || ( u 1 ,u 2 ) − ( u ⋆ 1 ,u ⋆ 2 )
||
|| ( u ∗ 1 ,u ∗ 2 )
|| , Finally, we measured the excess demand, which we took as the distance to market clearance, i.e., 1 T (cid:80)
Tt =1 ||
(cid:80) i ∈ [ n ] x ( t ) i − q ( t ) || .
The exploitability of the recursive competitive equilibrium computed by both nested GDA and max-oracle gradient descent in all market types is shown in Figure 2, while Figure 1 depicts the average value of the value function across all states as it varies with time.
We observe that value iteration converges for both nested GDA and max-oracle gradient descent in Cobb-Douglas markets.
Max-oracle gradient descent achieves a higher value, because the oracle does a better job at finding a global maximum for the inner player.
In linear markets, only the max-oracle gradient descent algorithm seems to converge.
This is most likely due to the non-differentiability of the payoffs in linear Fisher markets, making it difficult for nested GDA to find a global optimum.
In
Leontief markets, neither algorithm seems to affect the value function very much, which might lead one to conclude that value iteration converges.
In terms of exploitability, both algorithms seem to perform more or less as they do in the linear markets, which suggests that the initial value function is close to the optimal value function.
Surprisingly, even though value iteration converges in both settings for Cobb-Douglas markets, the allocation and savings computed seem to be far from optimal.
This is most likely due to the fact that we used linear regression to approximate the value function at each state, even though the value function associated with each individual buyer’s consumption/saving problem, and likewise the market’s value function, is strictly concave in Cobb-Douglas markets.
We note that the buyers’ value functions are linear in linear and Leontief markets, and hence compatible with our fitted linear regression, which might explain the greater exploitability of Cobb-Douglas markets even though value iteration seems to converge.
Conclusion
In this paper we proved the existence of recursive Stackelberg equilibria in zero-sum stochastic Stackelberg games, provided necessary and sufficient conditions for a policy profile to be a recursive Stackelberg equilibrium and showed that a Stackelberg equilibrium can be computed Presented at the Gamification in (weakly) polynomial time via value iteration.
Further, we provided a characterization of recursive competitive equilibria in stochastic Fisher markets with savings via zero-sum stochastic
Stackelberg games.
Finally, we used value iteration to solve for recursive competitive equilibria in stochastic Fisher markets.
Future work seeking to improve our solution methods for stochastic Fisher markets should aim to replace the linear regression step in our fitted value iteration method with concave regression, since the value function is guaranteed to be concave in budgets for stochastic Fisher markets.
Additionally, it is conceivable that policy-network-based deep reinforcement learning methods may be able to resolve the difficulties that our method has in solving for global optima in stochastic Fisher markets.
A Pseudo-Code for Algorithms
Inputs: S , X , Y , r, g , p, γ, T
Outputs: v ( T ) 1: Initialize v (0) arbitrarily, e.g. v (0) =2: for t = 1 , . . .
, T do 3: for s ∈ S do 4: v ( t +1)
( s ) =
min x ∈X max y ∈Y : g ( s , x , y ) ≥
E S ′ ∼ p ( ·| s , x , y ) (cid:104) r
( s , x , y )
+ γv ( t ) ( S ′ )
(cid:105) 5: end for 6: end for 7: return v ( T )
Inputs: v, b , η x , η y , T p , T X
Output: ( p ( t ) , X ( t ) )
Tt = 1: for t = 1 , . .
, T p do 2: 3: for s = 1 , . .
, T X do 4:
For all i ∈ [ n ] , 5: x ( s ) i =
Π { ( x i ,β i ) ∈
R m + × R + : x i p ( t − 1) + β i ≤ b i }   x ( s − 1) i + η X  
b i − β ( s − 1)
i u ( t ) i (cid:16) x
( t ) i (cid:17) ∇ x i u ( t ) i (cid:16) x
( t ) i (cid:17) 6: β ( s ) i
= Π { ( x i ,β i ) ∈
R m + × R + : x i p ( t − 1) + β i ≤ b i } (cid:18) β ( s − 1) i + η X (cid:18) − log( u i ( x i ( s − 1) ))
+ γ ∂v ( b ) ∂b i (cid:19)(cid:19) 7: end for 8: p ( t )
=
Π R m + (cid:16) p ( t − 1) − η p ( 1 − (cid:80) i ∈ [ n ] x i ) (cid:17) 9: end for 10: return ( p ( t ) , X ( t ) )
Tt =
Inputs: v, b , η x , η y , T p , T X
Output: ( p ( t ) , X ( t ) )
Tt = 1: for t = 1 , . .
, T p do 2: q ( p , X , β ) =
(cid:80) [Equation] q [Equation]
p [Equation] [Equation] [Equation]
[Equation] γ E ( t ′ [Equation]
[Equation] q ′ ) ∼ p ( ·| t , b , q , X , β ) [ v ( t ′ , b ′ + β , q ′ )] 3:
For all i ∈ [ n ] , x ( t ) i , β ( t ) i =
Simulated _
Annealing( q ) 4: x ( t )
[Equation]
, β ( t )
[Equation]
(cid:104)(cid:16) x ( t
[Equation]
( t
[Equation]
(cid:17)(cid:105) 5: end for 6: p ( t ) =
Π R m + (cid:16) p ( t − 1) + η p ( 1 − (cid:80) i ∈ [ n ] x i ) (cid:17) 7: return ( p ( t ) , X ( t ) )
Tt =
1: Initialize v (0) arbitrarily, e.g. v (0) =2: for k = 1 , . .
, T v do 3: for s ∈ S do 4:
[Equation]
i ∈ [ n ] ( b [Equation] − [Equation]
) log( u
[Equation] ( x i [Equation] + γv
[Equation]
( β ) 5: end for 6: end for
B Omitted Results and Proofs Section
We first note the following fundamental relationship between the state-value and action-value functions which is an analog of Bellman’s Theorem (Bellman, 1952) and which follows from their definitions:
Theorem B.1.
Given a stochastic min-max Stackelberg game ( S , X , Y , µ (0) , r, g , p, γ ) , for all v ∈
V , q ∈ Q , π x ∈ X S , and π y ∈ Y S , v = v π x π y and q = q π x π y iff:
[Equation]
[Equation]
Proof of Theorem B.1.
By the definition of the state value function we have v π x π y i
=
q i ( s , π x ( s ) , π y ( s )) , hence by Equation (7) we must have that v i =
v π x π y i .
Additionally, by
Equation (8) and the definition of the action-value functions this also implies that q i ( s , x , y ) =
q π x π y i
( s , x , y )
Lemma B.2.
Suppose that there exists ( π ∗ x ,
π ∗ y ) ∈
S X × S Y such that for all states s ∈ S :
[Equation]
Then, ( π ∗ x , π ∗ y ) is a recSE.
Lemma B
[Equation]
Suppose that there exists ( π ∗ x ,
π ∗ y ) ∈
S X × S Y such that for all states s ∈ S :
[Equation]
Then, ( π ∗ x , π ∗ y ) is a recSE.
Proof of Lemma B.2.
We prove the right hand side inequality first:
[Equation]
Since ( π ∗ x , π ∗ y ) is well-defined, it must mean that the same policy minimizes (resp.
maximizes) the value of each state for the x player ( y -player), hence the min-max operator can be pulled out of the sum, giving us:
[Equation]
Similarly, manipulating the the left hand side inequality , we obtain:
[Equation]
We will show, there always exists a policy profile which satisfies the conditions of Lemma B.2.
To show the existence of such a policy profile, we first re-state the condition given in Lemma B.2, in terms of action value functions.
Lemma 2.1. Consider a zero-sum stochastic Stackelberg game G (0) .
A policy profile ( π ∗ x , π ∗ y ) ∈
S X × S Y is a recSE if, for all s ∈ S , we have that: max y ∈Y : g ( s , π ∗ x
( s ) , y ) ≥
q π ∗ x π ∗ y ( s , π ∗ x ( s ) , y ) ≤ q π [Equation] x
[Equation]
( s , π [Equation] ( x ) , π ∗ y ( y ))
≤ min x ∈X max y ∈Y : g ( s , x , y ) ≥
q π ∗ x π ∗ y ( s , x , y ) .
Equivalently, a policy profile ( π ∗ x , π ∗ y ) is a recSE if ( π ∗ x ( s ) , π ∗ y ( s )) is a Stackelberg equilibrium: i.e., a solution to min x
[Equation]
max y
[Equation]
s
[Equation]
x
[Equation]
q π
[Equation]
x π ∗ y ( s , x , y ) at each s ∈ S .
Proof of Lemma 2.1.
[Equation]
By Lemma B.2, we must then have that ( π ∗ x , π ∗ y ) is a recursive Stackelberg equilibrium of ( S , X , Y , µ (0) , r, g , p, γ ) .
Theorem 2.2.
( π ∗ x , π ∗ y ) is a recSE of G (0) of v π x π y iff it induces a value function which is a fixed point of C : i.e., ( π ∗ x , π ∗ y ) is a Stackelberg equilbrium iff, for all s ∈ S , (cid:16)
Cv π ∗ x π ∗ y (cid:17) ( s ) =
v π ∗ x π ∗ y
( s ) .
Proof of Theorem 2.2.
(recursive Stackelberg equilibrium =
⇒ Fixed Point):
Suppose that ( π ∗ x , π ∗ y ) is a recursive Stackelberg equilibrium of ( S , X , Y , µ (0) , r, g , p, γ ) , then:
[Equation]
(Fixed Point = ⇒ recursive Stackelberg equilibrium)
Suppose that a value function v π ∗ x π ∗ y which is induced by a policy profile ( π ∗ x , π ∗ y ) is a fixed point of C , we then have for all states s ∈ S :
[Equation]
Hence, by Lemma 2.1, ( π ∗ x , π ∗ y ) is recursive Stackelberg equilibrium.
Lemma 2.3.
Suppose that f, h :
X × Y →
R , g :
X × Y → R d are continuous functions, and
X , Y are compact sets, we then have: [Equation] min [-x ∈X max y ∈Y : g ( x , y ) ≥ 
-][Equation]
max
[Equation] ≤ max [-y ∈Y : g ( x , y ) ≥ 
-][Equation] −
[Equation] y [Equation] | .
Proof of Lemma 2.3.
Let ( x ∗ , y ∗ ) be a
Stackelberg equilibrium of min x ∈X max y ∈Y : g ( x , y ) ≥
f ( x , y ) , and ( x ′ , y ′ ) be a
Stackelberg equilibrium of min x ∈X max y ∈Y : g ( x , y ) ≥
h ( x , y ) .
Additionally, let ¯ y ∈ arg max y ∈Y : g ( x ′ , y ) ≥
f ( x ′ , y ) , then by the the definition of a Stackelberg equilibrium, we have f ( x ∗ , y ∗ )
=
min
y ∈Y max x ∈X : g ( x , y ) ≥ 0 ≤ max y ∈Y : g ( x ′ , y ) ≥ 0 f ( x ′ , y )
=
f
( x ′ , ¯ y ) , and h ( x ′ , y ′ )
= max y ∈Y : g ( x ′ , y ) ≥
h ( x ′ , y ) ≥ h
( x ′ , ¯ y ) .
Suppose that min x ∈X max y ∈Y : g ( x , y ) ≥
f ( x , y ) ≥ min x ∈X max y ∈Y : g ( x , y ) ≥ 0 , h ( x , y ) this gives us: (cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
The opposite case follows similarly by symmetry.
Theorem 2.4.
Consider the operator C associated with a stochastic Stackelberg game G (0) .
If Assumption 1.1 holds, then C is a contraction mapping w.r
[Equation]
t. to the sup norm ∥ .
∥ ∞ with constant γ .
Proof of Theorem 2.4.
We will show that C is a contraction mapping, which then by Banach fixed point theorem establish the result.
Let v, v ′ ∈ V be any two state value functions and q
[Equation]
q ′ ∈ Q be the respective associated action-value functions.
We then have: ∥
[Equation] [Equation]
[Equation]
[Equation] [Equation]
≤ max
[Equation]
max
[Equation]
≤ max
[Equation]
≤ γ max s
[Equation]
Since γ ∈ (0 , 1) , C is a contraction mapping.
Theorem 2.5.
Consider a zero-sum stochastic Stackelberg game G (0) .
If Assumption 1.1 holds, then
G (0) has at least one recSE ( π ∗ x , π ∗ y ) with unique value function v π ∗ x π ∗ y .
Further, v π ∗ x π ∗ y can be computed by iteratively applying C to any initial state-value function v (0) ∈ V : lim t →∞ v ( t ) =
v π ∗ x π ∗ y .
Proof of Theorem 2.5.
By combining Theorem 2.4 and the Banach fixed point theorem Banach (1922), we obtain that a fixed point of C exists.
Hence, by Theorem 2.2, a recursive Stackelberg equilibrium of ( S , X , Y , µ (0) , r, g , p, γ ) exists and the value function induced by all
recursive
Stackelberg equilibria is the same, i.e., the optimal value function is unique.
Additionally, by the second part of the Banach fixed point theorem, we must then also have lim t →∞ v ( t ) → v π ∗ x
π ∗ y .
For any q ∈
Q , we define a greedy policy profile with respect to q as a policy profile ( π q x , π q y ) such that π q x ∈ arg min x ∈X max y ∈Y : g ( s , x , y ) ≥
q ( s , x , y ) and π q y ∈ arg max y ∈Y : g ( s , π q x ( x ) , y ) ≥
q
( s , π q x ( x ) , y ) .
The following lemma provides a progress bound for each iteration of value iteration which is expressed in terms of the value function associated with the greedy policy profile.
Lemma B.3.
Let ( π ∗ x , π ∗ y ) be the recSEof a zero-sum Stochastic Stackelberg game ( S , X , Y , µ (0) , r, g , p, γ ) .
For any q ∈ Q , let π q x , π q y denote the greedy policy with respect to q of x -player and y -player respectively.
Then, the following bound holds:
[Equation]
Using the above progress bound, we can then obtain a convergence rate for value iteration under
Assumption 1.1.
Proof of Lemma B.3.
For any state s , we have:
[Equation]
where the last line was obtained by the definition of a recursive Stackelberg equilibrium which ensures that q ( s , π q x ( s ) , π q
[Equation]
( s )
[Equation]
) ≥
q ( s , π ∗ x ( s ) , y ) .
Re-organizing terms, we have:
[Equation]
Taking the minimum of both sides over states, and re-collecting terms, we obtain:
[Equation]
Theorem 2.8.
[Convergence of Value Iteration] Suppose value iteration is run on input G .
If Assumption 1.1 holds, and if we initialize v (0) ( s ) = 0 , for all s ∈ S , then for k ≥ 11 − γ log 2 α ϵ (1 − γ ) , we have v ( k ) ( s ) − v π ∗ x π ∗ y ( s ) ≤ ϵ .
Proof of Theorem 2.8.
First note that by Assumption 1.1, we have that (cid:13)(cid:13)(cid:13) v π ∗ x
π ∗ y (cid:13)(cid:13)(cid:13) ∞ ≤ α .
Applying the operator C repeatedly and using the fact that v π ∗ x π ∗ y =
Cv π ∗ x π ∗ y from Theorem 2.2, we obtain ∥ v ( k ) − v π ∗ x π ∗ y ∥ ∞
=
∥ ( C ) k v [Equation] 0) − ( C ) k v ∗ ∥ ∞
[Equation]
≤ γ k α
Since 1 − x ≤ e − x for any x ∈ R
[Equation]
we have
[Equation]
Combining this bound with Equation (39) for all q ∈ Q :
[Equation]
Thus it suffices to solve for k such that
[Equation]
which concludes the proof.
C Omitted Results and Proofs Section
Our characterization of the subdifferential of the value function associated with a Stackelberg equilibrium w.r.t.
its parameters relies on a slightly generalized version of the subdifferential envelope theorem (Theorem C.1, Appendix C) of Goktas & Greenwald (2021), which characterizes the set of subdifferentials of parametrized constrained optimization problems, i.e., the set of subgradients w.r.t.
x of f ∗ ( x ) = max y ∈Y : h ( x , y ) ≥
f ( x , y ) .
In particular, we note that Goktas & Greenwald’s proof goes through even without assuming the concavity of f ( x , y ) , h 1 ( x , y ) , . . .
, h d ( x , y ) in y , for all x ∈ X .
Theorem C.
(Subdifferential Envelope Theorem) .
Consider the function f ∗ ( x ) = max y ∈Y : h ( x , y ) ≥
f ( x , y ) where f : X × Y → R , and h : X × Y → R d .
Let
Y ∗ ( x ) = arg max y ∈Y : h ( x , y ) ≥ 0 f ( x , y ) .
Suppose that 1.
f ( x , y ) , h 1 ( x , y ) , . . .
, h d ( x , y ) are continuous in ( x , y ) and convex in x ; 2.
∇ x f, ∇ x h 1 , . .
. , ∇ x h d exist for all x ∈ X , y ∈ Y ; 3.
Y is non-empty and compact, and 4.
(Slater’s condition) ∀ x ∈ X , ∃ (cid:98)
y ∈
Y s.t.
g k ( x , (cid:98) y ) > 0 , for all k = 1 , . . .
, d .
Then, f ∗ is subdifferentiable and at any point (cid:98) x ∈ X , ∂ x f ∗ ( (cid:98) x )
= conv   (cid:91)
[Equation]
where conv is the convex hull operator and λ ∗ ( (cid:98) x , y ∗ ( (cid:98) x ))
= ( λ ∗ 1 ( (cid:98) x , y ∗ ( (cid:98) x )) , . .
, λ ∗ d ( (cid:98) x , y ∗ ( (cid:98) x )))
T ∈ Λ ∗ ( (cid:98) x , y ∗ ( (cid:98) x )) are the optimal KKT multipliers associated with y ∗ ( (cid:98) x ) ∈ Y ∗ ( (cid:98) x ) .
We note the following known lemma which provides the necessary conditions for the Bellman equation associated with a recursive optimization problem to be continuous and convex in its parameters.
Lemma C.2.
Consider the Bellman equation associated with a parametric recursive stochastic optimization problem, where r : S × X × Y → R , with state space S and parameter set X , and γ ∈ (0 , 1) :
[Equation]
Suppose that Assumption 1.1 holds, and that 1.
for all s
[Equation]
∈
Y , r ( s , x , y ) , g 1 ( s , x , y ) , . .
g d ( s , x , y ) are concave in x , 2.
p ( s ′ | s , x , y ) is continuous (Feller property) and
CSD convex in x .
Then, v is continuous and convex in x .
Theorem C.3.
[Subdifferential Benveniste-Scheinkman Theorem] Consider the Bellman equation associated with a recursive stochastic optimization problem where r : S × X × Y → R , with state space S and parameter set X , and γ ∈ (0 , 1) :
[Equation]
Suppose that
Assumption 1. holds, and that 1.
for all s ∈
[Equation]
, x , y ) are concave in x , 2.
[Equation] [Equation]
s , x , y
[Equation]
p
[Equation]
s
[Equation]
| s , x , y ) exist for all s , s ′ ∈ S , x ∈ X , y ∈ Y , 3.
p ( s ′ | s , x , y ) is continuous CSD convex, and differentiable in x , and 4.
Slater’s condition holds for the optimization problem, i.e., ∀ x ∈ X , s ∈ S , ∃ (cid:98)
y ∈
Y s.t.
g k ( s , x , (cid:98) y ) > 0 , for all k = 1 , . .
, d .
Let Y ∗ ( s , x ) = max y
[Equation]
[Equation] [Equation]
( s , x , y ) + γ E s ′ ∼ p [Equation] ·| s , x , y )
[ v ( s ′ , x )] (cid:9) , then v is subdifferentiable and ∂ x v ( s , ˆ x ) = conv
[Equation]
Suppose additionally [Equation] that for all s , s ′ ∈ S , x ∈ X , y ∗ ( s , x ) ∈ Y ∗ ( s , x ) ∇ x p ( s ′ | s , x , y ∗ ( s , x )) > 0 , then ∂ x v ( s , ˆ x ) = conv
[Equation]
57) where conv is the convex hull operator and
[Equation]
=
[Equation]
) T ∈ Λ ∗ ( s , (cid:98) x , y ∗ ( s , (cid:98) x )) are the optimal KKT multipliers associated with y ∗ ( s , (cid:98) x ) ∈ Y ∗ ( s , (cid:98) x ) .
Proof of Theorem C.3.
First, note that by Lemma C.2, v is continuous and convex in x , hence it is subdifferentiable.
From Theorem C.1, we then obtain the first part of the theorem:
[Equation]
By the Leibniz integral rule Flanders (1973), the gradient of the expectation can instead be expressed as an expectation of the gradient under continuity of the function whose expectation is taken, in this case v .
In particular, if for all s , s ′ ∈ S , x ∈ X , y ∗ ( s , x ) ∈ Y ∗ ( s , x ) ∇ x p ( s ′ | s , x , y ∗ ( s , x )) >
we have:
[Equation]
Leibniz Integral Rule
[Equation]
This gives us ∂ x v ( s , ˆ x ) = conv
[Equation]
Note that in the special case that the probability transition function is representing a deterministic recursive parametrized optimization problem,
[Equation]
x [Equation] max y ∈Y : g ( s , x , y ) ≥ 0 { r ( s , x , y ) + γ [ v ( τ ( s , x , y ) , x )] } i.e., p ( s ′ | s , x , y ) ∈ { 0 , 1 } for all s , s ′ ∈ S , x ∈ X , y ∈ Y , and τ : S ×
X × Y → S is such that τ ( s , x , y )
=
s ′
iff p ( s ′ | s , x , y )
= 1 , the CSD convexity assumption reduces to the linearity of the deterministic state transition function τ (Proposition 1 of Atakan (2003)).
In this case, the subdifferential of the
Bellman equation reduces to
[Equation]
Theorem 3.1.
Consider a zero-sum stochastic Stackelberg game G , where X = { x ∈ R n | q 1 ( x ) ≤ 0 , . .
. , q p ( x ) ≤ 0 } and Y = { y ∈
R m | r 1 ( y ) ≥ 0 , . .
, r l ( y ) ≥ 0 } .
Let L s , x ( y , λ ) =
r ( s , x , y )
+ γ E S ′ ∼ p ( ·| s , x , y )
[ v ( S ′ , x )]
+ (cid:80) dk
=
λ k g
k
( s , x , y ) .
Suppose that Assumption 1.1 holds, and that 1.
for all s ∈
S , y ∈
Y , r ( s , x , y ) , g 1 ( s , x , y ) , . .
g d ( s , x , y ) are concave in x , 2.
∇ x r ( s , x , y ) , ∇ x g 1 ( s , x , y ) , . .
. , ∇ x g d ( s , x , y ) , ∇ y r ( s , x , y ) , ∇ y g
( s , x , y ) , . . .
, ∇ y g d ( s , x , y ) exist, for all s ∈ S , x ∈ X , y ∈ Y , 4.
p ( s ′ | s , x , y ) is continuous CSD convex and differentiable in ( x , y ) , and 5.
Slater’s condition holds, i.e., ∀ s ∈ S , x ∈ X , ∃ (cid:98)
y ∈
Y s.t.
g k ( s , x , (cid:98) y ) > 0 , for all k = 1 , . .
, d and r j ( (cid:98) y ) > 0 , for all j = 1 , . . .
, l , and ∃ x ∈ R n s.t.
q k ( x ) < 0 for all k = 1 . .
, p .
Then, there exists µ ∗ :
S → R p + , λ ∗ : S×X → R d + , and ν ∗ : S × X
→ R l + s.t.
a policy profile ( π ∗ x , π ∗ y ) ∈
X S
× Y S is a recSE of G only if it satisfies the following conditions, for all s ∈ S :
[Equation]
ν
[Equation]
s
[Equation]
π
[Equation] [Equation] [Equation]
[Equation]
Proof of Theorem 3.1.
By Theorem 2.2 and Theorem 2.4 we know that ( π ∗ x , π ∗ y ) is a recursive Stack- elberg equilibrium iff
[Equation]
Note that for any policy profile ( π ∗ x , π ∗ y ) that satisfies v
[Equation]
s ) by definition of
C we have that ( π ∗ x ( s ) ,
π ∗ y ( s )) is a Stackelberg equilibrium of
[Equation]
for all s ∈ S .
Fix a state s ∈ S , under the assumptions of the theorem, the conditions of Theorem C.3 are satisfied and u ∗ ( s , x ) = max y ∈Y : g ( s , x , y ) ≥ 0 (cid:8) r ( s , x , y )
+ γ E S ′ ∼ p ( ·| s , x , y )
[ v ( S ′ )]
(cid:9) is subdiffer- entiable in x .
Since u ∗ ( s , x ) is convex in x , and Slater’s condition are satisfied by the assumptions of the theorem, the necesssary and sufficient conditions for π ∗ x ( s ) to be an optimal solution to min x ∈X u ∗ ( s , x ) are given by the KKT conditions Kuhn & Tucker (1951) for min x ∈X u ∗ ( s , x ) .
Note that we can state the first order KKT conditions explicitly thanks to the subdifferential Benveniste-Scheinkman theorem (Theorem C.3).
That is, π ∗ x ( s ) is an optimal solution to min x ∈X u ∗ ( s , x ) if there exists µ ∗ ( s ) ∈ R p + such that:
[Equation]
= µ ∗ k ( s ) ∇ x q k
( π ∗ x ( s )) =(71) µ ∗ k ( s ) q k
( π ∗ x ( s ))
=
∀ k ∈ [ p ] (72) q k
( π ∗ x ( s ))
≤∀ k ∈
[ p ] (73) where y ∗ ( s , π ∗ x
( s )) ∈ arg max y ∈Y : g ( s , π ∗ x ( s ) , y ) ≥ 0 (cid:8) r ( s , π ∗ x ( s ) , y )
+ γ E S ′ ∼ p ( ·| s , π ∗ x ( s ) , y )
[ v ( S ′ )] (cid:9) and λ ∗ ( s , π ∗ x ( s ) , y ∗ ( s , π ∗ x ( s )))
= ( λ ∗ 1 ( s , π ∗ x ( s ) , y ∗ ( s , π ∗ x ( s ))) , . .
, λ ∗ d
( s , π ∗ x ( s ) , y ∗ ( s , π ∗ x ( s ))))
T ∈ Λ ∗ ( s , π ∗ x ( s ) , y ∗ ( s , π ∗ x ( s ))) are the optimal KKT multipliers associated with y ∗ ( s , π ∗ x ( s )) ∈
Y ∗ ( s , π ∗ x ( s )) which are guaranteed to exist since Slater’s condition is satisfied for max y ∈Y : g ( s , x , y ) ≥ 0 (cid:8) r
( s , x , y )
+ γ E S ′ ∼ p ( ·| s , x , y )
[ v ( S ′ )] (cid:9) .
Similarly, fix a state s ∈ S and action for the outer player x ∈ X , since Slater’s condition is satisfied for max y ∈Y : g ( s , x , y ) ≥ 0 (cid:8) r
( s , x , y )
+ γ E S ′ ∼ p ( ·| s , x , y )
[ v ( S ′ )] (cid:9) , the necessary conditions for π ∗ y ( s ) to be a Stackelberg equilibrium strategy for the inner player at state s are given by the KKT conditions for max y ∈Y : [Equation] s
[Equation]
x [Equation] y )
[Equation]
0 (cid:8)
[Equation]
s , x , y ) + γ E S ′ ∼ p ( ·| s , x , y )
[ v ( S ′ )] (cid:9) .
That is, there exists λ ∗ ( s , x ) ∈ R d
+ and ν ∗ ( s , x ) ∈ R l + such that:
[Equation]
Combining the necessary and sufficient conditions in Equations (71) to (73) with the necessary conditions in Equations (74) to (78), we obtain the necessary conditions for ( π ∗ x , π ∗ y ) to be a recursive
Stackelberg equilibrium.
Theorem C.4 (Recursive Stackelberg Equilibrium Necessary and Sufficient Optimality
Conditions) .
Consider a zero-sum Stochastic Stackelberg game ( S , X , Y , r, g , p, γ ) , where
X = { x ∈ R n | q 1 ( x ) ≤ 0 , . .
. , q p ( x ) ≤ 0 } and Y = { y ∈ R m | r 1 ( y ) ≥ 0 , . . .
, r l ( y ) ≥ 0 } .
Let L s , x ( y , λ ) =
r ( s , x , y )
+ γ E S ′ ∼ p ( ·| s , x , y )
[ v ( S ′ , x )]
+ (cid:80) dk
=
λ k g
k
( s , x , y ) .
Suppose that Assumption s ∈ S y ∈ Y r ( s x y ) g ( s x , y ) , . .
g d ( s , x , y ) are convex-concave in ( x , y ) , 2. [Equation] [Equation] [Equation]
[Equation]
[Equation] . , ∇ x g [Equation] ( s , x , y ) , ∇ y r ( s , x , y ) , ∇ y g
( s , x , y ) , . . .
, ∇ y g d ( s , x , y ) exist for all s ∈
S , x ∈
X , y ∈
Y , 4.
p ( s ′ | s , x , y ) is continuous, CSD convex in x , CSD concave in y , and differentiable in ( x , y ) , and 5.
Slater’s condition holds, i.e., ∀ s ∈ S , x ∈ X , ∃ (cid:98)
y ∈
Y s.t.
g k ( s , x , (cid:98) y ) > 0 , for all k = 1 , . .
, d and r j ( (cid:98) y ) > 0 for all j = 1 , . . .
, l , and ∃ x ∈ R n s.t.
q k ( x ) < 0 for all k = 1 . .
, p .
Then, there exists µ ∗ :
S → R p + , λ ∗ : S × X
→ R d + , and ν ∗ : S × X
→ R l + such that a policy profile ( π ∗ x , π ∗ y ) ∈
X S
× Y S is a recursive Stackelberg equilibrium of ( S , X , Y , r, g , p, γ ) only if it satisfies the following conditions for all s ∈ S :
[Equation]
[Equation] [Equation]
∀
[Equation]
[Equation] [Equation] ∀ k
[Equation]
[ p ] [Equation] [Equation] [ d ]
[Equation] [Equation]
[Table]
[Equation]
[Equation] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Equation] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
∀
[Equation]
[Equation]
D Omitted Results and Proofs Section
Before, we introduce the stochastic Stackelberg game whose recursive Stackelberg equilibria correspond to recursive competitive equilibria of an associate stochastic Fisher market, we introduce the following technical lemma, which provides the necessary and sufficient conditions for an allocation and saving system of a buyer to be expected utility maximizing.
Lemma D.1.
For any price system p ∈ R S× m + , a tuple ( x ∗ i , β ∗ i )
∈ R S× n × m + × R n × m
+ consisting of an allocation system and saving system for a buyer i , given by a continuous, and homogeneous utility function u
i : R m + × T → R representing a locally non-satiated preference is expected utility maximizing constrained by the saving and spending constrained, i.e., the first condition of a competitive equilibrium is satisfied only we have for all states s ∈ S , x ∗ i ( t , b , q ) · p ( t , b , q ) + β ∗ i ( t , b , q ) ≤ b i , and, x ∗ ij ( s ) > 0 = ⇒
[Equation]
If additionally, u i is concave, then the above condition become also sufficient.
Proof of Lemma D.1.
Let
[Equation]
γ
[Equation]
[Equation]
µ m +1 β
i be the Lagrangian associated with
[Equation]
Then, for any b i > 0 , Slater’s condition holds and as the objective is concave (by the Weierstass
M-test and the uniform limit theorem), the KKT first order necessary and sufficient optimality conditions for an allocation x ∗
i ∈ R m + , saving β ∗ i ∈ R + and associated Lagrangian multipliers λ ∗ ∈ R + , µ ∗ ∈
R m +1 to be optimal for any prices
p ∈ R m + are given by the following pair of equations:
[Equation]
Additionally, by the KKT complimentarity conditions, we have for all j ∈ j , µ ∗ j x ∗ ij = 0 and µ ∗ m +1 β ∗
i = 0 , which gives us:
[Equation]
Re-organizing expressions, yields:
[Equation]
Using the envelope theorem
[Equation]
we can also compute ∂ν i ∂β i ( s ) as follows:
[Equation]
[Equation]
Hence
[Equation]
combining the above with Equation (94) and Equation (95)
[Equation]
we get:
[Equation]
Finally, going back to Equation (90), multiplying by x ∗ ij ( s ) and summing up across all j ∈ [ m ]
[Equation]
we obtain:
[Equation]
j ∈ [ m ]
[Equation]
Combining the above conditions
[Equation]
with Equation (112)
[Equation]
and adding to it Equation (98)
[Equation]
and ensuring that the KKT primal feasibility conditions hold as well
[Equation]
we obtain the following necessary and sufficient conditions that need to hold for all states s ∈ S :
[Equation]
Theorem 4.1.
A stochastic Fisher market with savings ( S , U , b (0) , p, γ ) in which U is a vector of continuous and homogeneous utility functions has at least one recCE.
Additionally, the recSE ( p ∗ , X ∗ , β ∗ ) that solves the following Bellman equation corresponds to the recCE of ( S , U , b (0) , p, γ ) :
[Equation]
Proof of Theorem
[Equation]
Define
[Equation]
λ
[Equation]
−
[Equation]
p + β i ) .
By Theorem 3.1, the necessary optimality conditions for the stochastic Stackelberg game
[Equation]
t
[Equation]
are that for all states s ∈ S there exists µ ∗ ( s ) ∈ R n × ( m +1) , and ν ∗ ( s ) ∈ R n × ( m +1) + associated with the non-negativity constraints for ( X , β ) , and p respectively, and λ ∗ ( s
) ∈ R m + such that:
[Equation]
Note that by Theorem 3.1, we also have µ ∗ i ( m +1) ( s ) β ∗ i ( s )) = µ ∗ i +1) ( s ) x ∗ ij ( s ) = 0 which gives us: p
[Equation]
( s )
[Equation] [Equation]
⇒
[Equation]
− (cid:88) i ∈ [ n ] λ
[Equation] ( s [Equation] ij ( s )
[Equation]
∀ [ m ] ∈ [ m ] (109) x ∗ ij ( s ) > 0 =
⇒ b
i − β ∗
i ( s ) u i ( x ∗ i ( s ))
∂u i ∂x ij ( x ∗ i ( s )) − λ ∗ i
( s )
p j ( s ) =
∀ i ∈ [ n ] , j ∈ [ m ] (110) β ∗
i ( s ) > 0 =
⇒ − log ( u i ( x ∗ i ( s ); t i
[Equation]
Re-organizing expressions, we obtain:
[Equation]
Using the envelope theorem [Equation] we can compute ∂v∂b i as follows:
[Equation]
Re-organizing expressions, we get:
[Equation]
Combining Equation (116) and Equation (114), we obtain:
[Equation]
Going back to Equation (107)
[Equation]
multiplying both sides by x ∗ ij ( s ) and summing up across all j ∈ [ m ] , we get:
[Equation]
By Theorem 3.1, we have that λ ∗ i ( s ) (cid:16) b i − (cid:80) j ∈ [ m ] p j ( s ) x ∗ ij ( s ) − β ∗
i ( s ) (cid:17) = 0 , which gives us:
[Equation]
Combining the above with Equations (112) to (114) we obtain:
[Equation]
Since the utility functions are non-satiated, and by the second equation, the buyers are utility maximizing at state s over all allocations, we must also have that Walras’ law holds, i.e., p · (cid:16) q − (cid:80) i ∈ [ n ] x i (cid:17) − (cid:80) i ∈
[ n ] β i .
Walras’ law combined with the first equation above then imply the second condition of a recursive competitive equilibrium.
Finally, by Lemma D.1, the last two equations imply the first condition of recursive competitive equilibrium.
E Experiment Details
We initialized a stochastic Fisher market with n = 2 buyers and m = 5 goods.
To simplify the analysis, we assumed deterministic transitions such that the buyers do not get new budgets at each time period, and their types/valuations as well as the supply of goods does not change at each state, i.e., the type/valuation space and supply space has cardinality 1.
This reduced the market to a deterministic repeated market setting in which the amount of budget saved by the buyers differentiates different states of the market.
To initialize the state space of the market, we first fixed a range of [10 , 50] m for the buyers’ valuations and drew for all buyers i ∈ [ n ] valuations θ i from that range uniformly at random at the beginning of the experiment.
We have assumed the supply of goods is 1 m and that the budget space was [0 , 1] n .
This means that our state space for our experiments was S
= { ( θ 1 , θ 2 ) } × { 1 m } ×
[0 , 1] n .
We note that although the assumption that buyers valuations/type space has cardinality one does simplify the problem, the supply of the goods being 1 at each state is wlog because goods are divisible and the allocation of goods to buyers at each state can then be interpreted as the percentage of a particular good allocated to a buyer.
We assumed initial budgets of b (0)
= 1 n for both buyers.
Since the state space is continuous, the value function is also continuous in the stochastic Fisher market setting.
As a result, we had to use fitted variant of value iteration.
In particular, we assumed that the value function had a linear form at each state such that v ( t , b , q ; a , c )
=
a T b + c for some parameters a ∈ R n , c ∈ R , and we tried to approximate the value function at the next step of value iteration by using linear regression.
That is, at each value iteration step, we uniformly sampledbudget vectors from the range [0 , 1] n .
Next, for each sampled budget b , we solved the min-max step given that budget as a state.
This process gave us (budget, value) pairs on which we ran linear regression to approximate the value function at the next iterate.
To solve the generalized min-max operator at each step of value iteration, we used two different methods for comparison: nested gradient descent ascent (GDA) Goktas & Greenwald (2021) (Algorithm 2), which is not guaranteed to converge to a global optimum since the min-max Stackelberg game for stochastic Fisher markets is convex-non-concave and max-oracle gradient descent
Goktas & Greenwald (2021) where the max-oracle is the simulated annealing algorithm Bertsimas & Tsitsiklis (1993), a metaheuristic which probabiliticly aims to find a global maximum.
Although, simulated annealing annealing is not guaranteed to converge to a global maximum, we observed that it often performed better than nested GDA at finding a global optimum for the inner player’s strategy.
For both methods, we have run value iteration for 30 iterations.
We ran nested GDA with learning rates η X = 0 .
5 , η p = 0 .
02 for linear, η X = 0 .
5 , η p = 0 .
01 for leontief, and η X = 0 .
5 , η p = 0 .
0005 for Cobb-Douglas.
We also ran max-oracle gradient descent with a learning rate of η p = 0 .
The outer loop of nested GDA was run for T p = 30 iterations, while its inner loop was run for T X = 100 iterations, and the max-oracle gradient descent algorithm was run for T p =iterations.
We depict the trajectory of the average value of the value function at each iteration of value iteration, under nested GDA in Section 5, and under max-oracle gradient descent in Section 5.