On the convergence of policy gradient methods to Nash equilibria in general stochastic games
Abstract
Learning in stochastic games is a notoriously di ffi cult problem because, in addition to each other’s strategic decisions, the players must also contend with the fact that the game itself evolves over time, possibly in a very complicated manner.
Because of this, the convergence properties of popular learning algorithms – like policy gradient and its variants – are poorly understood, except in specific classes of games (such as potential or two-player, zero-sum games).
In view of this, we examine the long-run behavior of policy gradient methods with respect to Nash equilibrium policies that are second-order stationary (SOS) in a sense similar to the type of su ffi ciency conditions used in optimization.
Our first result is that SOS policies are locally attracting with high probability, and we show that policy gradient trajectories with gradient estimates provided by the R einforce algorithm achieve an O (1 / √ n ) distance-squared convergence rate if the method’s step-size is chosen appropriately. Subsequently, specializing to the class of deterministic Nash policies, we show that this rate can be improved dramatically and, in fact, policy gradient methods converge within a finite number of iterations in that case.
Introduction
Ever since they were introduced by Shapley [46] in the 1950’s, stochastic games have been one of the staples of non-cooperative game theory, with a range of pioneering applications to multiagent reinforcement learning [47], unmanned vehicles [45], general game-playing [33, 48, 53] etc.
Informally, a stochastic game unfolds in discrete time as follows:
At each point in time, the players are at a given state which determines the rules of the game for that stage.
The actions of the players in this state determine not only their instantaneous payo ff s (as defined by the stage game), but also the transition probabilities towards the next state of the process.
In this way, each player has to balance two distinct – and often competing – objectives: optimizing the payo ff s of today versus picking a possibly suboptimal action which could yield significant benefits tomorrow (i.e., by influencing the transitions of the process towards a more favorable state for the player).
Since all players in the game are involved in a similar dilemma, the decision-making problem for each player is a very complicated a ff air.
In particular, in addition to their changing strategic decisions, the players of the game must also contend with the fact that the stage game itself evolves over time.
Because of this, even the existence of a Nash equilibrium policy – viz.
a stationary Markovian policy that is stable to unilateral deviations [15] – is far more di ffi cult to prove compared to standard, stateless normal form games; for a comprehensive survey, cf. [37, 49] and references therein.
The question we seek to address in this paper is whether an ensemble of boundedly rational players can reach an equilibrium policy in a stochastic game.
Specifically, if players do not have su ffi cient information – or the computational resources required – to solve a high-dimensional Bellman equation [14, 50], it is not at all clear if they would somehow end up playing a Nash policy in the long run.
After all, the complexity of most games increases exponentially with the number of players, so the identification of a game’s equilibria quickly becomes prohibitively di ffi cult [25].
Our contributions in the context of related work.
This issue has sparked a vigorous literature with important ramifications for the range of applications mentioned above.
Nevertheless, these e ff orts must grapple with a series of strong lower bounds for computing even weaker solution concepts like coarse correlated equilibria in turn-based stochastic games [11, 25].
On that account, a recent line of work has focused on establishing convergence in specific subclasses of stochastic games, such as min-max [7, 10, 30, 43, 44, 54] and common interest potential games [12, 29, 57]. However, despite these encouraging results, the general case remains particularly elusive.
Our paper takes a complementary approach to the above and seeks to study the convergence landscape of a class of equilibrium policies – not games . For concreteness, we focus on the general class of policy gradient methods as pioneered by [26, 27, 51, 55], and we examine the methods’ convergence properties in general random stopping games – as opposed to ergodic stochastic games with an infinite horizon [30, 38].
Concretely, this means that the sequence of play evolves episode-by-episode: within each episode, the players commit a policy and play the game, and from one episode to the next, they use an iterative gradient step to update their policy and continue playing.
Our main contributions in this general context may be summarized as follows:
We introduce a flexible algorithmic template for the analysis of policy gradient methods which accounts for di ff erent information and update frameworks – from perfect policy gradients to valuebased estimates obtained on a per-episode basis, e g., via the R einforce algorithm [4, 51, 55].
Within this framework, we show that Nash policies that satisfy a certain strategic stability condition are locally attracting with arbitrarily high probability.
Moreover, to estimate the method’s rate of convergence, we focus on Nash policies that satisfy a second-order su ffi ciency condition similar to the type of su ffi ciency conditions used in optimization, and we show that such policies enjoy an O (1 / √ n ) squared distance convergence rate.
Finally, we also consider the method’s convergence to deterministic Nash policies – a special case of SOS policies – and we show that, generically, the above rate can be improved dramatically.
In particular, by a simple tweak to the method’s projection step, the induced sequence of play converges to equilibrium in a finite number of iterations, despite all the noise and uncertainty.
It is also worth noting that our analysis focuses squarely on the actual, episode-by-episode trajectory of play, not any “best-iterate” or time-averaged variant thereof.
In regards to the latter class of guarantees, the recent work of Jin et al.
[24] proposed an algorithm (called V-learning) which updates the policy π n of the n -th episode based on the observed rewards so far.
Thanks to the algorithm’s regret guarantees, Jin et al.
[24] showed that ( a ) in min-max games, the time-averaged policy ¯ π n = (1 / n ) (cid:80) nk = 1 π k converges to equilibrium at a rate of O (1 / √ n ); whereas ( b ) in general stochastic games, the empirical frequency of play converges to the game’s set of coarse correlated equilibria (a substantial relaxation of the notion of Nash equilibrium) at a rate of O (1 / √ n ).
By contrast, as we mentioned above, our paper focuses on the actual sequence of play, i.e., the policy π n employed at each episode of the game.
Moreover, the rates that we obtain all concern the convergence of the players’ policies to a Nash equilibrium – not a correlated equilibrium or other relaxation thereof.
In this regard, the best-iterate / ergodic convergence rates are incomparable to our own as they concern a weaker type of convergence (time-averaged instead of the actual sequence), and to a weaker solution concept (correlated equilibria instead of Nash equilibria).
This aspect of our results is especially relevant for multi-agent reinforcement learning scenarios where agents learn “on the fly”, and it has important ramifications for many of the practical applications of stochastic games.
From a technical standpoint, our analysis is based on mapping the problem of multi-agent policy learning to the problem of equilibrium learning in a class of continuous games characterized by the fact that first-order stationary points are necessarily Nash (itself a consequence of the so-called “gradient dominance” property of stochastic games).
By means of this reframing, we are able to leverage a series of recent techniques for establishing local convergence in (non-monotone) continuous games and variational inequalities [3, 22, 23, 31, 32, 41], which ultimately also yield convergence in our setting.
As a result, even though the unbounded variance of the R einforce estimator is a source of considerable complications, the resulting link between stochastic and continuous games is of particular technical interest because it opens up a wide array of stochastic approximation tools and techniques that can be used for the analysis of multi-agent learning in stochastic games.
Preliminaries
Setup of the game.
Throughout this work we consider N -player generic stochastic games where players repeatedly select actions in a shared Markov decision process (MDP) with the goal of maximizing their individual value functions.
Formally, we study the tabular version with random stopping of general stochastic games, which is specified by a tuple G =
( S , N , { A i , R i } i ∈ N , P , ζ, ρ ) with the following primitives:
• A finite set of agents i ∈ N = { 1 , 2 , . .
. , N } and a finite set of states S = { 1 , . .
. , S } .
• For each i ∈ N , a finite space of actions (or pure strategies )
A i indexed by α i = 1 , . .
A i =
|
A i | .
We will write A = (cid:81) i ∈ N A
i and A − i = (cid:81) j
(cid:44) i
A j for the action space of all agents and that of all agents other than i respectively.
In a similar vein, we will also write α =
( α i , α − i ) when we want to highlight the action α i of player i against the action profile α − i of i ’s opponents.
• For each i ∈ N , we will write R i : S ×
A → [ − 1 , 1] for the reward function of agent i ∈ N , i.e., R i ( s , α i , α − i ) will denote the value of the reward of agent i when the game is at state s ∈ S , the focal agent i ∈ N plays α
i ∈
A i , and all other agents take actions α − i ∈
A − i .
• The game transits from one state to another according to a Markov transition process, so that P ( s ′ | s , α ) denotes the probability of transitioning from s to s ′ when α ∈ A is the action profile chosen by the agents.
• Given an action profile α at state s , the process terminates with probability ζ s ,α > 0, i.e., ζ s ,α = 1 − (cid:80) s ′ ∈ S P ( s ′ | s , α ); for convenience, we will write ζ (cid:66) min s ,α { ζ s ,α } .
• ρ ∈ ∆ ( S ) is the distribution for the initial state of the game.
Episodic Setting.
We consider an episodic setting, where in each episode a realization of the game is completed.
At every time step t ≥ 0 of each episode, all agents observe the common state s t ∈ S , select actions α t and receive rewards { R i ( s t , α t ) } i ∈ N .
Then, with probability ζ s t ,α t the game terminates, and with probability 1 − ζ s t ,α t , it moves to the state s t
+ 1 , which is drawn according to P ( ·| s t , α t ).
Denoting the realized reward of player i at time t as r i , t (cid:66) R i ( s t , α t ), we will write τ = ( s t , α t , r t ) t ≤ T ( τ ) to denote the trajectory of the episode, where r t (cid:66) ( r i , t ) i ∈ N , and T ( τ ) the time the episode terminates.
Policies and value functions.
We consider stationary Markovian policies, i.e., policies that do not depend on the time-step and the history, given the current state of the game.
More specifically, for each agent i ∈ N , a policy π i : S → ∆ ( A i ) specifies a probability distribution over the actions of agent i in state s ∈ S , i.e., α i ∼ π i ( ·| s ) denotes the (random) action drawn by agent i at state s ∈ S according to π i , viewed here as an element of Π i (cid:66) ∆ ( A i ) S .
In addition, we will also write π = ( π i ) i ∈ N ∈ Π (cid:66) (cid:81)
i Π i and π − i =
( π j ) j (cid:44) i ∈
Π − i (cid:66) (cid:81) j
(cid:44) i Π j for the policy profile of all agents and all agents other than i , respectively.
The expected reward of agent i ∈ N if agents follow policy π , starting from initial state s ∈ S , defines the value function of agent i , denoted as V i , s ( π ), and is equal to
[Equation]
(1) where τ ∼ MDP denotes the randomness induced by the policy profile π , and the state-transition probabilities of the MDP.
Overloading the notation, we set V i ,ρ ( π ) (cid:66) (cid:133)
s ∼ ρ (cid:2)
V i , s ( π ) (cid:3) .
Although value functions are, in general, non-convex, they share similar smoothness properties with the payo ff functions of normal form games, namely bounded and Lipschitz gradients.
For precise statements, we defer to the paper’s supplement.
Visitation distribution and the mismatch coe ffi cient.
For a policy profile π ∈ Π and an arbitrary initial state distribution s 0 ∼ ρ , we define the discounted state visitation measure / distribution as
[Equation]
In the appendix, we prove formally that the above definition is well-posed for the random stopping episodic framework described above, i.e., ˜ d πρ ( s ) < ∞ , so Z πρ (cid:66) (cid:80) s ∈ S ˜
d πρ ( s ) is well-defined.
In our proofs, we will leverage a standard property of visitation distributions, namely the equivalence of the expected value of state-action function and the expected cumulative value over a random trajectory.
More precisely, we have:
Lemma 1.
[Conversion Lemma]
For an arbitrary state-action function f
: S × A → (cid:146) , a policy profile π and an initial state distribution s 0 ∼ ρ , we have
[Equation]
Finally, to quantify the di ffi culty of hard-to-reach states via a policy gradient method, we will follow the standard approach of [8, 13, 34, 35, 57] and use an appropriately-defined distribution “mismatch coe ffi cient”, generalizing the single-agent counterpart of Agarwal et al.
More precisely, for a stochastic game G , we define the mismatch coe ffi cient as C G (cid:66) max π,π ′ ∈ Π (cid:8) ∥
˜ d πρ / ˜
d π ′ ρ ∥ ∞ (cid:9) or, more simply, as C G (cid:66) max π, ∈ Π (cid:8) 1 ζ ∥
d
πρ /ρ ∥ ∞ (cid:9) .
Similar to prior work in this direction [1, 5, 10], we will assume C G is finite, which, equivalently, means that d πρ ( s ) > 0 for any policy π and state s .
Solution concepts.
The most widely used solution concept in game theory is that of a Nash equilibrium i.e., a strategy profile π ∗ ∈ Π that discourages unilateral deviations.
However, in stochastic games, the definition of a Nash policy is much more involved because of the existence of multiple states and steps, cf.
[15, 46, 49, 52] and references therein. Formally, we have:
Definition 1 (Nash policies) .
A policy π ∗ =
( π ∗ i ) i ∈ N ∈ Π is said to be a Nash policy for a given distribution of initial states ρ ∈ ∆ ( S ) if, for every player i ∈ N , we have
[Equation]
In contrast to general non-convex continuous games, stochastic games satsify a version of the wellknown Polyak-Łojasiewicz condition [40] but with linear gradient growth, also known as a gradient dominance property (GDP) [1, 5].
For the multi-agent case, Zhang et al. [57] and Daskalakis et al. [10] showed that a similar property holds even in the episodic setting:
Lemma 2 (Gradient dominance property) .
For any policy profile π = ( π i ) i ∈ N ∈ Π , we have that
[Equation]
V i ,ρ ( π i [Equation] π − i ) ≤ C G max ¯ π i ∈ Π i ⟨∇ i V i ,ρ ( π ) , ¯ π i − π i ⟩ (GDP) for any unilateral deviation π ′ i
∈ Π i of player i ∈ N .
Remark.
In the above and throughout our paper, we will write ∇ i to denote the gradient of the quantity in question with respect to π i , i.e., when π − i is kept fixed and only π i is varied.
For concision, we will write v i ( π ) =
∇
i
V i ,ρ ( π ) for the individual gradient of player
i ’s value function, and v ( π )
= ( v i ( π ))
i ∈ N for the ensemble thereof.
Thanks to (GDP), it is straightforward to check that first-order stationary (FOS) points of V are Nash.
Formally, as in [10, 29, 57], we have the following characterization:
Lemma 3 (First-order stationary policies are Nash) .
A policy π ∗ =
( π ∗ i ) i ∈ N ∈ Π is Nash
if and only if it satisfies the first-order stationary condition
[Equation]
Leonardos et al.
[29] and Zhang et al.
[57] proved a relaxation of the above lemma to the e ff ect that policies that satisfy (FOS) up to ε (i.e., in lieu of 0 in the RHS) are O ( ε )-Nash.
Going in the other direction, we will consider the following series of refinements of Nash policies which are particularly important from a learning standpoint [28, 49]:
Definition 2.
Let π ∗ =
( π ∗ i ) i ∈ N ∈ Π be a Nash policy.
We then say that:
• π ∗ is stable if ⟨ v ( π ) , π − π ∗ ⟩ < 0 for all π (cid:44)
π ∗ su ffi ciently close to π ∗ .
• π ∗ is second-order stationary if it satisfies the su ffi ciency condition
[Equation]
where Jac v ( π ∗ )
= ( ∇ j v i ( π ∗ ))
i , j ∈ N = ( ∇ j ∇ i
V i ( π ∗ ))
i , j ∈ N denotes the Jacobian of v at π ∗ .
• π ∗ is deterministic if it induces a deterministic selection rule π ∗ i : S
→ A i for all i ∈ N .
• π ∗ is strict if it is deterministic and (FOS) holds as a strict inequality whenever π (cid:44) π ∗ .
Remark 1 .
In the above and what follows, “su ffi ciently close” means that there exists a neighborhood
U of π ∗ in Π such that the stated inequality holds for all π ∈ U .
Unless mentioned otherwise, we will measure distances on Π relative to the Euclidean norm, but this choice does not impact our results.
Intuitively, the condition for equilibrium stability is a game-theoretic analogue of first-order KKT su ffi ciency condition, while the condition for second-order stationarity is the second-order version thereof.
In this regard, the distinction between first-order stationary, stable and second-order stationary points is formally analogous to the distinction between critical points, minimizers, and second-order minimum points in optimization.
As for deterministic policies, we should mention that, generically, deterministic policies are also strict, so we will use the two terms interchangeably.
Importantly, as we show in Appendix F, these refinements admit the following characterizations:
Proposition 1.
Let π ∗ =
( π ∗ i ) i ∈ N ∈ Π be a Nash policy.
Then:
a ) If π ∗ is second-order stationary, there exists some µ > 0 such that ⟨ v ( π ) , π − π ∗ ⟩ ≤ − µ ∥ π − π ∗ ∥for all π su ffi ciently close to π ∗ .
[Equation]
3a) b ) If π ∗ is strict, there exists some µ > 0 such that
[Equation]
for all π su ffi ciently close to π ∗
[Equation] 3b [Equation] In view of all the above, we get the following string of implications for equilibria in generic games:
[Equation]
For posterity, we should clarify here that, due to the highly complicated structure of the game’s value functions, it is not trivial to construct a concrete example where (3a) holds but (3b) does not.
Examples of strict Nash policies abound in the literature [28, 49], but we are not otherwise aware of an argument that could be used to close the gap between (3a) and (3b).
In view of this, our analysis will treat both cases concurrently (with the obvious anticipation that more refined solution concepts should enjoy stronger convergence guarantees).
Policy gradient methods
We now proceed to describe our general model for episodic learning in stochastic games.
To that end, we will consider a framework where agents follow a specific policy π n within each episode, and update it from one episode to the next with the objective of increasing their individual rewards.
Formally, our approach will adhere to the following inter-episode sequence of events:
At the beginning of each episode n = 1 , 2 , . . .
, every agent i ∈ N chooses a policy π i , n ∈ Π i .
Within the n -th episode, each player executes their chosen policy
π i , n , inducing in this way an intra-episode trajectory of play τ n = ( s ( n ) t , α ( n ) t , r ( n ) t ) t ≤ T ( τ n ) .
Once the episode terminates, agents update their policies and the process repeats.
In terms of feedback, we will treat several models, depending on what type of information is available to the agents during play.
More precisely, we will focus on the generic policy gradient (PG) template
[Equation]
where:
1: Input: ˆ π ∈ Π , τ = ( s t , α t , r t ) t ≤ T ( τ ) ∈ T 2: for i = 1 , . .
. , N do 3:
R i ( τ ) ← (cid:80) T ( τ ) t = 0 r i , t 4: Λ i ( τ ) ← (cid:80) T ( τ ) t = 0 ∇ i (cid:0) log ˆ π i ( α i , t | s t ) (cid:1) 5: ˆ v i ← R i
( τ ) · Λ i ( τ ) 6: end for 7: return { ˆ v i } i ∈ N
Algorithm 2: ε
1: Input: π 1 , { γ n } n ∈ (cid:142) , { ε n } n ∈
cid:142) 2: for n = 1 , 2 , . . .
do 3: ˆ π n ← (1 − ε n ) π n + ε n | A | 4:
Sample τ n ∼ MDP(ˆ π n | s 0 ) 5: ˆ v n ← R einforce (ˆ π n , τ n ) 6: π n + 1 ← proj Π ( π n + γ n ˆ v n ) 7: end for π n = ( π i , n ) i ∈ N ∈ Π denotes the player’s policy profile at each episode n = 1 , 2 , . . . ˆ v n = (ˆ v i , n ) i ∈ N ∈ (cid:81) i (cid:146)
A i × S is an estimate for the agents’ inidividual policy gradients. proj Π : (cid:81) i (cid:146) A i × S → Π denotes the Euclidean projection to the agents’ policy space Π . 4.
γ n > 0 is
the
method’s
step-size, for
which we will
assume throughout that (cid:80) n γ n = ∞ ; typically, (PG) is run with a step-size of the form γ n = γ/ ( n + m ) p for some γ > 0, m ≥ 0 and p ≥ 0.
Regarding the gradient signal ˆ v n , we will decompose it as
[Equation]
where [Equation] In the above, we treat π n , n = 1 , 2 , . . . , as a stochastic process on some complete probability space ( Ω , F , (cid:144) ), and we write F n (cid:66) F ( π 1 , . .
π n ) ⊆ F for the history (adapted filtration) of π n up to – and including – stage n .
By definition, (cid:133)
[ U n | F n ] =
and b n is F n -measurable, so U n can be intepreted as a random, zero-mean error relative to v ( π n ), whereas b n captures all systematic (non-zero-mean) errors.
To make this precise, we will further assume that b n and U n are bounded as
[Equation]
where the sequences B n and σ n , n = 1 , 2 , .
, are to be construed as deterministic upper bounds on the bias, fluctuations, and magnitude of the gradient signal ˆ v n .
Depending on these bounds, a gradient signal with B n = 0 will be called unbiased , and an unbiased signal with σ n = 0 will be called perfect .
More generally, we will assume that the above statistics are bounded as
[Equation]
for some ℓ b , ℓ σ > 0 which depend on the specific model under consideration.
For concreteness, we describe below three basic models that adhere to the above template for ˆ v n in order of decreasing information requirements:
Model 1 (Full gradient information) The
first model we will consider assumes that agents observe their full policy gradients ,
i
.e.
ˆ
[Equation]
implying in particular that U n = b n = 0.
This model is fully deterministic across episodes (though intra-episode play remains stochastic).
In particular, it tacitly assumes that agents know the game (and can observe their opponents’ policies) su ffi ciently well so as to calculate the full gradients of their individual value functions
V i ,ρ , cf.
[2, 29, 57] and references therein.
Model 2 (Learning with stochastic gradients) .
A relaxation of the above model which is particularly relevant for applications to deep reinforcement learning concerns the case where the player have access to stochastic policy gradients [56], i.e., unbiased gradient estimates of the form
[Equation] [Equation]
with (cid:133)
[ U n | F n ] =
(so we can formally take ℓ b = ∞ and ℓ σ = 0 in Eq.
(8) above).
Model 3 (Value-based learning) .
The last model we will consider concerns the case where agents only have access to their instantaneous rewards and need to reconstruct their individual gradients based on this information.
A widely used method to achieve this is via the R einforce subroutine, which we describe in pseudocode form in Algorithm 1.
In words, when employing R einforce , each agent i ∈ i commits to a sampling policy ˆ π i ∈ Π i and executes it in an episode of the stochastic game in play.
Then, at the end of the episode, players gather the total reward R i ( τ ) ← (cid:80) T ( τ ) t = 0 r
i , t associated to the intra-episode trajectory of play τ , and they estimate their policy gradients via the so-called “log-trick” [55] as
[Equation]
Lemma 4 below provides the vital statistics of the R einforce estimator:
Lemma 4.
Suppose that each agent i ∈ N follows a stationary policy π
i ∈ Π i .
Then:
[Equation]
where κ i = min s ∈ S ,α i ∈
A i π i ( α i | s ) .
Therefore, if R einforce is executed at ˆ π ← π n at each episode n = 1 , 2 , . .
, we will have
[Equation]
[Equation]
) and
[Equation]
In particular, this means that we will always have B n = 0 for the bias of the estimator, but its variance could be unbounded if π n gets close to the boundary of Π .
To avoid this, R einforce can be paired with an explicit exploration step that modifies the sampling policy of the n -th episode to
[Equation]
i.e., ˆ π i , n is the mixture between π i , n and the uniform distribution Unif A i over A i .
The resulting algorithm is known as ε -G reedy P olicy G radient ; for a pseudocode representation, see Algorithm 2.
Importantly, by calling R einforce at ˆ π n instead of π n , ˆ v n becomes biased (because of the di ff erence between ˆ π n and π n ), but its variance is bounded; in particular, by invoking Lemma 4, we have
[Equation]
where G is a constant that depends on the smoothness of V and the cardinalities of A and S .
In this way, Algorithm 2 can be seen as a special case of (PG) with B n = O ( ε n ) and σ 2 n =
O (1 /ε n ).
Convergence analysis and results
We are now in a position to state and discuss our main results.
For convenience, we will present our results in order of increasing structure, starting with stable policies, and then moving on to second-order stationary and deterministic Nash policies.
All proofs are deferred to the appendix.
Asymptotic convergence to stable Nash policies.
Our first convergence result concerns Nash policies that satisfy the stability requirement ⟨ v ( π ) , π − π ∗ ⟩ < 0 of Definition 2.
In this case, we have the following guarantee:
Theorem 1.
Let π ∗ be a stable Nash policy, and let π n be the sequence of play generated by (PG) with step-size γ n = γ/ ( n + m ) p , p ∈ (1 / 2 , 1] , and policy gradient estimates such that p + ℓ b > 1 and p − ℓ σ > 1 / 2 as per (8) .
Then there exists a neighborhood U of π ∗ in Π such that, for any given δ > 0 , we have
[Equation]
provided that γ is small enough ( or m large enough ) relative to δ .
Corollary 1.
Suppose that Models 1–3 are run with a step-size of the form γ n = γ/ ( n + m ) p , p > 1 / 2 , and if applicable, an exploration parameter ε n = ε/ ( n + m )
ℓ ε such that 1 − p <
ℓ ε < 2 p − 1 .
Then:
• For Models 1 and 2: the conclusions of Theorem 1 hold as stated.
• For Model 3: the conclusions of Theorem 1 hold as long as p > 2 / 3 .
We note here that Theorem 1 provides a trajectory convergence guarantee which is otherwise quite di ffi cult to obtain even in structured stochastic games.
For example, if we zoom in on the class of stochastic potential (or min-max) games, the existing guarantees in the literature concern the “best iterate” of the algorithm, cf.
[29, 57] and references therein.
Because of this, said guarantees do not apply to the actual trajectory of play generated by (PG); this makes them less suitable for agent-based learning where the players involved are learning “as they go”, as opposed to simulating the game in order to approximately compute an equilibrium policy o ffl ine.
We should also note that the convergence guarantees of Theorem 1 hold locally with arbitrarily high probability.
Without further assumptions, it is not possible to obtain global trajectory convergence guarantees that hold with probability 1, even in single-state games – that is, the case of learning in finite normal form games.
The reason for this locality is twofold: First, equilibrium policies are not unique in general, and gradient-based dynamics may also admit non-equilibrium attractors, such as limit cycles and the like [21].
As a result, in the presence of multiple equilibria / attractors, the best one can hope for is a local equilibrium convergence result, conditioned on the basin of attraction of said equilibrium (as per Theorem 1).
The second obstruction to a global, unconditional convergence result is probabilistic in nature, and has to do with the randomness that enters the learning process (e.g., in the estimation of policy gradients via the R einforce ).
In this case, no matter how close one starts to an equilibrium policy, there is always a finite, non-zero probability that an unlucky realization of the noise can drive the process away from its basin, possibly never to return.
This issue can only be overcome in games where Π is partitioned (up to a set of measure zero) into basins of attraction of equilibrium policies.
However, this can only occur in games with a su ffi ciently strong global structure, like potential stochastic games, two-player zero-sum games and the like; in complete generality, locality cannot be lifted, even in single-state problems [16].
Convergence to second-order stationary policies.
Albeit valuable as an asymptotic convergence guarantee, Theorem 1 does not provide an indication of how long it will take players to actually converge to a Nash policy.
Of course, in full generality, it is not plausible to expect to be able to derive such a convergence rate because the stability requirement provides no indication on how fast the players’ policy gradients stabilize near a solution.
This kind of estimate is provided by the second-order su ffi cient condition (SOS), which allows us to establish su ffi cient control over the sequence of play as indicated by the following theorem.
Theorem 2.
Let π ∗ be a second-order stationary policy, let B be a neighborhood of π ∗ such that (3a) holds on B , and let π n be the sequence of play generated by (PG) with step-size γ n = γ/ ( n + m ) p , p ∈ (1 / 2 , 1] , and policy gradient estimates such that p + ℓ b > 1 and p − ℓ σ > 1 / 2 as per (8) .
Then:
There exists a neighborhood U of π ∗ in Π such that, for any confidence level δ > 0 , the event
[Equation]
occurs with probability (cid:144) ( E | π 1 ∈ U ) ≥ 1 − δ if m is large enough relative to δ .
The sequence π n converges to π ∗ with probability 1 on E ; in particular, we have [Equation] converges
[Equation]
if m is large relative to δ .
Moreover, conditioned on E and taking q = min { ℓ b , p − 2 ℓ σ } , we have
[Equation]
Corollary 2.
Suppose that Models 1–3 are run with a step-size of the form γ n = γ/ ( n + m ) p , p > 1 / 2 , and if applicable, an exploration parameter ε n = ε/ ( n + m ) p / 2 .
Then:
• For Models 1 and 2: the conclusions of Theorem 2 hold with q = p; in particular, (19) gives an O (1 / n ) rate of convergence if p = 1 and 2 µγ > q. • For Model 3: the conclusions of Theorem 2 hold for p > 2 / 3 with q = p / 2 ; in particular, (19) gives an O (1 / √ n ) rate of convergence if p = 1 and 2 µγ > q.
Remark 2 .
Getting an explicit estimate for the constant in the O ( · ) guarantee of Theorem 2 is quite involved but, up to logarithmic and subleading factors, Chung’s lemma [9, 39] can be used to
show that a ) if 2 µγ > q , it scales as ( C b + C σ ) /
[(2 µγ − q )(1 − δ )] where C b = sup n γ n B n and C σ = sup n γ 2 n σ 2 n ; b ) if 2 µγ
= q , it scales as ( C b + C σ )(1 + max { (2 µγ ) 2 , 4 µγ } ) / (1 − δ ); and c ) if 2 µγ < q as ( C b + C σ )(1 + max { (2 µγ ) 2 , 4 µγ } ) /
[( q − 2 µγ )(1 − δ )].
Besides providing a general framework for achieving trajectory convergence, Theorem 2 gives the rates of convergence of the sequence of play to the Nash policy in question.
In particular, with this result in hand, one can confidently argue about the distance of the iterates of (PG) from equilibrium in a series of di ff erent environments.
More to the point, this convergence guarantee allows the algorithm designer to adapt the parameters of the learning process according to the complexity and limitations of the environment, a feature which further highlights the significance of this result.
We should also note the delicate interplay between the method’s step-size and the achieved convergence rate.
In the case of Model 1, Corollary 2 suggests a step-size of the form γ n = Θ (1 / n ), leading to a O (1 / n ) convergence rate.
As we show in the appendix, this rate can be improved: in the deterministic case with perfect gradient information, (PG) with a suitably chosen constant step-size achieves a geometric convergence rate, i.e., ∥ π n − π ∗ ∥
=
O (exp( − ρ n )) for some ρ > 0 (cf.
Proposition B.1 in Appendix B). By contrast, in the case of Model 2, the O (1 / n ) rate we provide cannot be improved, even if the quadratic minorant (3a) that characterizes SOS policies holds globally – and this because the learning process is running against standard lower bounds from convex optimization [6, 36].
Perhaps the most significant guarantee from a practical point of view is the O (1 / √ n ) convergence rate attained in Model 3 (cf.
Algorithms 1 and 2).
This guarantee amounts to a O (1 / n 1 / 4 ) convergence rate in terms of the (non-squared) distance to equilibrium which, mutatis mutandis, represents a notable improvement over the O (1 / n 1 / 6 ) guarantee of Leonardos et al.
[29] (expressed in norm values).
Of course, the latter guarantee is global – because the focus of [29] is stochastic potential games – but it also concerns the “best iterate” of the process (not its “last iterate”), so the two results are not immediately comparable.
However, a useful “best-of-both-worlds” heuristic that can be inferred by the combination of these works is that, given a budget of training episodes, Algorithm 2 can be run with a constant step-size as per [29] for a su ffi cient fraction of this budget, and then with a O (1 / n ) “cooldown” schedule for the rest.
In this way, after an aggressive “exploration” phase, the algorithm’s
O (1 / n 1 / 4 ) rate would kick in and supply faster stabilization to an SOS policy.
Convergence to deterministic Nash policies.
Our last series of results concerns the rate of convergence to deterministic Nash policies in generic stochastic games.
As we discussed in Section 2, deterministic Nash policies also satisfy (SOS), so the rate of convergence of (PG) to such policies can be harvested directly from Theorem 2.
However, as we show below, a simple projection tweak in (SOS) can improve this rate dramatically.
The tweak in question is inspired by the geometry of Π around a deterministic policy: by definition, such policies are corner points of Π , so any consistent drift towards them will cause π n to hit the boundary of Π in finite time.
Of course, under (PG), the process may rebound from the boundary and return to the interior of Π if the policy gradient estimate is not particularly good at a given iteration of the algorithm.
However, if we replace the projection step of (PG) with a “lazy projection” in the spirit of Zinkevich [58], the aggregation of gradient steps will eventually push the process far inside the normal cone of Π at π ∗ , so rebounds of this type can no longer occur.
Formally, we will consider the following lazy policy gradient (LPG) scheme:
[Equation]
where y n = ( y i , n ) i ∈ N ∈ (cid:81) i (cid:146)
A i × S is an auxiliary variable that maintains an aggregate of gradient steps before projecting them back to Π .
We then have the following convergence result:
Theorem 3.
Let π n be the sequence of play under (LPG) with step-size and policy gradient estimates such that p + ℓ b > 1 and p − ℓ σ > 1 / 2 as per (8) .
If π ∗ is a deterministic Nash policy, there exists an unbounded open set W ⊆ (cid:81) i (cid:146)
A i × S of initializations such that, for any δ > 0 , we have
[Equation]
provided that γ > 0 is small enough.
Moreover, conditioned on this event, π n converges to π ∗ at a finite number of iterations, i.e., there exists some n 0 such that π n = π ∗ for all n ≥ n 0 .
Corollary 3.
Suppose that Models 1–3 are run with parameters γ n = γ/ n p , p ∈ (1 / 2 , 1] , and if applicable, ε n = ε/ n ℓ ε with 1 − p <
ℓ ε < 2 p − 1 .
Then the conclusions of Theorem 3 hold.
Remark 3 .
Getting an explicit bound for n 0 is quite complicated, but the last part of the proof of Theorem 3 shows that n 0 scales in terms of the parameters of the game and the algorithm as n
= O (cid:16)(cid:0) MSAc γ (cid:1) 1 / (1 − p ) (cid:17) where c > 0 measures the minimum payo ff di ff erence between equilibrium and non-equilibrium strategies at π ∗ , M is a measure of the initial distance from π ∗ , and S and A is the number of states and pure strategies respectively.
Theorem 3 – and, by extension, Corollary 3 – are fairly unique because they provide a guarantee for convergence to an exact Nash equilibrium in a finite number of iterations.
To the best of our knowledge, the only comparable result in the literature is that of [57], where the authors provide a finite-time convergence guarantee to strict equilibria with perfect policy gradients (as per Model 1).
The result of Zhang et al.
[57] echoes the convergence properties of deterministic first-order algorithms around sharp minima of convex functions [39], but the fact that Theorem 3 applies to models with stochastic gradient feedback of unbounded variance (Models 2 and 3 respectively) is a major di ff erence.
As far as we are aware, this is the first guarantee of its kind in the literature on learning in stochastic games.
Concluding remarks
A key roadblock encountered by practical applications of multi-agent reinforcement learning is the lack of universal equilibrium convergence guarantees.
While the impossibility results of [19, 20] imply that unconditional convergence is not a reasonable aspiration without further assumptions on the game, the existence of local convergence results mitigates this deficiency as it provides a range of theoretically grounded stability and runtime guarantees.
In this regard, deterministic policies acquire particular importance, as the convergence of policy gradient methods is especially rapid and robust and this case.
Of course, this leaves open the question of non-tabular settings and parametrically encoded policies, e.g., as in the case of deep reinforcement learning; we defer these investigations to future work.
Another open issue of high practical relevance concerns policy gradient methods that do not rely on Euclidean projections to Π .
In the single-state case (i.e., learning in finite normal form games), the use of methods relying on softmax choice / exponential weights is very widely used because of its regret guarantees.
Whether the use of similar softmax techniques can lead to finer convergence guarantees in the context of general stochastic games is an important and intriguing question for future research.
Acknowledgments and Disclosure of Funding
Part of this work was done while the authors were visiting the Simons Institute for the Theory of
Computing.
P. Mertikopoulos gratefully acknowledges financial support by the French National
Research Agency (ANR) in the framework of the “Investissements d’avenir” program (ANR-15IDEX-02), the LabEx PERSYVAL (ANR-11-LABX-0025-01), MIAI@Grenoble Alpes (ANR-19P3IA-0003), and the bilateral ANR-NRF grant ALIAS (ANR-19-CE48-0018-01).
K. Lotidis and E. Vlatakis are grateful for financial support by the Onassis Foundation (F ZR 033-1 / 2021-2022, 010-1 / 2018-2019).
A. Giannou is grateful for financial support by ONR: “A Theoretically Principled
Framework for Learning by Pruning”.
E. V. Vlatakis-Gkaragkounis is grateful for financial support by the Google-Simons Fellowship, Pancretan Association of America and Simons Collaboration on
Algorithms and Geometry.
This project was completed while he was a visiting research fellow at the Simons Institute for the Theory of Computing.
Additionally, he would like to acknowledge the following series of NSF-CCF grants under the numbers 1763970 / 2107187 / 1563155 /
A Asymptotic convergence to stable Nash policies
Our goal in this appendix is to prove Theorem 1 and Corollary 1, which we restate below for convenience:
Theorem 1.
Let π ∗ be a stable Nash policy, and let π n be the sequence of play generated by (PG) with step-size γ n = γ/ ( n + m ) p , p ∈ (1 / 2 , 1] , and policy gradient estimates such that p + ℓ b > 1 and p − ℓ σ > 1 / 2 as per (8) .
Then there exists a neighborhood U of π ∗ in Π such that, for any given δ > 0 , we have
[Equation]
n converges
[Equation]
provided that γ is small enough ( or m large enough ) relative to δ .
Corollary 1.
Suppose that Models 1–3 are run with a step-size of the form γ n = γ/ ( n + m ) p , p > 1 / 2 , and if applicable, an exploration parameter ε n = ε/ ( n + m )
ℓ ε such that 1 − p <
ℓ ε < 2 p − 1 .
Then:
• For Models 1 and 2: the conclusions of Theorem 1 hold as stated.
• For Model 3: the conclusions of Theorem 1 hold as long as p > 2 / 3 .
Our proof strategy will comprise the following basic steps:
To begin with, we will show that the squared distance
[Equation]
can be seen as a “local Lyapunov function” for (PG) in the sense that it is locally decreasing near π ∗ , up to a series of error terms – both zero-mean and non-zero-mean.
Due to these errors, the evolution of the iterates D n (cid:66) D ( π n ) of D over time may exhibit significant jumps: in particular, a single “bad” realization of the noise could carry π n out of the basin of attraction of π ∗ , possibly never to return.
To exclude this event, our second step will be to show that the aggregation of these errors can be controlled with probability at least 1 − δ .
Conditioned on the above, we will show that, with probability at least 1 − δ , the iterates D n cannot grow more than a token value.
As a result, if (PG) is initialized close to π ∗ , it will remain in a neighborhood thereof for all n (again, with probability at least 1 − δ ).
Thanks to this “stochastic Lyapunov stability” result, we employ a series of martingale limit theory arguments to extract a subsequence converging to π ∗ .
Finally, we show that the increments of D n are summable; hence, by invoking the Gladyshev’s lemma [39, p. 49], we conclude that D n converges to some (finite) random variable D ∞ .
Combining this fact with the existence of a convergent subsequence, we obtain the desired conclusion that π n converges to π ∗ with probability at least 1 − δ .
In the sequel we make the above precise in a series of intermediate results
A.1.
Energy inequality.
We begin by establishing a “quasi-Lyapunov” inequality for the iterates
D n = ∥
π n − π ∗ ∥ 2 / 2 of (A.1).
Lemma A.1.
Let D n (cid:66) D ( π n ) .
Then, for all n = 1 , 2 , . .
, we have
[Equation]
where the error terms ξ n , χ n , and ψ n are given by
[Equation]
with ∥ Π ∥ (cid:66) max π,π ′ ∈ Π ∥ π − π ′ ∥ .
Proof.
By the definition of the iterates of (PG), we have
[Equation]
where we used the Cauchy-Schwarz inequality to bound the bias term as ⟨ b n , π n − π ∗ ⟩ ≤ ∥ b n ∥ · ∥ π n − π ∗ ∥ ≤ ∥ Π ∥ B n = χ n .
■
A
Error control and stability.
The second major step in our proof (and the most challenging one from a technical standpoint) is to establish a suitable measure of control over the error increments in (A.1), with the aim of showing that the process π n never leaves a neighborhood of π ∗ .
To make this idea precise, let B = { π ∈ Π : ∥
π − π ∗ ∥ ≤ ϱ } be a ball of radius ϱ based on π ∗ in Π so that ⟨ v ( π ) , π − π ∗ ⟩ < 0 for all π ∈ B \{ π ∗ } (without loss of generality, we can assume that B is maximal in that regard).
We will then examine the event that the aggregation of the error terms in (A.1) is not su ffi cient to drive π n to escape from B .
To that end, we will begin by aggregating the errors in (A.1) as
[Equation]
n
[Equation]
(cid:88)
[Equation]
Since (cid:133) [ ξ n | F n ] = 0, we have (cid:133)
[ M n | F n ] = M n − 1 , so M n is a martingale; likewise, (cid:133)
[ S n | F n ] ≥ S n − 1 , so S n is a submartingale.
Then, using a technique of Hsieh et al.
[22] that builds on an earlier idea by
Mertikopoulos and Zhou [31], we will also consider the “mean square” error process
[Equation]
and the associated indicator events
E n
[Equation]
{ π k
[Equation]
B for all k = 1 , 2 , . . . , n } and
[Equation]
A
[Equation]
7a) where, with a fair amount of hindsight, the error tolerance level a > 0 is such that 2 a + √ a < ϱ , and we are employing the convention E
= H
= Ω (since every statement is true for the elements of the empty set).
We will then assume that π 1 is initialized in a ball of radius √ 2 a centered at π ∗ , viz
[Equation]
With all this in hand, the key to showing that π n remains close to π ∗ with high probability is the following conditional estimate:
Lemma A.2.
Let π n be the sequence of play generated by (PG) initialized at π 1 ∈ U .
We then have:
E n + 1 ⊆ E n and H n + 1 ⊆ H n for all n = 1 , 2 , . . .
H n [Equation]
1 ⊆ E n for all n = 1 , 2 , . . .
3. Consider the “bad realization” event
[Equation]
and let ˜ W n = W n 1 H n − 1 be the cumulative error subject to the noise being “small”.
Then we have:
[Equation]
where, by convention, ˜ H 0 = ∅ and ˜ W 0 = 0 .
Remark.
In the above (and what follows), the notation 1 A is used to indicate the logical indicator of an event
A ⊆ Ω , i.e., 1 A ( ω ) = 1 if ω ∈ A and 1 A ( ω ) = 0 otherwise.
The proof of Lemma A.2 is quite technical, so we first proceed to derive an important stability result based on this estimate.
Proposition A.1.
Fix some confidence threshold δ > 0 and let π n be the sequence of play generated by (PG) with step-size and policy gradient estimates as per Theorem 1.
We then have:
[Equation]
provided that γ is small enough ( or m large enough ) relative to δ .
Proof.
We begin by bounding the probability of the “bad realization” event ˜
H n = H n − 1 \ H n .
Indeed, if π 1 ∈ U , we have:
[Equation]
n ] / a (A.12) where, in the penultimate step, we used the fact that W n ≥ 0 (so 1 { W n > a } ≤ W n / a ).
Telescoping (A.10) then yields
[Equation]
where we set
[Equation]
Hence, combining (A.12) and (A.13) and invoking our stated assumptions for γ n , B n and σ n , we get
[Equation]
for some C ≡ C ( γ, m ) > 0 with lim γ → 0 + C ( γ, m ) =
lim m →∞ C ( γ, m )
=
(since γ n = γ/ ( n + m ) p and p > 0).
Now, by choosing γ su ffi ciently small (or m su ffi ciently large), we can ensure that C / a < δ ; thus, given that the events ˜ H k are disjoint for all k = 1 , 2 , . .
, we get (cid:144) (cid:0)(cid:83)
nk = 1 ˜ H k
(cid:1)
= (cid:80) nk = 1 (cid:144)
( ˜ H k ) ≤ δ.
In turn, this implies that (cid:144) ( H n ) =
(cid:144) (cid:0) ˜
H c 1 ∩· · · ∩ ˜ H c n (cid:1) ≥ 1 − δ, and our assertion follows.
■
We conclude this appendix with the proof of our technical result on the events E n and H n :
Proof of Lemma A.2.
The first claim of the lemma is obvious.
For the second, we proceed inductively:
For the base case n = 1, we have E 1 = { π 1 ∈ B } ⊇ { π 1 ∈ U } =
Ω (recall that π 1 is initialized in U ⊆ B ).
Since H 0 = Ω , our claim follows.
Inductively, assume that H n − 1 ⊆ E n for some n ≥ 1.
To show that H n ⊆ E n + 1 , suppose that W k ≤ a for all k = 1 , 2 , . . . , n . Since H n ⊆ H n − 1 , this implies that E n also occurs, i
e., π k ∈ B for all k =
n ; as such, it su ffi ces to show that π n
+ 1 ∈ B .
To do so, given that π k ∈ U ⊆ B for all k = 1 , 2 ,
n , we readily obtain
[Equation]
for all
k =
[Equation]
[Equation]
n
[Equation]
A.16) and hence
, after telescoping over k = 1 , 2 , . . . , n , we
get
[Equation]
We conclude that D ( π n + 1 ) ≤ 2 a + √ a , i.e., π n + 1 ∈ B , as required for the induction.
For our third claim, note first that
[Equation]
so, after taking expectations, we get
[Equation]
i.e., W n is a submartingale.
To proceed, let ˜ W
[Equation]
=
[Equation]
where we used the fact that H n − 1 =
H n − 2 \
[Equation]
since H n −
[Equation]
⊆ H n −
[Equation]
Then, (A.18) yields
[Equation]
and hence, given that H n − 1 is F n -measurable, we get:
[Equation]
However, since H n − 1 and M n − 1 are both F n -measurable, we have the following estimates:
For the noise term in (A.22a), we have:
[Equation]
The term (A.22b) is where the reduction to H n − 1 kicks in; indeed, we have:
[Equation]
Finally, for the term (A.22c), we have:
[Equation]
where we used the bound ∥ v ( π ) ∥ ≤ G .
Likewise, χ n 1 H n − 1 ≤ ∥ Π ∥ B n , so
[Equation]
Thus, putting together all of the above [Equation] we obtain:
[Equation]
Going back to (A.20), we have W n − 1 > a if ˜ H n − 1 occurs, so the last term becomes
[Equation]
Our claim then follows by combining Eqs.
(A.20), (A.25), (A.26) and (A.28).
■
A.3.
Extraction of a convergent subsequence.
Our next step is to show that any realization π n of (PG) that is contained in B admits a subsequence π n k converging to π ∗ .
Proposition A.2.
Let π ∗ be a stable Nash policy, and let π n be the sequence of play generated by (PG) with step-size and policy gradient estimates such that p + ℓ b > 1 and p − ℓ σ > 1 / 2 as per (8) .
Then π n admits a subsequence π n k that converges to π ∗ with probability 1 on the event
E = (cid:84) n E n = { π n ∈ B for all n = 1 , 2 , . . . } .
Proof.
Let Q = { π n ∈ B for all n } ∩ { lim inf n ∥ π n − π ∗ ∥ > 0 } denote the event that π n is contained in B but the sequence π n does not admit a subsequence converging to π ∗ .
We will show that (cid:144) ( Q ) = 0.
Indeed, assume ad absurdum that (cid:144) ( Q ) > 0.
Hence, with probability 1 on Q , there exists some positive constant c > 0 (again, possibly random) such that ⟨ v ( π n ) , π n − π ∗ ⟩ ≤ − c < 0 for all n .
Thus, going back to (A.1), we get
[Equation]
so if we let τ n = (cid:80) nk = 1 γ k and telescope the above, we obtain the bound
[Equation]
with ξ n , χ n and ψ n given by (A.3), and M n = (cid:80) nk = 1 γ k ξ k , S n = (cid:80) nk = [-
-]1 [ γ
[Equation] χ k + γ 2 k ψ k ] defined as in (C.9).
Also, (7) readily gives
[Equation]
so, by the strong law of large numbers for martingale di ff erence sequences [18, Theorem 2.18], we conclude that M n /τ n converges to 0 with probability 1.
In a similar vein, for the submartingale S n we have
[Equation]
so, by (7) and the stated conditions for the method’s step-size and bias / noise parameters, it follows that
S n is bounded in L 1 .
Therefore, by Doob’s submartingale convergence theorem [18, Theorem 2.5], we further deduce that S n converges with probability 1 to some (finite) random variable S ∞ .
Going back to (A.30) and letting n → ∞ , the above shows that D n → −∞ with probability 1 on Q .
Since D is nonnegative by construction and (cid:144) ( Q ) > 0 by assumption, we obtain a contradiction and our proof is complete.
■
A.4.
Convergence of the energy values.
Our last auxiliary result concerns the convergence of the values of the dual energy function D .
We encode this as follows.
Proposition A.3.
If (PG) is run with assumptions as in Proposition A.1, there exists a finite random variable D ∞ such that
[Equation]
Proof.
Let E n = { π k ∈ B for all k = 1 , 2 , . . .
, n } be defined as in (A.7), and let ˜ D n = 1 E n D n .
Then, by the energy inequality (A.2) and the fact that E n + 1 ⊆ E n , we get
[Equation]
where we used the fact that that ⟨ v ( π k ) , π k − π ∗ ⟩ ≤ 0 for all k = 1 , 2 , . .
. , n if E n occurs.
Since E n is
F n -measurable, conditioning on F n and taking expectations yields
[Equation]
By our step-size assumptions, we have (cid:80) n γ 2 n (1 + B 2 n + σ 2 n ) < ∞ and (cid:80) n γ n B n < ∞ , which means that ˜
D n is an almost supermartingale with almost surely summable increments, i.e.,
[Equation]
Therefore, by Gladyshev’s lemma [39, p. 49], we conclude that ˜ D n converges almost surely to some (finite) random variable D ∞ .
Since 1 E n = 1 for all n if and only if π n ∈ B for all n , we conclude that (cid:144) (cid:0) D n converges (cid:12)(cid:12)(cid:12) π n ∈ B for all n (cid:1) =
(cid:144) ( ˜ D n converges)
= 1, and our claim follows.
■
A.5.
Putting everything together.
We are now in a position to prove Theorem 1 and Corollary 1.
Proof of Theorem 1.
Let E =
(cid:84) n E n = { π n ∈ B for all n } denote the event that π n lies in B for all n .
By Proposition A.1, if π 1 is initialized within the neighborhood U defined in (A.8), we have (cid:144)
( E | π 1 ∈ U ) ≥ 1 − a , noting also that the neighborhood U is independent of the required confidence level a .
Then, by Propositions A.2 and A.3, it follows that a ) lim inf n ∥ π n − π ∗ ∥ = 0; and b ) D n converges, both events occurring with probability 1 on the set E ∩ { π 1 ∈ U } .
We thus conclude that lim n →∞ D n = 0 and hence
[Equation]
and our proof is complete.
■
Proof of Corollary 1.
For Models 1 and 2, taking ℓ b = ∞ , ℓ σ = 0, we obtain p > 1 / 2.
Since we have that (cid:80) ∞ n = 1 γ n = ∞ , we get that p ≤ 1, i.e., p ∈ (1 / 2 , 1].
For Model 3, we have that B n = O ( ε n ) and σ n
= O (1 / √ ε n ), i.e., ℓ b = ℓ ε and ℓ σ = ℓ ε / 2.
Now, since p ≤ 1, p + ℓ b > 1 and p − ℓ σ > 1 / 2, we obtain that p ∈ (2 / 3 , 1] and (1 − p ) /
2 < r / 2 < p − 1 / 2.
■
B Rate of convergence to second-order stationary policies
We now proceed with the proof of Theorem 2, which we again restate below for convenience:
Theorem 2.
Let π ∗ be a second-order stationary policy, let B be a neighborhood of π ∗ such that (3a) holds on B , and let π n be the sequence of play generated by (PG) with step-size γ n = γ/ ( n + m ) p , p ∈ (1 / 2 , 1] , and policy gradient estimates such that p + ℓ b > 1 and p − ℓ σ > 1 / 2 as per (8) .
Then:
There exists a neighborhood U of π ∗ in Π such that, for any confidence level δ > 0 , the event
[Equation]
for all n = [Equation]
[Equation]
occurs with probability (cid:144) ( E | π 1 ∈ U ) ≥ 1 − δ if m is large enough relative to δ .
The sequence π n converges to π ∗ with probability 1 on E ; in particular, we have
[Equation]
if m is large relative to δ .
Moreover, conditioned on E and taking q = min { ℓ b p − 2 ℓ σ } we have [Equation]
[Equation]
Proof.
We will follow an approach similar to Theorem 1 for the first part of the theorem.
More precisely, let B = { π ∈ Π : ∥
π − π ∗ ∥ ≤ ϱ } be a ball of radius ϱ centered at π ∗ in Π such that (SOS) holds for all π ∈ B .
Then, for all π ∈ B \{ π ∗ } , we have ⟨ v ( π ) , π − π ∗ ⟩ ≤ − µ ∥ π − π ∗ ∥ < 0 by Proposition 1.
Hence, defining the events E n and H n as in Eq.
(A.7), and assuming that π 1 is initialized in a ball of radius √ 2 a centered at π ∗ , viz.
[Equation]
) then, by Lemma A.2 and Proposition A.1, we readily obtain that
[Equation]
which implies that
[Equation]
if m is large enough relative to δ .
For the second part, constraining Eq.
(A.2) on the event E n , we get:
[Equation]
where the last inequality comes from (SOS).
Therefore, taking expectations, we obtain:
[Equation]
where the equality in the third line comes from the fact that
[Equation]
Now, since 1 E n + 1 ≤ 1 E n
[Equation]
we further have
[Equation]
and hence, setting ¯ D n (cid:66) (cid:133) (cid:2) D n 1 E n [Equation] , we get
[Equation]
Therefore, taking γ n , B n , σ n as per the statement of the theorem and noting that the terms γ 2 n and γ 2 n B 2 n are respectively dominated by the terms γ 2 n σ 2 n and γ n B n , we obtain
[Equation]
for some C 1 , C 2 > 0, where q = min { ℓ b , p − 2 ℓ σ } , as per the theorem’s statement.
Therefore, by a straightforward modification of Chung’s lemma [9, Lemmas 2&3], [39, p. 45], we get
[Equation]
otherwise
[Equation]
Accordingly, letting n → ∞ and recalling that (cid:133)
[ D n 1 E ] ≤ (cid:133) [ D n 1 E n ] = ¯ D n
[Equation]
Then, by Fatou’s lemma [17], we obtain
[Equation]
which readily shows that (cid:133)
[lim inf n →∞ D n 1 E ] = 0.
Finally, since lim inf n →∞ D n 1 E ≥
(a.s.) and (cid:133)
[lim inf n →∞ D n 1 E ] = 0, we get that
[Equation]
Therefore, there exists a subsequence D n k that converges to 0 with probability 1 on the event E , i.e., π n k converges to π ∗ .
Hence, invoking Proposition A.3, we further deduce that D n converges to some
D ∞ with probability 1 on E , and thus, we obtain that lim n →∞ D n = 0 on E .
We thus get
[Equation]
as claimed.
For the last part of the theorem, note that
[Equation]
where we used the fact that (cid:133)
[ D n | σ ( E )] 1 E = (cid:133)
[ D n | E ] 1 E .
We thus conclude that
[Equation]
and hence
[Equation]
Proof of Corollary 2.
For Models 1 and 2, taking ℓ b = ∞ , ℓ σ =
we readily get that q = p and p > 1 / 2.
Since we require that (cid:80) ∞ n = 1 γ n = ∞ , we obtain that p ∈ (1 / 2 , 1].
Hence, for p = 1 and 2 µγ > 1 we obtain O (1 / n ) rate of convergence.
For Model 3, we have that B n = O ( ε n ) and σ n
= O (1 / √ ε n ), i.e., ℓ b = p / 2 and ℓ σ = p / 4, and, hence, we readily get that q = p / 2.
Now, since p ≤ 1, p + ℓ b > 1 and p − ℓ σ > 1 / 2, we obtain that p ∈ (2 / 3 , 1].
Hence, for p = 1 and µγ > 1, we obtain O (1 / √ n ) rate of convergence.
■
We conclude this appendix with a detailed statement and proof of the fact that, when run with perfect policy gradients (i.e., as per Model 1), the sequence of play generated by (PG) achieves a geometric convergence rate to Nash policies satisfying (SOS). The precise result is as follows:
Proposition B.1.
Let π ∗ be a second-order stationary policy, let B be a neighborhood of π ∗ such that (3a) holds on B , and let π n be the sequence of play generated by (PG) with a su ffi ciently small constant step-size γ > 0 and perfect policy gradients as per Model 1. Then, there exists a neighborhood U of π ∗ in Π and some ρ > 0 such that
[Equation]
Proof. The crucial part of the proof is the observation that, in the case of Model 1, the energy inequality (A.1) of Lemma A.1 takes the form
[Equation]
Indeed, that this is so follows from the standard development
[Equation]
where, in the penultimate line, we used the fact that π ∗ ∈ Π so ⟨ π n + 1 − ( π n + γv ( π n )) , π n + 1 − π ∗ ⟩ ≤ 0.
To proceed, consider a ball of radius R centered at π ∗ and contained in B , and assume that the step-size of (PG) is such that 2 γ sup π ∈ cl( B )
∥ v ( π ) ∥ < R .
Then, by (B.18), we conclude that if ∥ π n − π ∗ ∥ ≤ R , we will also have ∥ π n + 1 − π ∗ ∥ ≤ R , and hence, π n ∈ B for all n = 1 , 2 , . . .
if ∥ π 1 − π ∗ ∥ ≤ R .
Thus, given that π n ∈ B for all n = 1 , 2 , . . .
, Proposition 1 likewise guarantees that (3a) holds with π ← π n for all n .
Assuming now that γ is taken to be less than the Lipschitz constant of v over B , our assertion follows from standard arguments in the theory of strongly monotone operators.
■
C Rate of convergence to strict Nash policies
C.1.
Structural preliminaries.
To prove Theorem 3, we will first require some notions describing the geometry of Π near π ∗ .
Referring to [42] for a full treatment, we have:
Definition 3.
Let C be a convex set and let x ∈ C .
Then the tangent cone TC C ( x ) is defined as the set of all rays emanating from x and intersecting C to at least one other point di ff erent from x .
The polar cone PC C ( x ) to C at x is then defined PC C ( x ) =
{ y : ⟨ y, z ⟩ ≤ 0 for all z ∈ TC C ( x ) } , where y belong in the dual space of the vector space in which C is defined.
With these general definitions in hand, we proceed to characterize some further projections of
Euclidean projections on Π that will play an important role in the sequel.
For notational simplicity, we suppress the player and state indices in the statement and proof of the next lemma.
Lemma C.1.
x = proj
( y )
if and only if there exist µ ∈ (cid:146) and ν α ∈ (cid:146) + such that, for all α ∈ A , we have y α
=
x
α + µ − ν α with ν α ≥ 0 and x α ν α = 0 .
Proof.
Recall that proj( y ) =
arg min x ∈ ∆ ( A ) ∥
y − x ∥ 2 .
Our result then follows by applying the KKT conditions to this optimization problem and noting that, since the constraints are a ffi ne, the KKT conditions are su ffi cient for optimality.
Our Langragian is
[Equation]
where the set of constraints (i) of the statement of the lemma are the stationarity constraints, which in our case are ∇ L ( x , µ, ν ) = 0 ⇔ ∇ ( (cid:80)
α
[Equation]
A
[Equation]
y α − x α ) 2 ) = µ ∇ ( (cid:80) α ∈
A x
α − 1) − (cid:80) α ∈
A ν α ∇ x α , while the set of constraints (ii) of the statement of the lemmas are the complementary slackness constraints.
Note that complementary slackness implies ν α > 0 whenever α (cid:60) supp( x ), so our proof is complete.
■
Our next result is a concrete consequence of Proposition 1 which will be very useful in establishing the stability estimates required for the proof of Theorem 3.
Lemma C.2.
Let π ∗ =
( α ∗ i , s ) i ∈ N , s ∈ S be a strict Nash policy.
Then there exists a neighborhood U of π ∗ and constants
c i , s such that for each player i ∈ N and state s ∈ S , we have:
[Equation]
Proof.
Our claim is a consequence of the definition of strict Nash policies.
Specifically, from
Proposition 1 we have ⟨
[Equation]
for all
[Equation]
Let z = e i ,α i , s − e i ,α ∗ i , s , then we get that
[Equation]
where e i ,α i , s is the vector that has one only in the index and zero anywhere else.
By continuity there exists a neighborhood U ⊆
X
and c i , s > 0 for each player i ∈ N such that
[Equation]
Our final result is intimately tied to the lazy projection step in (LPG), and quantifies the relation between initializations in (cid:81) i (cid:146)
A i × S and Π .
Lemma C.3.
Let π ∗ =
( α ∗ i , s ) i ∈ N , s ∈ S , be a deterministic policy.
For each agent i ∈ N and each state s ∈ S , let y i ,α i , s − y i ,α ∗ i , s be the di ff erence of the aggregated gradients between the strategy of the equilibrium and any other strategy α ∗ i (cid:44) α
i ∈
A i .
Then for any ε > 0 such that U ε = { π : π i ,α ∗ i , s ≥ 1 − ε for all i ∈ N and s ∈ S } , there exist M i ,ε, s such that if W i , s = { y ∈ (cid:146)
A i : y i ,α i , s −
y
i ,α ∗ i , s < − M i ,ε, s } then (cid:81) i ∈ N , s ∈ S proj Π
i ( W i , s ) ⊆
U ε .
Proof.
Consider an arbitrary player i ∈ N , a state s ∈ S , and let W i ( M i ,ε, s ) be an open set as defined in the statement of the lemma.
For notational simplicity, we will drop the index s .
We will show that any M i ,ε > 1 − ε |
A i | > 0 satisfies our claim.
By using Lemma C.1 for a y i ∈ W i ( M i ,ε ) with π i = proj
( y i )
we have that
[Equation]
C.
[Equation]
with ν α i ≥ 0 and π i α i = [-
-]0 whenever ν α i > 0.
Notice that since M i ,ε > 1 − ε A
i we have that π
i [Equation] >
[Equation]
∗ i − ν [Equation] i [Equation]
or
[Equation] (C.
[Equation]
Hence, by summing over all strategies of player i we get the desired result.
■
C.2.
Proof of the main theorem.
We are now in a position to prove our main result on the rate of convergence towards strict Nash policies.
For ease of reference, we restate Theorem 3 below.
Theorem 3.
Let π n be the sequence of play under (LPG) with step-size and policy gradient estimates such that p + ℓ b > 1 and p − ℓ σ > 1 / 2 as per (8) .
If π ∗ is a deterministic Nash policy, there exists an unbounded open set W ⊆ (cid:81) i (cid:146)
A i × S of initializations such that, for any δ > 0 , we have
[Equation]
provided that γ > 0 is small enough.
Moreover, conditioned on this event, π n converges to π ∗ at a finite number of iterations, i.e., there exists some n 0 such that π n = π ∗ for all n ≥ n 0 .
Proof of Theorem 3.
We start by fixing a confidence level δ > 0 and all the parameters of the algorithm, such that all the assumptions stated in the theorem are satisfied and.
We will prove that for each agent i ∈ N , s ∈ S there exist M 1 , i , s > 0, W 1 , i , s = { y ∈ (cid:146)
A i : y i ,α
i − y
i ,α ∗ i < − M 1 , i , s for all α i ∈
A i , α i (cid:44) α ∗ i } , such that if y 1 ∈ W 1 (cid:66) (cid:81) i ∈ N , s ∈ S W 1 , i , s then the agents’ sequence of play, converge to the deterministic Nash policy, in finite number of iterations.
To simplify the notation, we will drop the indices s
and i referring to the states and agents, accordingly, and we will focus on a specific agent and a specific state.
From Lemma C.3, Lemma C.2 we have that there exist constants c , M , neighborhood U c
= { π ∈ Π : ∥ π − π ∗ ∥
≤ β } and open set W M such that
[Equation]
The first step is to prove that for an appropriate initialization for y 1 , we have y n ∈ W
( M c ) for all n = 1 , 2 , . . .
, with probability at least 1 − δ .
Assume that y k ∈ W ( M c ) for all k = 1 , . . .
, n ; then for the di ff erences of the scores at a round n + 1 between any α ∈ A and the equilibrium strategy α ∗ , we have
[Equation]
where ξ k =
( U α, k − U α ∗ , k ) and χ k = 2 ∥ b k ∥ .
Now, similarly to the proofs of Theorems 1 and 2 we will proceed to control the aggregate error terms
[Equation]
γ k ξ k and S n
[Equation]
n
[Equation]
Since (cid:133) [ ξ n | F n ] = 0, we have (cid:133)
[ R n | F n ] =
R n − 1 , so R n is a martingale; likewise, (cid:133)
[ S n | F n ] ≥ S n − 1 , so S n is a sub-martingale.
Furthermore from (7) we have:
I. (cid:133) [ ξ 2 n ] ≤ (cid:133)
[ ∥ U n ∥ 2 ] ≤ (cid:133) [ (cid:133)
[ ∥ U n ∥ 2 | F n ]] ≤ σ 2 n II.
(cid:133)
[ χ n ] = 2 (cid:133)
[ ∥ b n ∥ ] ≤ (cid:133)
[ (cid:133)
[ ∥ b n ∥ | F n ]] ≤ B n
Moreover, for any η 1 > 0, we get by Doob’s Maximal Inequality:
[Equation]
where ( a ) comes from the fact that (cid:133)
[ ξ i ξ j ] = 0 for i (cid:44) j .
Since γ n = γ/ n p , σ n = O ( n ℓ σ ) and p − ℓ σ > 1 / 2, there exists γ 1 su ffi ciently small such that if γ ≤ γ 1 then ∞
[Equation]
and so we automatically get that
[Equation]
Furthermore, notice that the term { S n } n ∈ (cid:142) is a sub-martingale, since (cid:133) [ | S n | | F n ] < ∞ and (cid:133)
[ S n + 1 | F n ] > S n , for all n .
As before, using Doob’s Maximal Inequality, we get for any η 2 > 0: (cid:144)
[Equation]
[Equation]
[Equation]
So, since p + ℓ b > 1 there exists γ 2 su ffi ciently small such that if γ ≤ γ 2 then (cid:80) nk = 1 γ k B k ≤ η 2 δ 4 which immediately implies that
[Equation]
[Equation]
By choosing γ ≤ min { γ 1 , γ 2 } we get that
[Equation]
[Equation] Notice now that by choosing M 1 > M c + η 1 + η 2 , from (C.8) we have that with probability at least 1 − δ , y α, n + 1 − y α ∗ , n + 1 < − M c , which implies that π n + 1 ∈ U c .
Defining the sequences of "good" events { E n } n ∈ (cid:142) and { E ′ n } n ∈ (cid:142) as E n (cid:66) { π k ∈ U c , ∀ k = 1 , . .
. , n } and
E ′ n (cid:66) (cid:110) sup 1 ≤ k ≤ n R k + S k ≤ η 1 + η 2 (cid:111) , accordingly, we get that E ′ n ⊆
E n for all n .
Because (cid:144) (cid:0) E ′ n (cid:1) ≥ 1 − δ , we get that (cid:144) ( E n ) ≥ 1 − δ and since { E n } n ∈
(cid:142) is a decreasing sequence converging to E (cid:66) { π n ∈ U c , ∀ n ∈ (cid:142) } , we obtain (cid:144) ( E ) ≥ 1 − δ.
i.e.,
[Equation] [Equation]
Notice that the above conclusions immediately imply convergence in finite time.
More specifically, constrained to the event E with probability at least 1 − δ , from Eq.
(C.8) we have
[Equation]
for all n = 1 , 2 , . . . .
Assume ad absurdum that there exists at least one strategy α (cid:44) α ∗ ,
α ∈
A such that lim sup n →∞ π α, n ≥ ε > 0.
for all su ffi ciently large n .
Recall also that for π ∈ U c , it holds that π α ∗ > 0 by construction.
Using Lemma C.
we get
[Equation]
Notice that the LHS of this inequality is bounded, while the RHS goes to −∞ , which is a contradiction.
Thus, with probability at least 1 − δ , π n → π ∗ as n → ∞ .
We can rewrite the previous inequality as
[Equation]
Now aggregating over all strategies, on the previous inequality, we get that
[Equation]
Thus, once (cid:80) nk = 1 γ k becomes at least (1 − M c ) / c , which occurs in finite time, the convergence is implied.
■
Proof of Corollary 3.
For Models 1 and 2, taking ℓ b = ∞ , ℓ σ =
we readily get that p > 1 / 2.
Since we require that (cid:80) ∞ n = 1 γ n = ∞ , we obtain that p ∈ (1 / 2 , 1].
For Model 3, we have that B n = O ( ε n ) and σ n
= O (1 / √ ε n ), i.e., ℓ b = r and ℓ σ = r / 2.
Now, since p ≤ 1, p + ℓ b > 1 and p − ℓ σ > 1 / 2, we obtain that p ∈ (2 / 3 , 1].
■
D Structural properties of policy gradient methods
In this appendix we will establish the required properties for the gradient of the players’ value function.
More precisely, we prove the following intermediate results:
• In Lemma D.1 we prove that in the random stopping episodic framework visitation the notion of discounted state visitation distribution is well-defined.
• In Lemma 1, we prove the conversion lemma, a standard lemma that connects a sample by visitation distribution and a random trajectory.
• In Lemma D.4, we establish di ff erent versions of Policy Gradient theorem via Q -value function for the random stopping episodic framework.
• In Lemmas D.5 and D.7, we establish the boundedness and the Lipschitz smoothness of policy gradient vector field,
i
e., v ( π ) =
( v i ( π ))
i ∈ N
where v i ( π ) = ∇ π i V i , s ( π )
For a policy profile π ∈ Π and an arbitrary initial state distribution s 0 ∼ ρ , let’s recall the definition of discounted state visitation measure / distribution as
[Equation]
To begin with, we prove formally that the above definition is well-posed for the random stopping episodic framework described above, i.e., ˜ d πρ ( s ) < ∞ , so Z πρ (cid:66) (cid:80) s ∈ S ˜
d πρ ( s ) is well-defined.
Lemma D.1.
For any s ∈ S , ˜ d πρ ( s ) < ∞ and Z πρ ≤ 1 ζ .
Proof.
For the sake of the proof, we define a new state s f , indicating that the game has stopped.
In other words, we have that (cid:144) ( s f | s , α ) =
ζ s ,α ≥ ζ > 0 for all α ∈ A , s ∈ S .
Hence, for s ∈ S we obtain:
[Equation]
■
Lemma 1.
[Conversion Lemma]
For an arbitrary state-action function f
: S × A → (cid:146) , a policy profile π and an initial state distribution s 0 ∼ ρ , we have
[Equation]
τ ∼ MDP
[Equation]
[Equation]
Proof.
(cid:133)
τ ∼ MDP (cid:20)(cid:88) T ( τ )
[Equation]
∞
[Equation]
[Equation]
where Z πρ (cid:66) (cid:133) s ∼ Unif( S ) (cid:2)
˜
d πρ ( s ) (cid:3) · | S | is well-defined by Lemma D.1.
■
A compact reformulation the aforementioned lemma is via the matrix representation of the discounted visitation distribution:
Lemma D.2 (Conversion Lemma (Matrix form)) .
For an arbitrary state-action function f
: S × A → (cid:146) and a policy profile π , we have
[Equation]
where T is a discounted visitation distribution (action-state)-matrix under poliy profile π i.e., [ T ( π )]
( α, s )
[Equation]
RowIndex
[Equation]
Proof.
By definition we have
[Equation]
Remark
[Equation]
Notice that T is a well-defined matrix.
Indeed
[Equation]
let’s us define P ( π ) as the state-action one step transition matrix:
[Equation]
Notice that P ( π ) is a substochastic matrix and therefore spectral( P ( π )) < 1 or equivalently ( I − P ( π )) −is invertible.
Thus using Neumann series we have that ( I − P
( π ))
− 1 = (cid:80) ∞ t =
P ( π ) t .
By induction, a folklore probabilistic-graph theoretic fact, we can show that (cid:80) ∞ t = 0 P ( π ) t =
T ( π ).
In order to analyze the gradient of MARL policy gradient methods, we will introduce the notions
Q , A and their per-player averages that are useful in the MDP analysis.
Definition 4.
For a state s ∈ S , a policy π and α =
( α 1 , . . .
, α N ) ∈ A , we define:
(i) The Q -value function of player i as:
[Equation]
(ii)
The advantage function of player i as:
[Equation]
We also define ¯ Q π i , adv π i to be the averaged for i -th player single MDP Q -value and advantage functions:
(i)
The averaged ¯ Q π i -value function of player i as:
[Equation]
(ii)
The averaged advantage function adv π i of player i as
[Equation]
By Remark
[Equation] we can rewrite the above notations using T [Equation] P .
Lemma D.3.
For a policy profile π , we have:
Q [Equation]
˜
[Equation]
Proof.
We separately have using Lemma D.3 and Remark 4.
[-Q π i ( s , α ) = (cid:133) τ ∼ MDP( π | s ) (cid:104)(cid:80) T ( τ ) t = 
 R i ( s t ( τ ) , α t ( τ )) | s 
-]For our first claim, a straightforward calculation gives:
[-= s , α 
-][Equation]
As for the second part of the lemma
we have:
[Equation]
■
Having defined the above notions
[Equation]
we are ready to provide equivalent forms of v ( π ) that will permit us to prove its boundedness and smoothness.
We start with the following versions of the policy gradient theorem for random stopping setting:
Lemma D.4.
For the independent gradient operator v ( π ) per player the following expressions are equal to v i ( π ) :
v i ( π ) = (cid:133)
τ ∼ MDP (cid:104)(cid:80) T ( τ ) t = 0 ∇ i (cid:0) log π i ( α i , t ( τ ) | s t ( τ )) (cid:1) ¯ Q π i ( s t ( τ ) , α i , t ( τ ))
(cid:105) 2.
v i ( π ) =
Z πρ (cid:133) s ∼ d
πρ (cid:133) α
i ∼ π i ( ·| s ) (cid:104) ∇ i (cid:0) log π i ( α i | s ) (cid:1) ¯ Q π i ( s , α i ) (cid:105) 3.
( v i ( π ))
α ◦ i , s ◦ =
∂ V i ,ρ ( π ) ∂π i ( α ◦ i | s ◦ ) =
˜
d πρ ( s ◦ ) ¯ Q π i ( s ◦ , α ◦ i ) =
Z πρ
d πρ ( s ◦ ) ¯ Q π i ( s ◦ , α ◦ i )
Proof.
Recall first that the independent gradient operator v ( π ) is given by v i ( π ) = ∇ i V i ,ρ ( π ) Accordingly
[Equation]
we will begin by showing that:
[Equation]
To that end
[Equation]
we will start with an arbitrary s
[Equation]
and by linearity of ∇ π i ( · ) and (cid:133) s 0 ∼ ρ [ · ], we will obtain the result.
Indeed, we have:
[Equation]
Thus
[Equation]
we can rewrite it as:
[Equation]
where ( a ) holds because (cid:144) ( T ( τ ) = ∞ ) =
[Equation]
and A ∞ is some limiting quantity
[Equation]
Hence
[Equation]
we readily obtain:
[Equation]
Now by Lemma [Equation] we further have [Equation]
D
[Equation]
Decoupling ∇ i per a state s ◦ and action α ◦ i , we get ∂ V i ,ρ ( π )
[Equation]
[Equation]
■
We are ready to bound the amplitude of the independent player gradient operator:
Lemma D.5.
For a given initial state distribution ρ , the independent player policy gradient operator v ( π ) is bounded.
More precisely,
[Equation]
Proof.
We start by analyzing ∥ v i ( π ) ∥ 2 using Lemma D.4.
Specifically
[Equation]
we have:
[Equation]
We thus conclude that
[Equation]
To prove the smoothness of the policy gradient operator [Equation] we have first to establish the performance lemma for our setting.
Respectively, we get
Lemma D.6 (Performance lemma) .
For any pair of policy profiles π = ( π i , π − i ) , π ′ =
( π ′ i , π ′− i ) , it holds
[Equation]
where MDP( π | ρ ) signifies that players follow π as policy profile with ρ as the initial state distribution
[Equation]
Proof.
We will initial prove the aforementioned result for an arbitrary deterministic initial state s 0 = s :
[Equation]
[Equation]
where in the last equation we recall the definition of the Advantage function and in the pre-last the equivalent definitions of Q π i ( s , α )
[Equation]
[Equation]
Applying the linearity of (cid:133) s ∼ ρ [ · ]
[Equation]
we get the desired result:
[Equation]
where the last expression comes from Lemma 1.
■
Before closing this section by proving the Lipschitz-smoothness of our operator, we describe a useful observation that will be helpful in the smoothness bounds.
Proposition D.1.
For any pair of policy profiles π = ( π i , π − i ) , π ′ =
( π ′ i , π ′− i ) and an arbitrary initial state distribution ρ and a subset M ⊆ N [Equation] it holds that:
[Equation]
where π M = ( π i )
i ∈ M and α M =
( α i ) i ∈ M , correspondingly
[Equation] Proof.
A series of direct calculations gives:
[Equation]
where d TV corresponds to the total variation distance, and the first inequality is a consequence of the triangle inequality for d TV .
■
Lemma D.7.
For a given initial state distribution ρ , the independent player policy gradient operator v ( π ) is lipschitz-smooth.
More precisely, for any pair of policy profiles π = ( π i , π − i ) , π ′ =
[Equation]
π ′ i , π ′− i ) , it holds
[Equation]
and consequently,
[Equation]
Proof.
For the proof, we will follow the approach of Zhang et al.
[57] and Agarwal et al.
Our first task is to bound the directional derivative of the i -th player’s value function.
To that end, let π, π ′ ∈ Π and pert ∈ S ×
A such that ∥ pert ∥ = 1, and consider λ -perturbed policies
[Equation]
We then have:
[Equation]
∂λ
[Equation]
∂λ
[Equation]
[Table]
[Equation]
where (D.26a) follows from Lemma D.6 and (D.26b) uses the fact that (cid:80)
α ∈
A π ( α | s )
adv π i ( s , α ) = , for all s ∈ S and the last one is derived by the definition
d πρ ( s ) (cid:66) ˜
d πρ ( s ) /
Z πρ .
By triangular inequality, the linearity of ∂ operator and Lemma D.1, we have:
[Equation]
∂λ
[Equation]
(D.27)
We will bound the following three terms separately:
[Equation]
For Term A
[Equation]
we will use Lemma D.3 in order to compute compactly the derivative:
[Equation]
i ( α ′− i | s ′ ) e s ′ ,α ′  ⊤ T ( π (cid:129) λ ) (cid:88)
α ∈ A e s ,α  +  (cid:88) s ′ ∈ S ρ ( s ′ ) (cid:88) α ′ ∈
A π (cid:129) λ
( α ′ | s ′ )
e s ′ ,α ′  ⊤ (cid:0) T ( π (cid:129) λ )
∂ P ( π (cid:129) λ ) ∂λ
T ( π (cid:129) λ ) (cid:1) (cid:88)
α ∈
A e s ,α  (D.29)
Thus for λ = 0, we get ∂ ˜
d π (cid:129) λ ρ ( s ) ∂λ (cid:12)(cid:12)(cid:12)(cid:12) λ
{+= 
+}
=  (cid:88) s ′ ∈ S
ρ ( s ′ ) (cid:88) α ′ ∈ A pert( α ′ i | s ′ ) · π − i ( α ′− i | s ′ ) e s ′ ,α ′ 
⊤
T ( π ) (cid:88)
α ∈
A e s ,α  +   (cid:88) s ′ ∈ S
ρ ( s ′ ) (cid:88)
α ′ ∈ A π ( α ′ | s ′ ) e s ′ ,α ′   ⊤ (cid:0) T ( π ) ∂ P ( π (cid:129) λ ) ∂λ (cid:12)(cid:12)(cid:12)(cid:12) λ
{+= 
 T ( π ) (cid:1) (cid:88) α ∈ A e s ,α   (D.30)+}
Notice that (cid:20) ∂
P
π (cid:129) λ ) ∂λ (cid:12)(cid:12)(cid:12)(cid:12)
λ =(cid:21) ( s ◦ ,α ◦ )
→ ( s ⋆ ,α ⋆ ) =
pert( α ⋆ i | s ⋆ ) · π − i ( α ⋆ − i | s
⋆ ) P ( s ⋆ | s ◦ , α ◦ ).
To compactify the notation let us call aux A (cid:66) (cid:104)(cid:80)
s ′ ∈ S
ρ ( s ′ ) (cid:80)
α
′ ∈
A pert( α ′ i | s ′ ) · π − i ( α ′− i | s ′ )
e s ′ ,α ′ (cid:105) , aux B (cid:66) (cid:2)(cid:80) s ′ ∈ S
ρ ( s ′ ) (cid:80) α ′ ∈
A π ( α ′ | s ′ )
e s ′ ,α ′ (cid:3) and aux C ( s ) (cid:66) (cid:80) α ∈
A e s ,α .
We thus get: Term
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
It is easy to see that ∥ aux A ∥ 1 ≤ √ A i , ∥ aux B ∥ 1 = 1. Indeed,
[Equation] [Equation]
Additionally by Conversion Lemma in Matrix form (See Lemma D.2)
[Equation]
we have that:
[Equation]
[Equation]
Similarly, for the matrix ∂ P ( π (cid:129) λ ) ∂λ (cid:12)(cid:12)(cid:12)(cid:12) λ = 0 , we have that
[Equation]
34) since ∥ pert ∥ 2 = 1.
Then, using (D.34) and (D.33) in (D.31) we get that :
Term
[Equation]
≤
[Equation]
[Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Equation]
where we used above the fact that Q function is bounded by 1 /ζ , ∥ pert ∥ = 1 and Proposition D.1 to bound the di ff erence of the policy profiles.
Moving forward, for Term B , we have:
[Equation]
[Equation]
where we used again the fact that Q function is bounded by 1 /ζ and Proposition D.1 to bound the di ff erence of the policy profiles.
Finally, for Term C , we have:
[Table]
[Equation]
D
[Equation]
[Table]
using again (D.34) and (D.33) and Proposition D
[Table]
Thus
[Table]
we are ready now to bound the gradient per player:
[Equation]
∂λ
[Equation]
where we recall that Z π (cid:129) ρ ≤ 1 ζ Since we prove it for an arbitrary perturbation vector pert for the directional derivative [Equation] for the independent player’s policy gradient it holds also that:
[Equation]
Finally for the concatenated gradient operator we get:
[Equation]
≤ (cid:115)(cid:88) i ∈ N 9 A i ζ(cid:0) (cid:88) j ∈ N (cid:112)
A j ∥
π j − π ′ j ∥ (cid:1) 2 ≤ (cid:115)(cid:88) i ∈ N 9 A i ζ(cid:88)
j ∈ N
A j (cid:88) j ∈ N ∥
π j − π ′ j ∥≤ζ(cid:115) ( (cid:88) i ∈ N
A i ) 2 ∥ π − π ′ ∥ 2 ≤ 3 A ζ 3 ∥
π − π ′ ∥ (D.40) which completes our proof.
■
Remark 5 .
In the proof above, we considered perturbations that may formally lie outside the game’s policy space.
However, it is not di ffi cult to see that for su ffi ciently small λ both V i ( π ( λ )) , ∇ V i ( π ( λ )) , Z π ( λ ) ρ are well-defined and bounded, for π (cid:129) λ ( α | s ) =
( π i + λ pert , π − i , π − i ).
In view of this, we may harmlessly assume that all functions considered above are defined in an open neighborhood of the players’ policy space.
E Statistics of R einforce
In this appendix, we prove the two fundamental properties of the R einforce Policy Gradient estimator that we stated in Lemma 4, namely:
a ) R einforce is an unbiased estimator of v ( π ).
b ) The variance of R einforce is bounded from above by O (1 / min s ∈ S ,
α i ∈ A i π i ( α i | s )) for each i ∈ N . We recall here that ∇ i denotes the gradient of the quantity in question with respect to π i , i.e., when π − i is kept fixed and only π i is varied.
For concision, we will write v i ( π ) =
∇
i
V i ,ρ ( π ) for the individual gradient of player
i ’s value function, and v ( π )
= ( v i ( π ))
i ∈ N for the ensemble thereof.
With all this said and done, we begin by restating Lemma 4 for convenience:
Lemma 4.
Suppose that each agent i ∈ N follows a stationary policy π
i ∈ Π i .
Then:
[Equation]
where κ i = min s ∈ S ,α i ∈
A i π i ( α i | s ) .
Proof.
Without loss of generality let’s assume that MDP ≡ MDP( π | ρ ) for some initial state distribution ρ .
Additionally, we denote (cid:144) π ( τ ) the induced probability of a random trajectory τ =
( s t , α t , r t ) t ≤ T ( τ ) .
[Equation]
[Equation]
where in the penultimate inequality we used the definition for the derivative of the logarithm.
We also note here that
[Equation]
For the variance of R einforce estimator we have that
[Equation]
or equivalently (cid:133)
[Equation]
∼
[Equation]
Therefore
[Equation]
we have that
[Equation]
and, after a series of – tedious but otherwise straightforward – calculations
[Equation]
we get:
[Equation]
[Equation] κ [Equation] where, to go from the first to the second inequality we used the boundeness by one of the rewards, while from the second to the third, we used Jensen’s inequality.
■
F Solution concepts
In this last appendix, we proceed to establish three important facts regarding the gradient characterization of stationary Nash policies.
More precisely, we prove the following:
• In Lemma 2, we prove the crucial property of Gradient Dominance for the multi-agent random stopping setting.
• In Lemma 3, we establish that any stationary point corresponds to Nash Equilibria.
• In Proposition 1, we prove the “drift” inequalities for all the di ff erent types of stationary points.
We begin with
the gradient dominance property of the game which we restate below for convenience
: Lemma
(Gradient dominance property) .
For any policy profile π = ( π i ) i ∈ N ∈ Π , we have that
[Equation]
for any unilateral deviation π ′ i ∈
Π i of player i ∈ N .
Proof.
We start by rewriting the LHS of (GDP) using Lemmas 1 and D.6 for π (cid:129) =
( π ′ i ; π − i ) and π (cid:130) =
( π i ; π − i ):
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Thus
[Table]
by a series of direct calculations
[Table]
we obtain: [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] (F.2) and our proof is complete
[Table]
■
Remark.
Notice that we have assumed that ˜ d π (cid:130) ρ > 0.
If this wasn’t the case we could take a trivial bound of ∞
[Table]
We now proceed to establish the link between (FOS) and (NE):
Lemma 3 (First-order stationary policies are Nash) .
A policy π ∗ =
( π [Equation] i ) i ∈ N ∈ Π is Nash if and only if it satisfies the first-order stationary condition
[Equation]
Proof.
By the definition of first-order stationarity applied to the policies π ∗ and π , it is straightforward to check that ⟨ v ( π ∗ ) , π ∗ {+− π ⟩ ≥ 
 if and only if max ¯ π i ∈ Π+}
i ⟨∇ i V i ,ρ ( π ∗ ) , π i − ¯ π ∗ i ⟩ ≤ 0.
However, by the gradient dominance property established in Lemma 2, we readily get
[Equation]
and hence
[Equation]
and our claim follows.
■
With all this in place, we are finally in a position to prove the characterization of second-order stationary and strict Nash policies that of Proposition 1.
For ease of reference, we restate the relevant claims below.
Proposition 1.
Let π ∗ =
( π ∗ i ) i ∈ N ∈ Π be a Nash policy.
Then:
a ) If π ∗ is second-order stationary, there exists some µ > 0 such that
[Equation]
b ) If π ∗ is strict, there exists some µ > 0 such that
[Equation]
Proof.
We begin with the characterization of second-order stationary policies.
To that end, let d = | S | (cid:80) i |
A i | denote the ambient dimension of (cid:81) i (cid:146)
A i × S and consider the mapping φ : (cid:146)
d × d → (cid:146) mapping H (cid:55)→ max { z ⊤ Hz : z ∈ TC( π ∗ ) , ∥ z ∥ = 1 } .
Clearly, φ is convex as the pointwise maximum of a set of linear – and hence convex – functions.
This in turn implies the continuity of φ as every convex function is continuous on the interior of its e ff ective domain.
Since π ∗ satisfies (SOS) by assumption, we have φ (Jac v ( π ∗ ))
< 0, so, by continuity and the convexity of Π , there exists some µ > 0 and a convex neighborhood U of π ∗ in Π such that φ (Jac v ( π ))
≤ − µ for all π ∈ U .
With this in mind, letting z = π − π ∗ ∈ TC( π ∗ ) for some π ∈ U , a straightforward Taylor expansion with integral remainder yields
[Equation]
and hence, setting π τ = π ∗ + τ z , we get
[Equation]
However, by (FOS), we have ⟨ v ( π ∗ ) , π − π ∗ ⟩ ≤ 0 which, combined with the above, yields ⟨ v ( π ) , π − π ∗ ⟩ ≤ − µ ∥ π − π ∗ ∥ 2 , as claimed.
For the second part of our lemma, pick some π (cid:44) π ∗ and let z =
( π − π ∗ ) /
∥ π − π ∗ ∥ , so z ∈ TC( π ∗ ) and ∥
z ∥ = 1.
Then, given that (FOS) is satisfied as a strict inequality for all π (cid:44) π ∗ , we readily get ⟨ v ( π ∗ ) , z ⟩ < 0 for all z ∈ TC( π ∗ ) with ∥ z ∥ = 1.
Thus, by the joint continuity of the function ⟨ v ( π ) , z ⟩ in π and z , there exists a compact convex neighborhood K of π ∗ in Π such that µ (cid:66) min {⟨ v ( π ) , z ⟩ : π ∈ K , z ∈ TC( π ∗ ) ,
∥ z ∥ = 1 } < 0.
Thus, letting z = ( π − π ∗ ) / ∥ π − π ∗ ∥ as above, we conclude that ⟨ v ( π ) , π − π ∗ ⟩ ≤ − µ ∥ π − π ∗ ∥ , as claimed.
■