G RADIENT PLAY IN STOCHASTIC GAMES : STATIONARY POINTS , CONVERGENCE , AND SAMPLE COMPLEXITY
Anonymous authors
Paper under double-blind review
A BSTRACT
We study the performance of the gradient play algorithm for stochastic games (SGs), where each agent tries to maximize its own total discounted reward by making decisions independently based on current state information which is shared between agents.
Policies are directly parameterized by the probability of choosing a certain action at a given state.
We show that Nash equilibria (NEs) and ﬁrst-order stationary policies are equivalent in this setting and give a local convergence rate around strict NEs.
Further, for a subclass of SGs called Markov potential games (which includes the cooperative setting with identical rewards among agents as an important special case), we design a sample-based reinforcement learning algorithm and give a non-asymptotic global convergence rate analysis for both exact gradient play and our sample-based learning algorithm.
Our result shows that the number of iterations to reach an (cid:15) -NE scales linearly, instead of exponentially, with the number of agents.
Local geometry and local stability are also considered, where we prove that strict NEs are local maxima of the total potential function and fully-mixed NEs are saddle points.
I NTRODUCTION
Multi-agent systems ﬁnd applications in a wide range of societal systems, e.g. electric grid, trafﬁc networks, smart building and smart cities etc.
Given the complexity of these systems, multi-agent reinforcement learning (MARL) has gained increasing attention in recent years (Daneshfar & Bevrani, 2010; Shalev-Shwartz et al., 2016; Vidhate & Kulkarni, 2017; Xu et al., 2020).
Among MARL algorithms, policy gradient-type methods are highly popular because of their ﬂexibility and capability to incorporate structured state and action spaces.
However, while many recent works (Zhang et al., 2018; Chen et al., 2018;
Wai et al., 2018; Li et al., 2019;
Qu et al., 2020) have studied the sample complexity of multi-agent policy gradient algorithms, due to a lack of understanding of the optimization landscape in these multi-agent learning problems, most works can only show convergence to a ﬁrst-order stationary point.
Deeper understanding of the quality of these stationary points is missing even in the simple identical-reward multi-agent RL setting.
In this paper, we examine this problem from a game-theoretic perspective.
We model the multi-agent system as a stochastic game (SG) where agents can have different reward functions, and study the dynamical behavior of ﬁrst-order (gradient-based) learning methods.
The study of SGs dates back to as early as the 1950s by Shapley (1953) with a series of followup works on developing NE-seeking algorithms, especially in the RL setting (e.g. (Littman, 1994; Bowling & Veloso, 2000;
Shoham et al., 2003; Bus¸oniu et al., 2010; Lanctot et al., 2017; Zhang et al., 2019a) and citations within).
While well-known classical algorithms for solving SGs are mostly value-based, such as Nash-Q learning (Hu & Wellman, 2003), Hyper-Q learning (Tesauro, 2003), and WoLF-PHC (Bowling & Veloso, 2001), gradient-based algorithms have also started to gain popularity in recent years due to their advantages as mentioned earlier (e.g. (Abdallah & Lesser, 2008; Zhang & Lesser, 2010; Foerster et al., 2017)).
In this work, we aim to gain a deeper understanding of the structure and quality of ﬁrst-order stationary points for these gradient-based methods, with a particular focus on answering the following questions: 1) How do the ﬁrst-order stationary points relate to the NEs of the underlying game?, 2) Do gradient-based algorithms guarantee convergence to a NE?, 3)
What is the stability of the individual NEs?
, and 4) How should agents learn using local samples from the environment?.
These questions have already been widely discussed in other settings, e.g., one-shot (stateless) ﬁniteaction games (Shapley, 1964; Crawford, 1985; Jordan, 1993; Krishna & Sj ¨ ostr ¨ om, 1998; Shamma &
Arslan, 2005; Kohlberg & Mertens, 1986; Van Damme, 1991), one-shot continuous games (Mazumdar et al., 2020), zero-sum linear quadratic (LQ) games Zhang et al.
(2019b), etc.
There are both negative and positive results depending on the settings.
For one-shot continuous games, (Mazumdar et al., 2020) proved a negative result suggesting that gradient ﬂow has stationary points (even local maxima) that are not necessarily NEs.
Conversely, Zhang et al.
(2019b) designed projected nested-gradient methods that provably converge to NEs in zero-sum LQ games.
However, much less is known in the tabular setting of SGs with ﬁnite state-action spaces.
Contributions.
In our paper, we consider the gradient play algorithm for the inﬁnite time-discounted reward SGs where an agent’s individual policy is directly parameterized by the probability of choosing an action from the agent’s own action space at a given state.
We focus on the tabular setting where state and action spaces are ﬁnite.
Through generalizing the gradient domination property in (Agarwal et al., 2020) to the multi-agent setting, we ﬁrst establish the equivalence of ﬁrst-order stationary policies and Nash equilibria.
We then show that strict NEs are locally asymptotically stable under gradient play and provide a local convergence rate analysis.
Additionally, we study the global convergence for a special class of SGs called Markov potential games (MPGs) (Gonz´alez-S´anchez & Hern´andez-Lerma, 2013; Macua et al., 2018;
Leonardos et al., 2021), which includes identical reward multi-agent RL (Tan, 1993; Claus & Boutilier, 1998; Panait &
Luke, 2005) as an important special case.
In this setting, we ﬁrst show that exact gradient play can ﬁnd an (cid:15) -Nash equilibrium within O (cid:16) |S| (cid:80) i |A
i | (cid:15)(cid:17) steps.
Then, we design a sample-based gradient play algorithm and show that it can
ﬁnd an (cid:15) -Nash equilibrium with high probability in a fullydecentralized manner using (cid:101) O (cid:16) n(cid:15) 6 poly (cid:16) 11 − γ , |S| , max i |A
i | (cid:17)(cid:17) samples, where |S| , |A
i | denote the size of the state space and action space of agent i respectively.
The convergence rate shows that the number of iterations to reach an (cid:15) -NE scales linearly with the number of agents.
In the sample-based learning, agents only need to observe the state, their own actions, and their own rewards.
The key enabler of the learning is the existence of an underlying averaged MDP for each agent when other agents’ policies are ﬁxed.
Our learning method can be viewed as a model-based policy evaluation method with respect to agents’ averaged MDPs.
This averaged MDP concept could be applied to design many other MARL algorithms, especially policy-evaluation-based methods.
We also study the local geometry around some special classes of equilibrium points, showing that strict NEs are local maxima of the total potential function and that fully mixed NEs are saddle points.
Lastly, all the algorithms studied in this paper have been numerical tested and results are provided in Appendix A.
Comparison to other MARL algorithms: For MPGs with continuous state and action spaces, there are studies about learning either the open-loop (Gonz ´ alez-S ´ anchez & Hern ´ andez-Lerma, 2013;
Zazo et al., 2016) or the closed-loop (Macua et al., 2018) NEs for MPGs.
These works generally assume full model information and solve the problem via optimal control.
There are two recent arXiv preprints (Mguni, 2020;
Leonardos et al., 2021) studying MPGs that are similar to our MPG setting.
In particular, Leonardos et al.
(2021) also studies gradient play for MPG.
Both of our papers share similar results on MPGs but the sample-based methods are designed from different perspectives.
In addition, our papers studies general SGs besides MPG.
Moreover, our concept of “averaged” MDPs could also serve as a useful tool for the design and analysis of other MARL algorithms.
Beyond these MPG works, decentralized Q-learning introduced in Arslan & Y¨uksel (2016) might be the closest to the setting considered in this paper.
They consider the identical interest case and only show asymptotic convergence to the set of NEs.
There are other recent works that also study learning for general-sum or zero-sum stochastic games.
However the settings they consider are different from our setting, for example, Daskalakis et al.
(2021) considers convergence to NE for two player zero-sum games, while Song et al.
(2021) considers convergence to correlated equilibrium for ﬁnite time horizon general-sum games.
On the other hand, Zhang et al.
; Li et al.
; Qu et al.
(2019) consider slightly different MARL settings, where agents collaboratively maximize the summation of agents’ reward with either full or partial state observation.
They also require communication between neighboring agents for a better global coordination.
P ROBLEM SETTING AND PRELIMINARIES
We consider a stochastic game (SG) M =
( N, S , A
=
A 1 × · · · × A n , P, r = ( r 1 , . . .
, r n ) , γ, ρ ) with n agents (Shapley, 1953) which is speciﬁed by an agent set N = { 1 , 2 , . . . , n } , a ﬁnite state space S , a ﬁnite action space
A i for each agent i ∈ N , a transition model P where
P ( s (cid:48) | s, a ) =
P ( s (cid:48) | s, a 1 , . . .
, a n ) is the probability of transitioning into state s (cid:48) upon taking action a :
= ( a 1 , . .
. , a n ) in state s where a i ∈
A i is action of agent i , agent
i ’s reward function r
i : S ×
A → [0 , 1] , a discount factor γ ∈ [0 , 1) , and an initial state distribution ρ over S .
A stochastic policy π : S → ∆( A ) (where ∆( A ) is the probability simplex over A ) speciﬁes a strategy in which agents choose their actions jointly based on the current state in a stochastic fashion, i.e. Pr( a t
| s t )
=
π ( a t | s t ) .
A distributed stochastic policy is a special subclass of stochastic policies, with π = π 1 × . . .
× π n , where π i : S → ∆( A i ) .
For distributed stochastic policies, each agent takes its action based on the current state s independently of other agents’ choices of actions, i.e.:
[Equation]
For notational simplicity, we deﬁne: π I ( a I | s ) :
= (cid:81) i ∈ I
π i ( a i | s ) , where I ⊆
N is an index set.
Further, we use the notation − i to denote the index set N \{ i } .
We consider direct distributed policy parameterization , where agent i ’s policy is parameterized by θ i :
[Equation]
For notational simplicity, we abbreviate π i,θ i ( a i | s ) as π θ i ( a i | s ) , and θ i, ( s,a i ) as θ s,a i .
Here θ i ∈ ∆( A i ) |S| , i.e. θ i is subject to the constraints θ s,a i ≥ 0 and (cid:80) a i ∈A i θ s,a i = 1 for all s ∈ S .
The global joint policy is given by: π θ ( a | s ) = (cid:81) ni =1 π θ i ( a i | s )
=
(cid:81) ni
=
θ s,a i .
We use X i :=
∆( A i ) |S| , X :=
X 1 × · · · × X n to denote the feasible region of θ i and θ .
Agent i ’s value function V θi : S → R , i ∈ N is deﬁned as the discounted sum of future rewards starting at state s via executing π θ , i.e.
[Equation]
where the expectation is with respect to the random trajectory τ = ( s t , a t , r i,t ) ∞ t =0 where a t ∼ π θ ( ·| s t ) , s t
+1 =
P ( ·| s t , a t ) .
We denote agent i ’s total reward starting from initial state s 0 ∼ ρ as:
[Equation]
In the game setting, Nash equilibrium is often used to characterize the performance of agents’ policies.
Deﬁnition 1.
(Nash equilibrium)
A policy θ ∗ = ( θ ∗ 1 , . .
. , θ ∗ n ) is called a Nash equilibrium (NE) if
[Equation]
The equilibrium is called a strict NE if the inequality holds strictly for all θ (cid:48)
i ∈
X i and i ∈ N .
The equilibrium is called a pure NE if θ ∗ corresponds to a deterministic policy.
The equilibrium is called a mixed NE if it is not pure.
Further, the equilibrium is called a fully mixed NE if every entry of θ ∗ is strictly positive, i.e.: θ ∗ s,a i > 0 , ∀ a i ∈
A i , ∀ s ∈ S , i ∈ N
We deﬁne the discounted state visitation distribution d θ of a policy π θ given an initial state distribution ρ as:
[Equation]
where Pr θ ( s t = s | s 0 ) is the state visitation probability that s t = s when executing π θ starting at state s 0 .
Throughout the paper, we make the following assumption on the SGs we study.
Assumption 1.
The stochastic game M satisﬁes: d θ ( s ) > 0 , ∀ s ∈ S , ∀ θ ∈ X .
Assumption 1 requires that every state is visited with positive probability, which is a standard assumption for convergence proofs in the RL literature (e.g. (Agarwal et al., 2020; Mei et al., 2020)).
Similar to centralized RL, we deﬁne agent i ’s Q -function Q θi : S ×A → R and its advantage function
A θi : S ×
A → R as:
[Equation]
‘Averaged’ Markov decision process (MDP): We further deﬁne agent i ’s ‘averaged’ Q-function Q
[Equation] → R and ‘averaged’ advantage-function A [Equation] : S
[Equation]
R as:
[Equation]
Similarly, we deﬁne agent i ’s ‘averaged’ transition probability distribution P θi : S × S
× A i
→ R , and ‘averaged’ reward r θi : S × A i → R as:
[Equation]
From its deﬁnition, the averaged Q-function satisﬁes the following Bellman equation:
Lemma 1.
Q θi satisﬁes:
[Equation]
Lemma 1 suggests that the averaged Q-function Q θi is indeed the Q-function for the MDP deﬁned on action space
A i , with r θi , P θi as its stage reward and transition probability respectively.
We deﬁne this MDP as the ‘averaged’ MDP of agent i , i.e., M θi =
( S , A i , P θi , r θi , γ, ρ ) .
The notion of an ‘averaged’ MDP will serve as an important intuition when designing the sample-based algorithm.
Note that the ‘averaged’ MDP is only well-deﬁned when the policies of the other agents θ − i are kept ﬁxed.
When this is indeed the case, agent i can be treated as an independent learner with respect to its own ‘averaged’ MDP.
Thus, various classical policy evaluation RL algorithms can then be applied.
E XACT G RADIENT P LAY FOR G ENERAL S TOCHASTIC G AMES
Under direct distributed parameterization, the gradient play algorithm is given by:
Exact Gradient Play:
[Equation]
Gradient play can be viewed as a ‘better response’ strategy, where agents update their own parameters by gradient ascent with respect to their own rewards.
A ﬁrst-order stationary point is deﬁned as such:
Deﬁnition 2.
(First-order stationary policy)
A policy θ ∗ = ( θ ∗ 1 , . .
. , θ ∗ n ) is called a ﬁrst-order stationary policy if ( θ (cid:48)
i − θ ∗ i ) (cid:62) ∇ θ i J i ( θ ∗ ) ≤ 0 , ∀ θ (cid:48) i ∈ X i , i ∈ N .
It is not hard to verify that θ ∗ is a ﬁrst-order stationary policy if and only if it is a ﬁxed point under gradient play (Equation (5)).
Comparing Deﬁnition 1 (of NE) and Deﬁnition 2, we know that NEs are ﬁrst-order stationary policies, but not necessarily vice versa.
For each agent i , ﬁrst-order stationarity does not imply that θ ∗ i is optimal among all possible θ i given θ ∗− i .
However, interestingly, we will show that NEs are equivalent to ﬁrst-order stationary policies due to a gradient domination property that we will show later.
Before that, we ﬁrst calculate the explicit form of the gradient ∇ θ
i J i .
Policy gradient theorem (Sutton et al., 1999) gives an efﬁcient formula for the gradient:
[Equation]
Applying Equation (6), the gradient ∇ θ
i
J i can be written explicitly as follows:
Lemma 2.
(Proof in Appendix D) For direct distributed parameterization (Equation (1)),
[Equation]
Gradient domination and the equivalence between NE and ﬁrst-order stationary policy.
Lemma 4.1 in Agarwal et al.
(2020) established gradient domination for centralized tabular MDP under direct parameterization.
We can show that a similar property still holds for stochastic games.
Lemma 3.
(Gradient domination, proof in Appendix E.) For direct distributed parameterization (Equation (1)), we have that for any θ =
( θ 1 , . .
, θ n ) ∈ X :
[Equation]
where (cid:13)(cid:13)(cid:13)
d θ (cid:48)
d θ (cid:13)(cid:13)(cid:13) ∞ :
= max s d θ (cid:48) ( s ) d θ ( s ) , and θ (cid:48) =
( θ (cid:48)
i , θ − i ) .
For the single-agent case ( n = 1 ), Equation (8) is consistent with the result in Agarwal et al.
(2020), i.e.: J ( θ (cid:48) )
− J ( θ ) ≤ (cid:13)(cid:13)(cid:13) d θ
[Equation]
d θ (cid:13)(cid:13)(cid:13) ∞ max θ ∈X ( θ − θ ) (cid:62) ∇ J ( θ ) .
However, when there are multiple agents, the condition is much weaker because the inequality requires θ − i to be ﬁxed.
When n = 1 , gradient domination rules out the existence of stationary points that are not global optima.
For the multi-agent case, the property can no longer guarantee the equivalence between ﬁrst-order stationarity and global optimality; instead, it links the stationary points with NEs as shown in the next theorem whose proof is in Appendix E.
Theorem 1.
Under Assumption 1, ﬁrst-order stationary policies and NEs are equivalent.
Local convergence for strict NEs Although the equivalence of NEs and stationary points under gradient play has been established, it is in fact difﬁcult to show that gradient play converges to these stationary points.
Even in the simpler static (stateless) game setup, gradient play might fail to converge (Shapley, 1964; Crawford, 1985; Jordan, 1993; Krishna & Sj ¨ ostr ¨ om, 1998).
One major difﬁculty is that the vector ﬁeld {∇ θ i J i ( θ ) } ni =1 is not a conservative vector ﬁeld.
Accordingly, its dynamics may display complicated behavior.
Thus, as a preliminary study, instead of looking at global convergence, we focus on the local convergence and restrict our study to a special subset of
NEs - the strict NEs.
We begin by giving the following characterization of strict NEs:
Lemma 4.
Given a stochastic game M , any strict NE θ ∗ is pure, meaning that for each i and s , there exist one a ∗ i ( s ) such that θ ∗ s,a i = 1 { a i = a ∗ i ( s ) } .
Additionally,
[Equation]
Based on this lemma, we deﬁne the following for studying the local convergence of a strict NE θ ∗ :
[Equation]
Theorem 2.
(Local ﬁnite time convergence around strict NE) Deﬁne the metric of policy parameters as: D ( θ || θ (cid:48) ) :
= max 1 ≤ i ≤ n max s ∈S (cid:107) θ i,s − θ (cid:48)
i,s (cid:107) 1 , where (cid:107) · (cid:107) 1 denote the (cid:96) 1 - norm.
Suppose θ ∗ is a strict Nash equilibrium, then for any θ (0) such that D ( θ (0)
||
θ ∗ )
≤ ∆ θ ∗ (1 − γ )8 n |S| ( (cid:80) ni
=1 |A
i | ) , running gradient play (Equation (5)) will guarantee D ( θ ( t +1)
||
θ ∗ ) ≤ max (cid:110) D ( θ ( t )
|| θ ∗ ) − η ∆ θ ∗ 2 , 0 (cid:111) , which means that gradient play is going to converge within (cid:100) 2 D ( θ (0)
||
θ ∗ )
η ∆ θ ∗ (cid:101) steps.
Proofs of Lemma 4 and Theorem 2 are provided in
Appendix F.
The convergence only requires a ﬁnite number of steps and the stepsize η can be chosen arbitrarily large so that exact convergence can happen in even just one step.
However, the caveat is that we need to assume that the initial policy is sufﬁciently close to θ ∗ .
For numerical stability considerations, one should pick reasonable stepsizes to run the algorithm to accommodate random initializations.
Theorem 2 also shows that the radius of region of attraction for strict NEs is at least
[Equation]
| ) , and thus θ ∗ with a larger ∆ θ ∗ , i.e., a larger value gap between the optimal action and other actions, will have a larger region of attraction.
We would like to further remark that Theorem 2 only focuses on the local convergence property, the way to interpret the theorem is that, if there exists a strict NE, then it is locally asymptotic stable under gradient play.
However, it does not claim to solve the global
G RADIENT PLAY FOR M ARKOV POTENTIAL GAMES
We have discussed that the main problem for the global convergence of gradient play for general
SGs is that the vector ﬁeld {∇ θ i J i ( θ ) } ni =1 is not conservative.
Thus, in this section, we restrict our analysis to a special subclass where the vector ﬁeld is conservative, which in turn enjoys global convergence.
This subclass is generally referred to as a Markov potential game (MPG) in the literature.
Deﬁnition 3.
(Markov potential game (Macua et al., 2018))
A stochastic game M is called a Markov potential game if there exists a potential function φ : S ×
A 1 × · · · × A n → R such that for any agent i and any pair of policy parameters ( θ (cid:48)
i , θ − i ) , ( θ i , θ − i ) :
[Equation]
As shown in the deﬁnition, the condition of a MPG is admittedly rather strong and difﬁcult to verify for general SGs.
Macua et al.
; Gonz ´ alez-S ´ anchez & Hern ´ andez-Lerma (2013) found that continuous MPGs can model applications such as the great ﬁsh war (Levhari & Mirman, 1980), the stochastic lake game (Dechert & O’Donnell, 2006), medium access control (Macua et al., 2018) etc.
There are also efforts attempting to identify conditions such that a SG is a MPG, e.g., Macua et al.
Leonardos et al.
(2021); Mguni (2020).
In Appendix B, we provide a more detailed discussion on MPGs, including a necessary condition (Lemma 5) of MPG, counterexamples of stage-wise potential games that are not MPG, sufﬁcient conditions for a SG to be a MPG, and application examples of MPG.
Nevertheless, identifying sufﬁcient and necessary conditions and broadening the applications of MPG are important furture directions.
Given a policy θ , we deﬁne the ‘total potential function’ Φ( θ ) : =
E s 0 ∼ ρ
[Equation]
[Equation]
∞
[Equation]
for a MPG.
The following proposition guarantees a MPG has at least one NE and it is a pure NE.
Proposition 1.
(Proof in Appendix G)
For a Markov potential game, there is at least one global maximum θ ∗ of the total potential function Φ , i.e.: θ ∗ ∈ arg max θ ∈X Φ( θ ) that is a pure NE.
From the deﬁnition of the total potential function we obtain the following relationship
[Equation]
[Equation]
Thus,
[Equation]
which means that gradient play (Equation (5)) is equivalent to running projected gradient ascent with respect to the total potential function Φ , i.e.: [Equation]
X [Equation] ( t ) i
[Equation] To measure the convergence to a NE, we deﬁne an (cid:15) -Nash equilibrium as follows:
Deﬁnition 4.
( (cid:15) -Nash equilibrium) Deﬁne the ‘NE-gap’ of a policy θ as:
[Equation]
A policy θ is an (cid:15) -Nash equilibrium if: NE-gap ( θ ) ≤ (cid:15).
We further assume that the MPG satisﬁes the following assumption.
Assumption 2.
For θ ∈ X , the total potential function Φ( θ ) is bounded by: Φ min ≤ Φ( θ ) ≤ Φ max .
4. E XACT GRADIENT PLAY - GLOBAL CONVERGENCE AND LOCAL GEOMETRY
In this section, we ﬁrst focus on the global convergence for exact gradient play (5), where the gradient ∇ θ
i
J i can be calculated exactly by agent i .
The convergence result is given as follows:
Theorem 3.
(Global convergence to Nash equilibria, proof in Appendix H.)
Given a MPG with potential function φ ( s, a ) , suppose the total potential function Φ satisﬁes Assumption 2.
Then with stepsize η =
[Equation]
|
[Equation] | , θ [Equation] t ) asymptotically converge to a NE under gradient play (Equation (5)), i.e., lim t →∞ NE-gap ( θ ( t ) )
= 0 .
Further, we have:
[Equation]
where M := max
[Equation]
[Equation]
(cid:13)(cid:13)(cid:13) ∞ (by Assumption 1, we know that this quantity is well-deﬁned).
The factor M is also known as the distribution mismatch coefﬁcient that characterizes how the state visitation varies with the policies.
Given an initial state distribution ρ that has positive measure on every state, M can be at least bounded by M ≤ 11 − γ max θ [Equation] d [Equation]
ρ [Equation] ∞ ≤ 11 − γ 1 min s ρ ( s ) .
Also note that
Inequality (12) on the average term 1 T (cid:80) 1 ≤ t ≤ T NE-gap ( θ ( t ) ) 2 could be translated to a constant probability guarantee on single NE-gap ( θ ( t ) ) .
For instance, if we randomly pick one θ ( t ) from 1 ≤ t ≤ T , then it guarantees that NE-gap ( θ ( t ) )
2 ≤ 3 · (cid:15) 2 with probability at least 23 .
As a comparison with centralized learning, if we parameterize the policy in a centralized way, the size of the action space will be |A| =
(cid:81) n i =1 |A
i | and the projected gradient ascent would need O (cid:16)
[Table]
(cid:17) steps to ﬁnd an (cid:15) -optimal policy (Agarwal et al., 2020); whereas we only need O (cid:16) |S|
[Table]
steps to ﬁnd an (cid:15) -NE, which scales linearly with n .
However, centralized parameterization can provably ﬁnd a global optimum, while distributed parameterization can only ﬁnd a NE.
Though gradient play is guaranteed to converge to a NE, the exact NE which it converges to is uncertain, and depends on the initial point as well as the local geometry around the various NEs.
As a preliminary study, we have the following characterization for two special types of NEs.
More future investigation is needed for general NEs.
Theorem 4.
For Markov potential game with Φ min <
Φ max (i.e., Φ is not a constant function):
• A strict NE θ ∗ is equivalent to a strict local maximum of the total potential function Φ , i
e.: ∃ δ, s.t. ∀ θ (cid:54) =
θ ∗ that satisﬁes (cid:107) θ − θ ∗ (cid:107) ≤ δ, θ ∈ X , we have Φ( θ ) <
Φ( θ ∗ ) .
• Any fully mixed NE θ ∗ is a saddle point with regard to the total potential function Φ , i
e.: ∀ δ > 0 , ∃ θ, s.t.
(cid:107) θ − θ ∗ (cid:107) ≤ δ and Φ( θ ) >
Φ( θ ∗ ) .
Theorem 4 implies that strict NEs are asymptotically stable under ﬁrst-order methods such as gradient play; while the fully mixed NEs are not stable under gradient play.
We remark that the conclusion about strict NE in Theorem 4 does not hold for settings other than tabular MPG; for instance, for continuous games, one can use quadratic functions to construct simple counterexamples (Mazumdar et al., 2020).
Also, similar to the remark after Theorem 2, this theorem focuses on the local geometry of the NEs but does not claim the global existence or convergence of either strict NEs or fully mixed
NEs.
4. S AMPLE - BASED LEARNING : ALGORITHM DESIGN AND SAMPLE COMPLEXITY
In this section, we no longer assume access to the exact gradient, but instead estimate it via samples.
Throughout the section, we make the following additional assumption on MPGs:
Assumption 3.
( ( τ, σ S ) -Sufﬁcient exploration on states)
There exist a positive integer τ and a σ S ∈ (0 , 1) such that for any policy θ and any initial state-action pair ( s, a i ) , ∀ i , we have
[Equation]
It says that every state has a positive probability of being visited after some time.
This assumption is common in proving ﬁnite time convergence (e.g. (Qu et al., 2019; Srikant & Ying, 2019)).
We further introduction the state transition probability under θ P θ S : S × S → R as:
[Equation]
We consider fully decentralized learning, where agent i ’s observation only includes state s t , its own action a i,t , and its own reward r i,t :
=
r i ( s t , a t ) at time t .
Such fully decentralized learning is plausible due to the fact that when θ − i is ﬁxed , agent i can be treated as an independent learner with the underlying MDP being the ‘averaged’ MDP described in Section 2.
With this key observation, we design a two-timescale ‘model-based’ on-policy learning algorithm, where agents perform policy evaluation in the inner loop and gradient ascent at the outer loop.
The algorithm is provided in
Algorithm 1.
Roughly, it consists of three main steps: 1) (Inner loop) Estimate the averaged transition probability and reward using on-policy samples P θi , r θi , P θ S . 2)
(Inner loop) Calculate averaged
Q-function Q θi and
discounted state visitation distribution d θ , and compute the estimated gradient accordingly, 3) (Outer loop)
Running projected gradient ascent with estimated gradients.
Before discussing our algorithm in more detail, we highlight that the idea of using the “averaged” MDP can be used to design other learning methods including model-free methods, e.g., using the temporal difference methods to perform policy evaluation.
One caveat is that the “averaged” MDP is only welldeﬁned when all the other agents use ﬁxed policies.
This makes it difﬁcult to extend the two-timescale framework to single-timescale settings, which is an interesting future direction.
Algorithm 1 Sample-based
Require: learning rate η , greedy parameter α , sample trajectory length T J , total iteration steps T G
For each agent i for k = 0 , 1 . . .
, T G − 1 do for t = 0 , 1 , . . .
, T J do
Implement policy θ ( k ) and collect trajectory D ( k ) i : D ( k ) i ←D ( k ) i ∪{ s t , a i,t , r i,t } , a i,t ∼ π θ ( k ) i ( ·| s ) end for
Estimate (cid:99) P θi , (cid:98) r θi , (cid:99) P θ S , (cid:100)
M θi by Equation (14), Equation (15), Equation (16) respectively.
Calculate (cid:99) Q θi , (cid:98) d θ by Equation (17), Equation (18) respectively.
Estimate the gradient by Equation (19):
Run projected gradient ascent as in Equation (20) end for
Step 1: empirical estimation of P θi , r θi , P θ S : Given a sequence { s t , a i,t , r i,t } T J t =0 generated by a policy θ :=
( θ i , θ − i ) , the empirical estimation (cid:99) P θi of P θi is given by:
[Equation]
Here we separately treat the special case where the state and action pair is not visited through the whole trajectory, i.e., (cid:80) T J − 1 t =1 1 { s t = s, a i,t = a i } = 0 to make (cid:99) P θ i well-deﬁned.
Similarly, the estimates (cid:98) r θi , (cid:99) P θ S of r θi , P θ S are given by:
[Equation]
Step 2: estimation of Q θi , d θ : We slightly abuse notation and use Q θi , r θi ∈ R |S||A i | to also denote the vectors corresponding to the averaged Q-function and reward function of agent i .
Similarly, ρ, d θ ∈ R |S| are used to denote the vectors for ρ ( s ) and d θ ( s ) .
Deﬁne M θi ∈ R |S||A i |×|S||A i | :
[Equation]
i [Equation] Then from Lemma 1, Q θi is given by: ( I − γM
[Equation]
The estimated averaged Q-function (cid:99) Q θi is given by:
[Equation]
, where [Equation] [Equation] Similarly, from Equation (2)
[Equation]
we have that d θ and (cid:98) d θ are given by (derivation see Appendix C): [Equation] [Equation] Then accordingly
[Equation]
the estimated gradient is computed as:
[Equation]
Step 3:
Projected gradient ascent onto the set of α -greedy policies: Let U n = [Equation]
1 n [Equation] n
[Equation]
∆
[Equation]
n ) denote the n dimensional uniform distribution. Deﬁne ∆ α ( n
[Equation]
+ αU n } .
We use X
[Equation]
denote the set of the α -greedy policies for θ i and θ respectively.
Every step after doing gradient ascent, the parameter θ will further be projected onto X α , i.e.:
[Equation]
The reason of projecting onto X α instead of X is to make sure that every action has positive possibility of being selected in order to get a relatively accurate estimation of averaged Q -function.
Intuitively, a larger α introduces a larger additional error in the NE-gap; however, a smaller α requires more samples to estimate the gradient.
Thus the choice of α is the tradeoff between the two effects.
Theorem 5.
(Sample complexity) Assume that the MPG satisﬁes Assumption 3.
Let M :=
max θ,θ (cid:48) ∈X (cid:13)(cid:13)(cid:13)
d θ d θ (cid:48) (cid:13)(cid:13)(cid:13) ∞ .
In Algorithm 1, for η ≤ (1 − γ ) 3 4 (cid:80) i |A i | , α = (1 − γ ) (cid:15) 6 M , and
T J
[Equation]
with probability at least 1 − δ , we have that: 1 T G (cid:80) T G k =1 NE-gap ( θ [Equation] k ) ) 2 ≤ (cid:15)
That is, the algorithm can ﬁnd an (cid:15) -NE with probability at least 1 − δ with [Equation] O [Equation]
poly
[Equation]
(21) samples, where ˜ O hides log factors.
Comparison with centralized learning: The best known sample complexity bound for singleagent/centralized MDP is ˜ O (cid:16) |S||A| (1 − γ )
(cid:15) 2 (cid:17) (Sidford et al., 2018).
Compared with Equation (21), the centralized bound scales better with respect to (cid:15), |S| , |A
i | , 1 1 − γ .
However, as argued in the previous subsection, the total action space |A| =
(cid:81) ni
=1 |A
i | in the centralized bound scales exponentially with the number of agent n , while our complexity bound only scales linearly.
Here, we brieﬂy state the fundamental difﬁculties of learning in the SG setting compared with centralized learning, which also explains why our bound scales worse with respect to the factors (cid:15), |S| , |A
i | , 11 − γ .
1) Firstly, the optimization landscape in the SG setting is more complicated.
For centralized learning, the gradient domination property is stronger and accelerated gradient methods (e.g. via natural policy gradient or entropy regularization) can speed up the convergence of exact gradient from O ( 1 (cid:15) 2 ) to
O ( 1 (cid:15) )
(Agarwal et al., 2020), or even O (log( 1 (cid:15) ))
(Mei et al., 2020).
In contrast, for multi-agent settings, due to the more complicated optimization landscape, these methods can no longer improve the dependency on (cid:15) , which thus makes the outer loop complexity T G larger.
Secondly, the behavior of other agents makes the environment non-stationary, i.e., the averaged Q-function Q θi as well as the averaged transition probability distribution P θi depends on the policy of other agent θ − i .
Thus, unlike centralized learning, where the state transition probability matrix can be estimated in an off-policy or even ofﬂine manner, i.e. using data samples from different policies, P θi can only be estimated in a online manner, using samples generated by exactly the same policy θ , which increases the inner loop complexity T J . 3)
Thirdly, the complicated interactions amongst agents necessitate more care during the learning process.
Algorithms designed for centralized learning that achieve near-optimal sample complexity are generally Q-learning type algorithms.
However, in SGs, it can be shown that having each agent maximize its own averaged Q-function may actually lead to non-convergent behavior.
Thus, we need to consider algorithms that update in a less aggressive manner, e.g. soft Q-learning, or policy gradient (which is considered in this paper).
Numerical Results.
Due to the space limit, numerical performance for algorithms studied in this paper are deferred to Appendix A.
C ONCLUSION AND D ISCUSSION
This paper studies the optimization landscape and convergence of gradient play for SGs.
For general
SGs, we establish some local convergence results; for MPGs, we establish the global convergence and design a sample-based method.
There are many future directions.
Firstly, the assumption of
MPGs is relatively strong compared with the notion of potential games in the one-shot setup, which might restrict its application to broader settings.
More effort would be needed to come up with other special types of SGs which facilitate learning.
It would also be meaningful to combine the insight from learning in stochastic games to investigate real-life applications, such as dynamic congestion and routing.
Secondly, the sample-based learning algorithm proposed in this paper is only one choice.
Other methods, such as actor-critic, natural policy gradient, Gauss-Newton methods, could also be considered, which might improve the sample complexity.
We note in passing that it is an interesting and important question to characterize the fundamental complexity of MARL.
Thirdly, this paper only considers direct policy parameterization.
To broaden the application of MARL and to strengthen the results, other types of parameterization such as softmax parameterization and general nonlinear parameterization should be investigated. EXAMPLES [Equation] s
[Equation] [Equation] [Equation] Game multi-stage prisoner’s dilemma
The ﬁrst example — multi-stage prisoner’s dilemma model(Arslan & Y ¨ uksel, 2016) — studies exact gradient play for general SG settings.
It is a 2 -agent
MDP, with S =
A 1 =
A 2 = { 1 , 2 } .
Assume that the reward for each agent r i ( s, a 1 , a 2 ) , i ∈ { 1 , 2 } is independent of state s and is given by Table 1.
The state transition probability is determined by agents’ previous actions:
[Equation]
Here action a i = 1 means that agent
i choose to cooperate and a i = 2 means betray .
The state s serves as a noisy indicator, with accuracy 1 − (cid:15) , of whether both agents cooperated ( s t = 1 ) or not ( s t = 2 ) in the previous stage t − 1 .
The single-stage game corresponds to the famous Prisoner’s Dilemma , and it is well-known that there is a unique NE ( a 1 , a 2 ) =
(2 , 2) , where both agent decide to betray.
The dilemma arises from the fact that there exists a joint non-NE strategy (1 , 1) such that both players obtain a higher reward than what they get under the NE.
However, in the multi-stage case, the introduction of an additional state s allows agents to make decisions based on whether they have cooperated before.
Intuitively, given that both agents cooperated at the previous stage, it is more beneﬁcial to keep this cooperation rather than destroy it by betraying.
It turns out that cooperation can be achieved in this manner given that the two agents are patient (i.e., γ is close to 1 ) and that the indicator s is accurate enough (i.e.
(cid:15) is close to 0 ).
Apart from the fully betray strategy, where both agents will betray regardless of s , there is another strict NE θ ∗ that is θ ∗ s =1 ,a i =1 = 1 , θ ∗ s =2 ,a i =1 = 0 , where agents will cooperate given that they have cooperated in previous stage, and betray otherwise.
We simulate gradient play for this model and mainly focus on the convergence to the cooperative equilibrium θ ∗ .
The initial policy is set as: θ (0) s =1 ,a i =1 = 1 − 0 .
4 δ
i , θ (0) s =2 ,a i =1 = 0 , where the δ i ’s are uniformly sampled from [0 , 1] .
The initialization implies that at the beginning, both agents are willing to cooperate to some extent given that they cooperated at the previous stage.
Figure 1 shows a trial converging to the NE starting from a randomly initialized policy.
The size of the region of attraction for θ ∗ can be reﬂected by the ratio of convergence ( # Trials that converge to θ ∗ # Total number of trials ) for multiple trials with different initial points.
An empirical estimate of the volume of the region is the convergence ratio times the volume of the uniform sampling area; the larger the ratio, the larger the region of attraction.
Table 3 demonstrates the change of the ratio with regard to different values of indicator error (cid:15) .
Intuitively speaking, the more accurately the state s represents the cooperation situation of the agents, the less incentive agents will have for betraying when observing s = 1 , that is, the larger ∆ θ ∗ ( s =1) will become, and thus the larger the convergence ratio will be.
This intuition matches the simulation result as well as the theoretical guarantees on the local convergence around a strict NE in
Theorem 2.
[Table] [Table] [Table] [Table]
4 ±
[Table] [Table] Here we ﬁx Convergence ratio is calculated by
[Table] we calculate by Game 2: coordination game Our second numerical example studies the empirical performance of exact gradient play for an identical-reward game which is a special class of Markov potential game.
Consider a 2 -agent identical reward coordination game problem with state space S
= S 1 × S 2 and action space
A
=
A 1 × A 2 , where S 1 = S 2 =
A 1 =
A 2 = { 1 , 2 } .
The state transition probability is given by:
[Equation]
The reward table is given by Table 2, where agents will only be rewarded if they are in the same state, and state 1 has a higher reward than state 2 .
Coordination games can be used to model the network effect in economics, where an agent reaps beneﬁts from being in the same network as other agents.
For this speciﬁc example, there are two networks with different utilities.
Agents can observe the occupancy of each network, and take actions to join one of the networks based on the observation.
There is at least one fully-mixed NE where both agents join network 1 with probability
[Equation]
) regardless of the current occupancy of networks, and there are 13 different strict NEs that can be veriﬁed numerically (computation of the NEs as well as detailed settings can be found in Appendix
N).
Figure 2 shows a gradient play trajectory whose initial point lies in a close neighborhood of the mixed NE.
As the algorithm progresses, we see that the trajectory in Figure 2 diverges from the mixed-NE, indicating that the fully-mixed NE is indeed a saddle point.
This corroborates our ﬁnding in Theorem 4.
Figure 3 shows the evolution of total reward J ( θ different random initial points θ (0) .
Different initial points converge to one of 13 different strict NEs each with a different total reward (some strict NEs with relatively small region of attraction are omitted in the ﬁgure).
We see the total reward is monotonically increasing for each initial point, which makes sense since gradient play runs projected gradient ascent on the total reward function J .
While the total rewards are different, as shown in Figure 4, we see that the NE-gap of each trajectory (corresponding to same initial points in Figure 3) converges to 0 .
This suggests that the algorithm is indeed able to converge to a NE.
Notice that the NE-gaps do not decrease monotonically.
[Figure] [Figure] Game 3: state-based coordination game Our third numerical example studies the empirical performance of the sample-based learning algorithm, Algorithm 1.
Here we consider a generalization of coordination game (Game 2) where the two players now try to coordinate on a 2D grid.
The two-player state-based coordination game on a 3 × 3 grid is deﬁned as follows: the state space is given by S = S 1 × S 2 , S 1 = S 2 =
S x × S y
=
{ x a , x b , x c } × { y a , y b , y c } , action space is given by
A =
A 1 × A 2 , A 1 =
A 2 = { Stay , Left , Right , Up , Down } , i.e., agent can choose to stay at current grid or move left/right/up/down to its neighboring grids.
We assume that there is random noise during the transition; for example, if agent 1 is staying at the middle grid ( x b , y b ) , then:
P ( s (cid:48) 1 ,x = x a | s 1 ,x = x b , a 1 = Stay ) = P ( s (cid:48) 1 ,x = x c | s 1 ,x = x b , a 1 = Stay ) =
(cid:15),
P ( s (cid:48) 1 ,x = x b | s 1 ,x = x b , a 1 = Stay ) =
1 − 2 (cid:15),
P ( s (cid:48) 1 ,y =
y a | s 1 ,y = x b , a 1 = Stay ) =
P ( s (cid:48) 1 ,y =
y c | s 1 ,y = x b , a 1 = Stay ) =
(cid:15),
P ( s (cid:48) 1 ,y =
y b | s (cid:48) 1 ,y = y b , a 1 = Stay ) = 1 − 2 (cid:15).
The reward is given by: r ( s 1 , s 2 ) =
{+1 { s 1 = s 
+}
= { x a , y a } or { x c , y c }} , i.e. the two agents are only rewarded if they stay at the upper-left or lower-right corner at the same time.
For numerical simulation, we take T G = 300 , T J = 10000 , α = 0 .
1 , η = 10 , (cid:15) = 0 .
1 ; the numerical results are as displayed in Figure 6 - 8.
Figure 6 shows that total reward increases as the number of iterations increase, and Figure 8 shows that the NE-gap converges to a value close to zero.
However, because we project the policy to the α -greedy set X α , the NE-gap cannot converge to exactly zero.
Figure 7 visualizes the discounted state visitation distribution.
To make the visualization more intuitive, we look at the marginalized discounted state visitation distribution d xθ , d yθ deﬁned below:
[Equation]
t
[Equation]
[Equation]
d yθ ( s 1 ,y , s 2 ,y )=
(cid:88) s 1 ,x ,s 2 ,x d θ ( s 1 ,x , s 1 ,y , s 2 ,x , s 2 ,y ) = E s 0 ∼ ρ (1 − γ ) ∞ (cid:88) t = γ t Pr( s 1 ,y,t = s 1 ,y , s 2 ,y,t = s 2 ,y | s 0 , π θ )
From Figure 7 we can see that most of the probability measure concentrates on { ( x a , x a ) , ( x c , x c ) } , { ( y a , y a ) , ( y c , y c ) } , indicating that the two agents are able to coordinate most of the time.
B M ORE ABOUT M ARKOV POTENTIAL GAMES
This section is dedicated to a more thorough understanding of Markov potential games, which includes necessary or sufﬁcient conditions for MPG and a few (counter)examples.
B.1. A necessary condition and counterexamples
Deﬁnition 5.
(Monderer & Shapley (1996)) Deﬁne the path in the parameter space as τ = ( θ (0) , θ (1) , . . .
, θ ( N ) ) , where θ t , θ t +1 differ only one component i t , i.e. θ ( t +1) =
( θ ( t +1)
i t , θ ( t ) − i t ) .
A closed path is a path such that θ (0)
=
θ ( N ) .
Deﬁne:
[Equation]
The following theorem is a direct generalization of Theorem 2.8 in Monderer & Shapley (1996) to
MPG setting:
Lemma 5.
For Markov potential games, I ( τ ) = 0 for any ﬁnite closed path τ .
Proof.
The proof is quite straightforward from the deﬁnition of MPG
I [Equation]
N (cid:88)
[Equation]
Although the proof of Lemma 5 is straightforward, it serves as a useful tool in proving that a game is not a MPG.
For example, applying the theorem we can get that the following conditions are not sufﬁcient conditions for MPG.
Proposition 2.
None of the following conditions on a SG necessarily imply that it is a MPG:
(1) There exists φ ( s, a ) such that at each s , r i ( s, a (cid:48)
i , a − i ) − r i ( s, a i , a − i ) =
φ ( s, a (cid:48)
i , a − i ) − φ ( s, a i , a − i ) ; (2) There exists φ ( s, a ) such that for every s, ˆ s , r i ( s, a (cid:48)
i , a − i ) − r i (ˆ s, a i , a − i ) =
φ ( s, a (cid:48)
i , a − i ) − φ
(ˆ s, a i , a − i ) ;
(3) Rewards r
i are independent of s , and they have a potential function,
i
e., r i ( a i , a − i ) − r
i ( a (cid:48) i , a − i ) =
φ ( a i , a − i ) − φ ( a (cid:48) i , a − i ) .
Proof.
(of Proposition 2)
A simple counterexample showing that the conditions in Equation (2) are not sufﬁcient is the multi-stage prisoner’s dilemma (Game 1) introduced in the numerical section (Appendix A).
Since the reward table for multi-stage prisoner’s dilemma is the same as the one-shot prisoner’s dilemma (which is known to be a potential game), Game 1 satisﬁes condition (3) in
Proposition 2, which implies condition (2), which in turn implies condition (1).
In the following we are going to use Lemma 5 to show that Game 1 is not a MPG.
We deﬁne the following individual policies:
[Equation]
Let:
[Equation]
and deﬁne a path τ by:
[Equation]
For the sake of easy calculation, we set (cid:15) = 0 and set initial state as s 0 = 1 in Game 1.
In this example, it is not hard to see that J i t ( θ ( t ) )
− J i t ( θ ( t − 1) ) >
for each t ∈ { 1 , 2 , 3 , 4 } , implying that
I ( τ ) > 0 .
This indicates that although Game 1 satisﬁes condition (3) (as well as conditions (1) and (2)), it is still not a MPG.
B.2. A sufﬁcient condition
Proposition 2 suggests that MPG is a quite restrictive assumption.
Even if the reward table for a
SG is the same as the reward table of a one-shot potential game, the SG may still not be a MPG.
Nevertheless, we can show that the following condition is sufﬁcient for a stochastic game to be a
MPG:
Lemma 6.
A stochastic game is a MPG if condition (1) in Proposition 2 is satisﬁed and that
P ( s (cid:48) | s, a ) =
P ( s (cid:48) | s ) .
Proof.
P ( s (cid:48) | s, a ) =
P ( s (cid:48) | s ) implies that the discounted state visitation distribution d θ does not depend on θ , and thus we denote it as d ( s ) instead.
Condition (1) implies that φ ( s, a i , a − i ) − r
i ( s, a i , a − i ) only depends on s and a − i but not a i , and so we denote the difference as δ i ( s, a − i ) , i.e.,
[Equation]
The total reward of agent i can be written as:
[Equation]
Similarly, total potential function can be written as:
[Equation]
Thus,
[Equation]
[Equation]
which does not depend on parameter θ i , i.e.,
[Equation]
which completes the proof.
B.3. MPG with local states and an application example
From Proposition 2, we see that it is difﬁcult for a SG to be a MPG even if the game is a potential game at each state.
Lemma 6 only presents a very special case where the action does not affect the state, meaning that this MPG is merely a collection of potential games.
To provide a MPG which is beyond the identical-interest case and the case in Lemma 6, inspired by the setting in Qu et al.
and Macua et al.
(2018), here we consider a special multi-agent setting where S = S 1 × · · · × S n and S i is the local state space of agent i .
In addition, the transition probability takes the decomposed form, P ( s (cid:48) | s, a ) =
(cid:81)
ni
=1 P ( s (cid:48)
i | s i , a i ) .
The rest of the SG setting is the same as the SG in Section 2.
Deviating slightly from the main text, we consider the localized policy where each agent take actions based on its own state,
[Equation]
with the localized direct parameterization :
[Equation]
We use X local i
[Equation]
|S i | to denote the feasibility region of θ i , and the feasibility region of θ is denoted as X local :=
X local i × · · · × X local n .
Lemma 7.
If there is a function φ ( s, a ) such that for every agent i , r i ( s i , s − i , a i , a − i ) =
φ ( s i , s −
i , a i , a − i ) + ψ i ( s − i , a − i ) where ψ i only depends on s − i , a − i , then this SG is a MPG, i.e., for any parameters ( θ (cid:48)
i , θ − i ) , ( θ i , θ − i ) ∈
X local , the equation in Deﬁnition 3 is satisﬁed.
The proof is straightforward given the local structure of the MDP and the localized policies.
This
MPG enjoys nontrivial multi-agent application examples such as medium access control (Macua et al., 2018), dynamic congestion control (Bertrand et al., 2020), etc.
Below we provide medium access control as one of the examples.
Real application - medium access control.
We consider the discretized version of the dynamic medium access control game introduced in (Macua et al., 2018), where each agent is a user that tries to transmit data via a single transmission medium by injecting power to the wireless network.
Each user’s goal is to maximize its data rate and battery lifespan.
If multiple users transmit at the same time, they will interfere with each other and decrease their data rate.
Here user
i ’s state is s i ∈ S
i = { 0 , 1 , . . .
, B i, max } , which denotes its own battery level, where B i, max is its initial battery level.
We use δ i to denote its discharging factor.
Its action a i ∈
A i = { 0 , 1 , . . .
, P i, max } denotes the power injected to the network at each time step, where P i, max is the maximum allowed power.
The state transition is deterministic, describing the discharging process of the battery proportional to the transmission power, which is given by:
[Equation]
t .
The stage reward of user i is given by:
[Equation]
where h i is the random fading channel coefﬁcient for user i .
By noticing that r i ( s, a ) = log (cid:0) 1 + (cid:80) [Equation] 1 | h i | 2 a i (cid:1) + α (cid:80)
i s i − log (cid:16)
+ (cid:80) j (cid:54) =
i | h
j | 2 a j (cid:17) − α (cid:80) j (cid:54) =
i s j , we can apply Lemma 7 to verify that the medium access control problem is indeed a MPG and that the potential function φ is given as:
[Equation]
C D ERIVATION OF E QUATION (18) ( CALCULATION OF d θ )
From the deﬁnition of d θ :
[Equation]
we have that:
[Equation]
D P ROOF OF L EMMA
Proof.
According to policy gradient theorem Equation (6):
[Equation]
Since for direct parameterization:
[Equation]
Thus we have that:
[Equation]
E P ROOFS OF L EMMA 3 AND T HEOREM
Before giving proofs for Lemma 3 and Theorem
[Equation]
we ﬁrst introduce the well-known the performance difference lemma (Kakade & Langford, 2002) in RL literature [Equation] which is helpful throughout.
A proof is also provided for completeness
[Equation]
Lemma 8.
(Performance difference lemma)
For policies π θ , π θ (cid:48) ,
[Equation]
Proof.
(of performance difference lemma)
From Bellman’s equation we have that:
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Equation]
Proof.
(Lemma 3)
According to performance difference lemma Equation (22):
[Equation]
According to the deﬁnition of ‘averaged’ advantage function:
[Equation]
which implies:
[Equation]
thus we have that:
[Equation]
We can rewrite 11 − γ (cid:80) s d θ ( s ) max a i ∈A i A θi ( s, a i ) as: 1 − γ (cid:88) s
[Equation]
[Equation] Substituting this into Equation (24), we may conclude that
[Equation]
and this completes the proof.
Proof.
(Theorem 1)
The deﬁnition of a Nash equilibrium naturally implies ﬁrst-order stationarity, because for any θ i ∈
X i :
[Equation]
Letting η → 0 gives the ﬁrst-order stationary condition:
[Equation]
It remains to be shown that all ﬁrst-order stationary policies are Nash equilibria.
From Assumptionwe know that for any pair of parameters θ (cid:48) , [Equation] ∞ < +
[Equation]
Take θ (cid:48)
=
( θ (cid:48)
i , θ ∗ − i ) , θ ∗ =
( θ ∗ i , θ ∗ − i ) .
According to Lemma 3, we have that for any ﬁrst-order stationary policy θ ∗ ,
[Equation]
which completes the proof.
F P ROOF OF T HEOREM 2 AND L EMMA
Proof.
(Lemma 4)
For a given strict NE θ ∗ randomly set:
[Equation]
and set θ i be:
[Equation]
And set θ :=
( θ i , θ ∗− i ) From performance difference lemma Equation (22):
[Equation]
Because θ ∗ is a strict NE, thus the inequality above forces θ ∗ i = θ , and that max a i A θ ∗ i ( s, a i ) = 0 .
The uniqueness of θ ∗ also implies uniqueness of a ∗ i ( s ) , and thus,
[Equation]
which completes the proof of the lemma.
The proof of Theorem 2 relies on the following auxiliary lemma, whose proof we defer to Appendix
M.
Lemma 9.
Let X denote the probability simplex of dimension n .
Suppose θ ∈ X , g ∈ R n and that there exists i ∗ ∈ { 1 , 2 , . .
. , n } and ∆ > 0 such that:
[Equation]
Let
[Equation]
, then:
[Equation]
Proof.
(Theorem 2) For a ﬁxed agent i and state s , the gradient play (Equation (5)) update rule of policy θ i,s is given by:
[Equation]
where ∆( |A i | ) denotes the probability simplex in |A
i | -th dimension and Q θ ( t ) i ( s, · )) is a |A
i | -th dimensional vector with a i -th element equals to Q θ ( t ) i ( s, a i )) .
We will show that this update rule satisﬁes the conditions in Lemma
[Equation]
which will then allow us to prove that
[Equation]
Letting a ∗ i ( s ) be the same deﬁnition as Equation (9)
[Equation]
we have that:
[Equation]
where Equation (27) to Equation (28) uses smoothness property in Lemma
[Equation]
We use proof of induction as supposed for (cid:96) ≤ t − 1 , we have:
[Equation]
thus
[Equation]
Then we can further conclude that:
[Equation]
Additionally, for D ( θ ( t ) || θ ∗ ) ≤ ∆
[Equation]
∗
[Equation]
1 − γ
[Equation] 8 n |S| [Equation] (cid:80) ni
=1 |A
i | ) , we may conclude:
[Equation]
then by applying Lemma 9 to Equation (26) we have:
[Equation]
which completes the proof.
G P ROOF OF P ROPOSITION
Proof.
First of all, from the deﬁnition of NE, the global maximum of the potential function is a NE.
We now show that this global maximum is a deterministic policy.
From classical results (e [Equation] g. (Sutton & Barto, 2018)) we know that there is an optimal deterministic centralized policy
[Equation]
that maximizes:
[Equation]
We now show that this centralized policy can also be represented by direct distributed policy parameterization.
Set θ ∗ as:
[Equation]
then:
[Equation]
i = π θ ∗ i ( a i | s ) Since π ∗ globally maximizes the discounted summation of potential function φ among centralized policies, which includes all possible direct distributedly parameterized policies, θ ∗ also maximizes the total potential function Φ globally among all direct distributed parameterization, which completes the proof.
H P ROOF OF T HEOREMH. U SEFUL O PTIMIZATION L EMMAS
Lemma 10.
Let Φ( θ ) be β -smooth in θ , deﬁne gradient mapping:
[Equation]
The update rule for projected gradient is:
[Equation]
Then:
[Equation]
Proof.
By a standard property of Euclidean projections onto a convex set, we get that
[Equation]
[Equation]
Lemma 11.
(Sufﬁcient ascent)
Suppose Φ( θ ) is β -smooth.
Let θ + =
Proj X
( θ + η ∇ Φ( θ )) .
Then for η ≤ 1 β ,
[Equation]
Proof.
From the smoothness property we have that:
[Equation]
Since θ + = Proj X ( θ + η ∇ Φ( θ )) , we have that:
[Equation]
take θ (cid:48)
= θ , we get:
[Equation]
Thus:
[Equation]
which completes the proof.
Lemma 12.
(Corollary of Lemma 11)
For Φ( θ ) that is β smooth and bounded Φ min ≤ Φ( θ ) ≤ Φ max , running projected gradient ascent:
[Equation]
with η = 1 β , will guarantee that:
[Equation]
Further, we have that:
[Equation]
Proof.
From Lemma 11 we have:
[Equation]
Thus Φ( θ ( t ) ) is non decreasing, and since it is bounded, we know that Φ( θ ( t ) ) asymptotically convergence to some value Φ ∗ , and thus show that
[Equation]
Additionally, from Equation (30),
[Equation]
which completes the proof.
H. P ROOF OF T HEOREM
Proof.
Recall the deﬁnition of gradient mapping:
[Equation]
From gradient domination property Equation (8) we have that:
[Equation]
where the last step follows as (cid:107) θ i − θ ( t +1)
i (cid:107) ≤ 2 (cid:112) |S| .
Thus
[Equation]
Then from Lemma 12 we have that:
[Equation]
and that:
[Equation]
we can get our required bound of (cid:15) if we set:
[Equation]
or equivalently
[Equation]
which completes the proof
[Equation]
I P ROOF OF T HEOREM
Proof.
(of the ﬁrst claim)
The proof requires knowledge of Lemma 4 in Section 3 thus we would recommend readers to ﬁrst go through Lemma 4 ﬁrst.
The lemma immediately leads to the conclusion that a strict NE θ ∗ should be deterministic.
Let a ∗
i ( s ) , ∆ θ ∗ i ( s ) , ∆ θ ∗ i be the same deﬁnition as
Equation (9) Equation (10) respectively.
For any θ ∈ X , Taylor expansion suggests that:
[Equation]
[Equation]
Thus for (cid:107) θ − θ ∗ (cid:107) sufﬁciently small,
[Equation]
∗
[Equation]
< 0 holds, this suggests that strict NEs are strict local maxima.
We now show that this also holds vice versa.
Strict local maxima satisfy ﬁrst-order stationarity by deﬁnition, and thus by Theorem 1 they are also
NEs, we only need to show that they are strict.
We prove by contradiction, suppose that there exists a local maximum θ ∗ such that it is non-strict NE, i.e., there exists θ (cid:48)
i ∈
X i , θ
[Equation]
i (cid:54) =
θ ∗ i such that:
[Equation]
According to Equation (25) and ﬁrst-order stationarity of θ ∗ :
[Equation]
Since max a i ∈A i A θi ( s, a i ) ≥ 0 for all θ , we may conclude:
[Equation]
We denote θ (cid:48) :
= ( θ (cid:48)
i , θ − i ∗ ) , according to Equation (23)
[Equation]
Since d θ (cid:48) ( s ) > 0 , ∀ s , this further implies that (cid:88)
[Equation]
i.e., π θ (cid:48)
i ( a i | s ) is nonzero only if A θ ∗ i
( s, a i )
= 0 .
Deﬁne θ ηi := ηθ (cid:48)
i + (1 − η ) θ ∗ i , then (cid:88)
a i π θ ηi ( a i | s ) A θ ∗ i ( s, a i )
= 0 , ∀ s ∈ S .
Thus let θ η := ( θ η i , θ ∗ − i )
J i ( θ ηi , θ ∗− i ) − J i ( θ ∗ i , θ ∗− i ) = 1 − γ (cid:88) s,a i d θ η ( s ) π θ ηi ( a i | s ) A θ ∗ i ( s, a i ) = 0 .
Since (cid:107) θ ηi − θ ∗ i (cid:107) → 0 as η → 0 , this contradicts the assumption that θ ∗ is a strict local maximum.
This suggests that all strict local maxima are strict NEs, which completes the proof.
Proof.
(of the second claim)
First, we deﬁne the corresponding value function, Q -function and advantage function for potential function φ .
[Equation]
For an index set I ⊆ { 1 , 2 , . . .
, n } we deﬁne the following averaged advantage potential function of index set I as:
[Equation]
We choose an index set I ⊆ { 1 , 2 , . . . , n } such that there exists s ∗ , a ∗I such that:
[Equation]
and that for any other index set I (cid:48) with smaller cardinality:
[Equation]
Because Φ is not a constant, this guarantees the existence of such an index set I .
Further, since
[Equation]
and that θ ∗ is fully-mixed, we have that:
[Equation]
We set θ :=
( θ I , θ ∗−I )
[Equation]
where θ I is a convex combination of θ ∗I , θ (cid:48)I ∈ X :
[Equation]
According to performance difference lemma Equation (22) we have:
[Equation]
a
[Equation] i [Equation] [Equation]
s
[Equation]
a [Equation] According to Equation (33), we know that:
[Equation]
, thus
[Equation]
Applying similar procedures recursively and using the fact that:
[Equation]
we get:
[Equation]
Set [Equation]
i ( a i | s ) as:
[Equation]
where s ∗ , a ∗ i are deﬁned in Equation (31).
Then:
[Equation]
which completes the proof
[Equation]
J B OUNDING THE GRADIENT ESTIMATION ERROR OF A LGORITHM
The accuracy of gradient estimation is essential in the sample-based algorithm 1.
In this section, we will give a high probability bound of the estimation error, which is stated in the following theorem:
Theorem 6.
(Error bound for gradient estimation)
Assume that the stochastic game that satisﬁes
Assumption 3.
In Algorithm 1, for
[Equation]
with probability at least 1 − δ , we have:
[Equation]
The proof of the theorem includes bounding the estimation error of Q θi (J.1) and d θ (J.2).
Let’s ﬁrst introduce the deﬁnition of ‘sufﬁcient exploration’ which is going to play an important role in this section.
In the main text Assumption 3 we have introduced ( τ, σ S ) -sufﬁcient exploration on states.
In this section we introduce a similar deﬁnition ( τ, σ ) -sufﬁcient exploration:
Deﬁnition 6.
( ( τ, σ ) -Sufﬁcient
Exploration)
A stochastic game and a policy θ is said to satisfy ( τ, σ ) -sufﬁcient
exploration condition if there exists positive integer τ and σ ∈ (0 , 1) such that for policy θ and any initial state-action pair ( s, a i ) , ∀ i , we have
[Equation]
Note that ‘ ( τ, σ ) -sufﬁcient exploration’ is a stronger condition compared with ‘ ( τ, σ S ) -sufﬁcient exploration on states’.
Additionally it is not hard to verify that for any stochastic game that satisﬁes ( τ, σ S ) -sufﬁcient exploration on states, and any θ ∈ X α , it will also satisfy ( τ, ασ S max i |A
i | ) -sufﬁcient exploration condition.
J. B OUNDING THE ESTIMATION ERROR OF THE AVERAGED -Q FUNCTION
We ﬁrst state the main theorem in this subsection:
Theorem 7.
(Estimation error of averaged Q-functions) Assume that the stochastic game with policy θ satisﬁes ( σ, τ ) -sufﬁcient exploration condition (Deﬁnition 6), then for a ﬁxed i , running Algorithm 1 will guarantee that:
[Equation]
further:
[Equation]
i.e., when
[Equation]
with probability at least
[Equation]
In the following, we will introduce some lemmas which will play an important role in bounding the estimation error of the averaged-Q function:
Lemma 13.
Assume that the stochastic game with policy θ satisﬁes ( σ, τ ) -sufﬁcient exploration condition (Deﬁnition 6), then ﬁx s (cid:48) , s, a i ,for (cid:15) ≤ 1 ,
[Equation] [Equation]
Proof.
According to the deﬁnition of (cid:99) P θi , we have that (cid:110)(cid:99) [Equation] [Equation] [Equation]

[Equation]
[Equation]
[Equation]
m [Equation] m = a i } (cid:17) ≥  [Equation]
 
[Equation]
k
[Equation] 1 { s [Equation] m =
s
[Equation]
a i,kτ
[Equation]
m =
a
[Equation]
} = 
Let:
A m :=
 
[Equation]
m [Equation] k [Equation]
[Equation]
[Equation]
s
[Equation]
m
[Equation]
, a
[Equation]
,kτ
[Equation]
m
[Equation]
a
[Equation]
[Equation]
[Equation]
[Equation]
(cid:48) | s
[Equation]
a i ) + (cid:15) ) 1 { s kτ + m = s, a i,kτ + m = a i } (cid:17) ≥ 
A (cid:48) m :=   (cid:98) T − 1 − m τ (cid:99) (cid:88) k
= 1 { s kτ + m =
s, a i,kτ + m = a i } = 
{+X k,m := 1 { s kτ + m +
+}
= s (cid:48) , s kτ + m = s,
a i,kτ + m = a i } − ( P θi ( s (cid:48) | s, a i ) + (cid:15) ) 1 { s kτ + m = s, a i,kτ + m = a i }
X (cid:48) k,m := 1 { s kτ + m = s, a i,kτ + m = a i }
Y k,m :=
X k,m − E [ X k,m |F
( k − 1)
τ + m ]
Y (cid:48) k,m := X (cid:48)
k,m − E
[ X (cid:48) k,m |F ( k − 1)
τ + m ]
Then { Y k,m } (cid:98) T − 1 − m τ (cid:99) k
= is a martingale difference sequence.
Because (cid:15) ≤ 1 , it is easy to verify that | X k,m | ≤ 2 , | X (cid:48)
k,m | ≤ 1 .
We have that: | Y k,m | ≤ | X k,m | + E [ | X k,m ||F ( k − 1) τ + m ] ≤ 4 , | Y (cid:48) k,m | ≤ | X (cid:48)
k,m | + E [ | X (cid:48) k,m ||F ( k − 1) τ + m ] ≤ 2 .
Further,
E
[Equation]
i }|F ( k − 1) τ + m ] ≥ σ, and that
[Equation]
To move from equation (34) to equation (35) [Equation] we used the fact that:
[Equation]
and the inequality in equation (35) is derived directly from Deﬁnition [Equation]
According to Azuma-Hoeffding inequality (Hoeffding, 1994; Azuma, 1967):
[Equation]
T − 1 − m τ (cid:99) (cid:88) k
=
X k,m ≥  =
Pr   (cid:98) T − 1 − m τ (cid:99) (cid:88) k
= Y k,m ≥ − (cid:98) T − 1 − m τ (cid:99) (cid:88) k = E [ X k,m |F ( k − 1) τ + m ] 
 ≤
Pr   (cid:98) T − 1 − m τ (cid:99) (cid:88) k = Y k,m ≥ (cid:22) T − 1 − m + τ τ (cid:23) (cid:15)σ   ≤ exp (cid:32) − (cid:15) 2 σ 2 (cid:4) Tτ (cid:5) (cid:33)
Similarly, from Azuma-Hoeffding inequality,
Pr( A (cid:48) m ) = Pr   (cid:98) T − 1 − m τ (cid:99) (cid:88) k = X (cid:48)
k,m =  =
Pr   (cid:98) T − 1 − m τ (cid:99) (cid:88) k = Y (cid:48) k,m = − (cid:98) T − 1 − m τ (cid:99) (cid:88) k
= E [ X (cid:48) k,m |F ( k − 1) τ + m ]   ≤
Pr   (cid:98) T − 1 − m τ (cid:99) (cid:88) k = Y (cid:48) k,m ≤ − (cid:22) T − 1 − m + τ τ (cid:23) σ   ≤ exp (cid:32) − σ 2 (cid:4) Tτ (cid:5)
[Equation]
Thus
[Equation]
Similarly
[Equation]
which completes the proof.
Lemma 14.
Assume that the stochastic game with policy θ satisﬁes ( σ, τ ) -sufﬁcient exploration condition (Deﬁnition 6), then ﬁx s (cid:48) , s, a i , for (cid:15) ≤ 1 ,
[Equation]
Proof.
The proof is similar to Lemma 13.
[Equation]
Let:
[Equation]

[Equation]

[Equation]
+ m ) − ( r θi ( s, a i ) + (cid:15) )
1 { s kτ + m = s, a i,kτ + m = a i }
Y k,m :=
X k,m − E [ X k,m |F
( k − 1)
τ + m ]
Then { Y k,m } (cid:98) T − 1 − m τ (cid:99) k
= is a martingale difference sequence.
Because (cid:15) ≤ 1 , it is easy to verify that | X k,m | ≤ 2 .
We have that:
[Equation]
Further,
[Equation]
the second line to the third line of the equation is derived by the fact that:
[Equation]
[Equation]
and the inequality in the third line is derived directly from Deﬁnition
[Equation]
According to Azuma-Hoeffding inequality:
[Equation] [Equation]
Same as Equation (36), we have that:
[Equation]
Thus
[Equation]
Similarly
[Equation]
which completes the proof
[Equation]
Lemma 13 and 14 lead to the following corollary:
Corollary
[Equation]
Proof.
We ﬁrst prove Equation (37)
[Equation]
Thus,
[Equation]
Then according to Lemma
[Equation]
Now we prove Equation (38).
Since
[Equation]
according to Lemma 14,
[Equation]
which completes the proof of the corollary.
We are now ready to prove Theorem
[Equation]
Proof.
(of Theorem 7)
From the deﬁnition of (cid:99) Q θi , Q θi ,
[Equation]
we have that
[Equation]
∞
[Equation]
Because both M θi and (cid:100) M θi are transition probability matrices
[Equation]
thus:
[Equation]
Thus,
[Equation]
[Table]
[Equation] Thus if
[Equation] we have that:
[Equation]
Thus from Corollary 1,
[-Pr (cid:16) (cid:107) (cid:99) Q θi − Q θi (cid:107) ∞ ≥ (cid:15) (cid:17) ≤ Pr (cid:16) (cid:107) (cid:98) r θi − r θi (cid:107) ∞ ≥ (1 − γ ) 2 (cid:15) (cid:17) + Pr (cid:16) (cid:107) (cid:100) M θi − M θi (cid:107) ∞ ≥ (1 − γ ) 2 (cid:15) (cid:17) ≤ 4 τ |S||A i | exp (cid:32) − (1 − γ ) 4 (cid:15) 
 σ 2 (cid:98) T τ (cid:99) (cid:33) + 4 τ |S| 2 |A i | exp (cid:32) − (1 − γ ) 4 (cid:15) 2 σ 2 (cid:98) T τ (cid:99) 32 |S|(cid:33) ≤ 8 τ |S| 2 |A i | exp (cid:32) − (1 − γ ) 4 (cid:15) 2 σ 2 (cid:98)-][Equation]
[Equation]
which completes the proof.
J. B OUNDING THE ESTIMATION ERROR OF d θ
We ﬁrst state our main result:
Theorem 8.
(Estimation error of d θ ) Under Assumption 3,
[Equation]
[Equation]
i.e., when
[Equation]
[Equation]
with probability at least 1 − δ , (cid:107) (cid:98) d θ − d θ (cid:107) 1 ≤ (cid:15) .
Similar to the previous section, the proof of the theorem begins by bounding the estimation error | (cid:99) P θ S ( s (cid:48) | s ) − P θ S ( s (cid:48) | s ) | .
Lemma 15.
Under Assumption 3, ﬁx s (cid:48) , s, a i ,for (cid:15) ≤ 1 ,
[Equation]
exp
[Equation]
[Equation]
Proof.
According to the deﬁnition of (cid:99) P θ S , we have that
[Equation]
[Equation]
[Equation]

[-= (cid:16) 1 { s kτ + m +
-][Equation]

[Equation]  
Let:
[-A m :=   (cid:98) T − 1 − m τ (cid:99) (cid:88) k = (cid:16) 1 { s kτ + m +
 = s (cid:48) , s kτ + m = s } − ( P θ S ( s (cid:48) | s ) + (cid:15) ) 1 { s kτ + m = s } (cid:17) ≥ -][Equation]
[-A (cid:48) m :=   (cid:98) T − 1 − m τ (cid:99) (cid:88) k = 1 { s kτ + m = s } =  X k,m := 1 { s kτ + m +
 = s (cid:48) , s kτ + m-]Y
[Equation]
Then { Y k,m } (cid:98) T [Equation] m τ (cid:99) k
= is a martingale difference sequence.
Because (cid:15) ≤ [Equation] it is easy to verify that | X k,m | ≤ 2 , | X k,m | ≤ 1 .
We have that:
[Equation]
Further,
E [
[Equation]
and that
[Equation]
the second line to the third line of the equation is derived by the fact that:
[Equation]
and the inequality in the third line is derived directly from Assumption 3.
According to Azuma-Hoeffding inequality:
[Equation]
Similarly, from Azuma-Hoeffding inequality,
[Equation]
= Pr   (cid:98) T − 1 − m τ (cid:99) (cid:88) k = Y (cid:48) k,m = − (cid:98) T − 1 − m τ (cid:99) (cid:88) k
= E [ X (cid:48) k,m |F ( k − 1) τ + m ]   ≤
Pr   (cid:98) T − 1 − m τ (cid:99) (cid:88) k = Y (cid:48) k,m
[Equation]
(cid:22) T − 1 − m + τ τ (cid:23)
[Equation]

 [Equation]
(cid:32)
[Equation]
(cid:33) Thus [Equation] Similarly
[Equation]
which completes the proof
[Equation]
Corollary
[Equation]
Proof.
[Equation]
Thus
[Equation]
Then according to Lemma 15,
[Equation]
[Equation]
which completes the proof of the corollary
[Equation]
Proof.
(Proof of Theorem 8)
[Equation]
[Table]
[Equation]
J. P ROOF OF T HEOREM
Proof.
Since the stochastic game satisﬁes ( τ
[Equation]
σ S ) -sufﬁcient exploration on states
[Equation]
then for any θ ∈ X α [Equation] we know that it satisﬁes ( τ, ασ S max i |A
i | ) -sufﬁcient exploration.
Substitute this into Theorem [Equation] we have that for
[Equation]
with probability at least 1 − δ 2 T G ,
[Equation]
Similarly
[Equation]
applying Theorem
[Equation]
we have that with probability at least 1 − δ 2 T G ,
[Equation]
Since:
[Equation]
Thus
[Equation]
with probability 1 − δ
[Equation]
K P ROOF OF T HEOREM
Notations: We deﬁne the following variables that will be useful in the analysis:
[Equation]
K. O PTIMIZATION L EMMAS
Lemma 16.
(Sufﬁcient ascent)
Suppose Φ( θ ) is β -smooth.
Let θ + =
Proj X α ( θ + η (cid:98) ∇ Φ( θ )) .
Then for η ≤ 12 β ,
[Equation]
Proof.
From the smoothness property we have that:
[Equation]
Since θ + = Proj X α ( θ + η (cid:98) ∇ Φ( θ )) , we have that:
[Equation]
take θ (cid:48)
= θ , we get:
[Equation]
Thus:
[Equation]
Thus from equation (29):
[Equation]
G η,α ( θ ) (cid:107) 2 − η 2 (cid:13)(cid:13)(cid:13) ∇ Φ( θ ) − (cid:98) ∇ Φ( θ ) (cid:13)(cid:13)(cid:13)which completes the proof.
Lemma 16 immediately results in the following corollary:
Corollary 3.
(of Lemma 16)
In Algorithm 1, suppose (cid:107) (cid:98) ∇ Φ( θ ( k ) )
− ∇ Φ( θ ( k ) )
(cid:107) ∞ ≤ (cid:15)
g holds for every 0 ≤ k ≤ T G − 1 , then running algorithm 1 will guarantee that:
[Equation]
Proof.
From Lemma 16 we have that:
[Equation]
Thus
[Equation]
Lemma 17.
(First-order stationarity and (cid:107) (cid:98) G η,α ( θ ) (cid:107) ) Suppose Φ( θ ) is β -smooth.
Let θ + =
Proj X α ( θ + η (cid:98) ∇ Φ( θ )) .
Then:
[Equation]
[Equation]
Further:
[Equation]
[Figure]
[Equation]
S| (cid:104) (1 + ηβ ) (cid:107) (cid:98) G η,α ( θ ) (cid:107) + (cid:107) (cid:98) ∇ Φ( θ ) − ∇ Φ( θ ) (cid:107) (cid:105) + 2 α 1 − γ (42)
Proof.
Since θ + =
Proj X α ( θ + η (cid:98) ∇ Φ( θ )) , we have:
[Equation]
which proves equation (41).
We now prove equation (42).
For any θ (cid:48)
i,s ∈ ∆( |A i | ) , we know that (1 − α ) θ (cid:48)
i,s + αU |A
i | ∈ ∆ α ( |A i | ) .
Let U i :=
[ U |A i | , . . .
, U |A
i | (cid:124) (cid:123)(cid:122) (cid:125) |S| times ] , then for any θ (cid:48)
i ∈
X i , (1 − α ) θ (cid:48)
i + αU i ∈
X αi .
Thus:
[Equation]
θ i Φ( θ + ) (cid:62) ( θ (cid:48) i − (1 − α ) θ (cid:48) i − αU i ) ≤ (cid:104) (1 + ηβ ) (cid:107) (cid:98) G η,α ( θ ) (cid:107) + (cid:107) (cid:98) ∇ Φ( θ ) − ∇ Φ( θ ) (cid:107) (cid:105) (cid:107) (1 − α ) θ (cid:48) i + αU i − θ
+ i (cid:107) + ∇ θ i Φ( θ + ) (cid:62) ( θ (cid:48) i − (1 − α ) θ (cid:48) i − αU i ) ≤ 2 (cid:112) |S| (cid:104) (1 + ηβ ) (cid:107) (cid:98) G η,α ( θ ) (cid:107) + (cid:107) (cid:98) ∇ Φ( θ ) − ∇ Φ( θ ) (cid:107) (cid:105) + α ∇ θ i Φ( θ + ) (cid:62) ( θ (cid:48) i − U i )
Since ∇
[Equation]
[Equation]
we have that:
[Equation]
K. P ROOF OF T HEOREM
Proof.
Recall that Φ is β -smooth with β = 2 (1 − γ ) 3 ( (cid:80) ni
=1 |A i | ) .
The step size η in Theoremsatisﬁes η ≤
[Equation]
Recall from gradient domination property:
[Equation]
Suppose (cid:107) (cid:98) ∇ Φ( θ ( k ) )
− ∇ Φ( θ ( k ) )
(cid:107) ∞ ≤ (cid:15) g , ∀ 0 ≤ k ≤ T G − 1 , recall from Lemma 17,
NE
[Equation]
Thus
[Equation]
[Table]
[Equation]
From Corollary
[Equation]
we have that
[Table]
[Equation]
Substitute
[Equation]
into the above inequality we get that:
[Equation]
Substitute the value of α, (cid:15) g in Equation (43) into Theorem 6 will give us:
[Equation]
which completes the proof
[Equation]
L S MOOTHNESS
Lemma 18.
(Smoothness for Direct Distributed Parameterization)
Assume that 0 ≤ r i ( s, a ) ≤ 1 , ∀ s, a, i = 1 , 2 , . .
. , n , then:
[Equation]
) where g ( θ ) = {∇ θ i
J i ( θ ) } [Equation]
The proof of Lemma 18 depends on the following lemma:
Lemma 19.
[Equation]
Lemma 18 is a simple corollary of Lemma
[Equation]
Proof.
(Proof of Lemma 18)
[Equation]
 n
[Equation]
=
[Equation]
| 
  
[Equation]
j
[Equation]

 =
[Equation]
i
[Equation]
| (cid:33)
[Equation]
(cid:107) θ (cid:48)
[Equation]
(cid:107)
[Equation]
which completes the proof
[Equation]
Lemma 19 is equivalent to the following lemma:
Lemma 20.
[Equation]
Proof.
(Lemma 20) Deﬁne:
[Equation]
According to cost difference lemma,
[Equation]
[Equation]
 Thus:
[Equation]
| u a i ,s | (cid:88) a − i (cid:12)(cid:12)(cid:12) π θ (cid:48)− i ( a − i | s )
− π θ − i ( a − i | s ) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) (48) ≤ 1 − γ (cid:32) max s (cid:88) a i | u a i ,s | (cid:33) (cid:88) s d (cid:48) θ ( s )2 d TV ( π θ (cid:48)− i ( ·| s ) || π θ − i ( ·| s )) (49) ≤ 1 − γ (cid:32) max s (cid:88) a i | u a i ,s | (cid:33) (cid:88) s d (cid:48)
θ ( s ) (cid:88) j (cid:54) = i 2 d TV ( π θ (cid:48)
{+j ( ·| s ) || π θ j ( ·| s )) (50) = 
 − γ (cid:32) max s (cid:88) a i | u a i ,s | (cid:33) (cid:88) s d (cid:48) θ ( s ) (cid:88)+}
j (cid:54) = i (cid:107) θ (cid:48) j,s − θ j,s (cid:107)(51) ≤ 1 − γ (cid:112) |A i | (cid:88) s d (cid:48)
θ ( s ) (cid:88) j (cid:54) = i (cid:113) |A j |(cid:107) θ (cid:48)
j,s − θ j,s (cid:107) (52) ≤ 1 − γ (cid:112) |A [Equation] [Equation] ≤
[Equation]
where Equation (47) to Equation (48) is derived from the fact that | Q θi ( s, a ) | ≤ 11 − γ .
Equation (49) to Equation (50) relies on the property of total variation distance:
[Equation]
Equation (51) to Equation (52) is derived from:
[Equation]
which can be immediately veriﬁed by applying Cauchy-Schwarz inequality.
Before looking into Part B, we ﬁrst deﬁne (cid:101) P ( α ) as the state-action under π α :
[Equation]
Then we have that:
[Equation]
For an arbitrary vector x :
[Equation]
[Equation]
a )
[Equation]
[Equation]
i
[Equation]
[Equation]
i |
[Equation]
∞
Thus: (cid:13)(cid:13)(cid:13)(cid:13) (cid:13) (cid:13) (cid:34) ∂
(cid:101) P ( α ) ∂α (cid:12)(cid:12) (cid:12) α
=0 x (cid:35) ( s,a ) (cid:13)(cid:13)(cid:13)(cid:13) (cid:13) (cid:13) ∞ ≤ (cid:112) |A
i |(cid:107) x (cid:107) ∞
Similarly we can deﬁne (cid:101) P ( α ) (cid:48) as the state-action under π (cid:48) α , and can easily check that (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:34)
[Equation]
(cid:12)(cid:12)(cid:12)
[Equation]
=
[Equation]
x (cid:35) ( s,a ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ∞ ≤ [Equation] (cid:107) ∞
Deﬁne:
[Equation]
− 1 .
Because:
M ( α
[Equation]
=
[Equation]
α
[Equation]
= ∞
[Equation] n = γ n [Equation] P ( α ) , which implies that every entry of M ( α ) is nonnegative and M ( α ) 1 = 11 − γ 1 , this implies:
[Equation]
and similarly
[Equation]
Now we are ready to bound Part B.
Because:
[Equation]
Thus
[Equation]
Part B = (cid:12)(cid:12)(cid:12)
[Equation]
∂Q αi ( s, a ) ∂α [Table] α = [Table]
≤ γ (1 − γ )(cid:112) |A i | (cid:88) s d (cid:48) θ ( s )2 d TV ( π θ (cid:48) ( ·| s ) || π θ ( ·| s ))
≤ γ (1 − γ )(cid:112) |A i | (cid:88) s d (cid:48) θ ( s ) (cid:88)
j 2 d TV ( π θ (cid:48) j ( ·| s ) || π θ j ( ·| s )) =
γ (1 − γ )(cid:112) |A i | (cid:88) s d (cid:48) θ ( s ) (cid:88)
j (cid:107) θ (cid:48) j,s − θ j,s (cid:107)≤ γ (1 − γ )(cid:112) |A i | (cid:88) s d (cid:48)
θ ( s ) n (cid:88) j = (cid:113) |A j |(cid:107) θ (cid:48) j,s − θ j,s (cid:107) ≤ γ (1 − γ )(cid:112) |A
i | n (cid:88)
j = (cid:113) |A j | (cid:115)(cid:88) s d (cid:48) θ ( s ) 2 (cid:115)(cid:88) s (cid:107) θ (cid:48)
j,s − θ j,s (cid:107)≤ γ (1 − γ )(cid:112) |A i | n (cid:88) j = (cid:113) |A j |(cid:107) θ (cid:48)
[Equation]
Now let’s look at Part C:
[Equation]

[Equation]
Note that v 1 , v 2 are constant vectors that are independent of the choice of s .
Additionally:
[Equation]
Thus:
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
Combining the above inequalities we get:
Part C ≤
[Equation]
[Table]
[Equation]
which completes the proof.
M A UXILIARY
We recall Lemma 9.
Lemma 9.
Let X denote the probability simplex of dimension n .
Suppose θ ∈ X , g ∈ R n and that there exists i ∗ ∈ { 1 , 2 , . .
. , n } and ∆ > 0 such that:
[Equation]
Let θ (cid:48)
=
Proj X (
[Equation]
g
[Equation]
, then:
[Equation]
Proof.
Let y
= θ + g , without loss of generality, assume that i ∗ = 1 and that:
[Equation]
Using KKT condition, one can derive an efﬁcient algorithm for solving Proj X ( y ) (Wang & CarreiraPerpi ˜ n ´ an, 2013), which consists of the following steps:
Find ρ := max { 1 ≤ j ≤ n : y j + 1 j (cid:16) 1 − (cid:80) ji =
y i (cid:17) > 0 } ; 2.
Set
λ := 1 ρ (1 − (cid:80) ρi
=1 y i ) ; 3.
Set θ (cid:48)
i = max { y i + λ, 0 } .
From the algorithm, we have that:
[Equation]
i =
[Equation] g
[Equation] If ρ ≥ 2 ,
[Equation]
If ρ = 1 ,
[Equation]
Thus:
[Equation]
which completes the proof.
N N UMERICAL S IMULATION D ETAILS
Veriﬁcation of the fully mixed NE in Game 2 We now verify that joining network 1 with probability 1 − 3 (cid:15) 3(1 − 2 (cid:15) ) ,i.e.: π
[Equation]
=
[Equation]
2 , is indeed a NE.
First, observe that
Pr
[Equation]
=
[Equation]
(cid:19)
[Equation]
Thus,
V
[Equation]
( s − i = 1) r ( s i
[Equation]
s − i = 1)
+ P ( s (cid:48)
i | a i )
Pr θ ( s − i = 2) r
( s i
[Equation]
s − i = 2))
[Equation]
[Equation] [Equation]
γ
[Equation]
= r ( s ) + γP ( s (cid:48)
i = 1 | a i )
[Equation]
s
[Equation]
i
[Equation] [Equation]
i
[Equation]
s
[Equation]
which implies that:
[Equation]
i.e. θ satisﬁes ﬁrst-order stationarity.
Since d θ ( s ) > 0 holds for any valid θ , by Theorem 1, θ is a NE.
Computation of strict NEs in Game
The computation of strict NEs is done numerically, using the criterion in Lemma 4.
We enumerate over all 2 8 possible deterministic policies and check whether the conditions in Lemma 4 hold.
For (cid:15) = 0 .
1 , γ = 0 .
95 , and an initial distribution set as:
[Equation]
} , the numerical calculation shows there exist 13 different strict NEs.