On the Global Convergence Rates of Decentralized Softmax Gradient Play in Markov Potential Games
Anonymous Author(s)
AffiliationAddress email
Abstract
Softmax policy gradient is a popular algorithm for policy optimization in singleagent reinforcement learning, particularly since projection is not needed for eachgradient update.
However, in multi-agent systems, the lack of central coordinationintroduces significant additional difficulties in the convergence analysis.
Even fora stochastic game with identical interest, there can be multiple Nash Equilibria(NEs), which disables proof techniques that rely on the existence of a unique globaloptimum.
Moreover, the softmax parameterization introduces non-NE policies withzero gradient, making it difficult for gradient-based algorithms in seeking NEs.
Inthis paper, we study the finite time convergence of decentralized softmax gradientplay in a special form of game, Markov Potential Games (MPGs), which includesthe identical interest game as a special case.
We investigate both gradient play andnatural gradient play, with and without log -barrier regularization.
The establishedconvergence rates for the unregularized cases contain a trajectory dependent constant that can be arbitrarily large , whereas the log -barrier regularization overcomesthis drawback, with the cost of slightly worse dependence on other factors suchas the action set size.
An empirical study on an identical interest matrix gameconfirms the theoretical findings.
Introduction
Multi-agent systems encounter vast application in real world scenarios, such as network routing[35, 8], social and economic decision making [36, 30] and robotic swarms [22, 14].
In these problems,a system consists of a group of agents interacting in a shared environment.
Given the recent successof reinforcement learning (RL), increasing attention has been drawn to the possibility of applying RLalgorithms, such as policy gradient, to multi-agent systems.
However, the theoretical foundations formulti-agent reinforcement learning (MARL) remain limited.
Unlike single-agent RL, the actions ofother agents affect the dynamics and the decision making outcome for each individual in the system,raising additional theoretical challenges when analyzing joint performance.
The stochastic game (SG) is a classical multi-agent model that has received extensive attention inrecent MARL studies.
In a stochastic game, the environment is represented by a state space thatevolves based on the joint actions of agents.
Each agent in a stochastic game tries to maximize its owntotal reward by making decisions independently , based on state information shared between agents.
The stochastic game model was first introduced in [31], with a series of followup works proposing
NE-seeking algorithms, particularly in the RL setting (e.g. [21, 5, 32, 6, 17, 38] and citations therein).
Given recent progress in the underlying theory of RL, many recent works have investigated finitetime iteration and sample complexity for learning NE or other general equilibria notions, such ascorrelated and coarse correlated equilibria (e.g. [33]).
There are different types of SGs, some with attributes that merit special attention; for example,two-player zero sum games [4, 9], which are widely used to model two player competitive games such as GO.
In this paper, we will focus on another type of SG, the Markov potential game (MPG) [23,27, 39, 18], which includes the identical interest game as a special case.
The structure of a MPGenables efficient learning through the use of gradient-based algorithms such as gradient play.
Recentwork [39, 18] has focused on the iteration and sample complexity of finding a NE in an MPG underthe direct policy parameterization, which is not practical in most real world scenarios, given thecost of projecting back to the probability simplex on every iteration.
This drawback has motivatedconsideration of the softmax parameterization, which bypasses the projection step in the gradientupdate, and is perhaps the most popular approach to parameterizing policies in practice.
[11] havestudied natural gradient play for MPG under softmax parameterization, but only address asymptoticbehavior and leave finite time complexity open.
From the perspective of analysis and practical performance, the extension from the direct to thesoftmax parameterization in policies is nontrivial.
Even in the single agent case, as shown by [2, 24],there are policies in the softmax parameterization that have near-zero gradient and yet are far frombeing globally optimal, which creates difficulty for a gradient-based algorithm to escape suboptimalpoints.
A similar issue exists for MPGs: due to the more complex interaction between agents, there iseven a greater set of policies that obtain small gradient norm but are far from a NE.
Based on ouranalysis and numerical results, even for natural gradient play—which is known to enjoy dimensionfree convergence in single agent learning [2]—we find in the multiagent setting that it can stillbecome stuck in these undesirable regions.
Such evidence suggests that preconditioning accordingto the Fisher information matrix [29, 3] is not sufficient to ensure fast convergence in multi-agentlearning.
A stronger form of regularization is required, which motivates the introduction of log -barrierregularization to avoid undesirable regions of policy space.
[Table]
[Equation]
direct parameterization [2] [39, 18]
Gradient play,
[Equation]
softmax parameterization [24]
Natural gradient play,
[Equation]
softmax parameterization [2]
Gradient play + log -barrier reg.,
[Equation]
softmax parameterization [2]
Natural gradient play + log -barrier reg. ,
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
Complexity bounds with ‘*’ depend on an additional assumption on the MPG (See Theorem 4 and 5).
The definitions of variables M and c appearing in some bounds can be found in (5) and (11).
Note that the definition of M is slightly different from the “distribution mismatch coefficient” D ∞ defined in [2] (see more details in descriptions that follows Assumption 1).
To make the complexity results more comparable, we slightly modify and re-derive the results in [2, 24, 39, 18].
Our contribution: In this paper, we provide finite time iteration complexity results for gradientplay and natural gradient play under the softmax parameterization, considering both unregularizedand log -barrier regularized dynamics.
We summarize the convergence rates and compare them toexisting results for the direct parameterization and to the corresponding single agent cases in Table 1.
These findings suggest that regularization is crucial for obtaining fast convergence to a NE under thesoftmax parameterization in a MPG.
In Table 1, the results for the two unregularized algorithms inthe multi-agent case rely on the assumption that the set of stationary policies is isolated (which isalso assumed in [11] when establishing the asymptotical convergence for natural policy gradient),and the corresponding complexity bounds contain an initialization dependent factor c .
By contrast,the log -barrier regularized algorithms overcome both drawbacks, but as a tradeoff, their boundsincur a slightly worse dependence on |A i | and M .
We observe numerically that the log -barrierregularized algorithms are indeed more robust against becoming trapped near undesirable non-NEstationary points.
To the best of our knowledge, the finite-time iteration complexity results are thefirst such results for MPGs under the softmax parameterization.
Though the analysis for the gradientplay follows their single-agent counterparts [2, 24], the results for natural gradient play are highly non-trivial, requiring very different analysis tools which have their own merits to the literature (see
Remark 2 and Remark 3 for more details on the technical novelty in the analysis).
Our results alsoconvey the following two messages.
First, finding the NE of a multi-agent MPG is harder thanfinding the global optimum for the single-agent case, because multi-agent learning suffers greaterrisk of becoming trapped near undesirable stationary points.
This is reflected in the dependence ofthe complexity bounds on ϵ in Table 1.
Second, natural gradient play outperforms gradient playcounterparts, suggesting that natural gradient play captures useful information about the geometry ofthe parameter space that accelerates the learning process.
Problem settings
We consider an infinite time horizon n -agent stochastic game (SG, [31])
M = ( N, S , A =
A 1 ×. . .
×A n , P, r = ( r 1 ,. . .
,r n ) , γ, ρ ) which is specified by an agent set N = { 1 , 2 , . . . , n } , a finitestate space S , a finite action space
A i for each agent i ∈ N , a transition model P (such that
P ( s ′ | s, a ) =
P ( s ′ | s, a 1 , . . .
, a n ) is the probability of transitioning into state s ′ upon taking actiona :
= ( a 1 , . .
. , a n ) in state s where a i ∈
A i is action of agent i ), a reward function r
i : S ×A → [0 , 1]for each agent
i , a discount factor γ ∈ [0 , 1) , and an initial state distribution ρ over S .
We use s ( t ) ∈
Sto denote the state at time step t , and a ( t ) =
( a 1 ( t ) , . . .
, a n ( t )) ∈
A to denote the total action.
A stochastic policy π : S → ∆( A ) (where ∆( A ) is the probability simplex over A ) specifies astrategy, where agents choose their actions jointly based on the current state in a stochastic fashion; i.e.
Pr( a ( t ) | s ( t ))
=
π ( a ( t ) | s ( t )) .
A decentralized stochastic policy is a special subclass of stochasticpolicies with π = π 1 × . . .
× π n , such that π i : S → ∆( A i ) , where π i is agent
i ’s own localpolicy.
For decentralized stochastic policies, each agent takes its action based on the current state sindependently of other agents’ action choices; i.e.,
[Equation]
For notation simplicity, we define π I ( a I | s ):= (cid:81) i ∈ I
π i ( a i | s ) , where I ⊆
N is an index set.
Further,we use the notation − i to denote the index set N \{ i } .
In this paper we focus on tabular softmaxparameterization for a policy, where policy π θ =
( π θ 1 , . . .
, π θ n ) is parameterized by a set ofparameters θ =
( θ 1 , . .
, θ n ) , with θ i
= { θ s,a i } s ∈S ,a i ∈A i , and where
[Equation]
We denote agent i ’s total reward starting from initial states s (0) ∼ ρ as:
J i ( θ ) :
= 101 E s (0) ∼ ρ (cid:2)(cid:80) ∞ t [-=
-]=0 γ t r
i ( s ( t ) , a ( t ))
(cid:12)(cid:12)
π θ , s (0) =
s (cid:3) .
Agent i ’s objective is to maximize its own total reward J i .
A Nash equilibrium (NE) is often used to characterize the equilibrium (a joint policy) whereno agent has a unilateral incentive to deviate from it.
Definition 1.
(Nash equilibrium)
A policy θ ∗ = ( θ ∗ 1 , . .
. , θ ∗ n ) is called a (Markov perfect) Nashequilibrium (NE) if
[Equation]
Further, we define the ‘NE-gap’ of a policy θ to be:
[Equation]
A policy θ is an ϵ -Nash equilibrium if: NE-gap ( θ ) ≤ ϵ.
We define the value function with respect to stage cost r i as:
[Equation]
We define agent i ’s Q -function and advantage function Q θi , A θi : S ×
A → R ,
Q θi ( s, a ):= E (cid:2)(cid:80) ∞ t
[Equation]
[Equation]
t
[Equation]
s
[Equation]
(cid:3) , A [Equation] We further define agent i ’s ‘averaged’ Q-function Q θi :
S × A i → R and ‘averaged’ advantagefunction
A θ i : S × A i
→ R as:
[Equation]
i , a − i ) . Finally, define the discounted state visitation distribution d θ of a policy π θ given an initial statedistribution ρ as:
[Equation]
where Pr θ ( s ( t ) = s | s (0)) is the state visitation probability that s ( t )
= s when executing π θ startingat state s (0) .
From the policy gradient theorem [34], we have that (proof given in Appendix B):
[Equation]
For the remainder of the paper, we make the following assumptions on the stochastic games we study.
Assumption 1.
The stochastic game M satisfies:
[Equation]
Assumption 1 requires that every state is visited with positive probability for any policy, which isa standard assumption for convergence proofs in the RL literature (e.g. [2, 24]).
We will use M todenote the following quantity
[Equation]
Note that M can be viewed as a measure of exploration sufficiency in the stochastic game, whichis slightly different from the “distributional mismatch coefficient” introduced in [2] defined bysup θ,θ ′ max s d θ ′ ( s ) d θ ( s ) ; however, both can be upper bounded by max s
[Equation]
We primarily focus on the following subclass of stochastic games in this paper:
Definition 2.
A stochastic game is called a Markov potential game (MPG, [37, 23, 39, 18, 26]) if 126 there exists a potential function ϕ : S ×
A 1 × · · · × A n → R such that for any agent i and any pairof policy parameters ( θ ′ i , θ − i ) , ( θ i , θ − i ) :
[Equation]
E (cid:34) ∞ (cid:88) t
= γ t r i ( s ( t ) , a ( t ))
(cid:12)(cid:12) π =(
θ i , θ − i ) , s (0)= s (cid:35) =
E (cid:34) ∞ (cid:88) t
= γ t ϕ ( s ( t ) , a ( t ))
(cid:12)(cid:12) π =( θ
i , θ − i ) , s (0)= s (cid:35) − E (cid:34) ∞ (cid:88) t
= γ t ϕ ( s ( t ) , a ( t )) (cid:12)(cid:12) π =(
θ i , θ − i ) , s (0)= s (cid:35) , ∀ s. (6)
Without loss of generality, we assume that ϕ min ≤ ϕ ( s, a ) ≤ ϕ max for all ( s, a ) .
The definition of
MPG is a generalization of the notion potential game in the one-shot setting [28].
Note that identicalreward game where agents share a same reward function naturally satisfies the above condition andserves as one important special case of MPG.
For non-identical reward settings, [23, 12] found thatcontinuous MPGs can model applications such as the great fish war [19], the stochastic lake game[10], medium access control [23] etc.
For tablular MPGs, [39, 18] also discuss necessary/sufficientconditions that implies a MPG, as well as its application and counterexamples.
Given a MPG, we define the total potential function Φ as:
[Equation]
Given the property in (6), it is straightforward to verify that the NE condition (2) is equivalent toΦ( θ ∗ i , θ ∗− i ) ≥ Φ( θ ′ i , θ ∗− i ) , ∀ θ ′
i , i ∈ N and that for the policy gradient, ∂J
i ( θ ) ∂θ s,ai = ∂
Φ( θ ) ∂θ s,ai for all i , s , a i .
Remark 1 ( Differences between MPG and single-agent/centralized MDP ) .
Because of the existence of the total potential function Φ , it is natural to ask whether MPG renders the multi-agent policygradient similar to single agent policy gradient and thus results and analysis tools developed for singleagent policy gradient in e.g., [2, 24] would be easily extended to the multiagent case.
Unfortunately,this is not the case.
To illustrate how it differs from single-agent/centralized case, we can focus on thespecial type of MPGs where every agent has the same reward function, namely the identical interestcase.
In the single agent/centralized case, there is a unique global optimal solution which correspondsto the convergent stationary policy.
However, in the multiagent case, even if the rewards are identical,because the policy is decentralized , i.e., agents taking independent policies π := π 1 × . . .
× π n ,we loose the connection between stationary policies and optimal policies.
As we shown later, theconvergent stationary policies are Nash equilibria, which are unfortunately non-unique even for theidentical interest case.
Moreover, a key condition that is used in establishing the convergence rate,Łojasiewicz condition (Lemma 1), is also much weaker for the multiagent case compared to singleagent [24]: the left hand side is the Nash gap max i,θ ∗ i Φ( θ ∗ i , θ − i ) − Φ( θ ) instead of the optimalitygap max θ ∗ Φ( θ ∗ ) − Φ( θ ) ).
Note that zero Nash gap does not imply zero optimality gap, as thereexists many NEs of different values.
These differences disable many proof technique used for singleagent case and make the analysis harder and lead to different performance results, as demonstrated inthe rest of the paper.
Relationship between first order stationary point and Nash equilibrium
Before studying convergence performance of gradient play algorithms, it is important to first understand the relationship between the stationary points and the NEs. Unfortunately, equivalence cannotbe established in this setting. Standard optimization theory guarantees that all NEs are stationarypoints, but unfortunately not vice versa.
Under softmax parameterization, there exist non-NE stationary points.
For example, from the gradient formulation (4), it can be shown that any non-NEdeterministic policies are also stationary points.
However, the notion of NE and stationarity areindeed closely related.
This section aims to characterize some differences between NE and non-NEstationary points.
This differentiation of the NE and non-NE stationary points is established by thenon-uniform Łojasiewicz condition (also known as gradient domination) for stochastic games.
Lemma 1.
(Non-uniform Łojasiewicz inequality; proof given in Appendix D) Define
[Equation]
Then we have that
[Equation]
The Łojasiewicz condition (gradient domination) implies that the NE-gap of a policy can be boundedby the norm of its gradient, whereas the term ‘non-uniform’ refers to the factor √ |A
[Equation]
| M
[Equation]
c
[Equation]
, whichcannot be bounded uniformly for all θ .
The counterpart of Lemma 1 for a single-agent MDPwas first introduced in [24, Lemma 8].
One major difference between Lemma 1 and [24, Lemma8] is how c ( θ ) is defined.
In [24], c ( θ ) :
=
min s π θ ( a ∗ ( s ) | s ) , where a ∗ ( s ) is the optimal actionon state s (i.e., a ∗ = argmax a Q ∗ ( s, a ) ), whereas in MPG, because there’s no globally defined
Q ∗ , the a ∗ i in (7) is chosen as the greedy optimal action of the current averaged Q -function (i.e.,a ∗ i ∈ argmax a i Q θi ( s, a i ) ).
Note that because c ( θ ) on the denominator can be zero for certain policies (e.g. one can verify thatany non-NE deterministic policy have c ( θ ) = 0 ), which implies that a θ with gradient norm close tozero is not necessarily near a NE.
Given this observation, we could differentiate the non-NE stationarypoints with NEs by whether c ( θ ∗ ) equals to zero, which is formally stated in the following lemma:
Lemma 2.
(Proof given in Appendix D) Suppose θ ∗ is a stationary point, i.e. ∥∇ Φ( θ ∗ ) ∥
=
0 , thenθ ∗ is a NE if and only if c ( θ ∗ )
= 1 , θ ∗ is not a NE if and only if c ( θ ∗ ) = 0 .
Unregularized gradient play
We first investigate the convergence to NE for gradient and natural gradient play, respectively.
Underthe softmax parameterization, the two schemes are given by
[Equation]
[Equation]
Natural Gradient
[Equation]
(9) where † denotes the Moore-Penrose inverse and F i ( θ ) is the Fisher information matrix for π θ i :
[Equation]
For notational simplicity, we abbreviate the variables
d θ ( t ) , A θ ( t ) i and A θ ( t ) i as d ( t ) , A ( t ) i and A ( t ) irespectively; and denote π θ ( t ) ( a | s ) and π θ ( t ) i ( a i | s ) as π ( t ) ( a | s ) and π ( t ) i ( a i | s ) respectively.
Forthe softmax parameterization, we can establish the equivalence of natural gradient play and soft
Q-learning [13], formally stated in the following lemma.
Lemma 3.
(Proof given in Appendix C) Natural gradient play is equivalent to
[Equation]
Asymptotic convergence to Nash Equilibrium.
As stated in Section 3, there exist stationary pointsthat are not NEs. It is not immediately obvious why running gradient methods can avoid convergingto these points, thus before studying convergence rate to NE, it is necessary to first examine whetherasymptotic convergence holds.
Moreover, the asymptotic convergence result is used to establish thefinite time convergence rate results later (see the subsection 4.1).
Theorem 4.
(Proof given in Appendix E) Suppose Assumption 1 holds and that the stationary policiesare isolated, gradient play (8) with η ≤ (1 − γ ) 3 6 n guarantees that lim t → + ∞ θ ( t ) = θ ( ∞ ) , where θ ( ∞ )is a NE.
The same argument also holds for natural gradient play (10) with η ≤
[Equation]
The proof of Theorem 4 resembles the technique used in [2] for the single agent case, where theadditional assumption on the isolated stationary policies is introduced due to some specific technicaldifficulties encountered in multi-agent learning (see more discussion in Appendix E, which is alsointroduced in [11] for establishing the asymptotic convergence of NPG.
We believe it is a conservativecondition for ensuring the asymptotic convergence.
It remains an interesting open question to establishconvergence without this assumption.
4.1 Finite time convergence rate
This section considers finite time convergence rate for gradient play and natural gradient play.
Corresponding results for the single-agent setting can be found in [24] (for gradient play) and[2, 16, 25] (for natural gradient play).
Some aspects of these analyses can be carried over to the multiagent MPG setting; however, as will be discussed later, there are several fundamental differences thatmake the multi-agent case more challenging.
Our convergence results rely on the observation from Section 3 and the asymptotic convergence to
NE.
Combining Theorem 4 and Lemma 2, we know that c ( θ ( t ) ) asymptotically converges to 1 for(natural) gradient play, and since c ( θ ( t ) ) > 0 for any softmax policy (because π θ i ( a i | s ) > 0 ),c :=
inf t c ( θ ( t ) ) > 0 .
We are now ready to give formal convergence rates for gradient and natural gradient play respectively.
Theorem 5.
(Gradient play and natural gradient play; proof given in F) Suppose Assumption 1 holdsand that the stationary policies are isolated, gradient play (8) with η = (1 − γ ) 3 6 n will guarantee thatfor all T
[Equation]
[Equation]
Natural gradient play (10) with η = (1 − γ
[Equation]
2 2 n ( ϕ max − ϕ min ) will guarantee that for all T ,
[Equation]
Here O ( · ) hides constant factors, M and c are defined as in (5) and (11) , respectively.
Remark 2 ( Proof sketch and novelty ) .
The proof for gradient play is relatively straightforwardfrom the non-uniform Łojasiewicz inequality and standard non-convex optimization results, whichwe refer readers to the appendix for more details. However, the proof for natural gradient play ismore involved and existing analysis on NPG cannot be generalized to this setting.
For single-agent MDP, the analysis on NPG leverages the unique existence of optimal value function V ∗ so thatsimilar analysis for mirror-descent can also carry over to NPG analysis, and thus obtain dimensionfree convergence.
However, in the multi-agent setting, there’s no well-defined V ∗ as NEs can benon-unique with different potential values, thus, we need to further deploy additional structures ofthe total potential function Φ .
Our analysis rely on the sufficient ascent lemma (Lemma 20) thatlower bounds the ascent amount Φ( θ ( t +1) )
− Φ( θ ( t ) ) for each natural gradient step (we would liketo further note that this sufficient ascent lemma cannot be trivially obtained by the smoothness ofΦ ). Then, we further lower bound the ascent amount in terms of NE-gap (Lemma 21). Lastly, thetheorem follows by conducting standard telescoping techniques.
Discussion on 1 c : The complexity results in Theorem 5 both depend on 1 c .
However, this term canbecome arbitrarily large.
In fact, [20] show that c can be exponentially small in terms of the numberof states |S| for a general finite MDP, even under uniform initialization, hence convergence can bevery slow.
This conclusion is also confirmed by numerical evidence.
As pointed out by [24], even forsingle agent settings, policy gradient can get stuck at regions with small gradient yet far from beingglobal optimal.
Similar or even worse phenomena can be observed for multi-agent MPG, as shownin Figure 1(a)-(c): even for a single state game ( |S| = 1 ) with uniform initialization, unregularizedgradient based algorithms can still enter regions with a relatively large NE-gap while the gradientnorm and c ( θ ) are close to zero. More comparison with learning for single-agent MDP: For gradient play, we have establishedan iteration complexity of O (cid:16) n
[Equation]
|
[Equation] [Equation] (cid:17) to find an ϵ -NE, whereas [24] show acomplexity of O (cid:16) ( ϕ max − ϕ
[Equation]
) M [Equation] (1 −
[Equation]
ϵ (cid:17) to reach an ϵ -global optimum for policy gradient in a singleagent MDP.
The dependence on 1 ϵ is better in the single agent case because of the existence of aglobal optimal policy π ∗ and optimal total reward V ∗ , which justify the definition of optimality gapδ t =
V ( θ ( t ) ) − V ∗ .
This, combined with the non-uniform Łojasiewicz condition which bounds δ tby the gradient norm, allows one to use techniques from convex smooth analysis to show that δ t ison the scale of 1 t .
By contrast, for multi-agent learning, there can be multiple NEs with differentvalues, hence δ t is ill-defined.
Further, note that the NE-gap is different from the optimality gap,hence gradient ascent no longer guarantees monotonic decreasing of NE-gap (Figure 1(a)), and wecan only exploit non-convex optimization techniques that yield O ( 1 ϵ 2 ) complexities.
For the same reason, the rate of convergence we obtain for natural gradient play is
O
[Equation]
(cid:17) , which is worse than the dimension free convergence rate of O (cid:16)
[Equation]
given in [2] for single-agent MDPs.
(A better exponential convergence rate for natural PG has alsobeen proved in [16, 25] with the exponential factor being problem dependent.)
Nevertheless, thedependence on 1 c , 11 − γ and M is better than gradient play, suggesting that the preconditioning ofnatural gradient play at least partially captures the geometry of the parameter space.
We also note thatthe quadratic dependence on ( ϕ max − ϕ min ) might be a proof artifact.
It remains an open questionwhether this can be reduced to a linear dependence.
Gradient play with log -barrier regularization
The previous section has shown that, for unregularized objectives, the convergence rate for gradientbased algorithms depends on a factor 1 c that can be arbitrarily large for bad initializations.
Thismotivates us to investigate regularization, in hopes of removing the dependence on 1 c .
For thispurpose, we consider log -barrier regularization:
[Equation]
Define:
[Equation]
It is not hard to verify that the gradient with respect to J i is:
[Equation]
Discussion on the choice of the regularizer: Before analyzing the resulting algorithm we firstdiscuss the motivation for this regularizer.
First, note that for each agent, the additional regularizeronly depends on an agent’s own local policy, which is desirable for multiagent RL.
As an alternative,one might impose regularization by choosing
[Equation]
i.e., so that the regularization weight imposed on a state s depends on the state visitation probabilityd θ ( s ) .
However, in this case the gradient of the i -th agent ∇ θ i (cid:101) Φ( θ ) will not only depend on its ownpolicy parameter θ i , but also on other parameters of the other agents’ policies θ − i .
Thus, runninggradient based algorithms with such a regularization scheme can no longer be executed in a fullydecentralized manner using local policy information.
Therefore, we prefer regularization (14) whichdoes not depend on d θ ( s ) .
Secondly, we adopt the log -barrier instead of entropy regularizationdue to technical rather than practical considerations.
Although entropy regularization achieves fastexponential convergence in single agent learning [7, 24], for multi-agent learning, we haven’t beenable to obtain results as strong as the log -barrier regularization. Intuitively, the log -barrier regularizedgradient field repels the trajectory from regions with small π i ( a i | s ) values (where the geometrybecomes close to singular) more strongly, which enables us to obtain our current analysis. However,we emphasize that our result does not imply that log-barrier is better than entropy regularization inpractice.
It remains future work to determine whether entropy regularization, or other methods suchas trust region based methods, can achieve the same, or even better convergence rates.
5.1 Gradient play
We first consider gradient play algorithm, i.e.,
[Equation]
i
[Equation]
Fortunately, similar analysis from [2] for single-agent MDP can be generalized to MPG with slightmodifications.
Here we only state the result and defer the proof to Appendix G.1.
Theorem 6.
Under Assumption 1, for η = (1 −
[Equation]
[Equation]
+2 λ max i |A
i | (1 − γ [Equation] 3 , and λ =
ϵ M max i |A
i | , let θ (0)be the uniform random policy, i.e., θ (0) = 0 , then running gradient play (15) for T steps, where T
[Equation]
(cid:17) will guarantee that min 0 ≤ t ≤ T − 1 NE-gap ( θ ( t ) ) ≤ ϵ.
Note that compared to the unregularized case in Theorem 5, it only requires Assumption 1, while theconvergence rate is accelerated by eliminating the dependence on 1 c .
However, as a (worthy) tradeoff,the dependence on the action space size max i |A
i | now becomes quadratic.
The key reason for thesedifferences is that log -barrier regularization assures that any policy with sufficiently small gradientnorm cannot be close to the boundary of the probability simplex where the non-uniform Łojasiewiczconstant is large.
5.2 Natural gradient play
In the unregularized setting, we have seen that natural gradient play enjoys a better convergence ratethan gradient play, which motivates us to consider whether a similar advantage still holds for theregularized case.
In this section we consider natural gradient play
[Equation]
(16) which is equivalent to (see the proof in Appendix C)
[Equation]
Theorem 7.
(Proof given in Appendix G.2)
Under Assumption 1, forη =
min
[Equation]
, the natural gradient play (17)will guarantee that (cid:80) T − 1 t =0 NE-gap ( θ ( t ) ) T ≤ 9 ( (cid:101) Φ( θ ( T ) ) − (cid:101) Φ
[Equation]
θ [Equation]
[Equation]
ηλT + λ max i |A i | M, Further, by settingλ = ϵ 2max i |A
i | M , θ [Equation]
= 0 , for T
≳
O (cid:16) n max i
[Equation]
A i | ( ϕ max − ϕ min ) M 2 (1 − γ ) 4 ϵ(cid:17) , we have (cid:80)
[Equation]
≤ ϵ.
Remark 3 .
( Proof sketch and novelty ) As also stated for unregularized natural gradient play, there’sno direct analysis tools we could borrow from literature for the analysis of natural gradient play. Ouranalysis depends on two key lemmas. The first is a sufficient ascent lemma on (cid:101)
Φ( θ ( t +1) ) − (cid:101) Φ( θ ( t ) )for each natural gradient step (Lemma 26). Another key lemma (Lemma 24) states that the algorithmimplicitly ensures that the policies never go near the boundary of the probability simplex, i.e., it canbe uniformly lower-bounded by π ( t ) i ( a i | s ) ≥ λ 4 (cid:16) λ |A
i | M + 1 (1 − γ )2 (cid:17) , ∀ t . Combining the two lemmas, itcan be concluded that the ascent value (cid:101) Φ( θ ( t +1) ) − (cid:101) Φ( θ ( t ) ) can be bounded by NE-gap ( θ ( t ) ) plusa λ max i |A i | M bias term (Lemma 27 and 28), thus the proof is finished by standard telescopingtechnique and choosing an appropriate λ .
Compared with gradient play, natural gradient play manages to reduce the time complexity by amax i |A
i | factor.
Further, gradient play only guarantees the minimal NE-gap smaller than ϵ , whilenatural gradient play guarantees the average NE-gap along the trajectory smaller than ϵ .
To the bestof our knowledge, this is the best time complexity bound for the softmax parameterization in a MPG.
An Illustrative example
This section aims to gain a better understanding of the four gradient play algorithms, (8), (10),(15), and (17).
To better justify our theoretical results and provide additional insights, we choose acarefully designed simple two-player game so that our theoretical results can be easily revealed from a 2 =1 a 2 = a 1 = - 0.
a 1 = 0. a 1 = 0.
[Figure]
the empirical observations.
However the four algorithms also works for settings with more agents.
Due to space limits, we defer the simulation with more agents in Appendix A.
The reward table as well as the performance of the four algorithms are shown in Figure 1.
Comparingthe log -barrier regularized algorithms to the unregularized counterparts, one can see that the regularized dynamics converge faster but with a bias induced by the regularizer.
This finding corroboratesthe analyses given in Theorem 6 and 7.
By contrast, the unregularized dynamics are able to find apolicy with zero NE-gap asymptotically, but tend to get stuck in regions where c ( θ ( t ) ) is very close tozero, as illustrated in Fig 1(a)(b).
Specifically unregularized natural gradient play gets stuck arounditeration 100-400 in a region where the gradient norm and c ( θ ( t ) ) are both close to zero while the
NE-gap is not.
This corroborates the finding in Lemma 1.
Similar behavior can be observed forgradient play if we keep running the algorithm.
In comparing the natural gradient play to gradientplay algorithms, natural gradient play generally converges faster, which matches with our complexityanalysis.
However, natural gradient play with log -barrier regularization can suffer from numericalinstability due to the 1 /π ( t ) i ( a i | s ) term in the exponential factor.
In this case, the stepsize needs to bechosen carefully.
To bypass the numerical instability, we truncate the update step of natural gradientplay with log -barrier regularization by a maximum absolute value of 1 for each entry.
Discussions and conclusions
We have established finite time iteration complexity bounds for gradient and natural gradient playunder the softmax parameterization, considering both unregularized and log -barrier regularizeddynamics, in the Markov potential game setting.
To our best knowledge, these are the first finitetime global convergence results for softmax gradient play for MPGs.
However, our work suffersfrom the following limitations: firstly, the paper mainly focuses on MPG settings, which limitsits application to general-sum Markov games; secondly, convergence results for the unregularizedcase relies on an extra assumption that the stationary points are isolated; thirdly, for the regularizedcase, we consider log -barrier regularization, which is admittedly a stronger regularization comparedwith entropy regularization which is more frequently used in practice.
Some limitations are due totechnical challenges, some might be caused by the fundamental difficulties of multi-agent learning.
Itremains interesting open questions to sharpen the analysis, derive similar or better bounds for otherregularizations, and to develop more fundamental understandings of multi-agent learning.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See the contribution paragraph inSection 1, assumptions in theorems, and Section 7: Discussions and Conclusions.(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] As this
work is primarily theoretical, we do not foresee any potentially negative societal impacts
of this work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] The URL is provided in Section
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?
[N/A] Our experiments are deterministic
(d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]