Finding Correlated Equilibrium of Constrained Markov Game: A Primal-Dual Approach
Anonymous Author(s)
AffiliationAddress email
Abstract
Constrained Markov game is a fundamental problem that covers many applications,where multiple agents compete with each other under behavioral constraints.
Theexisting literature has proved the existence of Nash equilibrium for constrained
Markov games, which turns out to be PPAD-complete and cannot be computedin polynomial time.
In this work, we propose a surrogate notion of correlatedequilibrium (CE) for constrained Markov games that can be computed in polynomial time, and study its fundamental properties.
We show that the modificationstructure of CE of constrained Markov games is fundamentally different fromthat of unconstrained Markov games.
Moreover, we prove that the corresponding
Lagrangian function has zero duality gap.
Based on these results, we develop thefirst primal-dual algorithm that provably converges to CE of constrained Markovgames.
In particular, we prove that both the duality gap and the constraint violationof the output policy converge at the rate O ( 1 √ T ) .
Moreover, when adopting the
V-learning algorithm as the subroutine in the primal update, our algorithm achievesan approximate CE with ϵ duality gap with the sample complexity
O ( H 9 SA 2 ϵ − 4 ) .
Introduction
Markov game [44] is a fundamental problem in game theory where multiple agents compete witheach other by interacting with a dynamic environment.
It has broad applications in diverse fields,including board games [45], inverse reinforcement learning [62], market pricing [35] etc.
In the existing literature, the goal of Markov game is often formulated as achieving Nash equilibrium(NE) – a product policy under which no agent can benefit via deviating from its own policy alone.
Although NE has been shown to exist for general Markov games [23], computing NE turns out to bea PPAD-complete problem that cannot be solved in polynomial time [18,33], except for some special
Markov games with zero-sum structure [12,17,22,64,65] or potential structure [36,59].
This furthermotivates researchers to consider other surrogate notions of equilibrium that are slightly weaker than
NE but are computationally tractable, and correlated equilibrium (CE) is such a classic and popularsurrogate notion [41] (see Section 2 for the formal definition) for which many convergent and efficientalgorithms have been developed very recently [32,38,47].
While Markov game has been well studied, its formulation has limited applicability to real-lifeapplications where the behavior of agents is usually subject to certain constraints.
For example, the users in a distributed power network have limited power budget [4], the users in a wirelesscommunication network are assigned to limited bandwidth [28,63], etc.
To overcome this limitation,constrained Markov game has been introduced where agents compete with each other under local behavioral constraints encoded by state value functions [5].
However, most existing works onconstrained Markov game only studied Nash equilibrium (NE) that is PPAD-complete to compute[3,5–7,21,25,28–31,46,49,52,53,56,60,61,63], and a tractable notion of correlated equilibrium(CE) has not been formally defined for constrained Markov games.
To the best of our knowledge, [27]is the only work that explored CE in constrained Markov games.
However, they define CE based onthe unconstrained Lagrangian function associated with the constrained Markov game, which doesnot correspond to an equilibrium of the original constrained Markov game.
Therefore, the goal ofthis work is to formally define a tractable notion of CE for constrained Markov games, study itsfundamental properties and develop provably convergent and computation-efficient algorithms forachieving such an equilibrium, as we further elaborate below.
To achieve this goal, we need to explore and address several fundamental problems.
First, whileone can generalize the definition of CE of Markov game to the constrained case, it may possessfundamentally different structures due to the presence of constraints.
Specifically, as we introducelater in Definition 2.1, modification (of actions) is a key structure of CE.
In standard Markovgames (without constraints), CE is equivalent under both stochastic and deterministic modifications.
However, it is unclear if such equivalence still holds in constrained Markov games, which criticallyaffects the formal definition of CE as well as the algorithm design.
Second, a classic and powerfultool to handle constrained problems is the strong duality of the associated Lagrangian, which hasbeen established for NE of constrained Markov games [31,63].
Thus, we are inspired to explore ifstrong duality holds for CE of constrained Markov games, which is a challenging problem due tothe complex modification structure of CE and the nonconvex Lagrangian.
Lastly, once we have acomprehensive understanding of the above two problems, we can hope for a provably convergentand efficient primal-dual algorithm for finding CE of constrained Markov games.
In particular,it is much desired to analyze the non-asymptotic convergence rate and sample complexity of theproposed algorithm, as the existing algorithms for constrained Markov games either lack convergenceguarantee [31] or only have asymptotic convergence guarantee [27,63].
1.1 Our Contributions
We define a notion of CE for constrained Markov games as a generalization of the CE for Markovgames.
It is defined as an equilibrium at which no feasible stochastic modification of the jointpolicy from any agent alone can improve the associated value function (see Definition 3.2).
Such ageneralized CE turns out to possess a structure of modification that is fundamentally different fromthe CE of Markov games.
Specifically, we prove in Theorem 1 that our proposed CE for constrained
Markov games is strictly stronger with stochastic modification than with deterministic modification.
As a comparison, the CE of Markov games is equivalent with both types of modifications.
This factmotivates defining CE with stochastic modification for constrained Markov games, which furtherlays the foundation for developing the strong duality result and the primal-dual algorithm later.
We study the Lagrangian function associated with the proposed CE for constrained Markov gamesand prove that it has zero duality gap (See Theorem 3).
Such a result is the key for developinga convergent primal-dual algorithm.
Compared with the strong duality established in the existingworks [2, 31, 43], the proof of our result for constrained Markov games requires new technicaldevelopments.
To elaborate, the existing works yield Lagrangian functions that are defined over theentire unconstrained policy space.
As a comparison, our Lagrangian function is defined over the setof stochastically modified policies, which corresponds to a subset of the policy space over which the
Lagrangian function is nonconvex.
To address this challenge, we transform the Lagrangian functionfrom the policy space to the corresponding space of probability measures on episodes induced by thestochastically modified policies.
We show that the transformed Lagrangian function is linear and thetransformed space is convex and compact, which further implies the desired strong duality.
Then, we propose the first primal-dual algorithm that provably converges to CE of constrained Markovgames.
In particular, the primal update of our algorithm requires to compute an approximate CE ofan unconstrained Markov game associated with the Lagrangian function, which can be efficientlysolved by many existing algorithms, e.g., V-learning [32,47] and Nash Value Iteration [38].
We prove that both the duality gap and the constraint violation of the output policy produced by our algorithmconverge at the rate O ( 1 √ T ) , where T denotes the number of iterations.
This rate is comparableto that of the primal-dual algorithm for single-agent constrained reinforcement learning [19, 20].
Moreover, when adopting the V-learning algorithm as the subroutine in the primal update, ouralgorithm achieves an approximate CE with ϵ duality gap and constrained violation with the samplecomplexity
O ( H 9 SA 2 ϵ − 4 ) , where H denotes the episode length, S, A denote the cardinality of thestate space and action space, respectively.
1.2 Other Related Work
Markov game: [54] introduced coarse correlated equilibrium (CCE) for Markov game that is weakerthan CE.
Many algorithms such as V-learning [32,40,47] and Nash Value Iteration [38] can achievean approximate CCE with finite time convergence guarantee.
Various other Markov game settingshave also been studied.
For example, [15,26] studied mean-field Markov game with a large number ofagents.
In a Stackelberg game [9,51], agents are partitioned into leaders and followers.
The followersselect the best response to the leaders’ actions, and the leaders’ goal is to achieve Stackelbergequilibrium which corresponds to the leaders’ optimal strategy.
Strategic game can be seen as aspecial case of Markov game without state and transition [42].
Extensive-form game [11,24,34,66]can be seen as a special case of Markov game where any state can only be reached from a uniquestate at the previous step due to the tree structure of the game.
Constrained reinforcement learning: Single-agent constrained reinforcement learning (RL) is asingle-agent RL problem with behavioral constraints.
It is usually formulated as constrained Markovdecision process [2], which can be seen as a special case of constrained Markov game with onlyone agent.
Various primal-dual algorithms have been proposed for single-agent constrained RL[1,2,19,20,37,48,50,57,58].
Other types of algorithms include Lyapunov-based algorithms that use
Lyapunov functions [13,14], interior point methods that use logarithmic barrier functions [39], policynetwork with a special layer to encode constraints [16], and primal algorithms that update the policyvia alternatively maximizing the objective function and minimizing the violated constraints [55].
Preliminaries of Markov Game
We consider M agents playing a Markov game over a finite time horizon of episode length H .
Specifically, at every time step h = 1 , ..., H , the agents observe a global state s h ∈ S of the environment, where S denotes the state space.
Then, the agents take a joint action a h =
[ a (1) h , . .
, a ( M ) h ]following a joint stochastic policy π h ( ·| s 1: h , a 1:( h − 1) ) , which corresponds to a distribution on thejoint action space A :
= (cid:81)
Mm =
1 A ( m ) and depends on the past states s 1: h := { s t } ht =1 and past actionsa 1:( h − 1) :=
{ a t } h − 1 t =1 .
After that, the global state of the environment transfers to a new state s h
+1 following the state transition kernel P ( ·| s h , a h ) , and each agent m receives a local reward r ( m ) 0 ,
h ( s h , a h )from the environment.
We note that here the joint policy π h can be a non-product (i.e., cannot befactorized into a product of independent policies associated with the agents) and non-Markov policy.
In the above Markov game, each agent m collects its own rewards over the episodes.
Denote ρ as theinitial state distribution, we can define the following value function associated with agent m .
[Equation]
(1) which corresponds to the expected cumulative reward received by agent m under joint policy π :={ π h } H h =1 and initial state s .
The goal of agent m is to optimize its own policy π ( m ) :
= { π ( m ) h } Hh =1in order to maximize its associated value function V ( m )( π ) .
However, since every agent’s valuefunction is also affected by the other agents’ policies and actions, the agents must compete with eachother to gain more rewards until they reach a certain equilibrium.
Here, we introduce two popularequilibrium notions that will be discussed throughout the paper.
Nash Equilibrium (NE).
A joint policy π is called an NE if the following two conditions are met:(i) for any time step h , the joint policy π h is a product policy that can be factorized into a productof independent policies, i.e., π h
= π (1) h × . . .
× π ( M ) h so that the agents take independent actions;(
ii)
For any agent m with any associated policy (cid:101)
π ( m
) , it holds that V (
m
)( π ) ≥ V ( m )( (cid:101) π ( m ) × π ( \ m ) ) .
Here, π ( \ m ) denotes the joint policy of all the other agents excluding the agent m , and ‘ × ’ meansthat (cid:101) π ( m ) is independent from π ( \ m ) .
In other words, at NE, no agent can improve its value function by deviating from its associated policy.
In the existing literature, it has been shown that computing NE is a PPAD-complete problem [18,33],for which it is not possible to develop polynomial-time algorithms.
This observation has motivatedresearchers to propose a surrogate correlated equilibrium (CE) notion [41].
Before introducing theformal definition of CE, we first define the following stochastic modification operator.
Definition 2.1 (Stochastic modification) .
For any time step h , denote a ( m ) h as agent m ’s actioninduced by joint policy π h .
Given the past states and actions s 1: h , a 1:( h − 1) , a stochastic modificationϕ ( m ) h (associated with agent m ) is a distribution that randomly maps a ( m ) h to another action (cid:101) a ( m ) h ,i.e., (cid:101) a ( m ) h ∼ ϕ
( m ) h ( ·| s 1: h , a 1: h − 1 , a ( m ) h ) .
Moreover, we denote ϕ ( m ) h ◦ π h as the joint policy modifiedby ϕ ( m ) h , i.e., π h first generates a joint action a h :=
[ a ( m ) h , a ( \ m ) h ] , and then ϕ ( m ) h modifies a ( m ) h toanother (cid:101) a ( m ) h at random.
We also denote ϕ ( m ) :
= { ϕ ( m ) h } Hh =1 , ϕ ( m ) ◦ π :=
{ ϕ ( m ) h ◦ π h } Hh =1 .
Correlated Equilibrium (CE).
A joint policy π is called a CE if for any agent m and any stochasticmodification ϕ ( m ) , it holds that V ( m )( π ) ≥ V ( m )( ϕ ( m ) ◦ π ) .
Intuitively, at CE, no agent can improve its value function by modifying its own action induced by thejoint CE policy.
Compare to NE policies, CE policies do not require joint independence among allthe agents.
In fact, it has been shown that any NE policy is guaranteed to be a CE policy [32,38,47],and hence CE is a weaker equilibrium notion than NE.
Moreover, CE can be reformulated as linearprogramming and therefore is computationally tractable.
Correlated Equilibrium of Constrained Markov Game
In this section, we introduce constrained Markov game – a generalization of the standard Markovgame with constraints on agents’ behavior.
We then propose and study a new notion of correlatedequilibrium for constrained Markov games.
3.1 Constrained Markov Game
In many real-life Markov games, the behavior of the agents is usually restricted by certain constraints,e.g., constraints on power, bandwidth, safety, etc.
This motivates us to consider a constrained versionof Markov game [2].
Constrained Markov game follows the same state/action transition as that of thestandard Markov game introduced in Section 2.
The key difference is that, after the state transition ateach time step h in a constrained Markov game, every agent m receives d m +1 number of rewards{ r ( m ) j,h
( s h , a h ) }
d m j =0 .
Here, the accumulation of the reward r ( m ) 0 ,h corresponds to the target rewardthat agent m aims to maximize, whereas the accumulations of the other rewards { r ( m ) j,h } d m j =1 encodethe constraints of the behavior of agent m .
More specifically, define the following value functionsassociated with the reward functions of agent m :
[Equation]
Every agent m aims to optimize its own policy π ( m ) via the following constrained Markov game.(Constrained Markov Game
[Equation] [Equation]
where { c ( m ) j } d m j =1 ∈ R denote the thresholds of the constraints.
Intuitively, the constraints on thevalue functions implicitly impose constraints on the joint policy π , and all the agents aim to maximizetheir own target value functions and achieve a certain equilibrium under their local constraints.
3.2 Nash Equilibrium and Correlated Equilibrium
For constrained Markov games, we can define the following notion of Nash equilibrium (NE) that issimilar to the standard Nash equilibrium but takes into account the constraints [3,28,46,52,56,61].
Definition 3.1 (NE of constrained Markov game) .
For a constrained Markov game, a joint policy πis called an NE if the following conditions are met: (i) for any time step h , the joint policy π h is aproduct policy that can be factorized into a product of independent policies; (ii) π is feasible, i.e., itsatisfies the constraints of all the agents; (iii) For any agent m with any associated policy (cid:101) π ( m ) suchthat (cid:101) π ( m ) × π ( \ m ) is feasible, it holds that V ( m )
[Equation]
Compared with the NE of Markov game defined in Section 2, NE of constrained Markov game furtherrequires the joint policies π and (cid:101) π ( m ) × π ( \ m ) to be feasible.
It has been shown that such an NE existsfor general constrained Markov games [5].
Moreover, since constrained Markov game generalizesthe Markov game, computing NE of constrained Markov game is also a PPAD-complete problem.
Therefore, we are motivated to propose the following surrogate notion of correlated equilibrium (CE).
Definition 3.2 (CE of constrained Markov game) .
For a constrained Markov game, a joint policy πis called a CE if the following conditions hold: (i) π is feasible, i.e., it satisfies the constraints of allthe agents; (ii) for any agent m and any stochastic modification ϕ ( m ) such that ϕ ( m ) ◦
π is feasible,it holds that V ( m )( π ) ≥ V ( m )( ϕ ( m ) ◦ π ) .
In particular, if π is a product policy, it can be seen that ϕ ( m ) ◦ π is also a product policy and then CEwould reduce to NE.
Therefore, the above definition of CE generalizes NE of constrained Markovgames.
We note that [27] also defined CE for constrained Markov games, but in a very differentaspect.
Specifically, they only defined CE based on the unconstrained Markov game associatedwith a Lagrange function (i.e., with surrogate reward defined in eq. (8)), which does not necessarilyrequire π and ϕ
( m ) ◦ π to be feasible but our Definition 3.2 requires. Therefore, the existing CEproposed in [27] essentially follows the CE of unconstrained Markov games, whose definition isstated after our Definition 2.1.
Moreover, [27] defined CE based on the state-action value functionand consider stationary Markov policies in the discounted infinite horizon setting, whereas we define CE based on the state value function and consider non-Markov policies in the finite horizon setting.
It may seem that the CE in Definition 3.2 is a straightforward generalization of that of the standard
Markov game.
However, it turns out to possess a fundamentally different structure of the modificationoperator at the equilibrium point due to the presence of constraints, as elaborated in the followingfundamental result.
Throughout the paper, we call a stochastic modification operator deterministic ifits pdf is supported on a single action.
Theorem 1 (Modification of CE) .
For unconstrained Markov games.
If a policy π satisfies that V ( m )( π ) ≥ V ( m )( ϕ ( m ) ◦ π ) for anyagent m and any deterministic modification ϕ ( m ) , then π must be a CE.
For constrained Markov games.
Even if a feasible policy π satisfies that V ( m )( π ) ≥ V ( m )( ϕ ( m ) ◦π ) for any agent m and any feasible deterministic modification ϕ ( m ) , π may not be a CE.
Theorem 1 shows that the structure of modification at the CE of Markov games is different fromthat of constrained Markov games.
To elaborate item 1, recall that a CE policy π of a Markovgame is defined to satisfy V ( m ) ( π ) ≥ V ( m ) ( ϕ ( m ) ◦ π ) for any stochastic modification ϕ ( m ) (see
Section 2).
Since stochastic modifications include deterministic modifications, item 1 proves that
CE of Markov games can always be achieved by a deterministic modification.
Therefore, it sufficesto define CE of Markov games using deterministic modifications, as adopted by many existingworks [32, 38, 47].
As a comparison, item 2 shows that for constrained Markov games, the CEdefined using deterministic modification is in general weaker than the CE defined using stochasticmodification (see Definition 3.2), which is the focus of this paper.
More importantly, as we showlater in Theorem 2, defining CE of constrained Markov games using stochastic modification allowsus to establish a strong duality result, which is the key to develop convergent primal-dual algorithms.
The proof of Theorem 1 is non-trivial and we provide a proof sketch here.
To prove item 1, ourstrategy is to construct a simple deterministic modification ϕ ( m ) (see Appendix C for the construction) for any optimal stochastic modification (cid:101) ϕ ( m ) and show that they achieve the same optimal valuefunction, i.e., V ( m )( ϕ ( m ) ◦ π )
=
V ( m )( (cid:101) ϕ
( m ) ◦ π ) .
To show this, we transform the value functionsfrom the policy space π to the space of probability measure p π of episodes induced by the associatedpolicy, i.e., V ( m )( π )
= (cid:101) V ( m )( p π ) (see eq.
(22)), so that the transformed value function (cid:101) V ( m )is linearin p π .
Then we consider the extrapolated probability measure p λ :=
λp ϕ ( m ) ◦ π
+ (1 − λ ) p (cid:101) ϕ
( m ) ◦ π andshow that there exists λ <
such that p λ is a measure induced by a modified policy ϕ ( m ) λ ◦ π forcertain proper stochastic modification ϕ ( m ) λ .
This together with the linearity of (cid:101) V ( m )proves item 1.
To prove item 2, we construct a counterexample of constrained Markov game such that its CE understochastic modification is shown to be strictly stronger than its CE under deterministic modification.
A Primal-Dual Algorithm for Constrained Markov Game
In this section, we develop a primal-dual algorithm for finding correlated equilibrium (CE) ofgeneral constrained Markov games.
Throughout this section, to simplify the notation, we denote
V ( m ) ( π ) :=
[ V ( m )( π ) , ..., V ( m )
d m ( π )] as the collection of all the value functions in the constraints,and denote c ( m ) :
=
[ c ( m ) 1 , ..., c ( m )
d m ] as the collection of all the thresholds of the constraints.
4.1 Strong Duality for Correlated Equilibrium
Consider any feasible policy π of a constrained Markov game, i.e., V ( m ) ( π ) ≥ c ( m ) holds entrywise.
The CE of constrained Markov game is related to the following constrained optimization problem.
For m = 1 , ..., M ,
[Equation]
where ‘ ≥ ’ applies entrywise and the maximization is taken over all stochastic modifications.
Tofurther explain, define the following duality gap for every agent m .
[Equation]
where a modification ϕ ( m ) is called feasible if ϕ ( m ) ◦ π satisfies the constraint in eq.
Clearly, wehave D ( m ) ( π ) ≥ 0 as the identity modification (i.e., ϕ ( m ) ◦ π = π ) is feasible.
Moreover, it can beseen that π is a CE of constrained Markov game if and only if max m D ( m ) ( π ) = 0 .
Since the problem in eq.
(4) takes a constrained form, it is natural to consider its equivalent Lagrangianformulation, which is written below.
[Equation]
where λ ( m ) ∈ R
d m + is the dual variable.
In order to develop convergent primal-dual algorithms forsolving the constrained problem, a fundamental problem that needs to be addressed is the dualitygap of the above Lagrangian function.
In conventional constrained convex optimization [10], strongduality can be achieved under a Slater’s condition, which requires the existence of a strictly feasiblepoint under the constraints.
Recently, this condition has also been introduced to establish strongduality for single-agent constrained reinforcement learning [19, 20, 43].
For constrained Markovgame, we introduce the following Slater-type condition to study its duality gap.
Assumption 1 (Slater’s condition) .
For any agent m and any joint policy π , there exists a stochasticmodification ϕ ( m ) such that V ( m ) ( ϕ ( m ) ◦ π ) − c ( m ) ≥ ξ ( m ) for some strictly positive ξ ( m ) .
The above Slater’s condition assumes that agents can always find a strictly feasible stochasticmodification, regardless of the joint policy.
Such a condition is closely related to the existing Slatertype condition for constrained Markov games [3,5,46], which replaces the modification operatorϕ ( m ) by an independent local policy (cid:101) π ( m ) and is used to study NE.
With the above Slater’s condition,we are able to establish the following strong duality result for constrained Markov game.
Theorem 2 (Strong duality) .
Let Assumption 1 hold.
Then, strong duality holds for constrained
Markov games, i.e., for any joint policy π and any agent m ,
[Equation]
To the best of our knowledge, such a strong duality result has not been established before for CEof constrained Markov games, and later we show that this is essential for developing a primal-dualalgorithm for finding CE of constrained Markov games.
In particular, compared to the strongduality results of single-agent constrained RL [2,43] and those for finding Nash equilibrium (NE) ofconstrained Markov game [31], the proof of Theorem 1 for finding CE of constrained Markov gameis substantially different and more challenging, as we elaborate below.
Technical novelty.
Note that the constrained settings considered in the aforementioned existingworks [2,31,43] lead to Lagrangian functions defined over the entire unconstrained policy space,which is clearly convex and helps establish the strong duality result.
As a comparison, in order tofind CE, our Lagrangian function L ( m ) is a nonlinear function in ϕ ( m ) and is defined over the set ofmodified policies { ϕ ( m ) ◦ π : ϕ ( m ) is stochastic modification } for a fixed π .
Such a set is a subsetof the entire policy space and is not easy to analyze directly.
To overcome this challenge, we firstrewrite L ( m ) ( ϕ ( m ) ◦ π, λ ( m ) ) as (cid:101) L ( m ) ( p ϕ ( m ) ◦ π , λ ( m ) )
(see eq.
(23)), which is a function of p ϕ ( m ) ◦ π– the probability measure of the episode ( s 1: H , a 1: H ) induced by the modified policy ϕ ( m ) ◦ π .
Since we showed that (cid:101) L ( m ) is linear in both of its arguments, strong duality would follow if we canprove that the domain of the first argument X := { p ϕ ( m ) ◦ π : ϕ ( m ) is stochastic modification } is aconvex and compact set, and the proof of this result requires substantial effort.
More specifically, toprove convexity we need to show that the convex combination λp ϕ ( m ) ◦ π + (1 − λ ) p (cid:101) ϕ
( m ) ◦ π for anyλ ∈ [0 , 1] and any modifications ϕ ( m ) , (cid:101) ϕ
( m ) is a valid probability measure of the episode induced by acertain modified policy, and to prove compactness we need to show the closedness of any convergingsequence { p ϕ ( m )
[ k ] ◦ π } k .
To prove these claims, we need to study the fundamental properties of themeasure p π and the modified measure p ϕ ( m ) ◦ π in Lemma A.1 and Lemma B.1.
4.2 A Primal-Dual Algorithm for Finding Correlated Equilibrium
Based on the strong duality result in Theorem 2, finding CE of constrained Markov game is equivalentto solving the minimax optimization problem in eq.
(7) for all agents m , which can be effectivelysolved by a primal-dual algorithm that alternatively updates the primal variable ϕ ( m ) and the dualvariable λ ( m ) .
While λ ( m ) can be updated using the standard projected dual descent update, theupdate of ϕ ( m ) requires further exploration as we elaborate below.
Specifically, with a fixed λ ( m ) , the primal update needs to solve the optimization problemmax ϕ ( m ) L ( m ) ( ϕ ( m ) ◦ π, λ ( m ) ) .
Denote r ( m ) ( s, a ) :=
[ r ( m )( s, a ) , . . .
, r ( m )
d m ( s, a )] ⊤ as the collectionof the rewards associated with the constraints, and define the following weighted surrogate reward
R ( m ) λ and its associated value function V ( m ) λ .
[Equation]
Then, the optimization problem of the primal update can be rewritten as follows.
[Equation]
Interestingly, the problem max ϕ ( m ) V ( m ) λ ( ϕ ( m ) ◦ π ) can be understood as finding CE of an unconstrained Markov game with associated reward R ( m ) λ and value function V ( m ) λ .
Therefore, to perform the primal update, we can apply any black-box algorithm for finding CE of unconstrained Markovgames to update the policy.
In particular, some existing popular algorithms include the V-learningalgorithm [32,47] and the Nash Value Iteration algorithm [38], both of which can output a policy thatachieves ϵ -duality gap (defined in eq.
(11)) with provable finite-time complexities.
We summarize this primal-dual algorithm in Algorithm 1, which alternatively updates the policy π tvia finding an approximated CE of an unconstrained Markov game (with fixed λ ( m ) t ), and updatesλ ( m ) t via projected dual descent (with fixed π t ).
In particular, the projection set Λ ( m ) ⊂
R d m + mustcontain the optimal dual variable and also be bounded to ensure a bounded surrogate reward R ( m ) λ .
We give a specific choice of Λ ( m ) in the next convergence analysis Section 5.
Input
Initialize
and obtain an approximate CE policy π t that satisfies
[Equation]
λ ( m ) t via the following projected dual descent step
[Equation]
end
Output: π (cid:101) t where (cid:101) t is sampled from { 0 , 1 , . .
, T − 1 } uniformly at random.
Comparison to the existing art.
Some existing works have developed other primal-dual algorithmswith different primal policy improvement updates for constrained Markov games.
For example, valueiteration (Algorithm 11.1 of [63]), simultaneous perturbation stochastic approximation (Algorithm11.2 of [63]) and policy gradient [31] are used as the primal update to find NE of constrained Markovgames, and Q-learning [27] is used as the primal update to find CE of the Lagrangian functionassociated with the constrained Markov game.
However, none of these works have established nonasymptotic convergence rate and sample complexity for the proposed algorithms.
As a comparison,our Algorithm 1 can use any black-box primal update that can compute an approximate CE ofunconstrained Markov games, and enjoys a provable non-asymptotic convergence rate and lowsample complexity as we show in the next section.
Convergence Analysis and Sample Complexity
In this section, we analyze the non-asymptotic convergence rate of the primal-dual Algorithm 1.
Then, we adopt the V-learning algorithm [32] as the primal update of Algorithm 1, and derive theoverall sample complexity required by Algorithm 1 to achieve an approximate CE with ϵ duality gap.
We adopt the following assumption of bounded reward that is commonly adopted in the analysis ofconstrained RL [19,20,43], Markov game [32,38,47] and constrained Markov game [27].
Assumption 2 (Bounded reward) .
For all agents m = 1 , ..., M , there exists constant r ( m ) j, max > 0such that 0 ≤ r ( m ) j,h ≤ r ( m ) j, max for all 1 ≤ h ≤ H and 0 ≤ j ≤ d m .
Under Assumptions 1 and 2, we choose the following projection set Λ ( m ) for Algorithm 1.
Inparticular, we prove in Lemma E.1 that such a set contains the optimal dual variable.
[Equation]
Hr
[Equation]
m ) 0 , max
[Equation]
max
[Equation]
max
[Equation]
Theorem 3 (Convergence rate) .
Let Assumptions 1 and 2 hold.
Run Algorithm 1 for T iterationswith stepsize η = 1 √ T and projection set Λ ( m ) defined in eq.
Then, the output policy π (cid:101) t achievesthe following duality gap (defined in eq.
[Equation]
where R ( m ) max := (cid:113)(cid:80)
d m j =1 ( r ( m ) j, max ) 2 .
Moreover, define the constraint violation of the output policy π (cid:101) tas W ( m ) ( π (cid:101) t ) :
= (cid:80) d m j =1 ( ξ ( m )
[Equation]
) − 1 (cid:0) c ( m ) j − V ( m ) j ( π (cid:101) t ) (cid:1) + , where ( a ) + := max { 0 , a } .
We obtain that
[Equation]
The above theorem shows that both the duality gap and the constraint violation associated with theoutput policy converge at the rate O ( 1 √ T + ϵ ) , where ϵ is the target duality gap of the unconstrained
Markov game in the primal update.
In particular, a larger slackness { ξ ( m ) j } j,m implies a fasterconvergence of the constraint violation.
These convergence rates are comparable to those of primaldual algorithms for single-agent constrained RL [19,20,37], and the analysis is a generalization ofthe single-agent case.
Specifically, the updated policy π t in the primal-dual algorithm for single-agentconstrained RL is an approximate maximizer of the value function under the constraints, whereas theπ t in our case corresponds to an approximate CE at which all agents can benefit little (up to ϵ ) bymodifying their own actions alone.
Particularly for each agent, the duality gap convergence criterionin eq.
(11) can be understood as a convergence criterion of single-agent constrained RL, and there theproof logic partially follows the analysis of single-agent case [20].
As our Algorithm 1 can adopt any existing algorithm for unconstrained Markov games in the primalupdate, we consider adopting the V-learning algorithm developed in [32].
This is a decentralizedalgorithm that incrementally updates the state value functions, which are further used to updatethe agents’ policies via adversarial bandit algorithms.
In particular, thanks to the use of state valuefunction, it requires much less memory than other algorithms based on state-action value functions[27].
In Theorem 7 of [32], it is proved that the V-learning algorithm achieves an approximate CEwith ϵ duality gap for any unconstrained Markov game with the state-of-the-art sample complexity(cid:101)
O ( H 5 SA 2 ϵ − 2 ) , where (cid:101)
O hides the logarithm factors and S = |S| , A = max m |A
( m ) | correspond tothe cardinality of the state space and action space, respectively.
We then obtain the following samplecomplexity result.
Corollary 5.1 (Sample complexity) .
Let Theorem 3 hold and assume r ( m ) 0 , max ≥ 1 H .
Apply the V-learning algorithm [32] to the primal update of Algorithm 1 and choose T = [Equation]
Then, the output policy π (cid:101) t achievesmax (cid:0) E
[ D ( m ) ( π (cid:101) t )] , E [ W ( m ) ( π (cid:101) t )] (cid:1) ≤ 2 ϵ with the sample complexity (cid:101) O ( H 9 SA 2 ϵ − 4 ) .
To the best of our knowledge, this is the first finite-time sample complexity result establsihed forconstrained Markov games.
Next, we compare our sample complexity with that of two popular RLalgorithms.
The first one is the V-learning algorithm [32] developed for finding CE of unconstrained
Markov games, and it achieves the state-of-the-art sample complexity (cid:101)
O ( H 5 SA 2 ϵ − 2 ) . The information theoretical lower bound is Ω( H 3 SAϵ − 2 ) [32]. As a comparison, the sample complexity ofour primal-dual V-learning algorithm is higher since we require additional projected dual descentsteps (12) to account for the constraints. The other one is the basic primal-dual algorithm for solvingsingle-agent constrained RL problems, which uses natural policy gradient in the primal updateand achieves the state-of-the-art sample complexity O ( ϵ − 4 ) [8].
Such a complexity has the samedependence on ϵ as the sample complexity of our algorithm.
Conclusion
In this work we propose a surrogate notion of correlated equilibrium (CE) for constrained Markovgames and show that it possesses a fundamentally different modification structure from that of CEof Markov game.
Moreover, we prove that the corresponding Lagrangian function has zero dualitygap.
Based on this result, we develop the first primal-dual algorithm that provably converges to
CE of constrained Markov games with the convergence rate O ( 1 √ T ) .
Moreover, when adopting the
V-learning algorithm as the subroutine in the primal update, our algorithm achieves an approximate
CE with ϵ duality gap with the sample complexity O
( H 9 SA 2 ϵ − 4 ) .
Limitations:
In this work,we only develop the basic primal-dual algorithm for constrained Markov games.
An interestingfuture direction is to develop other advanced algorithms such as Nesterov’s momentum-acceleratedprimal-dual algorithms.
Negative social impacts: This is a fundamental theoretical study and doesnot have any potential negative social impact.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See the “Conclusion” section.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See the “Conclusion” section.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[N/A]
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[N/A]
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e
g., type of GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]