Learning in Markov Games: can we exploit a general-sum opponent?
Abstract
In this paper, we study the learning problem in two-player general-sum Markov Games.
We consider the online setting where we control a single player, playing against an arbitrary opponent to minimize the regret.
Previous works only consider the zero-sum Markov Games setting, in which the two agents are completely adversarial.
However, in some cases, the two agents may have different reward functions without having conﬂicting objectives.
This involves a stronger notion of regret than the one used in previous works.
This class of games, called general-sum Markov Games is far to be well understood and studied.
We show that the new regret minimization problem is signiﬁcantly harder than in standard Markov Decision Processes and zero-sum Markov Games.
To do this, we derive a lower bound on the expected regret of any “good” learning strategy which shows the constant dependencies with the number of deterministic policies, which is not present in zerosum Markov Games and Markov Decision Processes.
Then we propose a novel optimistic algorithm that nearly matches the proposed lower bound.
Proving these results requires overcoming several new challenges that are not present in
Markov Decision Processes or zero-sum Markov
Games.
INTRODUCTION
Reinforcement Learning (RL) [Sutton and Barto, 2018] is an area of Machine Learning that studies sequential decisionmaking problems, where a learning agent interacts with an unknown environment to maximize its rewards.
In recent years, RL methods have made substantial progress in solving real-world problems (e g., beating the world champion player of Go
[Silver et al., 2017], solving real-time strategy games [Brockman et al., 2016] and Poker [Moravˇcík et al., 2017, Brown and Sandholm, 2018], in autonomous driving [Shalev-Shwartz et al., 2016], learning communications and emergent behaviours
[Foerster et al., 2016, Lowe et al., 2017, Bansal et al., 2018], providing solutions to robotic control problems [Lillicrap et al., 2015], and managing the power consumption of households [Chung et al., 2020]).
All of these challenging real-world problems can be framed in a Multi-Agent RL (MARL) context.
In Multi-Agent Reinforcement Learning (MARL), multiple agents act in the same environment, to optimize their objectives.
However, despite the empirical success of MARL algorithms, theoretical understanding of MARL is relatively rare.
The MARL framework is usually modeled as a Markov
Game (MG) [Shapley, 1953], which is an extension of
Markov Decision Processes (MDPs)
[Sutton et al., 1998].
In general, learning in MGs is harder than learning in MDPs.
The complications arise from the fact that all agents affect both the transitions and the rewards of the other agents, while the agents can have completely different, even conﬂicting objectives.
Moreover, the agents without knowledge of the transition model, have to estimate it by interactions, as in single-agent RL problems, but they need also to infer and learn the other agent’s policy.
In the literature [Xie et al., 2020, Wei et al., 2017, Tian et al., 2020], the learning problem in MGs has been divided into two settings: online and ofﬂine .
In the online setting, the algorithm has the control of only one agent, to maximize its rewards in a multi-agent environment.
On the other hand, the ofﬂine setting aims at providing self-play algorithms, i.e., algorithms that have the control of all the agents, or at least, it is assumed that all the agents use the same algorithm.
While the ofﬂine setting has received considerable attention, it fails at modeling many use-cases of practical interest.
For example, in many robot control problems, artiﬁcial agents interact with humans who are not-controllable agents; or in card/video games it is unrealistic for the opponent to use the same learning algorithm as our agent.
While the online setting is more suitable to model previous examples, it remains less studied and understood.
Moreover, the algorithm that takes into account the online setting usually assumes that the non-controllableagent has a conﬂicting objective, i.e. the problem that we are facing is a zero-sum Markov Game.
However, there are many real-world problems where the other agent can have only different objectives, but not completely adversarial.
In this paper, we consider the online learning problem in a two-player general-sum turn-based Markov Game.
More precisely, we consider the problem of learning in Markov
Games, where there is one agent that we can control and it can observe the interaction between the non-controllable agent and the environment.
We are interested in solving the following question left open in the literature: “Can we design a provably efﬁcient algorithm for Markov
Games exploiting a general-sum opponent?”
This problem was also raised out in [Xie et al., 2020] as an interesting open direction.
In that paper, the authors present the problem of learning in the presence of a weak opponent , i.e. we are facing an opponent that is not totally adversarial (as in zero-sum games).
As suggested by the authors, in this case, the guarantee involves a stronger notion of regret with respect to the minimax ones.
In this paper we answer partially afﬁrmative.
In fact, we show that the regret minimization problem in this context is more complex than in standard MDPs.
Our contributions can be summarized as follows:
First, we deﬁne a stronger regret deﬁnition for the problem since the minimax one [Xie et al., 2020, Tian et al., 2020] does not capture the nature of the interaction.
Second, we provide a novel lower bound for the regret minimization problem, that shows how the exploration problem in general-sum MGs is harder than in MDPs and in zero-sum MGs.
Finally, we provide an algorithm, called Turn-based Markov Game OPtimistic Value Iteration (TMG-OPVI) which nearly matches the proposed lower bound.
Paper outline
The paper is structured as follows.
We start by over-viewing some related works (Section 2).
Then we introduce the notation (Section 3) and we provide in Section 4 a formal introduction to the problem.
In Section 5, we derive a lower bound on the expected regret of any “good” learning strategy that captures the exploration challenges in this context.
In particular, the lower bound clearly shows that the regret minimization in Stochastic Games is signiﬁcantly more complex than in standard MDPs.
Then, in Section 6, we propose an algorithm that nearly matches the proposed lower bound.
Remark 1.1.
The considered setting is easier than the simultaneous MG setting and the one in which the agent can change its type over time.
We decided to consider this framework for two reasons: 1) it is similar to the online learning zero-sum MG considered in the literature, Xie et al.
[2020], and, most important, 2) we proved that even in this setting, which is a special case of the simultaneous one and of the one considered in Sessa et al.
[2020], the learning problem is hard.
RELATED WORKS
After the introduction of the concept of Markov
Games [Shapley, 1953], many RL algorithms were proposed to learn in this setting.
However, the theoretical study of this context is quite poor, compared to the empirical one (see the survey [Zhang et al., 2019, Da Silva and Costa, 2019, Hernandez-Leal et al., 2019, Papoudakis et al., 2019]).
Only recently, there has been a growing interest in providing algorithms with strong sample-complexity and regret guarantees for the two theoretical MARL settings: ofﬂine and online .
In the online setting, the algorithm controls only one agent, which has to maximize its own reward function.
Instead, in the ofﬂine setting, the algorithms control all the agents in the MG.
Ofﬂine setting Most recent works provide results in the zero-sum ofﬂine setting, where both model-free [Bai et al., 2020, Zhang et al., 2020b] and model-based [Bai and Jin, 2020, Sidford et al., 2020, Li et al., 2020, Liu et al., 2020,
Zhang et al., 2020a] algorithms were proposed with nearoptimal sample complexity and regret guarantees.
For the model-based setting, the prevalent approach is to assume to have access to a generative model, such as in [Sidford et al., 2020, Zhang et al., 2020a], where the authors provide nonasymptotic results on the number of queries to the generator.
However, in [Liu et al., 2020] the authors proposed a modelbased algorithm for the zero-sum setting without access to a generative model and which matches the informationtheoretic lower bound.
Furthermore, in this recent work, the authors also proposed the ﬁrst line of provably sampleefﬁcient algorithms for multi-player general-sum games.
In [Li et al., 2020] the authors introduced, instead, an algorithm to learn a Nash Equilibrium in the multi-player general-sum setting.
Very recently, the ﬁrst algorithm to deal with sample complexity in the general-sum games (not Markov) that achieves an (cid:15) -Stackelberg Equilibrium was introduced [Bai et al., 2021].
In this paper, the authors consider the bandit feedback setting i.e., they can see only the random samples of the rewards received by the two players.
The authors identify a fundamental gap between the exact value of the
Stackelberg equilibrium and its estimated version using ﬁnite samples.
This result gives insights into the hardness of learning in General-sum games even when the setting is stateless and the algorithm has the control of both the leader and the follower.
Online setting The online setting is only studied, as far as we know, in the zero-sum setting.
The ﬁrst work that analyzes the problem of online learning in Stochastic Games is [Brafman and Tennenholtz, 2002].
In this paper, the authors propose the famous R-MAX algorithm that deals with the zero-sum average-reward setting and provides the ﬁrst regret bound for the setting.
In Wei et al.
[2017] the authors provide an algorithm for zero-sum Stochastic games that extends UCRL2, but which works under strong reachability assumptions.
This algorithm signiﬁcantly improves the regret bound of R-MAX.
Xie et al.
[2020] propose an algorithm with a “weak” regret notion (the minimax deﬁned before), which is compatible with a zero-sum game.
This is the ﬁrst work that considers linear function approximation in Markov Games.
The authors analyze both the ofﬂine and online settings and their algorithms achieve near-optimal regret bounds.
Instead, Tian et al.
[2020] introduce the online setting with bandit-feedback, called also agnostic setting.
In this case, the agent cannot observe any interaction between the other agent and the environment.
The authors extend the method of Bai et al.
[2020] to deal with this setting.
In [Xie et al., 2020] the authors leave as open question how to construct an algorithm to achieve optimal regret exploiting a “weak opponent”, i.e., an opponent who is not totally adversarial (as in zero-sum games).
In our work, we give the ﬁrst solution to this problem.
Adversarial MDPs
The adversarial MDP problem is strictly related to the Markov Game setting.
Most of the works in this setting consider adversarial rewards [Even-Dar et al., 2009, Gergely Neu et al., 2010, Zimin and Neu, 2013,
Rosenberg and Mansour, 2019, Jin et al., 2020, Dick et al., 2014], i.e., the presence of an opponent who can change the received rewards.
This setting is substantially different from Markov Games as the opponent can affect only the rewards and not the transitions model.
Other works, instead, consider also adversarial transitions [Yu and Mannor, 2009,
Neu et al., 2012, Lykouris et al., 2019].
This setting is quite challenging and the algorithms to solve this problem do not provide a O (cid:16) √ T (cid:17) regret bound.
These approaches can be applied in the bandit setting where we cannot see any feedback from the other agent, with the scope of constructing an algorithm robust to these perturbations.
PRELIMINARIES
In this section, we formally describe the background that will be employed throughout the remainder of the paper.
3.1 TURN-BASED MARKOV GAMES: BACKGROUND AND NOTATION
We consider two-player ﬁnite-horizon Markov Game setting [Shapley, 1953, Xie et al., 2020, Bai et al., 2020]
MG = ( S , A 1 , A 2 , P , R 1 , R 2 , µ, H ) , where S is the ﬁnite state space, A 1 , A 2 are the ﬁnite action spaces respectively for the ﬁrst and the second player, P ∈ ∆ SS×A 1 ×A 2 is the transition kernel, R i : S
×
A 1 × A 2 → [0 , 1] is the reward function of the i -th player, µ is the initial state distribution and H is the horizon.
In a turn-based MG at each state only one player takes an action.
The state space S is partitioned into S
=
S
1 ∪ S 2 , S 1 ∩ S 2 = ∅ , where S i is the set of states where it is i ’s turn to play.
For each state s ∈ S , let
I ( s ) ∈ { 1 , 2 } be a function that indicates the current player to play.
A stochastic policy for the i-th ( i ∈ { 1 , 2 } ) player is a sequence of H functions π :=
( π h : S → ∆ A 1 ) h ∈ H .
We deﬁne as π 1 = ( π 1 , 1 , . . .
, π 1 ,K ) and π 2 = ( π 2 , 1 , . .
. , π 2 ,K ) the two sequences of policies that, respectively, are played by our agent (agent 1 ) and the other agent (agent 2 ).
Value Functions
The value function and action-value function, given policies π 1 and π 2 , are deﬁned for agent i ∈ { 1 , 2 } , for each time step h ∈
[1 , H ] , state s ∈ S , action a ∈ A , as follows:
[Equation]
where the a t ∼ π i ( s t ) and s t ∼ P ( ·| s t , a t ) .
Furthermore, we denote with V π 1 ,
[Equation]
s
[Equation]
] and V
[Equation]
( s )] the expected returns for the two agents.
The interaction between the two agents proceeds in episodes, where at the beginning of each episode the agents decide which policy to play.
We indicate with K the number of episodes played by the two agents.
The agent observes the states, the actions played by the two agents, and noisy feedback of the agents’ reward functions, i.e. (cid:101) r i,h sampled from a distribution with mean R i ( s h , a h ) .
PROBLEM STATEMENT
In this section, we introduce the online learning problem in Turn-based General-sum Markov Games.
We remark that in these games, at each step h the agent I ( s h ) has to decide the action a h to be taken and the two players receive respectively rewards R 1 ( s h , a h ) and R 2 ( s h , a h ) ; then, the system transitions to the next state s h +1 ∼ P ( ·| s h , a [Equation]
The algorithm controls only agent 1 .
We do not know the policy π 2 that agent 2 will play at iteration k as well as the reward function R 2 that agent 2 is optimizing.
The goal is to learn a sequence of policies π 1 = { π 1 , 1 , . . .
, π 1 ,k } that minimizes the total (expected) regret , deﬁned by:
[Equation]
where V (cid:63) 1 corresponds to the benchmarks couple of policies π (cid:63) 1 , π (cid:63) 2 used to compare our algorithm.
In literature, the common way of deﬁning V {+(cid:63) 
+} [Xie et al., 2020, Wei et al., 2017] is the minimax policy deﬁned as:
[Equation]
The two policies π 1 , π 2 correspond to the Nash Equilibrium of the MG = ( S , A 1 , A 2 , P , R 1 , −R 1 , µ, H ) , i.e. a zero-sum Markov Game where the agent 2 is minimizing agent 1 ’s reward function.
The minimax benchmark policy is suitable to account for adversarial settings, where the other agent can adversarially change its policy to maximize the regret, or when we are in a zero-sum game.
Furthermore, it was shown that in some cases, it is necessary to adopt the minimax benchmark since, otherwise, the regret minimization problem would be too difﬁcult.
An example is the agnostic setting when agent 1 cannot observe any information regarding the interaction between agent 2 and the environment [Tian et al., 2020].
However, in general, minimax policies do not capture the nature of the general-sum setting where the agent 2 just wants to maximize its own reward function.
In fact, in the latter setting, agent 1 could hope to perform better than when facing a non-Competitive (its reward is not the opposite of our reward function) opponent.
A benchmark that better ﬁts the general-sum setting is the Stackelberg Equilibrium of the game:
[Equation]
where br : Π 1 → Π 2 corresponds to a function that selects a best response policy for the agent 2 to each policy of the agent 1 .
Although this setting can not be applied in the agnostic case, in many real-world scenarios, it is plausible that the agent 1 can observe the interactions between agent 2 and the environment.
Moreover, in some cases, the agent 1 can observe also agent 2 rewards or it can at least recover them (for example, using IRL approaches).
In these cases, it is more reasonable to use the more challenging regret notion.
Since the uncontrollable agent is rationale, we can easily suppose that it plays the best response, as it is already done in literature [Balcan et al., 2015, Peng et al., 2019, Sessa et al., 2020].
More formally, the agent 2 , given the policy π i 1 ∈ Π 1 , follows the policy π ∗ ,i 2 such that:
[Equation]
i.e., it plays its best response .
This creates an inherently asymmetrical interaction: the ﬁrst agent can be seen as a leader , who decides the policy to be played in an episode, and the second agent can be seen as a follower , who can see the leader’s policy and adapts its response to it.
So, as in the game-theory literature [Balcan et al., 2015, Peng et al., 2019,
Sessa et al., 2020], we make the following assumption:
Assumption 4.1.
For every policy π 1 ∈ Π 1 the second uncontrollable agent will always play the same best response policy br ( π 1 ) , where br : Π 1 → Π 2 .
Furthermore, br ( π 1 ) is deterministic.
Under this assumption the goal of our agent is well-deﬁned and consists in ﬁnding the policy π 1 ∈ Π 1 that is optimal under the second agent’s best response policy:
[Equation]
This corresponds to ﬁnding the Stackelberg Equilibrium of the game.
We remark that agent 1 does not know the policies that the second agent will play, i.e., the br function is unknown.
From an online learning perspective, the regret that the algorithm has to minimize is deﬁned as:
[Equation]
V
[Equation]
[Equation]
Bandit vs Turn-based MG
Obviously, this problem can be seen as solving a stochastic multiarmed bandit problem [Lattimore and Szepesvári, 2020].
In this case, the arms are the policies, and the agent at each episode receives a random realization of its expected return.
So, this problem can be solved with standard bandit algorithms such as UCB [Auer et al., 2002].
However, as we will explain in the next section, this is not the best we can do.
In fact, the regret would not scale sublinearly with the number of possible policies, as it happens with standard bandit algorithms (where the regret is O (cid:16)(cid:112)
| Π 1 | K (cid:17) ).
However, we show in Section 5 that the regret has a constant dependence on the number of policies (i.e., not multiplicative of K).
In fact, we prove a lower bound on the regret and an upper bound such that the quantity O (cid:16) √ T (cid:17) does not scale with the number of possible policies, but with a constant dependence on the number of possible policies.
HARDNESS OF LEARNING IN TURN-BASED TWO-PLAYER MARKOV GAMES
In this section, we provide a lower bound on the expected regret deﬁned in Equation 4.
We remark that agent 1 , i.e., the one who is controlled by the algorithm, sees the actions taken by the second agent as well as its rewards.
We consider the Turn-based MG (TMG) shown in Figure 1.
That is, there are N +3 states, A actions with N, A ∈ N , and
[Figure]
H = N +1 .
The state space is deﬁned as S = S 1 ∪S 2 , S 1 ∩
S 2 = ∅ .
Agent 2 controls the starting state s 0 , S 2 = { s 0 } , identiﬁed in the ﬁgure with the orange color.
The state space of agent 1 , instead, is equal to S
= { s f , s p , s 1 , . . .
, s N } , i.e. it controls the blue states in Figure 1.
In state s 0 , agent 2 can choose between a good action a ∗ and a failure action a f .
The reward functions of the two agents are:
[Equation]
otherwise where R, R f ∈ R .
The transition model of the TMG is deﬁned as follows:
• In state s 0 , P ( s 1 | s 0 , a (cid:63) ) = 1 , P ( s f | s 0 , a f ) =
δ and P ( s 0 | s 0 ,
a f ) = 1 − δ .
Then, if the agent 2 takes the good action, the TMG transits to state s 1 , otherwise the next state is s f with probability δ and s 0 with probability 1 − δ .
• From state s f with any action we continue to stay in state s f , i.e., P ( s f | s f , a ) =
1 ∀ a ∈ A .
• From state s p with any action we continue to stay in state s p , i.e., P ( s p | s p , a ) = 1 ∀ a ∈ A .
• From all the other states s i with i ∈
[ N ] : P ( s i +
| s i , a (cid:63) ) =
δ + (cid:15) and P ( s p | s i , a (cid:63) )
= 1 − δ − (cid:15) ; instead, for any other action a ∈ A , P ( s i +
| s i , a ) =
δ and P ( s p | s i , a )
= 1 − δ .
The second agent has only two response functions: π 2 ,i ( s 0 )
= a 0 or π 2 ,i ( s 0 )
=
a f with i ∈ [1 , H ] 3 .
Obvi- ously, it depends on the policy that agent 1 decides to take at the beginning of the episode.
In the next proposition we prove that always exist two values for R and R f such that the only policies of agent 1 that induce the second agent to take action a ∗ are the ones such that
π 1 ,i ( a ∗ | s i )
=
for all i ∈ [2 , H ] .
We call this set of policies Π ∗ 1 .
Proposition 5.1.
For every δ, (cid:15) ∈ (0 , 1) , H > 3 , there exist two values R and R f such that agent 2 will play action a ∗ only if agent 1 plays a policy π ∈ Π ∗ 1 .
Intuition on lower bound From this construction, we show that agent 2 has the power to hide part of the MG.
In fact, in all cases where agent 1 plays a policy different from the optimal one, we cannot acquire any further information about the transition model since we only visit state s f and suntil the end of the episode.
Intuitively, we can notice that, in the worst case, we have to play all the policies in Π 1 before ﬁnding the policy that leads us to acquire information about the states in the chain.
In fact, only with a policy π ∈ Π ∗the agent 2 allows us to visit states other than s f .
In the following theorem, we will formally prove this intuition.
Theorem 5.1 (Lower bound for online Turn-based Stochastic Game) .
Let A be a“good” learning algorithm, where with “good” we indicate an algorithm such that its expected regret is upper bounded by O ( CK α ) with α < 1 in all
Turn-based Markov Games 4 .
Then we can create a Turnbased Markov Game such that the expected regret is lower either it prefers to gain R f or R . More details on it are provided in Appendix ?? . We note that algorithms that satisfy this assumption exist.
For instance, applying UCB over the set of policies Π 1 yields regret
O
[Equation]
bounded by:
[Equation]
Furthermore, we can create a Turn-based Markov Game with S states, A actions and horizon H
= S − 1 , and a speciﬁc initial distribution µ such that the expected regret of A after K steps:
[Equation]
5.1 PROOF SKETCH OF THEOREM 5.
In this part, we provide a sketch of the proof of the lower bound for the online Turn-based Markov Game problem.
The complete proof can be found in Supplementary ?? .
We start by proving Equation 5.
We can easily notice that a
Markov Decision Process is a special case of a Turn-based
Markov Game, in which the state space of the second agent
S 2 = ∅ .
Then from this consideration, we can state that the worst-case lower bound for MDPs can also be applied for
TMGs [Jaksch et al., 2010, Domingues et al., 2020]:
[Equation]
To prove Equation 6, instead, we rely on standard information-theoretic arguments used to prove lower bounds in episodic MDPs and bandit problems.
More precisely, we use the following lemma due to Simchowitz and Jamieson [2019] (lemma H.1).
Lemma 5.1 (Lower bound for online Turn-based Stochastic
Game) .
Let T MG
= ( S , A , H, R , µ, P ) and T MG (cid:48) =
( S , A , H, R , µ, P (cid:48) ) be two TMGs with the same state space
S , action space A , initial state distribution µ and horizon
H .
Fix a number of episodes K ≥ 1 and let F K be the ﬁltration generated by all rollouts up to episode K .
Then for any F K -measurable random variable Z ∈ [0 , 1] , [Equation]
[Equation]
P
[Equation] [Equation]
s, a ) , P
[Equation]
where kl ( x, y ) = x log (cid:16) xy (cid:17) + (1 − x ) log (cid:16) 1 − x 1 − y (cid:17) is the binary KL-divergence and KL ( · , · ) denotes the KLdivergence between two probability laws and N k ( s, a ) is the number of times the state-action pair ( s, a ) is visited till iteration k .
Then we construct an alternative MG (cid:48) such that MG (cid:48) coincides with the MG except in the transition from s 0 to s f :
[Equation]
Changing this transition the agent 2 will always play the action a f .
Knowing that the two games differ only in this transition, by construction, the sum of Lemma 5.1 reduces to consider in the left-hand side only the pair state s 0 and action a f .
From this consideration, we create two events such that, the KL divergence is greater than O (cid:0)
A SH (cid:1) .
5.2 DISCUSSION ON THEOREM 5.
Theorem 5.1 shows that learning in general-sum MGs is exponentially harder than learning in MDPs.
This result proves that when we are not in control of the environment it is hard to explore it in a smart way.
Furthermore, we would like to remark that the setting we are analyzing also supposes a strong assumption about the not-controllable agent behavior: it can answer only with the same deterministic optimal policy.
However, by removing this assumption, i.e., assuming agent 2 can choose any optimal policy, our result continues to apply.
The proof of the lower bound implicitly says that we can create very small sub-optimality gaps for the second agent and that the regret must scale with the inverse of them regardless of the suboptimality gaps of the ﬁrst agent.
Although we do not explicitly show this, it is intuitive to see why it can happen.
In fact, agent 1 does not pay for the small suboptimality gap of the not-controllable agent but for its gap that can be potentially very high.
We leave as future work to prove a problem-dependent lower bound for TMGs.
This lower bound is the ﬁrst one that states the difﬁculties in learning in general-sum Markov Games with the possibility to see the other agent’s reward function and actions.
Other lower bounds were derived for the general-sum setting.
In [Bai et al., 2020] the authors proposed a lower bound to underline the difﬁculties to learn against an adversarial opponent.
In [Tian et al., 2020], instead, the authors show the statistical hardness of learning with only bandit feedback, i.e., in an agnostic setting.
However, these two settings are harder than the one proposed in this section, and, for this reason, the results cannot be applied.
TMG OPTIMISTIC POLICIES VALUE ITERATION
In this section, we propose an algorithm, called Turn-based
Markov Game Optimistic Policies Value Iteration (TMGOPVI), that nearly matches the lower bound proposed in the previous section.
We assume that Π 1 is any set of policies (similarly to Abbasi-Yadkori et al.
[2013]), not necessarily corresponding to the full set of all deterministic Markov policies, and let M be the cardinality of the policy set Π 1 .
TMG-OPVI algorithm TMG-OPVI is a variant of Optimistic Value Iteration Azar et al.
[2013], an optimistic regret minimization algorithm for ﬁnite-horizon MDPs.
The algorithm proceeds as follows.
Given the set of policies Π 1 for the ﬁrst agent, it stores a table recording the policy played by the second player.
For every i ∈
[ M ] , k ∈ [ K ] , h ∈
[ H ] we denote with A ik,h ( s ) ⊆
A the set of plausible actions, i.e., the set of actions that can be played by the second agent, in state s at step h for policy π i ∈ Π 1 at the beginning of episode k .
Since, given agent 1 policy, the response policy of the other agent is deterministic and unique for assumption (see Section 3), when we play the policy π i and we observe in state s ∈ S 2 , at time step h , the policy π 2 ,h ( s ) , we can set A ik,h ( s ) =
{ π 2 ,h ( s ) } .
As common in optimistic value iteration algorithms [Azar et al., 2013], we shall build upper conﬁdence bounds to the value function of each policy by adding bonus terms based on conﬁdence intervals on the rewards and transition probabilities.
Formally, for every k ∈
[ K ] , state s ∈ S and action a ∈ A we derive the bonus term, based on Hoeffding’s concentration inequality, for the reward function and the expected value function:
[Equation]
Furthermore, we indicate with (cid:98) R 1 ,k
( s, a ) and (cid:98) P 1 ,k
( s (cid:48) | s, a ) the sample means of respectively the observed rewards and transitions up to (and not including) episode k .
Based on this, at the beginning of each episode k ∈ [ K ] the algorithm computes for each policy π i 1 with i ∈
[ M ] an optimistic approximation (cid:101) V i 1 ,k of the expected return
V i 1 .
Recursive for each h ∈
[ H ] , s ∈ S the optimistic approximation (cid:101) V i 1 ,k,h ( s ) of the value function V i 1 ,h ( s ) is equal to:
[Equation]
, if I ( s ) = 1 and, otherwise, is equal to: [Equation] s [Equation]
+ b k ( s, a )
[Equation]
where b k ( s, π i 1 ,h ( s ))
=
b rk ( s, π i 1 ,h ( s ))
+ b P k ( s, π i 1 ,h ( s )) .
Two levels of optimism Note that we use two levels of optimism: one for the unknown transition probabilities and rewards, and one for the unknown actions of the second agent.
More precisely, if we have already seen the action that the second agent will play in a state s with a policy π i 1 we use this information to estimate the value function, otherwise we act optimistically by taking the maximum overall plausible actions.
The pseudocode of TMG-OPVI is reported in Algorithm 1.
[Table]
. , K do 4: Compute (cid:101) V i 1 ,k for all i ∈
[ M ] 5: Play π I k 1 with I k ∈ arg max i ∈ [ M ] (cid:101) V i 1 ,k 6:
Observe ( s k, 1 , a k, 1 , . . .
, s k,H − 1 , a k,H − 1 , s k,H ) 7:
Compute the plausible actions for all s ∈ S , h ∈
[ H ] and i ∈
[ M ] :
A ik +1 ,h ( s ) =
(cid:40) { a k,h } if i = I
k
and s = s k,h
A ik,h ( s ) otherwise 8: end for 6.1 REGRET GUARANTEES In this section, we give a regret bound for the proposed algorithm. The result exploits the determinism of the other agent’s policies in order to match the lower bound derived in the previous section.
Estimation of the transition model The main idea behind the proof is that after having played a certain number of times every policy, we know in every state that is reachable what action the agent 2 will play.
At this point, we have reduced our problem to an MDP.
In fact, when we know the best response function of agent 2 , for every policy we can create a policy that is the union of the policy of agentand agent 2 .
At this point, the uncertainty comes only from the transition model and the reward function.
It is important to note that we do not need to know explicitly the set of reachable states, but the algorithm implicitly will estimate correctly the agent 2 policy after having visited all of them.
Before stating the result we need to introduce some quantities.
We deﬁne d i ( s ) as the probability of visiting state s playing policies π i 1 and br ( π i 1 ) .
Then, we deﬁne the set
S + ,i 2 ,h = { s ∈ S 2 such that d i h ( s ) > 0 } .
We deﬁne as d = min i ∈
[ M ] min h ∈
[ H ] min s ∈ S + ,i d ih ( s ) , i.e., the minimum probability of visiting a “reachable” state.
In the following theorem we provide an upper bound of the regret of TMG-OPVI algorithm.
Theorem 6.1 (Regret of TMG-OPVI) .
Let TMG =
( S , A , P , µ, R 1 , R 2 , H ) with S = S 1 ∪S 2 and S 1 ∩S 2 = ∅ be the ﬁnite-horizon TMG of our problem.
Then the expected regret of TMG-OPVI at every episode K > 0 is bounded by:
[Equation]
where ¯ K is the ﬁrst integer such that K >
[Equation]
6.2 PROOF SKETCH OF THEOREM 6.
In this section, we provide a proof sketch of Theorem 6.1.
The complete proof can be found in Appendix ?? .
We consider the fact that, at some point, if we play a certain policy π 1 we will observe every state that can be reached playing π i 1 and br ( π i 1 ) .
In fact, there exists an iteration K such that each state s ∈ S
+ ,i 2 ,h with i ∈
[ M ] has been visited at least one time.
After K , agent 1 has complete knowledge of the best response function br 5 .
From this iteration, the algorithm is facing a single-agent problem, where the joint policy is derived by the union of the policies of the two agents:
[Equation]
where in this case
A ik,h ( s ) is a singleton for every state s ∈ S 2 and policy π
i 1 with i ∈
[ M ] .
Then we can proceed with our proof considering this new single-agent problem.
6.3 DISCUSSION ON THEOREM 6.
The following regret nearly matches the proposed lower bound.
In fact, if we instantiate the set of policies of agentequal to all the possible deterministic policies, then M =
A HS where A is the cardinality of the action space and
S the cardinality of the state space and H is the horizon.
Instead, the second term of the regret is comparable with the worst-case lower bound for MDPs [Azar et al., 2013].
It is interesting to note that respect to agnostic MG [Tian et al., 2020] we achieve better regret guarantees in terms of
K in our setting where they achieve a regret upper bound of O (cid:16) K 23 (cid:17) .
We achieve also stronger regret guarantees with respect to adversarial MDPs [Abbasi-Yadkori et al., 2013] where transitions and rewards can change adversarially.
The two settings are quite similar, since also in the setting considered in this paper the transitions change adversarially , since agent 2 inﬂuences the transitions of the MG.
However, in this case, differently to ours, the regret that is achieved is O (cid:16)(cid:112) K log ( M ) (cid:17) while our regret does not depend on the number of policies in the K term.
On the other hand, their constant dependence on the number of policies is O (log( M )) , while we obtain a constant dependence on the number of policies.
Clearly, the two settings, are different, so it is hard to compare them, but our result shows that using the information of the not-controllable agent, we can achieve better performances.
CONCLUSIONS
Contributions In this paper, we propose the ﬁrst insights to the online learning problem in general-sum Stochastic
Games.
Although there are some recent results in solving the problem in the zero-sum (aka competitive) setting, there are no other works that take into account the problem or consider that we could face a non-Competitive opponent.
We have shown that the problem is much more complicated than in a zero-sum MG and
an MDP The main problems arise from the limited control on the environment’s exploration.
We underline this difﬁculty by providing a novel lower bound (Section 5), which proves that the regret scales constantly with the number of deterministic policies that can be played by the controllable agent.
This creates a big gap between what we can obtain learning in MDPs with respect to general-sum MGs.
Then we show how to build a provably efﬁcient algorithm in Section 6.
Our algorithm, TMG-OPVI, achieves optimal performance nearly matching the proposed lower bound.
We would like to underline that this is the ﬁrst paper that considers the online learning problem in the general-sum Markov Game, and we think that our ﬁndings help in the understanding of the MARL problem.
Future directions and discussion on learning in Markov
Games Currently, there is a need for a formal understanding of the online MARL problem to construct provably efﬁcient learning algorithms for this context.
As our result suggested, the MARL setting poses novel challenges, especially in the well-known exploration-exploitation dilemma, i.e., the trade-off between gathering new information and exploiting it: in a multi-agent environment, the agent needs to explore not only to understand the underlying environment but also to learn the other agents’ behaviors.
Moreover, from our ﬁndings, it is clear that the algorithm design and the resulting performance guarantees heavily depend on any knowledge about the opponents, either known as a priori or obtainable during the learning process.
Furthermore, there are open problems also in the agnostic setting, presented in [Tian et al., 2020], as it is possible to achieve better theoretical (regret) guarantees and construct algorithms with optimal sample complexity.
This scenario, having no assumptions on the opponents, is widely applicable to capture real-world problems.
On the other hand, assuming to have the possibility to observe other agents’ interactions with the environment or having some previous knowledge about the other agents (as having access to a ﬁnite set of opponents [Balcan et al., 2015] or considering a larger set of opponents’ classes with some regularity assumptions [Sessa et al., 2020]) we could hope to obtain better theoretical guarantees.
An unexplored, but promising future direction would be considering some structural relation between the best response of the agent 2 and agent 1 .
It could overcome the problem of the constant dependence on the number of deterministic policies.
Another interesting future direction is to better show the relationship between the sub-optimality gap of the noncontrollable agent and the one of the controllable agent.
To prove this it is necessary to prove a problem-dependent lower bound.
We leave this analysis as future work.
Acknowledgements
We thank Andrea Tirinzoni for valuable discussions.