Original: Sample-Efﬁcient Reinforcement Learning of Partially Observable Markov Games
Revised: Sample-Efﬁcient Reinforcement Learning of Partially Observable Markov Games

Original: Abstract
Revised: Abstract

Original: This paper considers the challenging tasks of Multi-Agent Reinforcement Learning (MARL) under partial observability, where each agent only sees her own individual observations and actions that reveal incomplete information about the underlying state of system.
Revised: This paper considers the challenging tasks of Multi-Agent Reinforcement Learning (MARL) under partial observability, where each agent only sees her own individual observations and actions that reveal incomplete information about the underlying state of system.

Original: This paper studies these tasks under the general model of multiplayer general-sum Partially Observable Markov Games (POMGs), which is signiﬁcantly larger than the standard model of Imperfect Information ExtensiveForm Games (IIEFGs).
Revised: This paper studies these tasks under the general model of multiplayer general-sum Partially Observable Markov Games (POMGs), which is signiﬁcantly larger than the standard model of Imperfect Information ExtensiveForm Games (IIEFGs).

Original: We identify a rich subclass of POMGs—weakly revealing
Revised: We identify a rich subclass of POMGs—weakly revealing

Original: POMGs—in which sample-efﬁcient learning is tractable.
Revised: POMGs—in which sample-efﬁcient learning is tractable.

Original: In the self-play setting, we prove that a simple algorithm combining optimism and Maximum Likelihood
Revised: In the self-play setting, we prove that a simple algorithm combining optimism and Maximum Likelihood

Original: Estimation (MLE) is sufﬁcient to ﬁnd approximate Nash equilibria, correlated equilibria, as well as coarse correlated equilibria of weakly revealing POMGs, in a polynomial number of samples when the number of agents is small.
Revised: Estimation (MLE) is sufﬁcient to ﬁnd approximate Nash equilibria, correlated equilibria, as well as coarse correlated equilibria of weakly revealing POMGs, in a polynomial number of samples when the number of agents is small.

Original: In the setting of playing against adversarial opponents, we show that a variant of our optimistic
Revised: In the setting of playing against adversarial opponents, we show that a variant of our optimistic

Original: MLE algorithm is capable of achieving sublinear regret when being compared against the optimal maximin policies.
Revised: MLE algorithm is capable of achieving sublinear regret when being compared against the optimal maximin policies.

Original: To our best knowledge, this work provides the ﬁrst line of sample-efﬁcient results for learning POMGs.
Revised: To our best knowledge, this work provides the ﬁrst line of sample-efﬁcient results for learning POMGs.

Original:  Introduction
Revised:  Introduction

Original: This paper studies Multi-Agent Reinforcement Learning (MARL) under partial observability , where each player tries to maximize her own utility via interacting with an unknown environment as well as other players.
Revised: This paper studies Multi-Agent Reinforcement Learning (MARL) under partial observability , where each player tries to maximize her own utility via interacting with an unknown environment as well as other players.

Original: In addition, each agent only sees her own observations and actions, which reveal incomplete information about the underlying state of system.
Revised: In addition, each agent only sees her own observations and actions, which reveal incomplete information about the underlying state of system.

Original: A large number of real-world applications can be cast into this framework: in Poker, cards in a player’s hand are hidden from the other players; in many real-time strategy games, players have only access to their local observations; in multi-agent robotic systems, agents with ﬁrst-person cameras have to cope with noisy sensors and occlusions.
Revised: A large number of real-world applications can be cast into this framework: in Poker, cards in a player’s hand are hidden from the other players; in many real-time strategy games, players have only access to their local observations; in multi-agent robotic systems, agents with ﬁrst-person cameras have to cope with noisy sensors and occlusions.

Original: While practical MARL systems have achieved remarkable success in a set of partially observable problems including Poker [9], Starcraft [40], Dota [6] and autonomous driving [36], the theoretical understanding of MARL under partial observability remains very limited.
Revised: While practical MARL systems have achieved remarkable success in a set of partially observable problems including Poker [9], Starcraft [40], Dota [6] and autonomous driving [36], the theoretical understanding of MARL under partial observability remains very limited.

Original: The combination of partial observability with multiagency introduces a number of unique challenges.
Revised: The combination of partial observability with multiagency introduces a number of unique challenges.

Original: The non-Markovian nature of the observations forces the agent to maintain memory and reason about beliefs of the system state, all while exploring to collect information about the environment.
Revised: The non-Markovian nature of the observations forces the agent to maintain memory and reason about beliefs of the system state, all while exploring to collect information about the environment.

Original: Consequently, well-known complexity-theoretic results show that learning and planning in partially observable environments is statistically and computationally intractable even in the single-agent setting [35, 30, 41, 29].
Revised: Consequently, well-known complexity-theoretic results show that learning and planning in partially observable environments is statistically and computationally intractable even in the single-agent setting [35, 30, 41, 29].

Original: The presence of interaction between multiple agents further complicates the partially observable problems.
Revised: The presence of interaction between multiple agents further complicates the partially observable problems.

Original: In addition to dealing with the adaptive nature of other players who can adjust their strategies according to the learner’s past behaviors, the learner is further required to discover and exploit the information asymmetry due to the separate observations of each agent.
Revised: In addition to dealing with the adaptive nature of other players who can adjust their strategies according to the learner’s past behaviors, the learner is further required to discover and exploit the information asymmetry due to the separate observations of each agent.

Original: As a result, prior theoretical works on partially observable MARL have been mostly focused on a small subset of problems with strong structural assumptions.
Revised: As a result, prior theoretical works on partially observable MARL have been mostly focused on a small subset of problems with strong structural assumptions.

Original: For instance, the line of works on Imperfect Information Extensive-Form Games (IIEFG)
Revised: For instance, the line of works on Imperfect Information Extensive-Form Games (IIEFG)

Original: [see, e.g., 45, 14, 13, 24] assumes treestructured transition with small depth 1 as well as a special type of emission which can be represented as information sets .
Revised: [see, e.g., 45, 14, 13, 24] assumes treestructured transition with small depth 1 as well as a special type of emission which can be represented as information sets .

Original: In contrast, this paper considers a more general mathematical model known as
Revised: In contrast, this paper considers a more general mathematical model known as

Original: Partially Observable Markov Games (POMGs).
Revised: Partially Observable Markov Games (POMGs).

Original: POMGs are the natural extensions of both Partially
Revised: POMGs are the natural extensions of both Partially

Original: Observable Markov Decision Processes (POMDPs)—the standard model for single-agent partially observable RL, and Markov Games [37]—the standard model for fully observable MARL.
Revised: Observable Markov Decision Processes (POMDPs)—the standard model for single-agent partially observable RL, and Markov Games [37]—the standard model for fully observable MARL.

Original: Despite the complexity barriers of learning partially observable systems apply to POMGs, they are of a worst-case nature, which do not preclude efﬁcient algorithms for learning interesting subclasses of
Revised: Despite the complexity barriers of learning partially observable systems apply to POMGs, they are of a worst-case nature, which do not preclude efﬁcient algorithms for learning interesting subclasses of

Original: POMGs.
Revised: POMGs.

Original: This motivates us to ask the following question:
Revised: This motivates us to ask the following question:

Original: Can we develop efﬁcient algorithms that learn a rich class of POMGs?
Revised: Can we develop efﬁcient algorithms that learn a rich class of POMGs?

Original: In this paper, we provide the ﬁrst positive answer to the highlighted question in terms of the sample efﬁciency .
Revised: In this paper, we provide the ﬁrst positive answer to the highlighted question in terms of the sample efﬁciency .

Original: 2 We identify a rich family of tractable POMGs— weakly revealing POMGs (see Section 3).
Revised: 2 We identify a rich family of tractable POMGs— weakly revealing POMGs (see Section 3).

Original: The weakly revealing condition only requires the joint observations of all agents to reveal certain amount of information about the latent states, which is satisﬁed in many real-world applications.
Revised: The weakly revealing condition only requires the joint observations of all agents to reveal certain amount of information about the latent states, which is satisﬁed in many real-world applications.

Original: The condition rules out the pathological instances where no player has any information to distinguish latent states, which prevents efﬁcient learninig in the worst case.
Revised: The condition rules out the pathological instances where no player has any information to distinguish latent states, which prevents efﬁcient learninig in the worst case.

Original: In the self-play setting where the algorithm can control all the players to learn the equilibria by playing against itself, this paper proposes a new simple algorithm— Optimistic Maximum Likelihood
Revised: In the self-play setting where the algorithm can control all the players to learn the equilibria by playing against itself, this paper proposes a new simple algorithm— Optimistic Maximum Likelihood

Original: Estimation for Learning Equilibria ( OMLE-Equilibrium ).
Revised: Estimation for Learning Equilibria ( OMLE-Equilibrium ).

Original: As the name suggests, it combines optimism, MLE principles with equilibria ﬁnding subroutines.
Revised: As the name suggests, it combines optimism, MLE principles with equilibria ﬁnding subroutines.

Original: The algorithm provably ﬁnds approximate
Revised: The algorithm provably ﬁnds approximate

Original: Nash equilibria, coarse correlated equilibria and correlate equilibria of any weakly-revealing POMGs using a number of samples polynomial in all relevant parameters.
Revised: Nash equilibria, coarse correlated equilibria and correlate equilibria of any weakly-revealing POMGs using a number of samples polynomial in all relevant parameters.

Original: In the setting of playing against adversarial opponents, we measure the performance of our algorithm by comparing against the optimal maximin policies.
Revised: In the setting of playing against adversarial opponents, we measure the performance of our algorithm by comparing against the optimal maximin policies.

Original: We ﬁrst prove that learning in this setting is hard if each player can only see her own observations and actions.
Revised: We ﬁrst prove that learning in this setting is hard if each player can only see her own observations and actions.

Original: Nevertheless, if the agent is allowed to access other players’ observations and actions after each episode of play (e.g., watch the replays of the games from other players’ perspectives afterwards), then we can design a new algorithm OMLE-Adversary which achieves sublinear regret.
Revised: Nevertheless, if the agent is allowed to access other players’ observations and actions after each episode of play (e.g., watch the replays of the games from other players’ perspectives afterwards), then we can design a new algorithm OMLE-Adversary which achieves sublinear regret.

Original: To our best knowledge, this is the ﬁrst line of provably sample-efﬁcient results for learning rich classes of POMGs.
Revised: To our best knowledge, this is the ﬁrst line of provably sample-efﬁcient results for learning rich classes of POMGs.

Original: Importantly, the classes of problems that can be learned in this paper are signiﬁcantly larger than known tractable classes of MARL problems under partial observability.
Revised: Importantly, the classes of problems that can be learned in this paper are signiﬁcantly larger than known tractable classes of MARL problems under partial observability.

Original: 1.1 Technical novelty
Revised: 1.1 Technical novelty

Original: This paper builds upon the recent progress in learning single-agent POMDPs [26], which identiﬁes the class of weakly revealing POMDPs and develops OMLE algorithm for learning the optimal policy.
Revised: This paper builds upon the recent progress in learning single-agent POMDPs [26], which identiﬁes the class of weakly revealing POMDPs and develops OMLE algorithm for learning the optimal policy.

Original: Besides obtaining a completely new set of results in the multi-agent setting, we here highlight a few contributions and technical novelties of this paper comparing to [26].
Revised: Besides obtaining a completely new set of results in the multi-agent setting, we here highlight a few contributions and technical novelties of this paper comparing to [26].

Original: • This paper rigorously formulates the models, related concepts and learning objectives of multiplayer general-sum POMGs, and provides the ﬁrst line of sample-efﬁcient learning results.
Revised: • This paper rigorously formulates the models, related concepts and learning objectives of multiplayer general-sum POMGs, and provides the ﬁrst line of sample-efﬁcient learning results.

Original: • Extending the weakly revealing conditions into the multiagent setting lead to two natural candidates: either (a) joint observations or (b) individual observations are required to weakly reveal the state information.
Revised: • Extending the weakly revealing conditions into the multiagent setting lead to two natural candidates: either (a) joint observations or (b) individual observations are required to weakly reveal the state information.

Original: This paper shows the former (the weaker assumption) sufﬁces to guarantee tractability.
Revised: This paper shows the former (the weaker assumption) sufﬁces to guarantee tractability.

Original: • Results in the self-play setting requires careful design of optimistic planning algorithms that effectively address the game-theoretical aspects of the problem under partial observability.
Revised: • Results in the self-play setting requires careful design of optimistic planning algorithms that effectively address the game-theoretical aspects of the problem under partial observability.

Original: We achieve this by Subroutine 1, which is even distinct from the standard techniques for learning MGs.
Revised: We achieve this by Subroutine 1, which is even distinct from the standard techniques for learning MGs.

Original: • The discussions and results in the setting of playing against adversarial opponents are completely new, and unique to the multiagent setup.
Revised: • The discussions and results in the setting of playing against adversarial opponents are completely new, and unique to the multiagent setup.

Original: The sample complexity of learning
Revised: The sample complexity of learning

Original: with respect to the
Revised: with respect to the

Original: of information
Revised: of information

Original: typically has an exponential growth
Revised: typically has an exponential growth

Original: in depth.
Revised: in depth.

Original: For computational efﬁciency, due to the inherent hardness of planning in POMDPs, all existing provable algorithms that learn large classes of POMDPs (single-agent version of POMGs) require super-polynomial time.
Revised: For computational efﬁciency, due to the inherent hardness of planning in POMDPs, all existing provable algorithms that learn large classes of POMDPs (single-agent version of POMGs) require super-polynomial time.

Original: We leave the challenge of computationally efﬁcient learning for future work.
Revised: We leave the challenge of computationally efﬁcient learning for future work.

Original: 1.2 Related Works
Revised: 1.2 Related Works

Original: Reinforcement learning has been extensively studied in the single-agent fully-observable setting [see, e.g., 1, 10, 18, 23, 44, 17, 20, and the references therein] .
Revised: Reinforcement learning has been extensively studied in the single-agent fully-observable setting [see, e.g., 1, 10, 18, 23, 44, 17, 20, and the references therein] .

Original: For the purpose of this paper, we focus on reviewing existing works on partially observable RL and multi-agent RL in the exploration setting.
Revised: For the purpose of this paper, we focus on reviewing existing works on partially observable RL and multi-agent RL in the exploration setting.

Original: Markov games
Revised: Markov games

Original: In recent years, there has been growing interest in studying Markov games [37] — the standard generalization of MDPs from the single-player setting to the multi-player setting.
Revised: In recent years, there has been growing interest in studying Markov games [37] — the standard generalization of MDPs from the single-player setting to the multi-player setting.

Original: Various sample-efﬁcient algorithms have been designed for either two-player zero-sum MGs [e.g., 8, 42, 3, 28, 5, 43, 22] or multi-player general-sum MGs [e.g., 28, 21, 38, 11].
Revised: Various sample-efﬁcient algorithms have been designed for either two-player zero-sum MGs [e.g., 8, 42, 3, 28, 5, 43, 22] or multi-player general-sum MGs [e.g., 28, 21, 38, 11].

Original: However, all these works rely on the states being fully observable, while the POMGs studied in this paper allow states to be only partially observable, which strictly generalizes MGs.
Revised: However, all these works rely on the states being fully observable, while the POMGs studied in this paper allow states to be only partially observable, which strictly generalizes MGs.

Original: POMDPs POMDPs generalize MDPs from the fully observable setting to the partially observable setting.
Revised: POMDPs POMDPs generalize MDPs from the fully observable setting to the partially observable setting.

Original: It is well-known that in POMDPs both planning [35, 41] and model estimation [29] are computationally hard in the worst case.
Revised: It is well-known that in POMDPs both planning [35, 41] and model estimation [29] are computationally hard in the worst case.

Original: Besides, reinforcement learning of POMDPs is also known to be statistically hard: [25] proved that ﬁnding a near-optimal policy of a POMDP in the worst case requires a number of samples that is exponential in the episode length.
Revised: Besides, reinforcement learning of POMDPs is also known to be statistically hard: [25] proved that ﬁnding a near-optimal policy of a POMDP in the worst case requires a number of samples that is exponential in the episode length.

Original: The hard instances are those pathological POMDPs where the observations contain no useful information for identifying the system dynamics.
Revised: The hard instances are those pathological POMDPs where the observations contain no useful information for identifying the system dynamics.

Original: Nonetheless, these hardness results are all in the worst-case sense and there are still many intriguing positive results on sample-efﬁcient learning of subclasses of POMDPs.
Revised: Nonetheless, these hardness results are all in the worst-case sense and there are still many intriguing positive results on sample-efﬁcient learning of subclasses of POMDPs.

Original: For example, [15, 2, 19] applied spectral methods to learning undercomplete POMDPs and [26] developed the optimistic MLE approach for learning both undercomplete and overcomplete POMDPs.
Revised: For example, [15, 2, 19] applied spectral methods to learning undercomplete POMDPs and [26] developed the optimistic MLE approach for learning both undercomplete and overcomplete POMDPs.

Original: We refer interested readers to [26] for a thorough review of existing results on POMDPs.
Revised: We refer interested readers to [26] for a thorough review of existing results on POMDPs.

Original: In terms of algorithmic design, our algorithms build upon the optimistic MLE methodology developed in [26].
Revised: In terms of algorithmic design, our algorithms build upon the optimistic MLE methodology developed in [26].

Original: Compared to [26], our main algorithmic contribution lies in the design of the optimistic equilibrium computation subroutine in the self-play setting and the optimistic maximin policy design in the adversarial setting.
Revised: Compared to [26], our main algorithmic contribution lies in the design of the optimistic equilibrium computation subroutine in the self-play setting and the optimistic maximin policy design in the adversarial setting.

Original: In terms of analysis, our proofs requrie new techniques tailored to controlling game-theoretic regret, in addition to the OMLE guarantees imported from [26].
Revised: In terms of analysis, our proofs requrie new techniques tailored to controlling game-theoretic regret, in addition to the OMLE guarantees imported from [26].

Original: For more detailed explanations of our technical contribution, please refer to Section 1.1.
Revised: For more detailed explanations of our technical contribution, please refer to Section 1.1.

Original: Imperfect-information extensive-form games In the literature on game theory
Revised: Imperfect-information extensive-form games In the literature on game theory

Original: , there is a long history of learning Imperfect-Information Extensive-Form Games with perfect recall (IIEFGs), [see, e.g., 45, 14, 12, 13, 24] and the references therein.
Revised: , there is a long history of learning Imperfect-Information Extensive-Form Games with perfect recall (IIEFGs), [see, e.g., 45, 14, 12, 13, 24] and the references therein.

Original: IIEFGs can be viewed as special cases of POMGs with tree-structured transition and deterministic emission (which are also known as information sets).
Revised: IIEFGs can be viewed as special cases of POMGs with tree-structured transition and deterministic emission (which are also known as information sets).

Original: As a result, IIEFGs can not efﬁciently represent POMGs with general transition and stochastic emission (see Appendix A), and thus sample-efﬁcient learning results for IIEFGs does not imply sample-efﬁcient learning of POMGs.
Revised: As a result, IIEFGs can not efﬁciently represent POMGs with general transition and stochastic emission (see Appendix A), and thus sample-efﬁcient learning results for IIEFGs does not imply sample-efﬁcient learning of POMGs.

Original: On the other hand, we show that all IIEFGs can be efﬁciently represented by 1 -weakly revealing POMGs (see Appendix A).
Revised: On the other hand, we show that all IIEFGs can be efﬁciently represented by 1 -weakly revealing POMGs (see Appendix A).

Original: Therefore, all algorithms and theoretical results developed in this paper can immediately used to learn IIEFGs with a polynomial sample complexity.
Revised: Therefore, all algorithms and theoretical results developed in this paper can immediately used to learn IIEFGs with a polynomial sample complexity.

Original: Decentralized POMDPs There is another classic model for studying multi-agent partially observable RL, named decentralized POMDPs [e.g., 32, 33], which is a special subclass of POMGs where all players share a common reward target.
Revised: Decentralized POMDPs There is another classic model for studying multi-agent partially observable RL, named decentralized POMDPs [e.g., 32, 33], which is a special subclass of POMGs where all players share a common reward target.

Original: Compared to general POMGs, decentralized POMDPs can only simulate cooperative relations among players, while general POMGs can model both cooperative and competitive relations.
Revised: Compared to general POMGs, decentralized POMDPs can only simulate cooperative relations among players, while general POMGs can model both cooperative and competitive relations.

Original: Besides, most works [e.g., 31, 7, 34, 39] along this direction mainly focus on the computational complexity of planning with known models or simulators instead of the sample efﬁciency in the exploration setting.
Revised: Besides, most works [e.g., 31, 7, 34, 39] along this direction mainly focus on the computational complexity of planning with known models or simulators instead of the sample efﬁciency in the exploration setting.

Original:  Preliminary
Revised:  Preliminary

Original: In this paper, we consider Partially Observable Markov Games (POMGs) in its most generic— multiplayer general-sum form.
Revised: In this paper, we consider Partially Observable Markov Games (POMGs) in its most generic— multiplayer general-sum form.

Original: Formally, we denote a tabular episodic POMG with n players by tuple ( H, S , {A i } ni =1 , {O i } ni =1 ; T , O , µ 1 ; { r i } ni =1 ) , where H denotes the length of each episode, S the state space with |S| = S ,
Revised: Formally, we denote a tabular episodic POMG with n players by tuple ( H, S , {A i } ni =1 , {O i } ni =1 ; T , O , µ 1 ; { r i } ni =1 ) , where H denotes the length of each episode, S the state space with |S| = S ,

Original: A
Revised: A

Original: i denotes the action space for the i th player with |A
Revised: i denotes the action space for the i th player with |A

Original: i | =
Revised: i | =

Original: A i .
Revised: A i .

Original: We denote by a :
Revised: We denote by a :

Original: =
Revised: =

Original: ( a 1 , · · · , a n ) the joint actions of all n players, and by A :
Revised: ( a 1 , · · · , a n ) the joint actions of all n players, and by A :

Original: = A 1 × . . .
Revised: = A 1 × . . .

Original: ×
Revised: ×

Original: A n the joint action space with |A| =
Revised: A n the joint action space with |A| =

Original: A = (cid:81)
Revised: A = (cid:81)

Original: i A
Revised: i A

Original: i .
Revised: i .

Original: T =
Revised: T =

Original: { T h } h ∈ [ H ] is the collection of transition matrices, so that
Revised: { T h } h ∈ [ H ] is the collection of transition matrices, so that

Original: T h ( ·| s, a ) ∈
Revised: T h ( ·| s, a ) ∈

Original: ∆ S gives the distribution of the next state if joint actions a are taken at state s at step h .
Revised: ∆ S gives the distribution of the next state if joint actions a are taken at state s at step h .

Original: µ 1 denotes the distribution of the initial state s 1 .
Revised: µ 1 denotes the distribution of the initial state s 1 .

Original: O i denotes the observation space for the i th player with |O
Revised: O i denotes the observation space for the i th player with |O

Original: i
Revised: i

Original: |
Revised: |

Original: =
Revised: =

Original: O i .
Revised: O i .

Original: We denote by o :=
Revised: We denote by o :=

Original: ( o 1 , . . .
Revised: ( o 1 , . . .

Original: , o n ) the joint observations of all n players, and by O :
Revised: , o n ) the joint observations of all n players, and by O :

Original: =
Revised: =

Original: O 1 × . . .
Revised: O 1 × . . .

Original: × O n with |O| =
Revised: × O n with |O| =

Original: O = (cid:81)
Revised: O = (cid:81)

Original: i O i .
Revised: i O i .

Original: O =
Revised: O =

Original: { O h } h ∈ [ H ] ⊆
Revised: { O h } h ∈ [ H ] ⊆

Original: R
Revised: R

Original: O × S is the collection of joint emission matrices, so that O h ( ·| s ) ∈
Revised: O × S is the collection of joint emission matrices, so that O h ( ·| s ) ∈

Original: ∆ O gives the emission distribution over the joint observation space O at state s and step h .
Revised: ∆ O gives the emission distribution over the joint observation space O at state s and step h .

Original: Finally r i =
Revised: Finally r i =

Original: { r i,h } h ∈
Revised: { r i,h } h ∈

Original: [ H ] is the collection of known reward functions for the i th player, so that r i,h ( o i ) ∈ [0 , 1] gives the deterministic reward received by the i th player if she observes o i at step h .
Revised: [ H ] is the collection of known reward functions for the i th player, so that r i,h ( o i ) ∈ [0 , 1] gives the deterministic reward received by the i th player if she observes o i at step h .

Original: 3 We remark that since the relation among the rewards of different players can be arbitrary, this model of POMGs subsumes both the cooperative and the competitive settings in partially observable MARL.
Revised: 3 We remark that since the relation among the rewards of different players can be arbitrary, this model of POMGs subsumes both the cooperative and the competitive settings in partially observable MARL.

Original: In a POMG, the states are always hidden from all players, and each player only observes her own individual observations and actions .
Revised: In a POMG, the states are always hidden from all players, and each player only observes her own individual observations and actions .

Original: That is, each player can not see the observations and actions of the other players.
Revised: That is, each player can not see the observations and actions of the other players.

Original: At the beginning of each episode, the environment samlpes s 1 from µ 1 .
Revised: At the beginning of each episode, the environment samlpes s 1 from µ 1 .

Original: At each step h ∈
Revised: At each step h ∈

Original: [ H ] , each player i observes her own observation
Revised: [ H ] , each player i observes her own observation

Original: o i,h where o h :=
Revised: o i,h where o h :=

Original: ( o 1 ,h , . . .
Revised: ( o 1 ,h , . . .

Original: , o n,h ) are jointly sampled from O h ( · | s h ) .
Revised: , o n,h ) are jointly sampled from O h ( · | s h ) .

Original: Then each player i receives reward r i,h
Revised: Then each player i receives reward r i,h

Original: ( o i,h ) and picks action
Revised: ( o i,h ) and picks action

Original: a i,h ∈
Revised: a i,h ∈

Original: A i simultaneously.
Revised: A i simultaneously.

Original: After that the environment transitions to the next state s h +1 ∼ T h ( ·| s h , a h ) where a h :=
Revised: After that the environment transitions to the next state s h +1 ∼ T h ( ·| s h , a h ) where a h :=

Original: ( a 1 ,h , . .
Revised: ( a 1 ,h , . .

Original: . , a n,h ) .
Revised: . , a n,h ) .

Original: The current episode terminates immediately once s H +1 is reached.
Revised: The current episode terminates immediately once s H +1 is reached.

Original: Policy, value function To deﬁne different types of polices, we extend the conventions in fully observable Markov games [21] to the partially observable settings.
Revised: Policy, value function To deﬁne different types of polices, we extend the conventions in fully observable Markov games [21] to the partially observable settings.

Original: A ( random ) policy π i of the i th player is a map
Revised: A ( random ) policy π i of the i th player is a map

Original: π i : Ω × (cid:83)
Revised: π i : Ω × (cid:83)

Original: Hh =1 (cid:0) ( O i × A i )
Revised: Hh =1 (cid:0) ( O i × A i )

Original: h − 1 ×
Revised: h − 1 ×

Original: O i (cid:1)
Revised: O i (cid:1)

Original: → A i , which maps a random seed ω from space Ω and a history of length h ∈
Revised: → A i , which maps a random seed ω from space Ω and a history of length h ∈

Original: [ H ] —say τ
Revised: [ H ] —say τ

Original: i,h :=
Revised: i,h :=

Original: ( o i, 1 , a i, 1 , · · · , o i,h ) , to an action in A i .
Revised: ( o i, 1 , a i, 1 , · · · , o i,h ) , to an action in A i .

Original: To execute policy π i , we ﬁrst draw a random sample ω at the beginning of the episode.
Revised: To execute policy π i , we ﬁrst draw a random sample ω at the beginning of the episode.

Original: Then, at each step h , the i th player simply takes action π i ( ω, τ i,h ) .
Revised: Then, at each step h , the i th player simply takes action π i ( ω, τ i,h ) .

Original: We note here ω is shared among all steps
Revised: We note here ω is shared among all steps

Original: h ∈
Revised: h ∈

Original: [ H ] .
Revised: [ H ] .

Original: ω encodes both the correlation among steps and the individual randomness of each step.
Revised: ω encodes both the correlation among steps and the individual randomness of each step.

Original: We further say a policy π i is deterministic if π i ( ω, τ i,h ) =
Revised: We further say a policy π i is deterministic if π i ( ω, τ i,h ) =

Original: π i ( τ i,h ) which is independent of the choice of ω .
Revised: π i ( τ i,h ) which is independent of the choice of ω .

Original: By deﬁnition, a random policy is equivalent to a mixture of deterministic policies because given a ﬁxed ω the decision of π i on any history is deterministic.
Revised: By deﬁnition, a random policy is equivalent to a mixture of deterministic policies because given a ﬁxed ω the decision of π i on any history is deterministic.

Original: With slight abuse of notation, we use π i ( ω, · ) to refer to the deterministic policy realized by policy π i and a ﬁxed ω .
Revised: With slight abuse of notation, we use π i ( ω, · ) to refer to the deterministic policy realized by policy π i and a ﬁxed ω .

Original: We denote the set of all policies of the i th player by Π i and the set of all deterministic ones by Π det i .
Revised: We denote the set of all policies of the i th player by Π i and the set of all deterministic ones by Π det i .

Original: A joint (potentially correlated) policy is a set of policies { π i } ni =1 , where the same random seed ω is shared among all agents, which we denote as π = π 1 (cid:12) π 2 (cid:12) . . .
Revised: A joint (potentially correlated) policy is a set of policies { π i } ni =1 , where the same random seed ω is shared among all agents, which we denote as π = π 1 (cid:12) π 2 (cid:12) . . .

Original: (cid:12) π n .
Revised: (cid:12) π n .

Original: We also denote π − i =
Revised: We also denote π − i =

Original: π 1 (cid:12) . . .
Revised: π 1 (cid:12) . . .

Original: π i − 1 (cid:12) π i +1 (cid:12) . . .
Revised: π i − 1 (cid:12) π i +1 (cid:12) . . .

Original: (cid:12) π n to be the joint policy excluding the i th player.
Revised: (cid:12) π n to be the joint policy excluding the i th player.

Original: A special case of joint policy is the product policy where the random seed has special form ω
Revised: A special case of joint policy is the product policy where the random seed has special form ω

Original: = ( ω 1 , . .
Revised: = ( ω 1 , . .

Original: . , ω n ) , and for any i ∈ [ n ]
Revised: . , ω n ) , and for any i ∈ [ n ]

Original: , π i only uses the randomness in ω i , which is independent of remaining { ω j } j (cid:54)
Revised: , π i only uses the randomness in ω i , which is independent of remaining { ω j } j (cid:54)

Original: =
Revised: =

Original: i , which we denote as π = π 1 × π 2 × . . .
Revised: i , which we denote as π = π 1 × π 2 × . . .

Original: × π n .
Revised: × π n .

Original: We deﬁne the value function V πi as the expected cumulative reward that the i th player will receive if all players follow joint policy π :
Revised: We deﬁne the value function V πi as the expected cumulative reward that the i th player will receive if all players follow joint policy π :

Original: [Equation]
Revised: [Equation]

Original: where the expectation is taken over the randomness in the initial state, the transitions, the emissions, and the random seed ω in policy π .
Revised: where the expectation is taken over the randomness in the initial state, the transitions, the emissions, and the random seed ω in policy π .

Original: Best response and strategy modiﬁcation For any strategy π − i , the best response of the i th player is deﬁned as a policy of the i th player, which is independent of the randomness in π − i and achieves the highest value for herself conditioned on all other players deploying π − i .
Revised: Best response and strategy modiﬁcation For any strategy π − i , the best response of the i th player is deﬁned as a policy of the i th player, which is independent of the randomness in π − i and achieves the highest value for herself conditioned on all other players deploying π − i .

Original: Formally, the best response is the maximizer of
Revised: Formally, the best response is the maximizer of

Original: [Equation]
Revised: [Equation]

Original: whose value we also denote as V † ,π − i i for simplicity.
Revised: whose value we also denote as V † ,π − i i for simplicity.

Original: By its deﬁnition, we know the best response can always be achieved by deterministic policies.
Revised: By its deﬁnition, we know the best response can always be achieved by deterministic policies.

Original: A strategy modiﬁcation for the i th player is a map
Revised: A strategy modiﬁcation for the i th player is a map

Original: φ
Revised: φ

Original: i : Π det i → Π det i , which maps a deterministic policy in Π det i to another one in it.
Revised: i : Π det i → Π det i , which maps a deterministic policy in Π det i to another one in it.

Original: For any such strategy modiﬁcation φ
Revised: For any such strategy modiﬁcation φ

Original: i , we can naturally extend its domain and image to include random policies, i.e., deﬁne its extension φ i : Π i → Π i as follows: by deﬁnition, a random policy π can be expressed as a mixture of deterministic policies, i.e., as π ( ω, · ) (a deterministic policy for a ﬁxed ω ) with a distribution over ω .
Revised: i , we can naturally extend its domain and image to include random policies, i.e., deﬁne its extension φ i : Π i → Π i as follows: by deﬁnition, a random policy π can be expressed as a mixture of deterministic policies, i.e., as π ( ω, · ) (a deterministic policy for a ﬁxed ω ) with a distribution over ω .

Original: Then if we apply map φ i on random policy π , we can deﬁne the resulting random policy (denoted as φ i (cid:5) π i ) as φ i ( π i ( ω, · ))
Revised: Then if we apply map φ i on random policy π , we can deﬁne the resulting random policy (denoted as φ i (cid:5) π i ) as φ i ( π i ( ω, · ))

Original: (again a deterministic policy for a ﬁxed ω ) with the same distribution over ω .
Revised: (again a deterministic policy for a ﬁxed ω ) with the same distribution over ω .

Original: For any joint policy π , we deﬁne the best strategy modiﬁcation of the i th player as the maximizer of max φ i V ( φ i (cid:5) π i ) (cid:12) π − i i .
Revised: For any joint policy π , we deﬁne the best strategy modiﬁcation of the i th player as the maximizer of max φ i V ( φ i (cid:5) π i ) (cid:12) π − i i .

Original: Different from the best response, which is completely independent of the randomness in π − i , the best strategy modiﬁcation changes the policy of the i th player while still utilizing the shared randomness among π i and π − i .
Revised: Different from the best response, which is completely independent of the randomness in π − i , the best strategy modiﬁcation changes the policy of the i th player while still utilizing the shared randomness among π i and π − i .

Original: Therefore, the best strategy modiﬁcation is more powerful than the best response: formally one can show that max
Revised: Therefore, the best strategy modiﬁcation is more powerful than the best response: formally one can show that max

Original: [Equation]
Revised: [Equation]

Original: for any policy π .
Revised: for any policy π .

Original: 2.1 Learning objectives
Revised: 2.1 Learning objectives

Original: We focus on three classic equilibrium concepts in game theory—Nash Equilibrium, Correlated
Revised: We focus on three classic equilibrium concepts in game theory—Nash Equilibrium, Correlated

Original: Equilibrium (CE) and Coarse Correlated Equilibrium (CCE).
Revised: Equilibrium (CE) and Coarse Correlated Equilibrium (CCE).

Original: First, a Nash equilibrium is deﬁned as a product policy in which no player can increase her value by changing only her own policy.
Revised: First, a Nash equilibrium is deﬁned as a product policy in which no player can increase her value by changing only her own policy.

Original: Formally,
Revised: Formally,

Original: Deﬁnition 1 (Nash Equilibrium) .
Revised: Deﬁnition 1 (Nash Equilibrium) .

Original: A product policy π is a Nash equilibrium if V † ,π − i
Revised: A product policy π is a Nash equilibrium if V † ,π − i

Original: i =
Revised: i =

Original: V πi for all i ∈ [ n ] .
Revised: V πi for all i ∈ [ n ] .

Original: A product policy π is an (cid:15) -approximate
Revised: A product policy π is an (cid:15) -approximate

Original: Nash equilibrium if V † ,π − i i ≤ V πi + (cid:15) for all i ∈ [ n ] .
Revised: Nash equilibrium if V † ,π − i i ≤ V πi + (cid:15) for all i ∈ [ n ] .

Original: The Nash-regret of a sequence of product policies is the cumulative violation of the Nash condition.
Revised: The Nash-regret of a sequence of product policies is the cumulative violation of the Nash condition.

Original: Deﬁnition 2 (Nash-regret) .
Revised: Deﬁnition 2 (Nash-regret) .

Original: Let π k denote the (product) policy deployed by an algorithm in the k th episode.
Revised: Let π k denote the (product) policy deployed by an algorithm in the k th episode.

Original: After a total of K episodes, the Nash-regret is deﬁned as
Revised: After a total of K episodes, the Nash-regret is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Second, a coarse correlated equilibrium is deﬁned as a joint (potentially correlated) policy where no player can increase her value by unilaterally changing her own policy.
Revised: Second, a coarse correlated equilibrium is deﬁned as a joint (potentially correlated) policy where no player can increase her value by unilaterally changing her own policy.

Original: Formally,
Revised: Formally,

Original: Deﬁnition 3 (Coarse Correlated Equilibrium) .
Revised: Deﬁnition 3 (Coarse Correlated Equilibrium) .

Original: A joint policy π is a CCE if V † ,π − i i ≤
Revised: A joint policy π is a CCE if V † ,π − i i ≤

Original: V πi for all i ∈ [ n ] .
Revised: V πi for all i ∈ [ n ] .

Original: A joint policy π is an (cid:15) -approximate
Revised: A joint policy π is an (cid:15) -approximate

Original: CCE if V † ,π − i i ≤ V πi + (cid:15) for all i ∈ [ n ] .
Revised: CCE if V † ,π − i i ≤ V πi + (cid:15) for all i ∈ [ n ] .

Original: The only difference between Deﬁnition 1 and Deﬁnition 3 is that a Nash equilibrium has to be a product policy while a CCE can be correlated.
Revised: The only difference between Deﬁnition 1 and Deﬁnition 3 is that a Nash equilibrium has to be a product policy while a CCE can be correlated.

Original: Therefore, CCE is a relaxed notion of Nash equilibrium, and a Nash equilibrium is always a CCE.
Revised: Therefore, CCE is a relaxed notion of Nash equilibrium, and a Nash equilibrium is always a CCE.

Original: Similarly, we can deﬁne the CCE-regret for a sequence of potentially correlated policies as the cumulative vilolation of the CCE condition.
Revised: Similarly, we can deﬁne the CCE-regret for a sequence of potentially correlated policies as the cumulative vilolation of the CCE condition.

Original: Deﬁnition 4 (CCE-regret) .
Revised: Deﬁnition 4 (CCE-regret) .

Original: Let π k denote the policy deployed by an algorithm in the k th episode.
Revised: Let π k denote the policy deployed by an algorithm in the k th episode.

Original: After a total of K episodes, the CCE-regret is deﬁned as
Revised: After a total of K episodes, the CCE-regret is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Finally, a correlated equilibrium is deﬁned as a joint (potentially correlated) policy where no player can increase her value by unilaterally applying any strategy modiﬁcation.
Revised: Finally, a correlated equilibrium is deﬁned as a joint (potentially correlated) policy where no player can increase her value by unilaterally applying any strategy modiﬁcation.

Original: Formally,
Revised: Formally,

Original: Deﬁnition 5 (Correlated Equilibrium) .
Revised: Deﬁnition 5 (Correlated Equilibrium) .

Original: A joint policy π is a CE if max φ i V ( φ i (cid:5) π i ) (cid:12) π − i
Revised: A joint policy π is a CE if max φ i V ( φ i (cid:5) π i ) (cid:12) π − i

Original: i =
Revised: i =

Original: V πi for all i ∈ [ n ] .
Revised: V πi for all i ∈ [ n ] .

Original: A joint policy π is an (cid:15) -approximate CE
Revised: A joint policy π is an (cid:15) -approximate CE

Original: if max φ i V ( φ i (cid:5) π i ) (cid:12) π − i i ≤
Revised: if max φ i V ( φ i (cid:5) π i ) (cid:12) π − i i ≤

Original: [Equation]
Revised: [Equation]

Original: πi
Revised: πi

Original: [Equation]
Revised: [Equation]

Original: for all i ∈ [ m ] .
Revised: for all i ∈ [ m ] .

Original: In Partially Observable Markov games, we always have that a Nash equilibrium is a CE, and a CE is a CCE.
Revised: In Partially Observable Markov games, we always have that a Nash equilibrium is a CE, and a CE is a CCE.

Original: Finally, we deﬁne the CE-regret to be the cumulative violation of the CE condition.
Revised: Finally, we deﬁne the CE-regret to be the cumulative violation of the CE condition.

Original: Deﬁnition 6 (CE-regret) .
Revised: Deﬁnition 6 (CE-regret) .

Original: Let π k denote the policy deployed by an algorithm in the k th episode.
Revised: Let π k denote the policy deployed by an algorithm in the k th episode.

Original: After a total of K episodes, the CE-regret is deﬁned as
Revised: After a total of K episodes, the CE-regret is deﬁned as

Original: [Equation]
Revised: [Equation]

Original:  Weakly Revealing Partially Observable Markov Games
Revised:  Weakly Revealing Partially Observable Markov Games

Original: In this section, we deﬁne the class of weakly revealing POMGs.
Revised: In this section, we deﬁne the class of weakly revealing POMGs.

Original: To begin with, we consider undercomplete POMGs where there are more observations than hidden states, i.e., O ≥ S .
Revised: To begin with, we consider undercomplete POMGs where there are more observations than hidden states, i.e., O ≥ S .

Original: Formally, the family of α -weakly revealing POMGs includes all POMGs, in which the S th singular value of each emission matrix
Revised: Formally, the family of α -weakly revealing POMGs includes all POMGs, in which the S th singular value of each emission matrix

Original: O h is lower bounded by α > 0 .
Revised: O h is lower bounded by α > 0 .

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Assumption 1 ( α -weakly revealing condition) .
Revised: Assumption 1 ( α -weakly revealing condition) .

Original: There exists α > 0 , such that min h σ S ( O h ) ≥ α .
Revised: There exists α > 0 , such that min h σ S ( O h ) ≥ α .

Original: Assumption 1 is simply a robust version of the condition that the rank of each emission matrix is
Revised: Assumption 1 is simply a robust version of the condition that the rank of each emission matrix is

Original: S , which guarantees that no two different latent state mixtures can generate the same observation distribution, i.e., O h ν 1 (cid:54) =
Revised: S , which guarantees that no two different latent state mixtures can generate the same observation distribution, i.e., O h ν 1 (cid:54) =

Original: O h ν 2 for any different ν 1 , ν 2 ∈ ∆ S .
Revised: O h ν 2 for any different ν 1 , ν 2 ∈ ∆ S .

Original: Intuitively, this guarantees that the joint observations of all agents contain sufﬁcient information to distinguish any two different state mixtures.
Revised: Intuitively, this guarantees that the joint observations of all agents contain sufﬁcient information to distinguish any two different state mixtures.

Original: We remark that this is much weaker than requiring the individual observations of each agent contain sufﬁcient information about the latent states.
Revised: We remark that this is much weaker than requiring the individual observations of each agent contain sufﬁcient information about the latent states.

Original: The weakly revealing condition is important in excluding those pathological POMGs where the observations contain no useful information for identifying the key parts of model dynamics.
Revised: The weakly revealing condition is important in excluding those pathological POMGs where the observations contain no useful information for identifying the key parts of model dynamics.

Original: Note that Assumption 1 never holds in the overcomplete setting ( S > O ) as it is impossible to distinguish any two latent state mixtures by only inspecting the observation distribution in a single step.
Revised: Note that Assumption 1 never holds in the overcomplete setting ( S > O ) as it is impossible to distinguish any two latent state mixtures by only inspecting the observation distribution in a single step.

Original: To address this issue, we can instead inspect the observations for m -consecutive steps.
Revised: To address this issue, we can instead inspect the observations for m -consecutive steps.

Original: To proceed, we deﬁne the m -step emission-action matrices
Revised: To proceed, we deﬁne the m -step emission-action matrices

Original: [Equation]
Revised: [Equation]

Original: as follows:
Revised: as follows:

Original: Given an observation sequence ¯ o of length m , initial state s and action sequence ¯ a of length m − 1 , we let [ M h ] (¯ a , ¯ o ) ,s be the probability of receiving ¯ o provided that the action sequence ¯ a is used from state s and step h :
Revised: Given an observation sequence ¯ o of length m , initial state s and action sequence ¯ a of length m − 1 , we let [ M h ] (¯ a , ¯ o ) ,s be the probability of receiving ¯ o provided that the action sequence ¯ a is used from state s and step h :

Original: [Equation]
Revised: [Equation]

Original: Similar to the undercomplete setting, the weakly-revealing condition in the over-complete setting simply assumes the S th singular value of each m -step emission-action matrix is lower bounded.
Revised: Similar to the undercomplete setting, the weakly-revealing condition in the over-complete setting simply assumes the S th singular value of each m -step emission-action matrix is lower bounded.

Original: Assumption 2 (multistep α -weakly revealing condition) .
Revised: Assumption 2 (multistep α -weakly revealing condition) .

Original: There exists m ∈ N , α > 0 such that min h σ S
Revised: There exists m ∈ N , α > 0 such that min h σ S

Original: ( M h ) ≥ α where M h is the m -step emission matrix deﬁned in (2) .
Revised: ( M h ) ≥ α where M h is the m -step emission matrix deﬁned in (2) .

Original: Assumption 2 ensures that m -step consecutive observations shall contain sufﬁcient information to distinguish any two different latent state mixtures.
Revised: Assumption 2 ensures that m -step consecutive observations shall contain sufﬁcient information to distinguish any two different latent state mixtures.

Original: Note that Assumption 1 is a special case of
Revised: Note that Assumption 1 is a special case of

Original: Assumption 2 with m = 1 .
Revised: Assumption 2 with m = 1 .

Original: Finally, we remark that the single-agent versions of Assumption 1 andwere ﬁrst identiﬁed in [19] and [26] as sufﬁcient conditions for sample-efﬁcient learning of single-step and multi-step weakly revealing POMDPs (the single-agent version of POMGs), respectively.
Revised: Finally, we remark that the single-agent versions of Assumption 1 andwere ﬁrst identiﬁed in [19] and [26] as sufﬁcient conditions for sample-efﬁcient learning of single-step and multi-step weakly revealing POMDPs (the single-agent version of POMGs), respectively.

Original:  Learning Equilibria with Self-play
Revised:  Learning Equilibria with Self-play

Original: In this section, we study the self-play setting where the algorithm can control all the players to learn the equilibria by playing against itself.
Revised: In this section, we study the self-play setting where the algorithm can control all the players to learn the equilibria by playing against itself.

Original: We propose a new algorithm — Optimistic Maximum
Revised: We propose a new algorithm — Optimistic Maximum

Original: Likelihood Estimation for Learning Equilibria ( OMLE-Equilibrium ) that can provably ﬁnd Nash equilibria, coarse correlated equilibria and correlate equilibria in any weakly-revealing partially observable Markov games using a number of samples polynomial in all relevant parameters.
Revised: Likelihood Estimation for Learning Equilibria ( OMLE-Equilibrium ) that can provably ﬁnd Nash equilibria, coarse correlated equilibria and correlate equilibria in any weakly-revealing partially observable Markov games using a number of samples polynomial in all relevant parameters.

Original: 4.1 Undercomplete partially observable Markov
Revised: 4.1 Undercomplete partially observable Markov

Original: games
Revised: games

Original: We ﬁrst present the algorithm and results for learning undercomplete POMGs under Assumption 1.
Revised: We ﬁrst present the algorithm and results for learning undercomplete POMGs under Assumption 1.

Original: We will see in the later section that with a minor modiﬁcation the same algorithm also applies to learning overcomplete POMGs under Assumption 2.
Revised: We will see in the later section that with a minor modiﬁcation the same algorithm also applies to learning overcomplete POMGs under Assumption 2.

Original: Subroutine
Revised: Subroutine

Original: 1: for i ∈ [ n ] do 2: let V i ∈ R | Π det 1 |×···×| Π det n | with its π th entry equal to sup ˆ θ ∈B V πi
Revised: 1: for i ∈ [ n ] do 2: let V i ∈ R | Π det 1 |×···×| Π det n | with its π th entry equal to sup ˆ θ ∈B V πi

Original: (ˆ θ ) for π ∈ Π 1 × · · · × Π n 3: return E QUILIBRIUM ( V 1 , . .
Revised: (ˆ θ ) for π ∈ Π 1 × · · · × Π n 3: return E QUILIBRIUM ( V 1 , . .

Original: , V n )
Revised: , V n )

Original: Algorithm description To condense notations, we will use θ =
Revised: Algorithm description To condense notations, we will use θ =

Original: ( T , O , µ 1 ) to denote the parameters of a POMG.
Revised: ( T , O , µ 1 ) to denote the parameters of a POMG.

Original: Given a policy π and a trajectory τ , we denote by V πi ( θ ) the i th player’s value and by P πθ ( τ ) the probability of observing trajectory τ , both under policy π in the POMG model parameterized by θ .
Revised: Given a policy π and a trajectory τ , we denote by V πi ( θ ) the i th player’s value and by P πθ ( τ ) the probability of observing trajectory τ , both under policy π in the POMG model parameterized by θ .

Original: We describe OMLE-Equilibrium in Algorithm 1.
Revised: We describe OMLE-Equilibrium in Algorithm 1.

Original: In each episode, the algorithm executes the following two key steps:
Revised: In each episode, the algorithm executes the following two key steps:

Original: • Optimistic equilibrium computation (Line 3)
Revised: • Optimistic equilibrium computation (Line 3)

Original: We ﬁrst invoke Optimistic_Equilibrium (Subroutine 1) with conﬁdence set B k to compute a joint (potentially correlated) policy π k .
Revised: We ﬁrst invoke Optimistic_Equilibrium (Subroutine 1) with conﬁdence set B k to compute a joint (potentially correlated) policy π k .

Original: Formally, subroutine Optimistic_Equilibrium ( B k ) consists of two components: – Optimisic value estimation (Line 1-2 of Subroutine 1) For each player i ∈ [ n ] and deterministic joint policy π ∈ Π det 1 × · · · × Π det n , we compute an upper bound V πi for the i th player’s value under policy π by using the most optimistic POMG model in the conﬁdence set B k .
Revised: Formally, subroutine Optimistic_Equilibrium ( B k ) consists of two components: – Optimisic value estimation (Line 1-2 of Subroutine 1) For each player i ∈ [ n ] and deterministic joint policy π ∈ Π det 1 × · · · × Π det n , we compute an upper bound V πi for the i th player’s value under policy π by using the most optimistic POMG model in the conﬁdence set B k .

Original: – Equilibira computation (Line 3 of Subroutine 1) Given the optimistic value estimates for all deterministic joint policies and all players, we can view the POMG as a normalform game where the i th player’s pure strategies consist of all her deterministic policies (i.e., Π det i ) and the payoff she receives under a joint deterministic policy π ∈ Π det 1 ×· · ·× Π det n is equal to the corresponding optimistic value estimate V πi .
Revised: – Equilibira computation (Line 3 of Subroutine 1) Given the optimistic value estimates for all deterministic joint policies and all players, we can view the POMG as a normalform game where the i th player’s pure strategies consist of all her deterministic policies (i.e., Π det i ) and the payoff she receives under a joint deterministic policy π ∈ Π det 1 ×· · ·× Π det n is equal to the corresponding optimistic value estimate V πi .

Original: Then we compute a E QUILIBRIUM π k for this normal-form game, which is a mixture of all the deterministic joint policies in Π det 1 × · · · × Π det n .
Revised: Then we compute a E QUILIBRIUM π k for this normal-form game, which is a mixture of all the deterministic joint policies in Π det 1 × · · · × Π det n .

Original: • Conﬁdence set update (Line 4-5)
Revised: • Conﬁdence set update (Line 4-5)

Original: We ﬁrst follow π k to collect a trajectory, and then utilize the newly collected data to update the model conﬁdence set via MLE principle.
Revised: We ﬁrst follow π k to collect a trajectory, and then utilize the newly collected data to update the model conﬁdence set via MLE principle.

Original: Here we highlight two algorithmic designs in OMLE-Equilibrium : the ﬂexibility of equilibrium computation and the MLE conﬁdence set construction.
Revised: Here we highlight two algorithmic designs in OMLE-Equilibrium : the ﬂexibility of equilibrium computation and the MLE conﬁdence set construction.

Original: In the step of equilibrium computation, we can choose E QUILIBRIUM to be Nash equilibrium or correlated equilibrium (CE) or corase correlated equilibrium (CCE) of the normal-form game depending on the target type of equilibrium we aim to learn for the POMG.
Revised: In the step of equilibrium computation, we can choose E QUILIBRIUM to be Nash equilibrium or correlated equilibrium (CE) or corase correlated equilibrium (CCE) of the normal-form game depending on the target type of equilibrium we aim to learn for the POMG.

Original: With regard to the conﬁdence set design, we adopt the idea from [26] to include all the POMG models whose likelihood on the historical data is close to the maximum likelihood.
Revised: With regard to the conﬁdence set design, we adopt the idea from [26] to include all the POMG models whose likelihood on the historical data is close to the maximum likelihood.

Original: This can be viewed as a relaxation of the classic MLE method, with the degree of relaxation controlled by parameter β .
Revised: This can be viewed as a relaxation of the classic MLE method, with the degree of relaxation controlled by parameter β .

Original: One important beneﬁt of this relaxation is that although the groundtruth POMG model is in general not a solution of MLE, its likelihood ratio is rather close to the maximal likelihood.
Revised: One important beneﬁt of this relaxation is that although the groundtruth POMG model is in general not a solution of MLE, its likelihood ratio is rather close to the maximal likelihood.

Original: By doing so, we can guarantee the true model is included in the conﬁdence set with high probability.
Revised: By doing so, we can guarantee the true model is included in the conﬁdence set with high probability.

Original: Finally, we remark that Algorithm 1 is computationally inefﬁcient in general due to the steps of optimistic value estimation and equilibrium computation.
Revised: Finally, we remark that Algorithm 1 is computationally inefﬁcient in general due to the steps of optimistic value estimation and equilibrium computation.

Original: Theoretical guarantees Below we present the main theorem for OMLE-Equilibrium .
Revised: Theoretical guarantees Below we present the main theorem for OMLE-Equilibrium .

Original: Theorem 7.
Revised: Theorem 7.

Original: (Regret of OMLE-Equilibrium )
Revised: (Regret of OMLE-Equilibrium )

Original: Under Assumption 1, there exists an absolute constant c such that for any δ ∈ (0 , 1] and K ∈ N , Algorithm 1 with β = c (cid:0) ( S 2 A + SO ) log( SAOHK )
Revised: Under Assumption 1, there exists an absolute constant c such that for any δ ∈ (0 , 1] and K ∈ N , Algorithm 1 with β = c (cid:0) ( S 2 A + SO ) log( SAOHK )

Original: + log( K/δ ) (cid:1)
Revised: + log( K/δ ) (cid:1)

Original: and
Revised: and

Original: E QUILIBRIUM being one of { Nash, CCE, CE } satisﬁes (respectively) that with probability at least 1 − δ ,
Revised: E QUILIBRIUM being one of { Nash, CCE, CE } satisﬁes (respectively) that with probability at least 1 − δ ,

Original: Regret { Nash , CCE , CE } ( k ) ≤ poly( S, A, O, H, α − 1 , log( Kδ − 1 )) · √ k for all k ∈
Revised: Regret { Nash , CCE , CE } ( k ) ≤ poly( S, A, O, H, α − 1 , log( Kδ − 1 )) · √ k for all k ∈

Original: [ K ] .
Revised: [ K ] .

Original: Theorem 7 claims that if all players follow OMLE-Equilibrium , then the cumulative {Nash,CCE,CE}- regret is upper bounded by ˜ O ( √ k ) for any weakly-revealing POMGs that satisfy Assumption 1, where the growth rate w.r.t k is optimal.
Revised: Theorem 7 claims that if all players follow OMLE-Equilibrium , then the cumulative {Nash,CCE,CE}- regret is upper bounded by ˜ O ( √ k ) for any weakly-revealing POMGs that satisfy Assumption 1, where the growth rate w.r.t k is optimal.

Original: By the standard online-to-batch conversion, it directly implies the following sample complexity result:
Revised: By the standard online-to-batch conversion, it directly implies the following sample complexity result:

Original: Corollary 8.
Revised: Corollary 8.

Original: (Sample Complexity of OMLE-Equilibrium )
Revised: (Sample Complexity of OMLE-Equilibrium )

Original: Under the same setting as Theorem 7, when K ≥ poly( S, A, O, H, α − 1 , log( (cid:15) − 1 δ − 1 )) · (cid:15) − 2 , then with probability at least 1 / 2 , π out is an (cid:15) - { Nash, CCE, CE } policy.
Revised: Under the same setting as Theorem 7, when K ≥ poly( S, A, O, H, α − 1 , log( (cid:15) − 1 δ − 1 )) · (cid:15) − 2 , then with probability at least 1 / 2 , π out is an (cid:15) - { Nash, CCE, CE } policy.

Original: 1: Initialize: B 1 = { ˆ θ ∈ Θ : min h σ S ( ˆ M h ) ≥ α } , D = {} 2: for do 3: compute
Revised: 1: Initialize: B 1 = { ˆ θ ∈ Θ : min h σ S ( ˆ M h ) ≥ α } , D = {} 2: for do 3: compute

Original: 4: for do 5: execute policy π k 1: h ◦ uniform ( A ) to collect a trajectory τ k h then add ( π k 1: h ◦ uniform ( A ) , τ k,h ) into D 6: update B
Revised: 4: for do 5: execute policy π k 1: h ◦ uniform ( A ) to collect a trajectory τ k h then add ( π k 1: h ◦ uniform ( A ) , τ k,h ) into D 6: update B

Original: +1 =
Revised: +1 =

Original: ˆ θ ∈ Θ : (cid:88) ( π,τ ) ∈D log P π
Revised: ˆ θ ∈ Θ : (cid:88) ( π,τ ) ∈D log P π

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ) ≥ max θ (cid:48) ∈
Revised: ) ≥ max θ (cid:48) ∈

Original: [Table]
Revised: [Table]

Original: (cid:88) (
Revised: (cid:88) (

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: log P
Revised: log P

Original: [Table]
Revised: [Table]

Original: (cid:48) (
Revised: (cid:48) (

Original: [Table] [Table]
Revised: [Table] [Table]

Original: (cid:27) (cid:92) B
Revised: (cid:27) (cid:92) B

Original: Here the dependence on the precision parameter (cid:15) is optimal.
Revised: Here the dependence on the precision parameter (cid:15) is optimal.

Original: Finally, notice that the upper bound in Theorem 7 depends polynomially on the inverse of α — a lower bound for the minimal singular value of the joint emission matrix O h in Assumption 1.
Revised: Finally, notice that the upper bound in Theorem 7 depends polynomially on the inverse of α — a lower bound for the minimal singular value of the joint emission matrix O h in Assumption 1.

Original: This dependence is shown to be unavoidable even in the single-player setting (POMDPs) [26].
Revised: This dependence is shown to be unavoidable even in the single-player setting (POMDPs) [26].

Original: 4.2 Overcomplete partially observable Markov games
Revised: 4.2 Overcomplete partially observable Markov games

Original: In this subsection, we extend OMLE-Equilibrium to the more challenging setting of learning overcomplete POMGs, where there can be much less observations than latent states.
Revised: In this subsection, we extend OMLE-Equilibrium to the more challenging setting of learning overcomplete POMGs, where there can be much less observations than latent states.

Original: We prove that a simple variant of OMLE-Equilibrium still enjoys polynomial sample-efﬁciency guarantee for learning any multi-step weakly revealing POMGs.
Revised: We prove that a simple variant of OMLE-Equilibrium still enjoys polynomial sample-efﬁciency guarantee for learning any multi-step weakly revealing POMGs.

Original: Algorithm description We describe the multi-step generalization of OMLE-Equilibrium in Algorithm 2, which inherits the key designs from Algorithm 1 and additionally makes two important modiﬁcations to address the challenge of insufﬁcient information from single-step observation.
Revised: Algorithm description We describe the multi-step generalization of OMLE-Equilibrium in Algorithm 2, which inherits the key designs from Algorithm 1 and additionally makes two important modiﬁcations to address the challenge of insufﬁcient information from single-step observation.

Original: The ﬁrst change is to utilize a more active sampling strategy for exploration.
Revised: The ﬁrst change is to utilize a more active sampling strategy for exploration.

Original: Instead of simply following the optimistic policy π k , we will iteratively execute H − m + 1 policies of form π k 1: h ◦ uniform ( A ) where the players ﬁrst follow policy π k from step 1 to step h , then pick actions uniformly at random to ﬁnish the remaining H − h steps.
Revised: Instead of simply following the optimistic policy π k , we will iteratively execute H − m + 1 policies of form π k 1: h ◦ uniform ( A ) where the players ﬁrst follow policy π k from step 1 to step h , then pick actions uniformly at random to ﬁnish the remaining H − h steps.

Original: Intuitively, by actively trying random action sequences after executing policy π k , the algorithm can acquire more information about the system dynamics corresponding to those latent states that are frequently visited by π k , and therefore help address the challenge of lacking sufﬁcient information from single-step observation.
Revised: Intuitively, by actively trying random action sequences after executing policy π k , the algorithm can acquire more information about the system dynamics corresponding to those latent states that are frequently visited by π k , and therefore help address the challenge of lacking sufﬁcient information from single-step observation.

Original: The second change made by Algorithm 2 is that in constructing the conﬁdence set, we require the minimal singular value of the multistep emission matrix to be lower bounded, which enforces the multistep weakly revealing condition in Assumption 2.
Revised: The second change made by Algorithm 2 is that in constructing the conﬁdence set, we require the minimal singular value of the multistep emission matrix to be lower bounded, which enforces the multistep weakly revealing condition in Assumption 2.

Original: Theoretical guarantee Below we present the main theorem for multi-step OMLE-Equilibrium .
Revised: Theoretical guarantee Below we present the main theorem for multi-step OMLE-Equilibrium .

Original: Theorem 9.
Revised: Theorem 9.

Original: (Total suboptimality of multi-step OMLE-Equilibrium )
Revised: (Total suboptimality of multi-step OMLE-Equilibrium )

Original: Under Assumption 2, there exists an absolute constant c such that for any δ ∈ (0 , 1] and K ∈ N , Algorithm 1 with β = c (cid:0) ( S 2 A + SO ) log( SAOHK )
Revised: Under Assumption 2, there exists an absolute constant c such that for any δ ∈ (0 , 1] and K ∈ N , Algorithm 1 with β = c (cid:0) ( S 2 A + SO ) log( SAOHK )

Original: + log( K/δ ) (cid:1)
Revised: + log( K/δ ) (cid:1)

Original: and
Revised: and

Original: E QUILIBRIUM being one of { Nash, CCE, CE } satisﬁes (respectively) that with probability at least 1 − δ ,
Revised: E QUILIBRIUM being one of { Nash, CCE, CE } satisﬁes (respectively) that with probability at least 1 − δ ,

Original: Regret { Nash , CCE , CE } ( k ) ≤ poly( S, A m , O, H, α − 1 , log( Kδ − 1 )) · √ k for all k ∈
Revised: Regret { Nash , CCE , CE } ( k ) ≤ poly( S, A m , O, H, α − 1 , log( Kδ − 1 )) · √ k for all k ∈

Original: [ K ] , where the regret is computed for policy π 1 , . . . ,
Revised: [ K ] , where the regret is computed for policy π 1 , . . . ,

Original: π k .
Revised: π k .

Original: Theorem 9 claims that the total {Nash,CCE,CE}-“regret” (that are computed on policy π 1 , . .
Revised: Theorem 9 claims that the total {Nash,CCE,CE}-“regret” (that are computed on policy π 1 , . .

Original: . , π k ) of multi-step OMLE-Equilibrium is upper bounded by ˜ O ( √ k ) for any multi-step weakly revealing
Revised: . , π k ) of multi-step OMLE-Equilibrium is upper bounded by ˜ O ( √ k ) for any multi-step weakly revealing

Original: POMGs satisfying Assumption 2.
Revised: POMGs satisfying Assumption 2.

Original: We remark that, strictly speaking, Theorem 9 is not a standard regret guarantee since the policies executed by multi-step OMLE-Equilibrium are compositions of π 1 , . . .
Revised: We remark that, strictly speaking, Theorem 9 is not a standard regret guarantee since the policies executed by multi-step OMLE-Equilibrium are compositions of π 1 , . . .

Original: , π k and random actions, instead of purely π 1 , . . .
Revised: , π k and random actions, instead of purely π 1 , . . .

Original: , π k .
Revised: , π k .

Original: Nevertheless, we can still utilize the standard online-to-batch conversion to obtain the following sample complexity guarantee:
Revised: Nevertheless, we can still utilize the standard online-to-batch conversion to obtain the following sample complexity guarantee:

Original: Corollary 10.
Revised: Corollary 10.

Original: (sample complexity of multi-step OMLE-Equilibrium )
Revised: (sample complexity of multi-step OMLE-Equilibrium )

Original: Under the same setting as
Revised: Under the same setting as

Original: Theorem 9, when K ≥ poly( S, A m , O, H, α − 1 , log( (cid:15) − 1 δ − 1 )) · (cid:15) − 2 , then with probability at least 1 / 2 , π out is an (cid:15) - { Nash, CCE, CE } policy.
Revised: Theorem 9, when K ≥ poly( S, A m , O, H, α − 1 , log( (cid:15) − 1 δ − 1 )) · (cid:15) − 2 , then with probability at least 1 / 2 , π out is an (cid:15) - { Nash, CCE, CE } policy.

Original: =
Revised: =

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: (cid:88)
Revised: (cid:88)

Original: [Table]
Revised: [Table]

Original: Here the dependence on the precision parameter (cid:15) is optimal up to poly-logarithmic factors.
Revised: Here the dependence on the precision parameter (cid:15) is optimal up to poly-logarithmic factors.

Original: Finally, observe that the sample complexity has A m dependency that is exponential in m , which is unavoidable in general even in the single-player setting (POMDPs) [26].
Revised: Finally, observe that the sample complexity has A m dependency that is exponential in m , which is unavoidable in general even in the single-player setting (POMDPs) [26].

Original: Nonetheless, in order to make min h rank( M h ) =
Revised: Nonetheless, in order to make min h rank( M h ) =

Original: S possible (Assumption 2), we only need to make ( OA ) m
Revised: S possible (Assumption 2), we only need to make ( OA ) m

Original: (cid:38) S , i.e., m (cid:38) log S
Revised: (cid:38) S , i.e., m (cid:38) log S

Original: which is very small.
Revised: which is very small.

Original: In this paper, when we claim the sample complexity is polynomial, we consider m to be small enough so that A m ≤ poly( S, A, O, H, α − 1 ) .
Revised: In this paper, when we claim the sample complexity is polynomial, we consider m to be small enough so that A m ≤ poly( S, A, O, H, α − 1 ) .

Original:  Playing against Adversarial Opponents
Revised:  Playing against Adversarial Opponents

Original: In this section, we turn to the online setting where the learner only controls a single player and the remaining players can execute arbitrary strategies.
Revised: In this section, we turn to the online setting where the learner only controls a single player and the remaining players can execute arbitrary strategies.

Original: In this setting, we no longer target at learning game-theoretic equilibria because if other players keep playing some highly suboptimal policies then the learner may never be able to explore the environment thoroughly and thus lacks sufﬁcient information to compute equilibria.
Revised: In this setting, we no longer target at learning game-theoretic equilibria because if other players keep playing some highly suboptimal policies then the learner may never be able to explore the environment thoroughly and thus lacks sufﬁcient information to compute equilibria.

Original: Instead, we consider the standard goal for online setting, which is to achieve low regret in terms of cumulative rewards even if all other players play adversarially against the learner.
Revised: Instead, we consider the standard goal for online setting, which is to achieve low regret in terms of cumulative rewards even if all other players play adversarially against the learner.

Original: Without loss of generality, we assume the learner only controls the 1 st player throughout this section.
Revised: Without loss of generality, we assume the learner only controls the 1 st player throughout this section.

Original: 5.1 Statistical hardness for the standard setting
Revised: 5.1 Statistical hardness for the standard setting

Original: We ﬁrst consider the standard POMG setting where each player can only observe her own observations and actions.
Revised: We ﬁrst consider the standard POMG setting where each player can only observe her own observations and actions.

Original: We prove that achieving low regret in this setting is impossible in general even if (i) the POMG is two-player zero-sum and satisﬁes Assumption 1 with α = 1 , (ii) the opponent keeps playing a ﬁxed deterministic policy known to the learner, and (iii) the only parts of the model unknown to the learner are the emission matrices.
Revised: We prove that achieving low regret in this setting is impossible in general even if (i) the POMG is two-player zero-sum and satisﬁes Assumption 1 with α = 1 , (ii) the opponent keeps playing a ﬁxed deterministic policy known to the learner, and (iii) the only parts of the model unknown to the learner are the emission matrices.

Original: Theorem 11.
Revised: Theorem 11.

Original: For any L, k ∈ N + , there exist (i) a two-player zero-sum POMG of size S, A, O, H =
Revised: For any L, k ∈ N + , there exist (i) a two-player zero-sum POMG of size S, A, O, H =

Original: O ( L ) and satisfying Assumption 1 with α = 1 , and (ii) a ﬁxed opponent who keeps playing a known deterministic policy π 2 , so that with probability at least 1 /
Revised: O ( L ) and satisfying Assumption 1 with α = 1 , and (ii) a ﬁxed opponent who keeps playing a known deterministic policy π 2 , so that with probability at least 1 /

Original: [Equation]
Revised: [Equation]

Original: where π t 1 is the policy played by the learner in the t th episode.
Revised: where π t 1 is the policy played by the learner in the t th episode.

Original: Theorem 11 claims that when the learner is not allowed to access the opponent’s observations and actions, there exists exponential regret lower bound for competing with the max-min value (i.e., Nash value in two-player zero-sum POMGs) even in the very benign scenario as described above.
Revised: Theorem 11 claims that when the learner is not allowed to access the opponent’s observations and actions, there exists exponential regret lower bound for competing with the max-min value (i.e., Nash value in two-player zero-sum POMGs) even in the very benign scenario as described above.

Original: We remark that this lower bound directly implies competing with the best ﬁxed policy in hindsight is also hard because the max-min value is always no larger than the value of the best-response to π 2 :
Revised: We remark that this lower bound directly implies competing with the best ﬁxed policy in hindsight is also hard because the max-min value is always no larger than the value of the best-response to π 2 :

Original: [Equation]
Revised: [Equation]

Original: 5.2 Positive results for the game-replay setting
Revised: 5.2 Positive results for the game-replay setting

Original: In this section, we consider the game-replay setting where after each episode of play, every player will reveal their observations and actions in this episode to other players.
Revised: In this section, we consider the game-replay setting where after each episode of play, every player will reveal their observations and actions in this episode to other players.

Original: In other words, every player is able to observe the whole trajectory τ k =
Revised: In other words, every player is able to observe the whole trajectory τ k =

Original: ( o k 1 , a k 1 , . . .
Revised: ( o k 1 , a k 1 , . . .

Original: , o kH , a kH ) after the k th episode is ﬁnished.
Revised: , o kH , a kH ) after the k th episode is ﬁnished.

Original: The motivation for considering this setting is in many real-world games, e.g., Dota, StarCraft and
Revised: The motivation for considering this setting is in many real-world games, e.g., Dota, StarCraft and

Original: Poker, players are usually allowed to watch the replays of the games they have played, in which they can freely view other players’ observations and actions.
Revised: Poker, players are usually allowed to watch the replays of the games they have played, in which they can freely view other players’ observations and actions.

Original: Below, we show that a simple variant of
Revised: Below, we show that a simple variant of

Original: OMLE-Equilibrium enjoys sublinear regret when playing against adversarial opponents.
Revised: OMLE-Equilibrium enjoys sublinear regret when playing against adversarial opponents.

Original: Algorithm description We provide the formal description of OMLE-Adversary in Algorithm 3.
Revised: Algorithm description We provide the formal description of OMLE-Adversary in Algorithm 3.

Original: Same as OMLE-Equilibrium , OMLE-Adversary utilizes the relaxed MLE approach to construct the conﬁdence set.
Revised: Same as OMLE-Equilibrium , OMLE-Adversary utilizes the relaxed MLE approach to construct the conﬁdence set.

Original: The key modiﬁcation lies in the computation of player 1 ’s (stochastic) policy π k(Line 3).
Revised: The key modiﬁcation lies in the computation of player 1 ’s (stochastic) policy π k(Line 3).

Original: Speciﬁcally, the learner will compute the most optimistic model θ k in the conﬁdence set B k by examining player 1 ’s max-min value in each model.
Revised: Speciﬁcally, the learner will compute the most optimistic model θ k in the conﬁdence set B k by examining player 1 ’s max-min value in each model.

Original: Then choose player 1 ’s policy to be the one with the highest value under θ k , assuming all other players jointly play against player 1 .
Revised: Then choose player 1 ’s policy to be the one with the highest value under θ k , assuming all other players jointly play against player 1 .

Original: Finally, we remark that although the conﬁdence set construction in Line 6 seems to involve the joint policies π of all players, the conﬁdence set itself is in fact independent of the joint policies π .
Revised: Finally, we remark that although the conﬁdence set construction in Line 6 seems to involve the joint policies π of all players, the conﬁdence set itself is in fact independent of the joint policies π .

Original: Therefore, the 1 st player can still construct the conﬁdence set without knowledge of other players’ policies.
Revised: Therefore, the 1 st player can still construct the conﬁdence set without knowledge of other players’ policies.

Original: This is because the dependency of the loglikelihood function on policy π are equal on both sides of (3), and thus they cancel with each other.
Revised: This is because the dependency of the loglikelihood function on policy π are equal on both sides of (3), and thus they cancel with each other.

Original: Formally, for any ˆ θ, θ (cid:48)
Revised: Formally, for any ˆ θ, θ (cid:48)

Original: ∈ Θ , we have
Revised: ∈ Θ , we have

Original: [Equation]
Revised: [Equation]

Original: Theoretical guarantees Below we present the main theorem for OMLE-Adversary .
Revised: Theoretical guarantees Below we present the main theorem for OMLE-Adversary .

Original: Theorem 12.
Revised: Theorem 12.

Original: (Regret of OMLE-Adversary )
Revised: (Regret of OMLE-Adversary )

Original: Under Assumption 1, there exists an absolute constant c such that for any δ ∈ (0 , 1] and K ∈ N , Algorithm 3 with β = c (cid:0) ( S 2 A + SO ) log( SAOHK )
Revised: Under Assumption 1, there exists an absolute constant c such that for any δ ∈ (0 , 1] and K ∈ N , Algorithm 3 with β = c (cid:0) ( S 2 A + SO ) log( SAOHK )

Original: + log( K/δ ) (cid:1) satisﬁes that with probability at least 1 − δ ,
Revised: + log( K/δ ) (cid:1) satisﬁes that with probability at least 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: for all k
Revised: for all k

Original: [Equation]
Revised: [Equation]

Original: [ K ] .
Revised: [ K ] .

Original: Theorem 12 claims that the regret of OMLE-Adversary is upper bounded by ˜ O ( √ k ) in any weakly revealing POMGs that satisfy Assumption 1, no matter what adversarial strategies other players might take.
Revised: Theorem 12 claims that the regret of OMLE-Adversary is upper bounded by ˜ O ( √ k ) in any weakly revealing POMGs that satisfy Assumption 1, no matter what adversarial strategies other players might take.

Original: Here the regret is deﬁned by comparing the cumulative rewards received by player 1 to the max-min value max ˜ π 1 min ˜ π 2 V ˜ π 1 × ˜ π 2that is the largest value she could receive if all other players jointly play against her.
Revised: Here the regret is deﬁned by comparing the cumulative rewards received by player 1 to the max-min value max ˜ π 1 min ˜ π 2 V ˜ π 1 × ˜ π 2that is the largest value she could receive if all other players jointly play against her.

Original: Notice that this regret is weaker than the typical version of regret considered in online learning literature, which typically competes with the best response in hindsight, i.e.,
Revised: Notice that this regret is weaker than the typical version of regret considered in online learning literature, which typically competes with the best response in hindsight, i.e.,

Original: [Equation]
Revised: [Equation]

Original: Therefore, it is natural to ask whether we can obtain similar sublinear regret in terms of the above regret deﬁnition.
Revised: Therefore, it is natural to ask whether we can obtain similar sublinear regret in terms of the above regret deﬁnition.

Original: Unfortunately, previous work [27] proved that there exists exponential regret lower bound for competing with the best response in hindsight even in fully observable two-player zero-sum
Revised: Unfortunately, previous work [27] proved that there exists exponential regret lower bound for competing with the best response in hindsight even in fully observable two-player zero-sum

Original: Markov games, which are special cases of POMGs satisfying Assumption 1 with α = 1 .
Revised: Markov games, which are special cases of POMGs satisfying Assumption 1 with α = 1 .

Original: As a result, achieving low regret in the above sense is also intractable in POMGs.
Revised: As a result, achieving low regret in the above sense is also intractable in POMGs.

Original: Generalization to multi-step weakly revealing POMGs So far, we only derive the positve result (Theorem 12) for single-step weakly revealing POMGs.
Revised: Generalization to multi-step weakly revealing POMGs So far, we only derive the positve result (Theorem 12) for single-step weakly revealing POMGs.

Original: A reader might wonder whether similar results can be obtained in the more general setting of multi-step weakly revealing POMGs.
Revised: A reader might wonder whether similar results can be obtained in the more general setting of multi-step weakly revealing POMGs.

Original: Unfortunately, this generalization turns out to be impossible in general, even if (i) the POMG satisﬁes
Revised: Unfortunately, this generalization turns out to be impossible in general, even if (i) the POMG satisﬁes

Original: Assumption 2 with m = 2 and α ≥ 1 , and (ii) the learner can directly observe the opponents’ actions and observations.
Revised: Assumption 2 with m = 2 and α ≥ 1 , and (ii) the learner can directly observe the opponents’ actions and observations.

Original: We defer the formal statement of this hardness result and its proof to Appendix
Revised: We defer the formal statement of this hardness result and its proof to Appendix

Original: D.3.
Revised: D.3.

Original: Acknowledgements
Revised: Acknowledgements

Original: Csaba Szepesvári gratefully acknowledges the funding from Natural Sciences and Engineering
Revised: Csaba Szepesvári gratefully acknowledges the funding from Natural Sciences and Engineering

Original: Research Council (NSERC) of Canada, “Design.
Revised: Research Council (NSERC) of Canada, “Design.

Original: R AI-assisted CPS Design” (DARPA) project and the Canada CIFAR AI Chairs Program for Amii.
Revised: R AI-assisted CPS Design” (DARPA) project and the Canada CIFAR AI Chairs Program for Amii.

Original: Chi Jin gratefully acknowledges the Project X innovation fund from Princeton.
Revised: Chi Jin gratefully acknowledges the Project X innovation fund from Princeton.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] (c) Did you discuss any potential negative societal impacts of your work?
Revised: [Yes] (c) Did you discuss any potential negative societal impacts of your work?

Original: [No] This work is purely theoretical and we do not anticipate any potential negative social impacts.
Revised: [No] This work is purely theoretical and we do not anticipate any potential negative social impacts.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] (b) Did you include complete proofs of all theoretical results?
Revised: [Yes] (b) Did you include complete proofs of all theoretical results?

Original: [Yes] We provide all the proofs in the appendix.
Revised: [Yes] We provide all the proofs in the appendix.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: [N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: [N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: [N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [N/A] 4.
Revised: [N/A] 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [N/A] (b) Did you mention the license of the assets?
Revised: [N/A] (b) Did you mention the license of the assets?

Original: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] (d) Did you discuss whether
Revised: [N/A] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?
Revised: and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Original: [N/A] 5.
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: A On Relation between IIEFGs and Weakly Revealing POMGs

Original: In this section, we consider imperfect-information extensive-form games with perfect recall, which we call IIEFGs for simplicity.

Original: In this section, we will show that any IIEFG (of a polynomial size) is also a weakly revealing POMG (of a polynomial size) that satisﬁes Assumption 1 with α = 1 .

Original: As a result, all the algorithms and the polynomial sample complexity results developed in this paper immediately apply to learning IIEFGs using polynomial samples.

Original: We note that the reverse is not true, we can easily construct a weakly-revealing POMGs of a polynomial size that can not be represented by any IIEFGs with a polynomial size, due to the restriction of tree-structured transition and deterministic transition in IIEFGs.

Original: Therefore, polynomial sample complexity for learning IIEFGs does not imply polynomial sample complexity results for learning POMGs.

Original: A.1 Representing IIEFGs as 1-Weakly Revealing POMGs

Original: We ﬁrst introduce the deﬁnition of IIEFGs.

Original: There are many equivalent formulations of IIEFGs and here we adopt the formulation used in [24], which allows a clearer comparison to POMGs.

Original: Deﬁnition 13.

Original: An imperfect information extensive-form game with perfect recall 4 is a

Original: POMG ( H, S , {A i } n i =1 , {O i } ni =1 ; T , O , µ 1 ; { r i } ni =1 ) that additionally satisﬁes the followings:

Original: • Tree-structured transition : for each s ∈ S and h ∈ [ H − 1] , there is at most one state-action pair ( s (cid:48) , a (cid:48) )

Original: ∈ S ×

Original: A such that T h ( s | s (cid:48) , a (cid:48) )

Original: (cid:54) = 0 .

Original: In other words, for any s h , there is a unique history sequence ( s 1 , a 1 , . . .

Original: , s h − 1 , a h − 1 ) that leads to s h .

Original: [-• Deterministic emission and perfect-recall : for each s ∈ S and h ∈ [ H ] , (cid:107) O h ( · | s ) (cid:107) 
-]
Revised: [-• Deterministic emission and perfect-recall : for each s ∈ S and h ∈ [ H ] , (cid:107) O h ( · | s ) (cid:107) 
-]

Original: = 1 .

Original: That is, no state can emit two different observations.

Original: Moreover, for each player i and x ∈ O i , there is a unique history ( o i, 1 , a i, 1 , . . .

Original: , o i,h = x ) up to x from player

Original: i ’s perspective.

Original: This means player i can always retrieve her previous observations and actions solely from her current-step observation.

Original: In IIEFGs, the observations are usually referred to as information sets .

Original: • Delayed and state-action-dependent reward : different from our deﬁnition of reward in Section 2, now each r i,h is a random function from S × A to [0 , 1] , and the rewards are revealed to each learner only at the end of each episode.

Original: In other words, player i gets to observe r ki, 1 , . . .

Original: , r ki,H after the k th episode is ﬁnished.

Original: We now show that any IIEFG can be represented by 1 -weakly-revealing POMGs.

Original: Theorem 14.

Original: Any IIEFG ( H, S , {A i } ni =1 , {O i } ni =1 ; T , O , µ 1 ; { r i } ni =1 ) can be represented as a

Original: POMG with (cid:81) i

Original: |O i | states, the same action space, the same observation space, stochastic rewards which depend on the joint observation and action, and satisfying the single-step weakly revealing condition (Assumption 1) with α = 1 .

Original: Theorem 14 shows that any IIEFG of a polynomial size can be efﬁciently represented by 1 -weaklyrevealing POMGs with a polynomial size.

Original: Here, we consider the number of player n as constant when discussing polynomial versus exponential.

Original: Proof of Theorem 14.

Original: We consider an equivalent POMG formulation denoted as

Original: [Equation]

Original: where we highlight the modiﬁed parts in blue and deﬁne them as following:

Original: • State and transition.

Original: Notice that the joint observation in IIEFGs always satisﬁes the Markov property because of the perfect-recall emission structure:

Original: [Equation]

Original: Therefore we can view the original joint observation space as the new state space ˜ S :

Original: = (cid:81)

Original: i O i and deﬁne the transition as

Original: [Equation]

Original: And the initial distribution is deﬁned as ˜ µ 1 :=

Original: P ( o 1 = · | s 1 ∼ µ 1 ) .

Original: • Emission.

Original: We deﬁne the emission so that player i always observes [˜ s h

Original: i

Original: (the i th entry in ˜ s h ) with probability 1 at step h .

Original: Formally for all h ∈

Original: [ H ] and ( o , ˜ s ) ∈ O × ˜ S

Original: ˜ O h

Original: ( o | ˜ s )

Original: = 1 ( o = ˜ s ) .

Original: Clearly, in this case the joint emission is identity and therefore satisﬁes the single-step weakly revealing condition (Assumption 1) with α = 1 . • Reward.

Original: As for the reward function, we let ˜ r i,h :=

Original: for h ≤ H − 1 , and deﬁne ˜

Original: r i,H

Original: ( o H , a H ) to be a random variable taking value (cid:80)

Original: Hh =1 r i,h ( s h , a h ) with s 1: H sampled from P

Original: [Equation]

Original: Therefore, the reward ˜ r i,H is a random function of the joint observation and action ( o H , a H ) at step H .

Original: It is direct to see any policy induces the same distribution over o 1 , a 1 , . . .

Original: , o H , a H and enjoys the same value in this new formulation as in the original IIEFG.

Original: As a result, any algorithms designed for weakly revealing POMGs also apply to learning IIEFGs.

Original: Stochastic reward depending on the joint observation and action Recall when deﬁning POMGs in Section 2, we let the reward to be a deterministic function of individual observations.

Original: Nonetheless, one can easily verify all our results in this paper still hold without non-trivial modiﬁcations when the reward functions are stochastic and depend on the joint observation and action.

Original: As a result, we conclude that any IIEFG with O observations, S latent states and A actions can be represented as a 1 -weakly revealing POMG with O observations, O latent states and A actions, to which all our algorithms and theoretical guarantees directly apply.

Original: Regarding the curse of multi-player Note that all the sample complexity results proved in this paper scale exponentially with respect to n , the number of players.

Original: Therefore, they suffer from the curse of multi-players when n is large.

Original: In particular, when specializing these results to the setting of IIEFGs, we obtain sample complexity scaling with (cid:81) i ∈ [ n ] |O i | instead of (cid:80) i ∈ [ n ] |O i | where the latter is achievable by algorithms specially designed for learning IIEFGs [e.g., 4].

Original: Nonetheless, we observe that the 1 -weakly revealing POMG presentation of IIEFGs derived in Theorem 14 possesses additional benign structures: the state space is factored and the emission is identity, which could potentially be exploited to overcome the curse of dimensionality with sharper analysis or different algorithm design (e.g., incorporate the idea of V-learning style algorithms [21, 38, 11]).

Original: A.2 On Inefﬁciency of Representing POMGs using IIEFGs

Original: We prove two theorems for representing POMGs using IIEFGs:

Original: • First we show POMGs can be represented by IIEFGs with an exponentially large size.

Original: (Exponential large model is prohibitive in practice, more relevant question is for the polynomial size).

Original: • Then we prove a lower bound showing that there exists weakly revealing POMGs of constant size, which can not be represented by any IIEFGs with a polynomial size.

Original: This implies IIEFGs can not efﬁciently represent POMGs and polynomial results for learning IIEFGs can not translate into efﬁciency guarantees for learning POMGs.

Original: Theorem 15.

Original: A POMG ( H, S , {A i } ni =1 , {O i } ni =1 ; T , O , µ 1 ; { r i } ni =1 ) can be represented as an IIEFG with ( (cid:81) [Table]
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: Proof of Theorem 15.

Original: We consider an equivalent IIEFG formulation denoted as

Original: [Equation]

Original: where we highlight the modiﬁed parts in blue and deﬁne them as following:

Original: • State and transition.

Original: We view the entire interaction history as the state of IIEFG, that is, ˜ s

Original: h =

Original: ( o 1 , a 1 , . . .

Original: , o h ) .

Original: Under such choice of latent state, the transition is clearly tree structured and satisﬁes

Original: for any ˜ s h

Original: = ( o 1 , a 1 , . . .

Original: , o h ) and ˜ s h

Original: +1 =

Original: ( o

Original: (cid:48)

Original: 1 , a

Original: (cid:48) 1 , . . .

Original: , o (cid:48)

Original: h +1 )

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: h ) .

Original: • Emission.

Original: We deﬁne the emission so that player i always observes [˜ s h

Original: i

Original: (the i th entry in ˜ s h ) with probability 1 at step h .

Original: Formally for all h ∈

Original: [ H ] , if the environment is at state ˜ s h

Original: = ( o 1 , a 1 , . . .

Original: , o h ) , then each player i will observe ( o i, 1 , a i, 1 , . . .

Original: , o i,h ) with probability 1 .

Original: By the deﬁnition of state and transition, [˜ s h ]

Original: i is exactly equal to the interaction history of player i .

Original: Therefore, such emission structure satisﬁes the perfect-recall condition.

Original: • Reward.

Original: As for the reward function, we let ˜ r i,h :=

Original: for h ≤ H − 1 .

Original: At step H , for any ˜ s H

Original: =

Original: ( o 1 , a 1 , . . .

Original: , o H ) and a H , we deﬁne

Original: [Equation]

Original: h

Original: = r

Original: i,h

Original: ( o i,h , a i,h

Original: [Equation]

Original: Theorem 16.

Original: There exists an Ω(1) -weakly revealing POMG of size O (1) , which is not equivalent to any perfect-recall IIEFG with max i ∈ [ n ] |O i | ≤ 4 H − 1 .

Original: Proof of Theorem 16.

Original: It sufﬁces to prove the above theorem for the single-agent case, i.e., POMDPs.

Original: Consider a POMDP with 2 states, 2 actions and 2 observations.

Original: The emission and transition is deﬁned as

Original: [Equation]

Original: where { α h,i } ( h,i ) ∈

Original: [ H ] × [2] and { β h } h ∈ [ H ] are i.i.d.

Original: sampled from [0 , 1 / 2] .

Original: To represent the above

Original: POMDP as IIEFG with perfect recall, the size of the observation space must be at least 4 H − 1 since there are 4 H − 1 different possible trajectories of form ( o 1 , a 1 , . . .

Original: , o H − 1 , a H − 1 ) .

Original: B Notations

Original: We ﬁrst introduce some notations that will be frequently used in the remainder of appendix.

Original: • We will use µ ∈ Π det to refer to a deterministic joint policy, and use µ i ∈ Π det i to refer to a deterministic policy of player i .

Original: • Since each stochastic joint policy π ∈ Π is equivalent to a distribution over all the deterministic joint polices Π det , with slight abuse of notation, we denote by µ ∼ π the process of sampling a deterministic joint policy µ from the policy distribution speciﬁed by π .

Original: We can similarly deﬁne µ i ∼ π i for any stochastic policy π

Original: = i of player i .

Original: • Given a policy π and a POMG model θ

Original: , denote by P πθ the distribution over trajectories (i.e., τ H ) produced by executing policy π in a POMG parameterized by θ .

Original: Since the reward per trajectory is bounded by H , we always have

Original: [Equation]

Original: for any policy π , POMG models θ, ˆ θ , and player i .

Original: Denote by θ (cid:63) the parameters of the groundtruth POMG model we are interacting with.

Original: C Proofs for the Self-play Setting C.1 Proof of Theorem

Original: In this section, we prove Theorem 7 with a speciﬁc polynomial dependency as stated in the following theorem.

Original: Theorem 17.

Original: (Regret of OMLE-Equilibrium )

Original: Under Assumption 1, there exists an absolute constant c such that for any δ ∈ (0 , 1] and K ∈ N , Algorithm 1 with β = c (cid:0) H ( S 2 A + SO ) log( SAOHK )

Original: + log( K/δ ) (cid:1)

Original: and

Original: E QUILIBRIUM being one of { Nash, CCE, CE } satisﬁes (respectively) that with probability at least 1 − δ ,

Original: Regret { Nash , CCE , CE } ( k ) ≤ ˜ O (cid:16) S 2 AOα 2 (cid:112) k

Original: ( S 2 A + SO ) × poly( H ) (cid:17) for all k ∈

Original: [ K ] .

Original: The proof consists of three steps:

Original: First we rewrite Algorithm 1 in an equivalent form that is perfectly compatible with the analysis in [26]. 2.

Original: After that we can directly import the theoretical guarantees from [26] and obtain a sublinear upper bound for the cumulative error of density estimation.

Original: Finally, we combine the game-theoretic analysis tailored for POMGs with the density estimation guarantee derived in the second step, which gives the desired sublinear gametheoretic regret.

Original: C.1.1 Step

Original: To begin with, we make the following observations about Algorithm 1:

Original: • The sampling procedure in each episode k is equivalent to: ﬁrst sample a deterministic joint policy µ k from π k and then execute µ k to collect a trajectory

Original: τ k . •

Original: In constructing the conﬁdence set B k , we can replace π k with µ k without making any difference, because the dependency of the log-likelihood function on policy π are equal on both sides of the inequality in B k and thus they cancel with each other.

Original: Formally, for any ˆ θ, θ (cid:48)

Original: ∈ Θ , we have k (cid:88) t

Original: [Equation]

Original: (cid:16) log P

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: log P

Original: [Equation]

Original: (cid:48)

Original: [Equation]

Original: Based on the above two observations, Algorithm 1 can be equivalently written in the form of

Original: Algorithm 4 where we highlight the modiﬁed parts in blue.

Original: Remark 18.

Original: The technical reason for rewriting Algorithm 1 in the form of Algorithm 4 is that in the optimistic equilibrium subroutine (Subroutine 1) we utilize the optimistic value estimate for each deterministic joint policy to construct the optimistic normal-form game and compute the optimistic game-theoretic equilibria.

Original: As a result, in order to control the cumulative regret due to over-optimism, we need guarantees on the accuracy of optimistic value estimates for deterministic joint policies.

Original: This is why we want to explicitly insert the “dummy” deterministic policy µ k in each episode.

Original: C.1.2 Step

Original: Now we can directly instantiate the analysis of optimistic MLE (Appendix E in [26]) on Algorithm 4, which gives the following theoretical guarantee:

Original: Theorem 19.

Original: Under Assumption 1 and the same choice of β as in Theorem 7, with probability at least 1 − δ , Algorithm 4 satisﬁes that for all k ∈ [ K ] and all θ 1 ∈ B 1 ,..., θ K ∈ B K • θ (cid:63)

Original: ∈

Original: B

Original: k ,

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: We comment that there are two differences between the optimistic MLE algorithm in [26] and the

Original: Algorithm 4 here: (i) the former one is designed for single-player POMGs, i.e., POMDPs while the latter one is for multi-player POMGs; (ii) µ t is computed using different criteria.

Original: Nonetheless, we can still reuse their theoretical guarantees proved in their Appendix E without making any change because: (i) in the self-play setting, multi-player POMGs can be viewed as POMDPs with a single meta-player whose action space is of cardinality A

Original: =

Original: A 1 × · · · × A n and observation space is of cardinality O

Original: =

Original: O 1 × · · · × O n ; (ii) when proving the second statement in Theorem 19, [26] only use the fact that τ t is sampled from µ t but allow both µ t and θ t ∈ B t to be arbitrarily chosen.

Original: (The only place [26] need to use how µ t and θ t is computed is in relating the regret to (cid:80) kt =1 (cid:107) P µ t θ t − P µ t θ (cid:63) (cid:107) 1 , which has nothing to do with the proof of Theorem 19.)

Original: C.1.3 Step

Original: Now let us prove Theorem 17 conditioning on the two relations stated in Theorem 19 being true.

Original: To proceed, we deﬁne

Original: [Equation]

Original: for any ( µ, k,

Original: i

Original: [Equation]

Original: [Equation]

Original: Note that conditioning on the ﬁrst relation in Theorem 19, we always have V k,µi ≥ V µi for all ( µ,

Original: k, i ) ∈ Π det × [ K ] × [ n ] because by deﬁnition V µ i =

Original: V µ i ( θ (cid:63) ) .

Original: Nash equilibrium When we choose E QUILIBRIUM in Subroutine 1 to be Nash equilibrium, by the deﬁnition of Nash-regret,

Original: [Equation]

Original: [Equation]

Original: where the ﬁnal equality uses the fact that π k is a Nash equilibrium of the normal-form game deﬁned by ( V k 1 , .

Original: , V kn ) as described in Subroutine 1.

Original: By Jensen’s inequality and Azuma-Hoeffding inequality,

Original: [Equation]

Original: where the last equality uses the deﬁnition of V k and the last inequality uses the fact that the reward is an H -bounded function of the trajectory.

Original: Finally, we complete the proof by using the second relation in Theorem 19, which upper bounds (cid:80) k max ˆ θ ∈B k

Original: [Equation]

Original: − P µ

Original: [Equation]

Original: 1 by

Original: [Equation]

Original: Coarse correlated equilibrium When we choose E QUILIBRIUM in Subroutine 1 to be CCE, the proof is exactly the same as for Nash equilibrium, except that the last equality in Equation (4) becomes “no larger than” by the deﬁnition of CCE.

Original: Correlated equilibrium

Original: When we choose E QUILIBRIUM in Subroutine 1 to be CE, by the deﬁnition of CE-regret,

Original: [Equation]

Original: where the second equality uses the deﬁnition of strategy modiﬁcation, and the ﬁnal equality uses the fact that π k is a CE of the normal-form game deﬁned by ( V k 1 , . .

Original: , V k n ) as described in Subroutine 1.

Original: The remaining steps are the same as of the proof for Nash-regret.

Original: C.2 Proof of Theorem

Original: In this section, we prove Theorem 9 with a speciﬁc polynomial dependency as stated in the following theorem.

Original: Theorem 20.

Original: (Total suboptimality of multi-step OMLE-Equilibrium )

Original: Under Assumption 2, there exists an absolute constant c such that for any δ ∈ (0 , 1] and K ∈ N , Algorithm 2 with

Original: [Equation]

Original: (cid:1) and E QUILIBRIUM being one of { Nash, CCE, CE } satisﬁes (respectively) that with probability at least 1 − δ ,

Original: [Equation]

Original: for all k ∈ [ K ] , where the regret is computed for policy π 1 , . . .

Original: , π k .

Original: 1: Initialize: B 1 = { ˆ θ ∈ Θ : min h σ S ( ˆ M h ) ≥ α } , D = {} 2: for k = 1 , . . .

Original: , K do 3: compute π k = Optimistic_Equilibrium

Original: ( B k ) and sample µ k from π k 4: for h = 0 , . . .

Original: , H − m do 5: execute policy µ k 1: h ◦ uniform ( A ) to collect a trajectory τ k,h then add ( µ k 1: h ◦ uniform ( A ) , τ k,h ) into D 6: update

Original: [Equation]

Original: =

Original: [Equation]

Original: [Equation]

Original: (cid:27)

Original: [Equation]

Original: The proof of Theorem 20 follows basically the same arguments as in the undercomplete setting, except that we replace Algorithm 4 with Algorithm 5 6 and Theorem 19 with Theorem 21 in the ﬁrst two steps.

Original: And the third step is exactly the same.

Original: To avoid noninformative repetitive arguments, here we only state Algorithm 5 and Theorem 21, while one can directly verify all the proofs in Section C. still hold after we make the aforementioned replacements.

Original: Theorem 21.

Original: Under Assumption 2 and the same choice of β as in Theorem 9, with probability at least 1 − δ , Algorithm 5 satisﬁes that for all k ∈ [ K ] and all θ 1 ∈ B 1 ,..., θ K ∈ B K

Original: • θ (cid:63)

Original: ∈

Original: B

Original: k

Original: , • (cid:80) kt =1 (cid:107) P µ t θ t − P µ t θ (cid:63) (cid:107) 1 ≤ ˜ O (cid:16)

Original: [-S 
-]
Revised: [-S 
-]

Original: A 3 m − 2 α

Original: [Equation]

Original: k

Original: [Equation]

Original: S 2 A

Original: +

Original: [Equation]

Original: × poly

Original: [Equation]

Original: H )

Original: [Equation]

Original: We remark that Theorem 21 follows directly from instantiating the analysis of multi-step optimistic

Original: MLE (Appendix F in [26]) on Algorithm 5.

Original: D Proofs for Playing against Adversarial Opponents D.1 Proof of Theorem

Original: In this section, we prove Theorem 12 with a speciﬁc polynomial dependency as stated in the following theorem.

Original: Theorem 22.

Original: (Regret of OMLE-Adversary )

Original: Under Assumption 1, there exists an absolute constant c such that for any δ ∈ (0 , 1] and K ∈ N , Algorithm 3 with β

Original: = c

Original: [Equation]

Original: satisﬁes that with probability at least

Original: [Equation]

Original: for all k ∈ [ K ]

Original: [Equation]

Original: To begin with, for the same reasons as explained in Section C.1.2, we can directly instantiate the guarantees for optimistic MLE (Appendix E in [26]) on Algorithm 3 and obtain:

Original: Theorem 23.

Original: Under Assumption 1 and the same choice of β as in Theorem 12, with probability at least 1 − δ , Algorithm 3 satisﬁes that for all k ∈ [ K ] and all θ 1 ∈ B 1 ,..., θ K ∈ B K

Original: • θ (cid:63) ∈

Original: B k , • (cid:80) k

Original: t =1 (cid:107) P π t θ t − P π t θ (cid:63) (cid:107) 1 ≤ ˜ O (cid:16) S 2 AO α 2 (cid:112) k

Original: ( S 2 A + SO ) × poly( H ) (cid:17) .

Original: Now conditioning on the two relations in Theorem 23 being true, we have

Original: [Equation]

Original: by the deﬁnition of π k=

Original: [Equation]

Original: reward per episode ∈ [0 , H ]

Original: [Equation]

Original: Theorem

Original: [Equation]

Original: D.2 Proof of Theorem

Original: [Equation]

Original: up low

Original: [Equation]

Original: The hard instance is best illustrated by Figure 1 where we sketch the transition dynamics of the

Original: POMG.

Original: Below we elaborate the construction based on Figure 1.

Original: • States and actions.

Original: Each circle and rectangle in Figure 1 represents a state.

Original: Each player has two actions denoted by A = { a 0 , a 1 } and B = { b 0 , b 1 } respectively.

Original: • Observations.

Original: The max-player can always directly observe the current latent state.

Original: The min-player observes the same dummy observation o null in any black circle while directly observes the current state in any red circle and any rectangle.

Original: It is easy to verify by deﬁnition min h σ min

Original: ( O h ) = 1 because the emission structure is a bijection between the joint observation space and the latent state space. • Reward. Only the upper rectangle emits an observation containing reward 1 for the maxplayer (thus reward − 1 for the min-player).
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: All other states emit observations with zero reward.

Original: • Transitions.

Original: At the beginning of each episode, the environment starts from u 1 or luniformly at random.

Original: The transition dynamics from step 1 to step H only depend on the actions of the max-player while in the ﬁnal step ( H → H +1 ) the transitions are determined by the min-player.

Original: Formally, – When the environment is in the upper half of the POMG: for each step h ∈ [ H − 1] , the environment will transition to u h +1 , 0 if the max-player takes action a 0 and transition to u h +1 , 1 if the max-player takes action a 1 .

Original: At step H , the agent will transition to the upper rectangle if the min-player takes action b 1 and transition to the lower one if the min-player picks b 0 .

Original: – When the environment is in the lower half of the POMG: for each step

Original: h ∈

Original: [ H − 1] , the environment will transition to l h +1 , 0 if the max-player takes action a 0 and transition to l h +1 , 1 if the max-player takes action a 1 .

Original: At step H , the agent will transition to the upper rectangle if the min-player takes action b 0 and transition to the lower one if the min-player picks b 1 .

Original: Min-player’s optimal strategy.

Original: It is direct to see the min-player’s optimal strategy is to take action b 0 in the upper half of the POMG and action b 1 in the lower half, at step H .

Original: This stratety will lead to zero-reward for the max-player.

Original: However, implementing this strategy requires the min-player to infer which half the environment is in from her observations, which is possible only when the environment has visited some red circles in the ﬁrst H steps.

Original: This is because the min-player directly observes the current state in red circles while observes the same observation o null in all black circles.

Original: Max-player’s optimal strategy.

Original: To prevent the min-player from discovering which half the environment currently lies in, the max-player’s optimal strategy is to avoid visiting any red circles.

Original: Hardness.

Original: However, hardness happens if (a) the max-player cannot access the observations of the min-player and (b) for each h ∈ { 2 , . . .

Original: , H } , we uniformly at random pick one of { u h 0 , u h 1 } and one of { l h 0 , l h 1 } to be red circles, and set the remaining ones to be black.

Original: From the perspective of the max-player, she cannot directly tell which state is red or black because (a) the difference between black circles and red circles only appear in the min-player’s observations, and (b) the max-player cannot see what the min-player observes.

Original: As a result, the only useful information for the max-player to ﬁgure out which circles are red is the action picked by the min-player in the ﬁnal step.

Original: Now suppose the min-player will play the optimal strategy when she knows which half the environment is in, and pick action b 0 when she does not.

Original: In this case, for the max-player, identifying all the red circles is as hard as learning a bandit with Ω(2 H ) arms where only one arm has reward 1 / 2 and all other arms has reward 0 .

Original: Therefore, by using standard lower bound arguments for bandits, we can show the max-player’s cumulative rewards in the ﬁrst K = Θ(2 H ) episodes is 0 with constant probability.

Original: In comparison, the optimal strategy, which avoids visiting all red circles, can collect K/rewards with high probability.

Original: As a result, we obtain the desired Ω(min { 2 H , K } ) regret lower bound for competing against the Nash value.

Original: D.3 Playing against adversary in multi-step weakly-revealing POMGs is hard

Original: In this section, we prove that competing with the max-min value is statistically hard even if (i)

Original: the

Original: POMG is two-player zero-sum and satisﬁes Assumption 2 with m = 2 and α = 1 , (ii) the opponent keeps playing a ﬁxed action, and (iii) the player can directly observe the opponents’ actions and observations.

Original: Theorem 24.

Original: Assume the player can directly observe the opponents’ actions and observations.

Original: For any L, k ∈ N + , there exist (i) a two-player zero-sum POMG of size S, A, O, H

Original: = O ( L ) and satisfying Assumption 2 with m =

Original: 2 and α = 1 , and (ii) an opponent who keeps playing a ﬁxed action ˆ a 2 , so that with probability at least 1 /

Original: [Equation]
Revised: [N/A]

Original: where π t 1 is the policy played by the learner in the t th episode.

Original: Proof.

Original: The hard instance is constructed as following:

Original: • States and actions : There are four states: p 0 , p 1 and q 0 , q 1 .

Original: Each player has two actions, denoted by { a 0 , a 1 } and { b 0 , b 1 } respectively.

Original: • Emission and reward : There are three different observations:

Original: o dummy , o 1 and o 0 .

Original: At step h ∈

Original: [ H − 1] , p 1 and p 0 emit the same observation o dummy .

Original: At step H , p 1 emits o 1 while pemits o 0 .

Original: Regardless of h , q 0 always emits o 0 and q 1 always emits o 1 .

Original: Importantly, all players share the same observation.

Original: The reward function is deﬁned so that r ( o dummy ) =

Original: r ( o 0 )

Original: =and r ( o 1 )

Original: = 1 for the max-player.

Original: Since the game is zero-sum, the reward function for the min-player is simply − r ( · ) .

Original: • Transition : Let x 1 , . . .

Original: , x h be a binary sequence sampled independently and uniformly at random from standard Bernoulli distribution.

Original: At step h = 1 , the POMG always starts from state p 1 .

Original: For each step h ∈

Original: [ H − 1] : – If the current state is p i , then the environment will transition to p 1 if and only if i = 1 , the max-player plays action a x h , and the min-player plays b 0 .

Original: Otherwise, if the minplayer plays b 1 , then the environment will transition to q i .

Original: Otherwise, the environment will transition to p 0 .

Original: – If the current state is q i , the next state will be q 1 regardless of players’ actions.

Original: We have the following observations

Original: : • If the min-player keeps playing b 0 , then from the perspective of the max-player the POMG essentially reduces to a multi-arm bandit problem with 2 H − 1 arms because in this case the only useful feedback for the max-player is the reward observed at step H .

Original: • The max-min (Nash) value is equal to 1 , which is attained when the max-player picks a x h at step h with probability 1 .

Original: • The 2 -step emission-action matrix at each step

Original: h ∈

Original: [ H − 1] is rank 4 and has minimum singular value no

Original: smaller than 1 , because we can always exactly identify the current state (for step h ∈

Original: [ H − 1] ) by the current-step observation and the next-step observation if the min-player picks action b 1 in the current step.

Original: Based on the ﬁrst two observations above, we immediately obtain a Θ(min { 2 H , k } ) lower bound for competing with the max-min (Nash) value.

Original: Using the third observation, we know the POMG is 2 -step weakly revealing with α = 1 , which completes the proof.