Original: Sample-Efﬁcient Reinforcement Learning of Partially Observable Markov Games
Revised: Sample-Efﬁcient Reinforcement Learning of Partially Observable Markov Games

Original: AfﬁliationAddress
Revised: AfﬁliationAddress

Original: Abstract
Revised: Abstract

Original: This paper considers the challenging tasks of Multi-Agent Reinforcement Learning(MARL) under partial observability, where each agent only sees her own individualobservations and actions that reveal incomplete information about the underlying state of system.
Revised: This paper considers the challenging tasks of Multi-Agent Reinforcement Learning(MARL) under partial observability, where each agent only sees her own individualobservations and actions that reveal incomplete information about the underlying state of system.

Original: This paper studies these tasks under the general model ofmultiplayer general-sum Partially Observable Markov Games (POMGs), which issigniﬁcantly larger than the standard model of Imperfect Information ExtensiveForm Games (IIEFGs).
Revised: This paper studies these tasks under the general model ofmultiplayer general-sum Partially Observable Markov Games (POMGs), which issigniﬁcantly larger than the standard model of Imperfect Information ExtensiveForm Games (IIEFGs).

Original: We identify a rich subclass of POMGs—weakly revealing
Revised: We identify a rich subclass of POMGs—weakly revealing

Original: POMGs—in which sample-efﬁcient learning is tractable.
Revised: POMGs—in which sample-efﬁcient learning is tractable.

Original: In the self-play setting,we prove that a simple algorithm combining optimism and Maximum Likelihood
Revised: In the self-play setting,we prove that a simple algorithm combining optimism and Maximum Likelihood

Original: Estimation (MLE) is sufﬁcient to ﬁnd approximate Nash equilibria, correlatedequilibria, as well as coarse correlated equilibria of weakly revealing POMGs, in apolynomial number of samples when the number of agents is small.
Revised: Estimation (MLE) is sufﬁcient to ﬁnd approximate Nash equilibria, correlatedequilibria, as well as coarse correlated equilibria of weakly revealing POMGs, in apolynomial number of samples when the number of agents is small.

Original: In the settingof playing against adversarial opponents, we show that a variant of our optimistic
Revised: In the settingof playing against adversarial opponents, we show that a variant of our optimistic

Original: MLE algorithm is capable of achieving sublinear regret when being comparedagainst the optimal maximin policies.
Revised: MLE algorithm is capable of achieving sublinear regret when being comparedagainst the optimal maximin policies.

Original: To our best knowledge, this work providesthe ﬁrst line of sample-efﬁcient results for learning POMGs.
Revised: To our best knowledge, this work providesthe ﬁrst line of sample-efﬁcient results for learning POMGs.

Original:  Introduction
Revised:  Introduction

Original: This paper studies Multi-Agent Reinforcement Learning (MARL) under partial observability , whereeach player tries to maximize her own utility via interacting with an unknown environment aswell as other players.
Revised: This paper studies Multi-Agent Reinforcement Learning (MARL) under partial observability , whereeach player tries to maximize her own utility via interacting with an unknown environment aswell as other players.

Original: In addition, each agent only sees her own observations and actions, whichreveal incomplete information about the underlying state of system.
Revised: In addition, each agent only sees her own observations and actions, whichreveal incomplete information about the underlying state of system.

Original: A large number of real-worldapplications can be cast into this framework: in Poker, cards in a player’s hand are hidden from theother players; in many real-time strategy games, players have only access to their local observations;in multi-agent robotic systems, agents with ﬁrst-person cameras have to cope with noisy sensors andocclusions.
Revised: A large number of real-worldapplications can be cast into this framework: in Poker, cards in a player’s hand are hidden from theother players; in many real-time strategy games, players have only access to their local observations;in multi-agent robotic systems, agents with ﬁrst-person cameras have to cope with noisy sensors andocclusions.

Original: While practical MARL systems have achieved remarkable success in a set of partiallyobservable problems including Poker [10], Starcraft [52], Dota [9] and autonomous driving [47], thetheoretical understanding of MARL under partial observability remains very limited.
Revised: While practical MARL systems have achieved remarkable success in a set of partiallyobservable problems including Poker [8], Starcraft [38], Dota [5] and autonomous driving [34], thetheoretical understanding of MARL under partial observability remains very limited.

Original: The combination of partial observability with multiagency introduces a number of unique challenges.
Revised: The combination of partial observability with multiagency introduces a number of unique challenges.

Original: The non-Markovian nature of the observations forces the agent to maintain memory and reasonabout beliefs of the system state, all while exploring to collect information about the environment.
Revised: The non-Markovian nature of the observations forces the agent to maintain memory and reasonabout beliefs of the system state, all while exploring to collect information about the environment.

Original: Consequently, well-known complexity-theoretic results show that learning and planning in partiallyobservable environments is statistically and computationally intractable even in the single-agentsetting [44, 43, 53, 42].
Revised: Consequently, well-known complexity-theoretic results show that learning and planning in partiallyobservable environments is statistically and computationally intractable even in the single-agentsetting [33, 28, 39, 27].

Original: The presence of interaction between multiple agents further complicates thepartially observable problems.
Revised: The presence of interaction between multiple agents further complicates thepartially observable problems.

Original: In addition to dealing with the adaptive nature of other players whocan adjust their strategies according to the learner’s past behaviors, the learner is further required todiscover and exploit the information asymmetry due to the separate observations of each agent.
Revised: In addition to dealing with the adaptive nature of other players whocan adjust their strategies according to the learner’s past behaviors, the learner is further required todiscover and exploit the information asymmetry due to the separate observations of each agent.

Original: As a result, prior theoretical works on partially observable MARL have been mostly focused ona small subset of problems with strong structural assumptions.
Revised: As a result, prior theoretical works on partially observable MARL have been mostly focused ona small subset of problems with strong structural assumptions.

Original: For instance, the line of workson Imperfect Information Extensive-Form Games (IIEFG)
Revised: For instance, the line of workson Imperfect Information Extensive-Form Games (IIEFG)

Original: [see, e.g., 61, 18, 15, 31] assumes treestructured transition with small depth 1 as well as a special type of emission which can be representedas information sets .
Revised: [see, e.g., 43, 13, 12, 22] assumes treestructured transition with small depth 1 as well as a special type of emission which can be representedas information sets .

Original: In contrast, this paper considers a more general mathematical model known as
Revised: In contrast, this paper considers a more general mathematical model known as

Original: Partially Observable Markov Games (POMGs).
Revised: Partially Observable Markov Games (POMGs).

Original: POMGs are the natural extensions of both Partially
Revised: POMGs are the natural extensions of both Partially

Original: Observable Markov Decision Processes (POMDPs)—the standard model for single-agent partiallyobservable RL, and Markov Games [48]—the standard model for fully observable MARL.
Revised: Observable Markov Decision Processes (POMDPs)—the standard model for single-agent partiallyobservable RL, and Markov Games [35]—the standard model for fully observable MARL.

Original: Despitethe complexity barriers of learning partially observable systems apply to POMGs, they are of aworst-case nature, which do not preclude efﬁcient algorithms for learning interesting subclasses of
Revised: Despitethe complexity barriers of learning partially observable systems apply to POMGs, they are of aworst-case nature, which do not preclude efﬁcient algorithms for learning interesting subclasses of

Original: POMGs.
Revised: POMGs.

Original: This motivates us to ask the following question:
Revised: This motivates us to ask the following question:

Original: Can we develop efﬁcient algorithms that learn a rich class of POMGs?
Revised: Can we develop efﬁcient algorithms that learn a rich class of POMGs?

Original: In this paper, we provide the ﬁrst positive answer to the highlighted question in terms of the sampleefﬁciency .
Revised: In this paper, we provide the ﬁrst positive answer to the highlighted question in terms of the sampleefﬁciency .

Original: We identify a rich family of tractable POMGs— weakly revealing POMGs (see Section3).
Revised: We identify a rich family of tractable POMGs— weakly revealing POMGs (see Section3).

Original: The weakly revealing condition only requires the joint observations of all agents to reveal certainamount of information about the latent states, which is satisﬁed in many real-world applications.
Revised: The weakly revealing condition only requires the joint observations of all agents to reveal certainamount of information about the latent states, which is satisﬁed in many real-world applications.

Original: Thecondition rules out the pathological instances where no player has any information to distinguishlatent states, which prevents efﬁcient learninig in the worst case.
Revised: Thecondition rules out the pathological instances where no player has any information to distinguishlatent states, which prevents efﬁcient learninig in the worst case.

Original: In the self-play setting where the algorithm can control all the players to learn the equilibria byplaying against itself, this paper proposes a new simple algorithm— Optimistic Maximum Likelihood
Revised: In the self-play setting where the algorithm can control all the players to learn the equilibria byplaying against itself, this paper proposes a new simple algorithm— Optimistic Maximum Likelihood

Original: Estimation for Learning Equilibria ( OMLE-Equilibrium ).
Revised: Estimation for Learning Equilibria ( OMLE-Equilibrium ).

Original: As the name suggests, it combines optimism, MLE principles with equilibria ﬁnding subroutines.
Revised: As the name suggests, it combines optimism, MLE principles with equilibria ﬁnding subroutines.

Original: The algorithm provably ﬁnds approximate
Revised: The algorithm provably ﬁnds approximate

Original: Nash equilibria, coarse correlated equilibria and correlate equilibria of any weakly-revealing POMGsusing a number of samples polynomial in all relevant parameters.
Revised: Nash equilibria, coarse correlated equilibria and correlate equilibria of any weakly-revealing POMGsusing a number of samples polynomial in all relevant parameters.

Original: In the setting of playing against adversarial opponents, we measure the performance of our algorithmby comparing against the optimal maximin policies.
Revised: In the setting of playing against adversarial opponents, we measure the performance of our algorithmby comparing against the optimal maximin policies.

Original: We ﬁrst prove that learning in this setting ishard if each player can only see her own observations and actions.
Revised: We ﬁrst prove that learning in this setting ishard if each player can only see her own observations and actions.

Original: Nevertheless, if the agent isallowed to access other players’ observations and actions after each episode of play (e.g., watchthe replays of the games from other players’ perspectives afterwards), then we can design a newalgorithm OMLE-Adversary which achieves sublinear regret.
Revised: Nevertheless, if the agent isallowed to access other players’ observations and actions after each episode of play (e.g., watchthe replays of the games from other players’ perspectives afterwards), then we can design a newalgorithm OMLE-Adversary which achieves sublinear regret.

Original: To our best knowledge, this is the ﬁrst line of provably sample-efﬁcient results for learning rich classesof POMGs.
Revised: To our best knowledge, this is the ﬁrst line of provably sample-efﬁcient results for learning rich classesof POMGs.

Original: Importantly, the classes of problems that can be learned in this paper are signiﬁcantlylarger than known tractable classes of MARL problems under partial observability.
Revised: Importantly, the classes of problems that can be learned in this paper are signiﬁcantlylarger than known tractable classes of MARL problems under partial observability.

Original: 1.1 Technical novelty
Revised: 1.1 Technical novelty

Original: This paper builds upon the recent progress in learning single-agent POMDPs [37], which identiﬁesthe class of weakly revealing POMDPs and develops OMLE algorithm for learning the optimal policy.
Revised: This paper builds upon the recent progress in learning single-agent POMDPs [24], which identiﬁesthe class of weakly revealing POMDPs and develops OMLE algorithm for learning the optimal policy.

Original: Besides obtaining a completely new set of results in the multi-agent setting, we here highlight a fewcontributions and technical novelties of this paper comparing to [37].
Revised: Besides obtaining a completely new set of results in the multi-agent setting, we here highlight a fewcontributions and technical novelties of this paper comparing to [24].

Original: • This paper rigorously formulates the models, related concepts and learning objectives of multi-
Revised: • This paper rigorously formulates the models, related concepts and learning objectives of multi-

Original: player general-sum POMGs
Revised: player general-sum POMGs

Original: and provides the ﬁrst line of sample-efﬁcient learning results.
Revised: and provides the ﬁrst line of sample-efﬁcient learning results.

Original: • Extending the weakly revealing conditions into the multiagent setting lead to two natural candidates:
Revised: • Extending the weakly revealing conditions into the multiagent setting lead to two natural candidates:

Original: either (a) joint observations or (b) individual observations are required to weakly reveal the state
Revised: either (a) joint observations or (b) individual observations are required to weakly reveal the state

Original: information.
Revised: information.

Original: This paper shows the former (the weaker assumption) sufﬁces to guarantee tractability.
Revised: This paper shows the former (the weaker assumption) sufﬁces to guarantee tractability.

Original: • Results in the self-play setting requires careful design of optimistic planning algorithms thateffectively address the game-theoretical aspects of the problem under partial observability.
Revised: • Results in the self-play setting requires careful design of optimistic planning algorithms thateffectively address the game-theoretical aspects of the problem under partial observability.

Original: Weachieve this by Subroutine 1, which is even distinct from the standard techniques for learning MGs.•
Revised: Weachieve this by Subroutine 1, which is even distinct from the standard techniques for learning MGs.•

Original: The discussions and results in the setting of playing against adversarial
Revised: The discussions and results in the setting of playing against adversarial

Original: We leave the challenge of computationally efﬁcient learning for future work
Revised: We leave the challenge of computationally efﬁcient learning for future work

Original: 1.2 Related works
Revised: 1.2 Related works

Original: Due to the space limit, we postpone a detailed discussion of related works to Appendix A.
Revised: Due to the space limit, we postpone a detailed discussion of related works to Appendix A.

Original:  Preliminary
Revised:  Preliminary

Original: In this paper, we consider Partially Observable Markov Games (POMGs) in its most generic—multiplayer general-sum form.
Revised: In this paper, we consider Partially Observable Markov Games (POMGs) in its most generic—multiplayer general-sum form.

Original: Formally, we denote a tabular episodic POMG with n players bytuple
Revised: Formally, we denote a tabular episodic POMG with n players bytuple

Original: ( H, S , {A i } ni
Revised: ( H, S , {A i } ni

Original: =1 , {O i } ni =1 ; T , O , µ 1 ; { r i } ni
Revised: =1 , {O i } ni =1 ; T , O , µ 1 ; { r i } ni

Original: =1 ) , where H denotes the length of each episode, Sthe state space with |S| = S , A
Revised: =1 ) , where H denotes the length of each episode, Sthe state space with |S| = S , A

Original: i denotes the action space for the i th player with |A
Revised: i denotes the action space for the i th player with |A

Original: i | =
Revised: i | =

Original: A i .
Revised: A i .

Original: Wedenote by a := ( a 1 , · · · , a n ) the joint actions of all n players, and by A :
Revised: Wedenote by a := ( a 1 , · · · , a n ) the joint actions of all n players, and by A :

Original: = A 1 × . . .
Revised: = A 1 × . . .

Original: ×
Revised: ×

Original: A n the jointaction space with |A| =
Revised: A n the jointaction space with |A| =

Original: A = (cid:81)
Revised: A = (cid:81)

Original: i A i .
Revised: i A i .

Original: P =
Revised: T =

Original: { P h } h ∈ [ H ] is the collection of transition matrices, so that 93 P h ( ·| s, a ) ∈ ∆ S gives the distribution of the next state if joint actions a are taken at state s at steph .
Revised: { T h } h ∈ [ H ] is the collection of transition matrices, so that 93 T h ( ·| s, a ) ∈ ∆ S gives the distribution of the next state if joint actions a are taken at state s at steph .

Original: µ 1 denotes the distribution of the initial state s 1 .
Revised: µ 1 denotes the distribution of the initial state s 1 .

Original: O i denotes the observation space for the i thplayer with |O i
Revised: O i denotes the observation space for the i thplayer with |O i

Original: |
Revised: |

Original: =
Revised: =

Original: O i .
Revised: O i .

Original: We denote by o :=
Revised: We denote by o :=

Original: ( o 1 , . . .
Revised: ( o 1 , . . .

Original: o n ) the joint observations of all n players, andby O :
Revised: o n ) the joint observations of all n players, andby O :

Original: =
Revised: =

Original: O 1 × . . .
Revised: O 1 × . . .

Original: × O n with |O| =
Revised: × O n with |O| =

Original: O = (cid:81)
Revised: O = (cid:81)

Original: i O i .
Revised: i O i .

Original: O =
Revised: O =

Original: { O h } h ∈ [ H ] ⊆
Revised: { O h } h ∈ [ H ] ⊆

Original: R
Revised: R

Original: O × S is the collection of jointemission matrices, so that O h ( ·| s ) ∈
Revised: O × S is the collection of jointemission matrices, so that O h ( ·| s ) ∈

Original: ∆ O gives the emission distribution over the joint observationspace O at state s and step h .
Revised: ∆ O gives the emission distribution over the joint observationspace O at state s and step h .

Original: Finally r i =
Revised: Finally r i =

Original: { r i,h } h ∈
Revised: { r i,h } h ∈

Original: [ H ] is the collection of known reward functionsfor the i th player, so that r i,h ( o i ) ∈ [0 , 1] gives the deterministic reward received by the i th player ifshe observes o i at step h .
Revised: [ H ] is the collection of known reward functionsfor the i th player, so that r i,h ( o i ) ∈ [0 , 1] gives the deterministic reward received by the i th player ifshe observes o i at step h .

Original: 3 We remark that since the relation among the rewards of different playerscan be arbitrary, this model of POMGs subsumes both the cooperative and the competitive settings inpartially observable MARL.
Revised: 3 We remark that since the relation among the rewards of different playerscan be arbitrary, this model of POMGs subsumes both the cooperative and the competitive settings inpartially observable MARL.

Original: In a POMG, the states are always hidden from all players, and each player only observes her ownindividual observations and actions .
Revised: In a POMG, the states are always hidden from all players, and each player only observes her ownindividual observations and actions .

Original: That is, each player can not see the observations and actions ofthe other players.
Revised: That is, each player can not see the observations and actions ofthe other players.

Original: At the beginning of each episode, the environment samlpes s 1 from µ 1 .
Revised: At the beginning of each episode, the environment samlpes s 1 from µ 1 .

Original: At each steph ∈ [ H ] , each player i observes her own observation
Revised: At each steph ∈ [ H ] , each player i observes her own observation

Original: o i,h where o h :=
Revised: o i,h where o h :=

Original: ( o 1 ,h , . . .
Revised: ( o 1 ,h , . . .

Original: , o n,h ) are jointlysampled from O h ( · | s h ) .
Revised: , o n,h ) are jointlysampled from O h ( · | s h ) .

Original: Then each player i receives reward r i,h
Revised: Then each player i receives reward r i,h

Original: ( o i,h ) and picks action a i,h ∈
Revised: ( o i,h ) and picks action a i,h ∈

Original: A isimultaneously.
Revised: A isimultaneously.

Original: After that the environment transitions to the next state s h +1 ∼ P h ( ·| s h , a h ) wherea h :=
Revised: After that the environment transitions to the next state s h +1 ∼ T h ( ·| s h , a h ) wherea h :=

Original: ( a 1 ,h , . .
Revised: ( a 1 ,h , . .

Original: . , a n,h ) .
Revised: . , a n,h ) .

Original: The current episode terminates immediately once s H +1 is reached.
Revised: The current episode terminates immediately once s H +1 is reached.

Original: Policy, value function To deﬁne different types of polices, we extend the conventions in fullyobservable Markov games [29] to the partially observable settings.
Revised: Policy, value function To deﬁne different types of polices, we extend the conventions in fullyobservable Markov games [19] to the partially observable settings.

Original: A ( random ) policy π i of the i thplayer is a map
Revised: A ( random ) policy π i of the i thplayer is a map

Original: π i : Ω × (cid:83)
Revised: π i : Ω × (cid:83)

Original: Hh =1 (cid:0) ( O i × A i )
Revised: Hh =1 (cid:0) ( O i × A i )

Original: h − 1 ×
Revised: h − 1 ×

Original: O i (cid:1)
Revised: O i (cid:1)

Original: → A i , which maps a random seed ω fromspace Ω and a history of length h ∈
Revised: → A i , which maps a random seed ω fromspace Ω and a history of length h ∈

Original: [ H ] —say τ
Revised: [ H ] —say τ

Original: i,h :=
Revised: i,h :=

Original: ( o i, 1 , a i, 1 , · · · , o i,h ) , to an action in A i .
Revised: ( o i, 1 , a i, 1 , · · · , o i,h ) , to an action in A i .

Original: Toexecute policy π
Revised: Toexecute policy π

Original: i , we ﬁrst draw a random sample ω at the beginning of the episode.
Revised: i , we ﬁrst draw a random sample ω at the beginning of the episode.

Original: Then, at each steph , the i th player simply takes action π i ( ω, τ i,h ) .
Revised: Then, at each steph , the i th player simply takes action π i ( ω, τ i,h ) .

Original: We note here ω is shared among all steps
Revised: We note here ω is shared among all steps

Original: h ∈
Revised: h ∈

Original: [ H ] .ω encodes both the correlation among steps and the individual randomness of each step.
Revised: [ H ] .ω encodes both the correlation among steps and the individual randomness of each step.

Original: We furthersay a policy π i is deterministic if π i ( ω, τ i,h ) =
Revised: We furthersay a policy π i is deterministic if π i ( ω, τ i,h ) =

Original: π i ( τ i,h ) which is independent of the choice of ω .
Revised: π i ( τ i,h ) which is independent of the choice of ω .

Original: By deﬁnition, a random policy is equivalent to a mixture of deterministic policies because givena ﬁxed ω the decision of π i on any history is deterministic.
Revised: By deﬁnition, a random policy is equivalent to a mixture of deterministic policies because givena ﬁxed ω the decision of π i on any history is deterministic.

Original: With slight abuse of notation, we useπ i ( ω, · ) to refer to the deterministic policy realized by policy π i and a ﬁxed ω .
Revised: With slight abuse of notation, we useπ i ( ω, · ) to refer to the deterministic policy realized by policy π i and a ﬁxed ω .

Original: We denote the set ofall policies of the i th player by Π i and the set of all deterministic ones by Π det i .
Revised: We denote the set ofall policies of the i th player by Π i and the set of all deterministic ones by Π det i .

Original: A joint (potentially correlated) policy is a set of policies { π i } ni =1 , where the same random seedω is shared among all agents, which we denote as π = π 1 (cid:12) π 2 (cid:12) . . .
Revised: A joint (potentially correlated) policy is a set of policies { π i } ni =1 , where the same random seedω is shared among all agents, which we denote as π = π 1 (cid:12) π 2 (cid:12) . . .

Original: (cid:12) π n .
Revised: (cid:12) π n .

Original: We also denoteπ − i =
Revised: We also denoteπ − i =

Original: π 1 (cid:12) . . .
Revised: π 1 (cid:12) . . .

Original: π i − 1 (cid:12) π i +1 (cid:12) . . .
Revised: π i − 1 (cid:12) π i +1 (cid:12) . . .

Original: (cid:12) π n to be the joint policy excluding the i th player.
Revised: (cid:12) π n to be the joint policy excluding the i th player.

Original: A specialcase of joint policy is the product policy where the random seed has special form ω
Revised: A specialcase of joint policy is the product policy where the random seed has special form ω

Original: = ( ω 1 , . .
Revised: = ( ω 1 , . .

Original: . , ω n ) ,and for any i ∈ [ n ]
Revised: . , ω n ) ,and for any i ∈ [ n ]

Original: , π i only uses the randomness in ω i , which is independent of remaining { ω j } j (cid:54)
Revised: , π i only uses the randomness in ω i , which is independent of remaining { ω j } j (cid:54)

Original: =
Revised: =

Original: i ,which we denote as π = π 1 × π 2 × . . .
Revised: i ,which we denote as π = π 1 × π 2 × . . .

Original: × π n .
Revised: × π n .

Original: We deﬁne the value function V πi as the expected cumulative reward that the i th player will receive ifall players follow joint policy π :
Revised: We deﬁne the value function V πi as the expected cumulative reward that the i th player will receive ifall players follow joint policy π :

Original: [Equation]
Revised: [Equation]

Original: (1) where the expectation is taken over the randomness in the initial state, the transitions, the emissions,and the random seed ω in policy π .
Revised: (1) where the expectation is taken over the randomness in the initial state, the transitions, the emissions,and the random seed ω in policy π .

Original: the observation.
Revised: the observation.

Original: Best response and strategy modiﬁcation For any strategy π − i , the best response of the i th playeris deﬁned as a policy of the i th player, which is independent of the randomness in π − i and achievesthe highest value for herself conditioned on all other players deploying π − i .
Revised: Best response and strategy modiﬁcation For any strategy π − i , the best response of the i th playeris deﬁned as a policy of the i th player, which is independent of the randomness in π − i and achievesthe highest value for herself conditioned on all other players deploying π − i .

Original: Formally, the bestresponse is the maximizer of max π (cid:48)
Revised: Formally, the bestresponse is the maximizer of max π (cid:48)

Original: i V π (cid:48)
Revised: i V π (cid:48)

Original: i × π − i i whose value we also denote as V † ,π − i i for simplicity.
Revised: i × π − i i whose value we also denote as V † ,π − i i for simplicity.

Original: By its deﬁnition, we know the best response can always be achieved by deterministic policies.
Revised: By its deﬁnition, we know the best response can always be achieved by deterministic policies.

Original: A strategy modiﬁcation for the i th player is a map
Revised: A strategy modiﬁcation for the i th player is a map

Original: φ
Revised: φ

Original: i : Π det i → Π det i , which maps a deterministicpolicy in Π det i to another one in it.
Revised: i : Π det i → Π det i , which maps a deterministicpolicy in Π det i to another one in it.

Original: For any such strategy modiﬁcation φ
Revised: For any such strategy modiﬁcation φ

Original: i , we can naturally extend itsdomain and image to include random policies, i.e., deﬁne its extension φ i : Π i
Revised: i , we can naturally extend itsdomain and image to include random policies, i.e., deﬁne its extension φ i : Π i

Original: → Π i as follows: bydeﬁnition, a random policy π can be expressed as a mixture of deterministic policies, i.e., as π ( ω, · )(a deterministic policy for a ﬁxed ω ) with a distribution over ω .
Revised: → Π i as follows: bydeﬁnition, a random policy π can be expressed as a mixture of deterministic policies, i.e., as π ( ω, · )(a deterministic policy for a ﬁxed ω ) with a distribution over ω .

Original: Then if we apply map φ i on randompolicy π , we can deﬁne the resulting random policy (denoted as φ i (cid:5) π i ) as φ
Revised: Then if we apply map φ i on randompolicy π , we can deﬁne the resulting random policy (denoted as φ i (cid:5) π i ) as φ

Original: i ( π i ( ω, · ))
Revised: i ( π i ( ω, · ))

Original: (again adeterministic policy for a ﬁxed ω ) with the same distribution over ω .
Revised: (again adeterministic policy for a ﬁxed ω ) with the same distribution over ω .

Original: For any joint policy π , we deﬁnethe best strategy modiﬁcation of the i th player as the maximizer of max φ i V ( φ i (cid:5) π i ) (cid:12) π − i i .
Revised: For any joint policy π , we deﬁnethe best strategy modiﬁcation of the i th player as the maximizer of max φ i V ( φ i (cid:5) π i ) (cid:12) π − i i .

Original: Different from the best response, which is completely independent of the randomness in π − i , the beststrategy modiﬁcation changes the policy of the i th player while still utilizing the shared randomnessamong π i and π − i .
Revised: Different from the best response, which is completely independent of the randomness in π − i , the beststrategy modiﬁcation changes the policy of the i th player while still utilizing the shared randomnessamong π i and π − i .

Original: Therefore, the best strategy modiﬁcation is more powerful than the best response:formally one can show that max φ i V ( φ i (cid:5)
Revised: Therefore, the best strategy modiﬁcation is more powerful than the best response:formally one can show that max φ i V ( φ i (cid:5)

Original: [Equation]
Revised: [Equation]

Original: for any policy π .
Revised: for any policy π .

Original: 2.1 Learning objectives
Revised: 2.1 Learning objectives

Original: We focus on three classic equilibrium concepts in game theory—Nash Equilibrium, Correlated
Revised: We focus on three classic equilibrium concepts in game theory—Nash Equilibrium, Correlated

Original: Equilibrium (CE) and Coarse Correlated Equilibrium (CCE).
Revised: Equilibrium (CE) and Coarse Correlated Equilibrium (CCE).

Original: First, a Nash equilibrium is deﬁned as aproduct policy in which no player can increase her value by changing only her own policy.
Revised: First, a Nash equilibrium is deﬁned as aproduct policy in which no player can increase her value by changing only her own policy.

Original: Formally,
Revised: Formally,

Original: Deﬁnition 1 (Nash Equilibrium) 
Revised: Deﬁnition 1 (Nash Equilibrium) .

Original: A product policy π is a Nash equilibrium if V † ,π − i
Revised: A product policy π is a Nash equilibrium if V † ,π − i

Original: i =
Revised: i =

Original: V πi for alli ∈ [ n ] .
Revised: V πi for alli ∈ [ n ] .

Original: A product policy π is an (cid:15) -approximate
Revised: A product policy π is an (cid:15) -approximate

Original: Nash equilibrium if V † ,π − i i [Equation] for all i ∈ [ n ] .
Revised: Nash equilibrium if V † ,π − i i ≤ V πi + (cid:15) for all i ∈ [ n ] .

Original: The Nash-regret of a sequence of product policies is the cumulative violation of the Nash condition.
Revised: The Nash-regret of a sequence of product policies is the cumulative violation of the Nash condition.

Original: Deﬁnition 2 (Nash-regret)  Let π k denote the (product) policy deployed by an algorithm in the k th episode.
Revised: Deﬁnition 2 (Nash-regret) . Let π k denote the (product) policy deployed by an algorithm in the k th episode.

Original: After a total of K episodes, the Nash-regret is deﬁned as
Revised: After a total of K episodes, the Nash-regret is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Second, a coarse correlated equilibrium is deﬁned as a joint (potentially correlated) policy where noplayer can increase her value by unilaterally changing her own policy.
Revised: Second, a coarse correlated equilibrium is deﬁned as a joint (potentially correlated) policy where noplayer can increase her value by unilaterally changing her own policy.

Original: Formally,
Revised: Formally,

Original: Deﬁnition 3 (Coarse Correlated Equilibrium) 
Revised: Deﬁnition 3 (Coarse Correlated Equilibrium) .

Original: A joint policy π is a CCE if V † ,π − i i ≤ V πi for alli ∈ [ n ] .
Revised: A joint policy π is a CCE if V † ,π − [Equation] i ≤ V πi for alli ∈ [ n ] .

Original: A joint policy π is an (cid:15) -approximate
Revised: A joint policy π is an (cid:15) -approximate

Original: CCE if V † ,π − i i ≤ V πi + (cid:15) for all i ∈ [ n ] .
Revised: CCE if V † ,π − i i ≤ V πi + (cid:15) for all i ∈ [ n ] .

Original: The only difference between Deﬁnition 1 and Deﬁnition 3 is that a Nash equilibrium has to be aproduct policy while a CCE can be correlated.
Revised: The only difference between Deﬁnition 1 and Deﬁnition 3 is that a Nash equilibrium has to be aproduct policy while a CCE can be correlated.

Original: Therefore, CCE is a relaxed notion of Nash equilibrium,and a Nash equilibrium is always a CCE.
Revised: Therefore, CCE is a relaxed notion of Nash equilibrium,and a Nash equilibrium is always a CCE.

Original: Similarly, we can deﬁne the CCE-regret for a sequence ofpotentially correlated policies as the cumulative vilolation of the CCE condition.
Revised: Similarly, we can deﬁne the CCE-regret for a sequence ofpotentially correlated policies as the cumulative vilolation of the CCE condition.

Original: Deﬁnition 4 (CCE-regret)  Let π k denote the policy deployed by an algorithm in the k th episode.
Revised: Deﬁnition 4 (CCE-regret) . Let π k denote the policy deployed by an algorithm in the k th episode.

Original: After a total of K episodes, the CCE-regret is deﬁned as
Revised: After a total of K episodes, the CCE-regret is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Finally, a correlated equilibrium is deﬁned as a joint (potentially correlated) policy where no playercan increase her value by unilaterally applying any strategy modiﬁcation.
Revised: Finally, a correlated equilibrium is deﬁned as a joint (potentially correlated) policy where no playercan increase her value by unilaterally applying any strategy modiﬁcation.

Original: Formally,
Revised: Formally,

Original: Deﬁnition 5 (Correlated Equilibrium) 
Revised: Deﬁnition 5 (Correlated Equilibrium) .

Original: A joint policy π is a CE if max φ i V ( φ i (cid:5) π i ) (cid:12) π − i
Revised: A joint policy π is a CE if max φ i V ( φ i (cid:5) π i ) (cid:12) π − i

Original: i =
Revised: i =

Original: V πi forall i ∈ [ n ] .
Revised: V πi for alli ∈ [ n ] .

Original: A joint policy π is an (cid:15) -approximate CE
Revised: A joint policy π is an (cid:15) -approximate CE

Original: if max φ i V ( φ i (cid:5) π i ) (cid:12) π − i i ≤ V πi + (cid:15) for all i ∈ [ m ] .
Revised: if max φ i V ( φ i (cid:5) π i ) (cid:12) π − i i ≤ V πi + (cid:15) for all i ∈ [ m ] .

Original: In Partially Observable Markov games, we always have that a Nash equilibrium is a CE, and a CE isa CCE.
Revised: In Partially Observable Markov games, we always have that a Nash equilibrium is a CE, and a CE isa CCE.

Original: Finally, we deﬁne the CE-regret to be the cumulative violation of the CE condition.
Revised: Finally, we deﬁne the CE-regret to be the cumulative violation of the CE condition.

Original: Deﬁnition 6 (CE-regret)  Let π k denote the policy deployed by an algorithm in the k th episode.
Revised: Deﬁnition 6 (CE-regret) . Let π k denote the policy deployed by an algorithm in the k th episode.

Original: After a total of K episodes, the CE-regret is deﬁned as
Revised: After a total of K episodes, the CE-regret is deﬁned as

Original: [Equation]
Revised: [Equation]

Original:  Weakly Revealing Partially Observable Markov Games
Revised:  Weakly Revealing Partially Observable Markov Games

Original: In this section, we deﬁne the class of weakly revealing POMGs.
Revised: In this section, we deﬁne the class of weakly revealing POMGs.

Original: To begin with, we considerundercomplete POMGs where there are more observations than hidden states, i.e., O ≥ S .
Revised: To begin with, we considerundercomplete POMGs where there are more observations than hidden states, i.e., O ≥ S .

Original: Formally,the family of α -weakly revealing POMGs includes all POMGs, in which the S th singular value ofeach emission matrix O h is lower bounded by α > 0 .
Revised: Formally,the family of α -weakly revealing POMGs includes all POMGs, in which the S th singular value ofeach emission matrix O h is lower bounded by α > 0 .

Original: Assumption 1 ( α -weakly revealing condition) 
Revised: Assumption 1 ( α -weakly revealing condition) .

Original: There exists α > 0 , such that min h σ S ( O h ) ≥ α .
Revised: There exists α > 0 , such that min h σ S ( O h ) ≥ α .

Original: Assumption 1 is simply a robust version of the condition that the rank of each emission matrix is
Revised: Assumption 1 is simply a robust version of the condition that the rank of each emission matrix is

Original: S , which guarantees that no two different latent state mixtures can generate the same observationdistribution, i.e., O h ν 1 (cid:54) =
Revised: S , which guarantees that no two different latent state mixtures can generate the same observationdistribution, i.e., O h ν 1 (cid:54) =

Original: O h ν 2 for any different ν 1 , ν 2 ∈ ∆ S .
Revised: O h ν 2 for any different ν 1 , ν 2 ∈ ∆ S .

Original: Intuitively, this guarantees that thejoint observations of all agents contain sufﬁcient information to distinguish any two different statemixtures.
Revised: Intuitively, this guarantees that thejoint observations of all agents contain sufﬁcient information to distinguish any two different statemixtures.

Original: We remark that this is much weaker than requiring the individual observations of each agentcontain sufﬁcient information about the latent states.
Revised: We remark that this is much weaker than requiring the individual observations of each agentcontain sufﬁcient information about the latent states.

Original: The weakly revealing condition is importantin excluding those pathological POMGs where the observations contain no useful information foridentifying the key parts of model dynamics.
Revised: The weakly revealing condition is importantin excluding those pathological POMGs where the observations contain no useful information foridentifying the key parts of model dynamics.

Original: Note that Assumption 1 never holds in the overcomplete setting ( S > O ) as it is impossible todistinguish any two latent state mixtures by only inspecting the observation distribution in a singlestep.
Revised: Note that Assumption 1 never holds in the overcomplete setting ( S > O ) as it is impossible todistinguish any two latent state mixtures by only inspecting the observation distribution in a singlestep.

Original: To address this issue, we can instead inspect the observations for m -consecutive steps.
Revised: To address this issue, we can instead inspect the observations for m -consecutive steps.

Original: Toproceed, we deﬁne the m -step emission-action matrices
Revised: Toproceed, we deﬁne the m -step emission-action matrices

Original: [Equation]
Revised: [Equation]

Original: as follows:
Revised: as follows:

Original: Given an observation sequence ¯ o of length m , initial state s and action sequence ¯ a oflength m − 1 , we let [ M h ] (¯ a , ¯ o ) ,s be the probability of receiving ¯ o provided that the action sequence ¯ ais used from state s and step h :
Revised: Given an observation sequence ¯ o of length m , initial state s and action sequence ¯ a oflength m − 1 , we let [ M h ] (¯ a , ¯ o ) ,s be the probability of receiving ¯ o provided that the action sequence ¯ ais used from state s and step h :

Original: [Equation]
Revised: [Equation]

Original: Similar to the undercomplete setting, the weakly-revealing condition in the over-complete settingsimply assumes the S th singular value of each m -step emission-action matrix is lower bounded.
Revised: Similar to the undercomplete setting, the weakly-revealing condition in the over-complete settingsimply assumes the S th singular value of each m -step emission-action matrix is lower bounded.

Original: Assumption 2 (multistep α -weakly revealing condition) 
Revised: Assumption 2 (multistep α -weakly revealing condition) .

Original: There exists m ∈ N , α > 0 such thatmin h σ S ( M h ) ≥ α where M h is the m -step emission matrix deﬁned in (2) .
Revised: There exists m ∈ N , α > 0 such thatmin h σ S ( M h ) ≥ α where M h is the m -step emission matrix deﬁned in (2) .

Original: Assumption 2 ensures that m -step consecutive observations shall contain sufﬁcient information todistinguish any two different latent state mixtures.
Revised: Assumption 2 ensures that m -step consecutive observations shall contain sufﬁcient information todistinguish any two different latent state mixtures.

Original: Note that Assumption 1 is a special case of
Revised: Note that Assumption 1 is a special case of

Original: Assumption 2 with m = 1 .
Revised: Assumption 2 with m = 1 .

Original: Finally, we remark that the single-agent versions of Assumption 1 and 2were ﬁrst identiﬁed in [27] and [37] as sufﬁcient conditions for sample-efﬁcient learning of single-stepand multi-step weakly revealing POMDPs (the single-agent version of POMGs), respectively.
Revised: Finally, we remark that the single-agent versions of Assumption 1 and 2were ﬁrst identiﬁed in [17] and [24] as sufﬁcient conditions for sample-efﬁcient learning of single-stepand multi-step weakly revealing POMDPs (the single-agent version of POMGs), respectively.

Original:  Learning Equilibria with Self-play
Revised:  Learning Equilibria with Self-play

Original: In this section, we study the self-play setting where the algorithm can control all the players tolearn the equilibria by playing against itself.
Revised: In this section, we study the self-play setting where the algorithm can control all the players tolearn the equilibria by playing against itself.

Original: We propose a new algorithm — Optimistic Maximum
Revised: We propose a new algorithm — Optimistic Maximum

Original: Likelihood Estimation for Learning Equilibria ( OMLE-Equilibrium ) that can provably ﬁnd Nashequilibria, coarse correlated equilibria and correlate equilibria in any weakly-revealing partiallyobservable Markov games using a number of samples polynomial in all relevant parameters.
Revised: Likelihood Estimation for Learning Equilibria ( OMLE-Equilibrium ) that can provably ﬁnd Nashequilibria, coarse correlated equilibria and correlate equilibria in any weakly-revealing partiallyobservable Markov games using a number of samples polynomial in all relevant parameters.

Original: 4.1 Undercomplete 
Revised: 4.1 Undercomplete partially observable Markov games

Original: partially

Original: We ﬁrst present the algorithm and results for learning undercomplete POMGs under Assumption 1.
Revised: We ﬁrst present the algorithm and results for learning undercomplete POMGs under Assumption 1.

Original: We will see in the later section that with a minor modiﬁcation the same algorithm also applies tolearning overcomplete POMGs under Assumption 2.Algorithm [Equation]
Revised: We will see in the later section that with a minor modiﬁcation the same algorithm also applies tolearning overcomplete POMGs under Assumption 2.

Original: Subroutine 1 Optimistic_Equilibrium ( B ) 1: i

Original: [ n ] do 2: let V i ∈ R | Π det 1 |×···×| Π det n | with its π th entry equal to sup ˆ θ ∈B V πi (ˆ θ ) for π ∈ Π 1 × · · · × Π n 3:

Original: QUILIBRIUM

Original: V n ) Algorithm description To condense notations, we will use θ =
Revised: Algorithm description To condense notations, we will use θ =

Original: ( T , O , µ 1 ) to denote the parametersof a POMG.
Revised: ( T , O , µ 1 ) to denote the parametersof a POMG.

Original: Given a policy π and a trajectory τ , we denote by V πi ( θ ) the i th player’s valueand by P πθ ( τ ) the probability of observing trajectory τ , both under policy π in the POMG modelparameterized by θ .
Revised: Given a policy π and a trajectory τ , we denote by V πi ( θ ) the i th player’s valueand by P πθ ( τ ) the probability of observing trajectory τ , both under policy π in the POMG modelparameterized by θ .

Original: We describe OMLE-Equilibrium in Algorithm 1.
Revised: We describe OMLE-Equilibrium in Algorithm 1.

Original: In each episode, the algorithmexecutes the following two key steps:
Revised: In each episode, the algorithmexecutes the following two key steps:

Original: • Optimistic equilibrium computation (Line 3)
Revised: [Equation] [Equation] [Equation] [Equation]

Revised: Subroutine 1 Optimistic_Equilibrium ( B ) 1: for i ∈ [ n ] do 2: let V i ∈ R | Π det 1 |×···×| Π det n | with its π th entry equal to sup ˆ θ ∈B V πi

Revised: (ˆ θ ) for π ∈ Π 1 × · · · × Π n 3: return

Revised: QUILIBRIUM

Revised: V

Revised: V n )

Revised: • Optimistic equilibrium computation (Line 3)

Original: We ﬁrst invoke Optimistic_Equilibrium(Subroutine 1) with conﬁdence set B k to compute a joint (potentially correlated) policy π k .Formally, subroutine Optimistic_Equilibrium ( B k ) consists of two components:
Revised: We ﬁrst invoke Optimistic_Equilibrium (Subroutine ) with conﬁdence set B k to compute a joint (potentially correlated) policy π k . Formally, subroutine Optimistic_Equilibrium ( B k ) consists of two components:

Original: – Optimisic value estimation (Line 1-2 of Subroutine 1) For each player i ∈ [ n ] anddeterministic joint policy π ∈ Π det 1 × · · · × Π det n , we compute an upper bound V πi for
Revised: – Optimisic value estimation (Line 1-2 of Subroutine 1) For each player i ∈ [ n ] anddeterministic joint policy π ∈ Π det 1 × · · · × Π det n , we compute an upper bound V πi forthe i

Original: the i th player’s value under policy π by using the most optimistic POMG model in theconﬁdence set B k .
Revised: th player’s value under policy π by using the most optimistic POMG model in theconﬁdence set B k .

Original: – Equilibira computation (Line 3 of Subroutine 1) Given the optimistic value estimatesfor all deterministic joint policies and all players, we can view the POMG as a normalform game where the i th player’s pure strategies consist of all her deterministic policies(i.e.
Revised: – Equilibira computation (Line 3 of Subroutine 1) Given the optimistic value estimatesfor all deterministic joint policies and all players, we can view the POMG as a normal- form game where the i th player’s pure strategies consist of all her deterministic policies

Original: , Π det i ) and the payoff she receives under a joint deterministic policy π ∈ Π det 1 ×· · ·× Π det n is equal to the corresponding optimistic value estimate V πi .
Revised: (i.e., Π det i ) and the payoff she receives under a joint deterministic policy π ∈ Π det 1 ×· · ·×Π det n is equal to the corresponding optimistic value estimate V πi .

Original: Then we compute a E QUILIBRIUM π k for this normal-form game, which is a mixture of all the deterministic joint policies in Π det 1 × · · · × Π det n .
Revised: Then we compute aE QUILIBRIUM π k for this normal-form game, which is a mixture of all the deterministicjoint policies in Π det 1 × · · · × Π det n .

Original: • Conﬁdence set update (Line 4-5)
Revised: • Conﬁdence set update (Line 4-5)

Original: We ﬁrst follow π k to collect a trajectory, and then utilize
Revised: We ﬁrst follow π k to collect a trajectory, and then utilize

Original: the newly collected data to update the model conﬁdence set via MLE principle.
Revised: the newly collected data to update the model conﬁdence set via MLE principle.

Original: Here we highlight two algorithmic designs in OMLE-Equilibrium : the ﬂexibility of equilibriumcomputation and the MLE conﬁdence set construction.
Revised: Here we highlight two algorithmic designs in OMLE-Equilibrium : the ﬂexibility of equilibrium computation and the MLE conﬁdence set construction.

Original: In the step of equilibrium computation, wecan choose E QUILIBRIUM to be Nash equilibrium or correlated equilibrium (CE) or corase correlatedequilibrium (CCE) of the normal-form game depending on the target type of equilibrium we aim tolearn for the POMG.
Revised: In the step of equilibrium computation, we can choose E QUILIBRIUM to be Nash equilibrium or correlated equilibrium (CE) or corase correlated equilibrium (CCE) of the normal-form game depending on the target type of equilibrium we aim to learn for the POMG.

Revised: With regard to the conﬁdence

Revised: set

Revised: design,

Original: With regard to the conﬁdence set design, we adopt the idea from [37] to includeall the POMG models whose likelihood on the historical data is close to the maximum likelihood.
Revised: we adopt the idea from [ ] to include all the POMG models whose likelihood on the historical data is close to the maximum likelihood.

Revised: This can be

Revised: viewed

Original: This can be viewed as a relaxation of the classic MLE method, with the degree of relaxation controlledby parameter β .
Revised: as a relaxation of the classic MLE method , with the degree of relaxation controlledby parameter β .

Original: One important beneﬁt of this relaxation is that although the groundtruth POMGmodel is in general not a solution of MLE, its likelihood ratio is rather close to the maximal likelihood.
Revised: One important beneﬁt of this relaxation is that although the groundtruth POMGmodel is in general not a solution of MLE, its likelihood ratio is rather close to the maximal likelihood.

Original: By doing so, we can guarantee the true model is included in the conﬁdence set with high probability.
Revised: By doing so, we can guarantee the true model is included in the conﬁdence set with high probability.

Original: Finally, we remark that Algorithm 1 is computationally inefﬁcient in general due to the steps ofoptimistic value estimation and equilibrium computation.
Revised: Finally, we remark that Algorithm 1 is computationally inefﬁcient in general due to the steps ofoptimistic value estimation and equilibrium computation.

Original: Theoretical guarantees Below we present the main theorem for OMLE-Equilibrium .
Revised: Theoretical guarantees Below we present the main theorem for OMLE-Equilibrium .

Revised: Theorem 7.

Original: 1: Initialize: B 1  = { ˆ θ ∈ Θ : min h σ S ( ˆ M h ) ≥ α } , D = {} 2: for k = 1 , . . .
Revised: (Regret of OMLE-Equilibrium ) Under Assumption 1, there exists an absolute constant c such that for any δ ∈ (0 , 1] and K ∈ N , Algorithm 1 with β = c (cid:0) ( S 2 A + SO ) log( SAOHK ) + log( K/δ ) (cid:1) and

Revised: E QUILIBRIUM being one of { Nash, CCE, CE } satisﬁes (respectively) that with probability at least 1 − δ ,

Revised: Regret { Nash , CCE , CE } ( k ) ≤ poly( S, A, O, H, α − 1 , log( Kδ − 1 )) · √ k for all k ∈

Revised: [ K ] .

Revised: 1: Initialize: B 1 = { ˆ θ ∈ Θ : min h σ S ( ˆ M h ) ≥ α } , D = {} 2: for k = 1 , . . .

Original: , K do 3: compute π k = Optimistic_Equilibrium 
Revised: , K do 3: compute π k = Optimistic_Equilibrium (

Original: ( B k ) 4: for h = 0 ,. .
Revised: ) 4: for h = 0 ,

Original:  , H − m do 5: execute policy π k 1: h ◦ uniform ( A ) to collect a trajectory τ k,h then add ( π k 1: h ◦ uniform ( A ) , τ k,h ) into D 6: update B k +1 = (cid:26) ˆ θ ∈ Θ : (cid:88) ( π,τ ) ∈D log P
Revised: . , H − m do 5: execute policy π k 1: h ◦ uniform ( A ) to collect a trajectory τ k,h then add ( π k 1: h ◦ uniform ( A ) , τ k,h ) into D 6: update B k +1 = (cid:26) ˆ θ ∈ Θ : (cid:88) ( π,τ ) ∈D log P

Original: π ˆ θ ( τ ) ≥ max θ (cid:48) ∈ Θ (cid:88)
Revised: π ˆ θ ( τ ) ≥ max θ (cid:48) ∈ Θ (cid:88)

Original: ( π,τ ) ∈D log P πθ (cid:48)
Revised: ( π,τ ) ∈D log P πθ (cid:48)

Original: ( τ ) − β (cid:27) (cid:92) B7: K ]
Revised: ( τ ) − β (cid:27) (cid:92) B

Original: Theorem 7 (Regret of OMLE-Equilibrium )

Original: Under Assumption 1, there exists an absolute constant c such that for any δ ∈ (0 , 1] and K ∈

Original: N , Algorithm 1 with β = c (cid:0) ( S 2 A + SO ) log( SAOHK ) + log( K/δ ) (cid:1) and E QUILIBRIUM being one of { Nash, CCE, CE } satisﬁes (respectively) that with probability at least 1 − δ ,

Original: Regret { Nash , CCE , CE } ( k ) ≤ poly( S, A, O, H, α − 1 , log( Kδ − 1 )) · √ k for all k ∈

Original: [ K ] .

Original: Theorem 7 claims that if all players follow OMLE-Equilibrium , then the cumulative {Nash,CCE,CE}-regret is upper bounded by ˜ O ( √ k ) for any weakly-revealing POMGs that satisfy Assumption 1,where the growth rate w.r.t k is optimal.
Revised: Theorem 7 claims that if all players follow OMLE-Equilibrium , then the cumulative {Nash,CCE,CE}-regret is upper bounded by ˜ O ( √ k ) for any weakly-revealing POMGs that satisfy Assumption 1,where the growth rate w.r.t k is optimal.

Original: By the standard online-to-batch conversion, it directlyimplies the following sample complexity result:
Revised: By the standard online-to-batch conversion, it directlyimplies the following sample complexity result:

Original: Corollary 8 (Sample Complexity of OMLE-Equilibrium )
Revised: Corollary 8. (Sample Complexity of OMLE-Equilibrium )

Original: Under the same setting as Theorem 7,when K ≥ poly( S, A
Revised: Under the same setting as Theorem 7,when K ≥ poly( S, A

Original: O, H, α − 1 , log( (cid:15) − 1 δ − 1 )) · (cid:15) − 2 , then with probability at least 1 / 2 , π out is an(cid:15) - { Nash, CCE, CE } policy.
Revised: O, H, α − 1 , log( (cid:15) − 1 δ − 1 )) · (cid:15) − 2 , then with probability at least 1 / 2 , π out is an(cid:15) - { Nash, CCE, CE } policy.

Original: Here the dependence on the precision parameter (cid:15) is optimal.
Revised: Here the dependence on the precision parameter (cid:15) is optimal.

Original: Finally, notice that the upper boundin Theorem 7 depends polynomially on the inverse of α — a lower bound for the minimal singularvalue of the joint emission matrix O h in Assumption 1.
Revised: Finally, notice that the upper boundin Theorem 7 depends polynomially on the inverse of α — a lower bound for the minimal singularvalue of the joint emission matrix O h in Assumption 1.

Original: This dependence is shown to be unavoidableeven in the single-player setting (POMDPs) [37].
Revised: This dependence is shown to be unavoidableeven in the single-player setting (POMDPs) [24].

Original: 4.2 Overcomplete partially observable Markov games
Revised: 4.2 Overcomplete partially observable Markov games

Original: In this subsection, we extend OMLE-Equilibrium to the more challenging setting of learning overcomplete POMGs, where there can be much less observations than latent states.
Revised: In this subsection, we extend OMLE-Equilibrium to the more challenging setting of learning overcomplete POMGs, where there can be much less observations than latent states.

Original: We prove thata simple variant of OMLE-Equilibrium still enjoys polynomial sample-efﬁciency guarantee forlearning any multi-step weakly revealing POMGs.
Revised: We prove thata simple variant of OMLE-Equilibrium still enjoys polynomial sample-efﬁciency guarantee forlearning any multi-step weakly revealing POMGs.

Original: Algorithm description We describe the multi-step generalization of OMLE-Equilibrium in Algorithm 2, which inherits the key designs from Algorithm 1 and additionally makes two importantmodiﬁcations to address the challenge of insufﬁcient information from single-step observation.
Revised: Algorithm description We describe the multi-step generalization of OMLE-Equilibrium in Algorithm 2, which inherits the key designs from Algorithm 1 and additionally makes two importantmodiﬁcations to address the challenge of insufﬁcient information from single-step observation.

Original: Theﬁrst change is to utilize a more active sampling strategy for exploration.
Revised: Theﬁrst change is to utilize a more active sampling strategy for exploration.

Original: Instead of simply followingthe optimistic policy π k , we will iteratively execute H − m + 1 policies of form π k 1: h ◦ uniform ( A )where the players ﬁrst follow policy π k from step 1 to step h , then pick actions uniformly at random to ﬁnish the remaining H − h steps.
Revised: Instead of simply followingthe optimistic policy π k , we will iteratively execute H − m + 1 policies of form π k 1: h ◦ uniform ( A )where the players ﬁrst follow policy π k from step 1 to step h , then pick actions uniformly at random to ﬁnish the remaining H − h steps.

Original: Intuitively, by actively trying random action sequencesafter executing policy π k , the algorithm can acquire more information about the system dynamicscorresponding to those latent states that are frequently visited by π k , and therefore help address thechallenge of lacking sufﬁcient information from single-step observation.
Revised: Intuitively, by actively trying random action sequencesafter executing policy π k , the algorithm can acquire more information about the system dynamicscorresponding to those latent states that are frequently visited by π k , and therefore help address thechallenge of lacking sufﬁcient information from single-step observation.

Original: The second change madeby Algorithm 2 is that in constructing the conﬁdence set, we require the minimal singular value ofthe multistep emission matrix to be lower bounded, which enforces the multistep weakly revealingcondition in Assumption 2.
Revised: The second change madeby Algorithm 2 is that in constructing the conﬁdence set, we require the minimal singular value ofthe multistep emission matrix to be lower bounded, which enforces the multistep weakly revealingcondition in Assumption 2.

Original: Theoretical guarantee Below we present the main theorem for multi-step OMLE-Equilibrium .
Revised: Theoretical guarantee Below we present the main theorem for multi-step OMLE-Equilibrium .

Original: Theorem 9 (Total suboptimality of multi-step OMLE-Equilibrium )
Revised: Theorem 9. (Total suboptimality of multi-step OMLE-Equilibrium )

Original: Under Assumption 2, there exists an absolute constant c such that for any δ ∈ (0 , 1] and K ∈ N , Algorithm 1 with β = c (cid:0) ( S 2 A + SO ) log( SAOHK )
Revised: Under Assumption 2, there exists an absolute constant c such that for any δ ∈ (0 , 1] and K ∈ N , Algorithm 1 with β = c (cid:0) ( S 2 A + SO ) log( SAOHK )

Original: + log( K/δ ) (cid:1)
Revised: + log( K/δ ) (cid:1)

Original: and
Revised: and

Original: E QUILIBRIUM being one of { Nash, CCE, CE } satisﬁes (respectively) that with probability at least 1 − δ ,
Revised: E QUILIBRIUM being one of { Nash, CCE, CE } satisﬁes (respectively) that with probability at least 1 − δ ,

Original: Regret { Nash , CCE , CE } ( k ) ≤ poly( S, A m , O, H, α − 1 , log( Kδ

Original: [Equation]
Revised: [Equation]

Original: 1 )) · √ k for all k ∈ [ K ] , where the regret is computed for policy π 1 , . ..
Revised: for all [Equation] ∈ [ K ] , where the regret is computed for policy π 1 , . .

Original: , π k .
Revised: π k .

Original: Theorem 9 claims that the total {Nash,CCE,CE}-“regret” (that are computed on policy π 1 , . .
Revised: Theorem 9 claims that the total {Nash,CCE,CE}-“regret” (that are computed on policy π 1 , . .

Original: . , π k )of multi-step OMLE-Equilibrium is upper bounded by ˜ O ( √ k ) for any multi-step weakly revealing
Revised: . , π k )of multi-step OMLE-Equilibrium is upper bounded by ˜ O ( √ k ) for any multi-step weakly revealing

Original: POMGs satisfying Assumption 2.
Revised: POMGs satisfying Assumption 2.

Original: We remark that, strictly speaking, Theorem 9 is not a standardregret guarantee since the policies executed by multi-step OMLE-Equilibrium are compositions ofπ 1 , . . .
Revised: We remark that, strictly speaking, Theorem 9 is not a standardregret guarantee since the policies executed by multi-step OMLE-Equilibrium are compositions ofπ 1 , . . .

Original: , π k and random actions, instead of purely π 1 , . . .
Revised: , π k and random actions, instead of purely π 1 , . . .

Original: , π k .
Revised: , π k .

Original: Nevertheless, we can still utilize thestandard online-to-batch conversion to obtain the following sample complexity guarantee:
Revised: Nevertheless, we can still utilize thestandard online-to-batch conversion to obtain the following sample complexity guarantee:

Original: Corollary 10 (sample complexity of multi-step OMLE-Equilibrium )
Revised: Corollary 10. (sample complexity of multi-step OMLE-Equilibrium )

Original: Under the same setting as
Revised: Under the same setting as

Original: Theorem 9, when K ≥ poly( S, A m , O, H, α − 1 , log( (cid:15) − 1 δ − 1 )) · (cid:15) − 2 , then with probability at least1 / 2 , π out is an (cid:15) - { Nash, CCE, CE } policy.
Revised: Theorem 9, when K ≥ poly( S, A m , O, H, α − 1 , log( (cid:15) − 1 δ − 1 )) · (cid:15) − 2 , then with probability at least1 / 2 , π out is an (cid:15) - { Nash, CCE, CE } policy.

Original: Here the dependence on the precision parameter (cid:15) is optimal up to poly-logarithmic factors.
Revised: Here the dependence on the precision parameter (cid:15) is optimal up to poly-logarithmic factors.

Original: Finally,observe that the sample complexity has A m dependency that is exponential in m , which is unavoidable in general even in the single-player setting (POMDPs) [37].
Revised: Finally,observe that the sample complexity has A m dependency that is exponential in m , which is unavoidable in general even in the single-player setting (POMDPs) [24].

Original: Nonetheless, in order to makemin h rank( M h ) =
Revised: Nonetheless, in order to makemin h rank( M h ) =

Original: S possible (Assumption 2), we only need to make ( OA ) m
Revised: S possible (Assumption 2), we only need to make ( OA ) m

Original: (cid:38) S , i.e., m (cid:38) log
Revised: (cid:38) S , i.e., m (cid:38) log

Original: Swhich is very small.
Revised: Swhich is very small.

Original: In this paper, when we claim the sample complexity is polynomial, we considerm to be small enough so that A m ≤ poly( S, A, O, H, α − 1 ) .
Revised: In this paper, when we claim the sample complexity is polynomial, we considerm to be small enough so that A m ≤ poly( S, A, O, H, α − 1 ) .

Original:  Playing against Adversarial Opponents
Revised:  Playing against Adversarial Opponents

Original: In this section, we turn to the online setting where the learner only controls a single player and theremaining players can execute arbitrary strategies.
Revised: In this section, we turn to the online setting where the learner only controls a single player and theremaining players can execute arbitrary strategies.

Original: In this setting, we no longer target at learninggame-theoretic equilibria because if other players keep playing some highly suboptimal policiesthen the learner may never be able to explore the environment thoroughly and thus lacks sufﬁcientinformation to compute equilibria.
Revised: In this setting, we no longer target at learninggame-theoretic equilibria because if other players keep playing some highly suboptimal policiesthen the learner may never be able to explore the environment thoroughly and thus lacks sufﬁcientinformation to compute equilibria.

Original: Instead, we consider the standard goal for online setting, whichis to achieve low regret in terms of cumulative rewards even if all other players play adversariallyagainst the learner.
Revised: Instead, we consider the standard goal for online setting, whichis to achieve low regret in terms of cumulative rewards even if all other players play adversariallyagainst the learner.

Original: In this section, we focus on the undercomplete setting.

Original: Without loss of generality,we assume the learner only controls the 1 st player throughout this section.
Revised: Without loss of generality, we assume the learner only controls the 1 st playerthroughout this section.

Original: 5.1 Statistical hardness for the standard setting
Revised: 5.1 Statistical hardness for the standard setting

Original: We ﬁrst consider the standard POMG setting where each player can only observe her own observationsand actions.
Revised: We ﬁrst consider the standard POMG setting where each player can only observe her own observationsand actions.

Original: We prove that achieving low regret in this setting is impossible in general even if (i)the POMG is two-player zero-sum and satisﬁes Assumption 1 with α = 1 , (ii) the opponent keepsplaying a ﬁxed deterministic policy known to the learner, and (iii) the only parts of the model unknownto the learner are the emission matrices.
Revised: We prove that achieving low regret in this setting is impossible in general even if (i)the POMG is two-player zero-sum and satisﬁes Assumption 1 with α = 1 , (ii) the opponent keepsplaying a ﬁxed deterministic policy known to the learner, and (iii) the only parts of the model unknownto the learner are the emission matrices.

Original: Theorem 11 For any L, k ∈ N + , there exist (i) a two-player zero-sum POMG of size S, A, O, H =
Revised: Theorem 11. For any L, k ∈ N + , there exist (i) a two-player zero-sum POMG of size S, A, O, H =

Original: O ( L ) and satisfying Assumption 1 with α = 1 , and (ii) a ﬁxed opponent who keeps playing a known deterministic policy π 2 , so that with probability at least 1 /
Revised: O ( L ) and satisfying Assumption 1 with α = 1 , and (ii) a ﬁxed opponent who keeps playing a known deterministic policy π 2 , so that with probability at least 1 /

Original: [Equation]
Revised: [Equation]

Original: where π t 1 is the policy played by the learner in the t th episode.
Revised: where π t 1 is the policy played by the learner in the t th episode.

Original: Theorem 11 claims that when the learner is not allowed to access the opponent’s observations and actions, there exists exponential regret lower bound for competing with the max-min value (i.e., Nash value in two-player zero-sum POMGs) even in the very benign scenario as described above.
Revised: Theorem 11 claims that when the learner is not allowed to access the opponent’s observations and actions, there exists exponential regret lower bound for competing with the max-min value (i.e., Nash value in two-player zero-sum POMGs) even in the very benign scenario as described above.

Original: We remark that this lower bound directly implies competing with the best ﬁxed policy in hindsight is also hard because the max-min value is always no larger than the value of the best-response to π 2 :
Revised: We remark that this lower bound directly implies competing with the best ﬁxed policy in hindsight is also hard because the max-min value is always no larger than the value of the best-response to π 2 :

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: 5.2 Positive results for the game-replay setting
Revised: 5.2 Positive results for the game-replay setting

Original: In this section, we consider the game-replay setting where after each episode of play, every playerwill reveal their observations and actions in this episode to other players.
Revised: In this section, we consider the game-replay setting where after each episode of play, every playerwill reveal their observations and actions in this episode to other players.

Original: In other words, every playeris able to observe the whole trajectory τ k =
Revised: In other words, every playeris able to observe the whole trajectory τ k =

Original: ( o k 1 , a k 1 , . . .
Revised: ( o k 1 , a k 1 , . . .

Original: , o kH , a kH ) after the k th episode is ﬁnished.
Revised: , o kH , a kH ) after the k th episode is ﬁnished.

Original: The motivation for considering this setting is in many real-world games, e.g., Dota, StarCraft and
Revised: The motivation for considering this setting is in many real-world games, e.g., Dota, StarCraft and

Original: Poker, players are usually allowed to watch the replays of the games they have played, in which theycan freely view other players’ observations and actions.
Revised: Poker, players are usually allowed to watch the replays of the games they have played, in which theycan freely view other players’ observations and actions.

Original: Below, we show that a simple variant of
Revised: Below, we show that a simple variant of

Original: OMLE-Equilibrium enjoys sublinear regret when playing against adversarial opponents.
Revised: OMLE-Equilibrium enjoys sublinear regret when playing against adversarial opponents.

Revised: Algorithm

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Original:  Algorithm description We provide the formal description of OMLE-Adversary in Algorithm 3.
Revised: [Equation] [Equation] [Equation] Algorithm description We provide the formal description of OMLE-Adversary in Algorithm 3.

Original: Same as OMLE-Equilibrium , OMLE-Adversary utilizes the relaxed MLE approach to constructthe conﬁdence set.
Revised: Same as OMLE-Equilibrium , OMLE-Adversary utilizes the relaxed MLE approach to constructthe conﬁdence set.

Original: The key modiﬁcation lies in the computation of player 1 ’s (stochastic) policy π k 1(Line 3).
Revised: The key modiﬁcation lies in the computation of player 1 ’s (stochastic) policy π k 1(Line 3).

Original: Speciﬁcally, the learner will compute the most optimistic model θ k in the conﬁdence set B kby examining player 1 ’s max-min value in each model.
Revised: Speciﬁcally, the learner will compute the most optimistic model θ k in the conﬁdence set B kby examining player 1 ’s max-min value in each model.

Original: Then choose player 1 ’s policy to be the onewith the highest value under θ k , assuming all other players jointly play against player 1 .
Revised: Then choose player 1 ’s policy to be the onewith the highest value under θ k , assuming all other players jointly play against player 1 .

Original: Finally, we remark that although the conﬁdence set construction in Line 6 seems to involve the joint policies π of all players, the conﬁdence set itself is in fact independent of the joint policies π .
Revised: Finally, we remark that although the conﬁdence set construction in Line 6 seems to involve the joint policies π of all players, the conﬁdence set itself is in fact independent of the joint policies π .

Original: Therefore, the 1 st player can still construct the conﬁdence set without knowledge of other players’ policies.
Revised: Therefore, the 1 st player can still construct the conﬁdence set without knowledge of other players’ policies.

Original: This is because the dependency of the loglikelihood function on policy π are equal on both sides of (3), and thus they cancel with each other.
Revised: This is because the dependency of the loglikelihood function on policy π are equal on both sides of (3), and thus they cancel with each other.

Original: Formally, for any ˆ θ, θ (cid:48)
Revised: Formally, for any ˆ θ, θ (cid:48)

Original: ∈ Θ , we have
Revised: ∈ Θ , we have

Original: [Equation]
Revised: [Equation]

Original: Theoretical guarantees Below we present the main theorem for OMLE-Adversary .
Revised: Theoretical guarantees Below we present the main theorem for OMLE-Adversary .

Original: Theorem 12 (Regret of OMLE-Adversary )
Revised: Theorem 12. (Regret of OMLE-Adversary )

Original: Under Assumption 1, there exists an absolute constant c such that for any δ ∈ (0 , 1] and K ∈ N , Algorithm 3 with β
Revised: Under Assumption 1, there exists an absolute constant c such that for any δ ∈ (0 , 1] and K ∈ N , Algorithm 3 with β

Original: = c(cid:0) [Equation] S
Revised: = c

Original: [Equation]
Revised: [Equation]

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original: SO
Revised: SO

Original: [Equation]
Revised: [Equation]

Original: log( SAOHK ) + log( K/δ ) (cid:1) satisﬁes that with probability at least 1 − δ ,
Revised: log

Original: [Equation]
Revised: [Equation]

Original: −
Revised: SAOHK

Original: [Equation]
Revised: [Equation]

Original: ≤ poly( S, A, O, H,
Revised: + log( K/δ

Original: [Equation]
Revised: [Equation]

Original:  ,log( Kδ
Revised: (cid:1) satisﬁes that with probability at least 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: 1 )) ·√ k for all k ∈ [ K ] .
Revised: for all k ∈ [ K ] .

Original: Theorem 12 claims that the regret of OMLE-Adversary is upper bounded by ˜ O ( √ k ) in any weakly revealing POMGs that satisfy Assumption 1, no matter what adversarial strategies other players might take.
Revised: Theorem 12 claims that the regret of OMLE-Adversary is upper bounded by ˜ O ( √ k ) in any weakly revealing POMGs that satisfy Assumption 1, no matter what adversarial strategies other players might take.

Original: Here the regret is deﬁned by comparing the cumulative rewards received by player 1 to the max
Revised: Here the regret is deﬁned by comparing the cumulative rewards received by player 1 to the max-min value [Equation] ˜

Original: [Equation]
Revised: [Equation]

Original: that is the largest value she could receive if all other players jointly play against her.
Revised: 2that is the largest value she could receive if all other players jointly play against her.

Original: Notice that this regret is weaker than the typical version of regret considered in online learning literature, which typically competes with the best response in hindsight, i.e.,
Revised: Notice that this regret is weaker than the typical version of regret considered in online learning literature, which typically competes with the best response in hindsight, i.e.,

Original: [Equation]
Revised: [Equation]

Original: Therefore, it is natural to ask whether we can obtain similar sublinear regret in terms of the aboveregret deﬁnition.
Revised: Therefore, it is natural to ask whether we can obtain similar sublinear regret in terms of the aboveregret deﬁnition.

Original: Unfortunately, previous work [38] proved that there exists exponential regret lowerbound for competing with the best response in hindsight even in fully observable two-player zero-sum
Revised: Unfortunately, previous work [25] proved that there exists exponential regret lowerbound for competing with the best response in hindsight even in fully observable two-player zero-sum

Original: Markov games, which are special cases of POMGs satisfying Assumption 1 with α = 1 .
Revised: Markov games, which are special cases of POMGs satisfying Assumption 1 with α = 1 .

Original: As a result,achieving low regret in the above sense is also intractable in POMGs.
Revised: As a result,achieving low regret in the above sense is also intractable in POMGs.

Revised: Generalization to multi-step weakly revealing POMGs So far, we only derive the positveresult (Theorem 12) for single-step weakly revealing POMGs.

Revised: A reader might wonder whethersimilar results can be obtained in the more general setting of multi-step weakly revealing POMGs.

Revised: Unfortunately, this generalization turns out to be impossible in general, even if (i) the POMG satisﬁes

Revised: Assumption 2 with m = 2 and α ≥ 1 , and (ii) the learner can directly observe the opponents’ actionsand observations.

Revised: We defer the formal statement of this hardness result and its proof to Appendix

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂectthe paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reﬂect

Revised: the

Revised: paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [No] This work
Revised: [No] This work

Original: is purely theoretical and we do not anticipate any potential negative social impacts.
Revised: is purely theoretical and we do not anticipate any potential negative social impacts.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] We provide all theproofs in the appendix.
Revised: [Yes] We provide all the proofs in the appendix.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [N/A]
Revised: [N/A]

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]