Original: A S ELF -P LAY P OSTERIOR S AMPLING A LGORITHM FOR Z ERO -S UM M ARKOV G AMES
Revised: A S ELF -P LAY P OSTERIOR S AMPLING A LGORITHM FOR Z ERO -S UM M ARKOV G AMES

Original: Anonymous authors

Original: Paper under double-blind review

Original:  BSTRACT
Revised: A BSTRACT

Original: Existing studies on provably efficient algorithms for Markov games (MGs) almost exclusively build on the “optimism in the face of uncertainty” (OFU) principle.
Revised: Existing studies on provably efficient algorithms for Markov games (MGs) almost exclusively build on the “optimism in the face of uncertainty” (OFU) principle.

Original: This work focuses on a different approach of posterior sampling, which is celebrated in many bandits and reinforcement learning settings but remains under-explored for MGs.
Revised: This work focuses on a different approach of posterior sampling, which is celebrated in many bandits and reinforcement learning settings but remains under-explored for MGs.

Original: Specifically, for episodic two-player zero-sum MGs, a novel posterior sampling algorithm is developed with general function approximation.
Revised: Specifically, for episodic two-player zero-sum MGs, a novel posterior sampling algorithm is developed with general function approximation.

Original: Theoretical analysis demonstrates that the posterior sampling algorithm admits a √ T -regret bound for problems with a low multi-agent decoupling coefficient, which is a new complexity measure for MGs, where T denotes the number of episodes.
Revised: Theoretical analysis demonstrates that the posterior sampling algorithm admits a √ T -regret bound for problems with a low multi-agent decoupling coefficient, which is a new complexity measure for MGs, where T denotes the number of episodes.

Original: When specialized to linear MGs, the obtained regret bound matches the state-of-the-art results.
Revised: When specialized to linear MGs, the obtained regret bound matches the state-of-the-art results.

Original: To the best of our knowledge, this is the first provably efficient posterior sampling algorithm for MGs with frequentist regret guarantees, which enriches the toolbox for MGs and promotes the broad applicability of posterior sampling.
Revised: To the best of our knowledge, this is the first provably efficient posterior sampling algorithm for MGs with frequentist regret guarantees, which enriches the toolbox for MGs and promotes the broad applicability of posterior sampling.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Multi-agent reinforcement learning (MARL) focuses on the sequential decision making problem involving more than one agent, each of which aims to optimize her own long-term return by interacting with the environment and other agents Zhang et al.
Revised: Multi-agent reinforcement learning (MARL) focuses on the sequential decision making problem involving more than one agent, each of which aims to optimize her own long-term return by interacting with the environment and other agents Zhang et al.

Original: Today, MARL has a diverse set of realworld applications, including Go Silver et al.
Revised: Today, MARL has a diverse set of realworld applications, including Go Silver et al.

Original: (2016; 2017), autonomous driving Shalev-Shwartz et al.
Revised: (2016; 2017), autonomous driving Shalev-Shwartz et al.

Original: (2016), Poker Brown & Sandholm (2019), and Dota Berner et al.
Revised: (2016), Poker Brown & Sandholm (2019), and Dota Berner et al.

Original: (2019), just to name a few.
Revised: (2019), just to name a few.

Original: Due to the large state space of these practical problems, function approximation (with neural networks) is often used in these applications for the generalization across different state-action pairs.
Revised: Due to the large state space of these practical problems, function approximation (with neural networks) is often used in these applications for the generalization across different state-action pairs.

Original: While there is a long line of related works on the theoretical understanding of single-agent RL with general function approximation Jiang et al.
Revised: While there is a long line of related works on the theoretical understanding of single-agent RL with general function approximation Jiang et al.

Original: (2017); Sun et al.
Revised: (2017); Sun et al.

Original: ; Wang et al.
Revised: ; Wang et al.

Original: (2020); Jin et al.
Revised: (2020); Jin et al.

Original: (2021a);
Revised: (2021a);

Original: Du et al.
Revised: Du et al.

Original: ; Dann et al.
Revised: ; Dann et al.

Original: (2021), the theory of MARL with general function approximation is substantially less explored.
Revised: (2021), the theory of MARL with general function approximation is substantially less explored.

Original: In this paper, we aim to explore this topic in the context of two-player zero-sum Markov games (MGs) Shapley (1953);
Revised: In this paper, we aim to explore this topic in the context of two-player zero-sum Markov games (MGs) Shapley (1953);

Original: Littman (1994).
Revised: Littman (1994).

Original: The goal of learning in a two-player zero-sum MG is to learn the Nash equilibrium at which the policy of each player maximizes her own cumulative rewards, provided that the policies of other agents are fixed.
Revised: The goal of learning in a two-player zero-sum MG is to learn the Nash equilibrium at which the policy of each player maximizes her own cumulative rewards, provided that the policies of other agents are fixed.

Original: Intuitively speaking, Nash equilibrium characterizes the point from which no agent will deviate.
Revised: Intuitively speaking, Nash equilibrium characterizes the point from which no agent will deviate.

Original: Since the reward and the state transition are determined jointly by the actions of both agents, in addition to the unknown environment, each agent must also handle the dynamics from other strategic agents.
Revised: Since the reward and the state transition are determined jointly by the actions of both agents, in addition to the unknown environment, each agent must also handle the dynamics from other strategic agents.

Original: Due to this game-theoretical feature, algorithms designed for MDP cannot be directly extended to the MARL case.
Revised: Due to this game-theoretical feature, algorithms designed for MDP cannot be directly extended to the MARL case.

Original: However, recent studies Jin et al.
Revised: However, recent studies Jin et al.

Original: (2021b)
Revised: (2021b)

Original: ; Huang et al.
Revised: ; Huang et al.

Original: (2021) have shown that with an innovative asymmetrical structure, similar theoretical results can be established for the two-player zero-sum MG with general function approximation.
Revised: (2021) have shown that with an innovative asymmetrical structure, similar theoretical results can be established for the two-player zero-sum MG with general function approximation.

Original: Nevertheless, despite a handful of recent progress on the theory of the two-player zero-sum MG with general function approximation, the existing works are mainly confined to algorithms based on the optimism in the face of uncertainty (OFU) principle.
Revised: Nevertheless, despite a handful of recent progress on the theory of the two-player zero-sum MG with general function approximation, the existing works are mainly confined to algorithms based on the optimism in the face of uncertainty (OFU) principle.

Original: In contrast, the theory of posteriorsampling-based algorithms is less developed (in the frequentist setting).
Revised: In contrast, the theory of posteriorsampling-based algorithms is less developed (in the frequentist setting).

Original: Recent works in the context  of contextual multi-armed bandit and single-agent RL demonstrate that there is no statistical efficiency gap between OFU and posterior sampling algorithms Dann et al.
Revised: Recent works in the context and of contextual multi-armed bandit and single-agent RL demonstrate that there is no statistical efficiency gap between OFU and posterior sampling algorithms Dann et al.

Original: ; Zhang (2021).
Revised: ; Zhang (2021).

Original: However, whether we can design model-free posterior sampling algorithms in MARL that achieve similar theoretical guarantees remains open.
Revised: However, whether we can design model-free posterior sampling algorithms in MARL that achieve similar theoretical guarantees remains open.

Original: In this paper, we are interested in the application of posterior sampling in the two-player zero-sum
Revised: In this paper, we are interested in the application of posterior sampling in the two-player zero-sum

Original: MG with general function approximation and self-play (which means that the learning agent can control both players).
Revised: MG with general function approximation.

Original: Our main result indicates that, similar to the single-agent case, posterior sampling algorithms can achieve comparable theoretical guarantees as the OFU-based algorithms.
Revised: Our main result indicates that, similar to the single-agent case, posterior sampling algorithms can achieve comparable theoretical guarantees as the OFU-based algorithms.

Original: Our contributions are summarized as follows:
Revised: Our contributions are summarized as follows:

Original: • A provably efficient posterior sampling algorithm is designed under the self-play framework for the two-player zero-sum MG with general function approximation.
Revised: • A provably efficient posterior sampling algorithm is designed under the self-play framework for the two-player zero-sum MG with general function approximation.

Original: To the best of our knowledge, this is the first posterior sampling algorithm with frequentist regret guarantee in the context of Markov games; • The single-agent complexity measure of decoupling coefficient , first introduced in Dann et al.
Revised: To the best of our knowledge, this is the first posterior sampling algorithm with frequentist regret guarantee in the context of Markov games; • The single-agent complexity measure of decoupling coefficient , first introduced in Dann et al.

Original: (2021), is extended to the multi-agent setting, Moreover, a number of examples with provably small multi-agent decoupling coefficients are identified; • The proposed algorithm is rigorously proved to obtain a √ T -regret for problems with low multi-agent decoupling coefficient, where T is the number of episodes.
Revised: (2021), is extended to the multi-agent setting, Moreover, a number of examples with provably small multi-agent decoupling coefficients are identified; • The proposed algorithm is rigorously proved to obtain a √ T -regret for problems with low multi-agent decoupling coefficient, where T is the number of episodes.

Original: It is noted that while the sampling procedure of the proposed algorithm may not be computationally efficient, the lack of computational tractability also appears in the works of Jin et al.
Revised: It is noted that while the sampling procedure of the proposed algorithm may not be computationally efficient, the lack of computational tractability also appears in the works of Jin et al.

Original: (2021b)
Revised: (2021b)

Original: ; Huang et al.
Revised: ; Huang et al.

Original: (2021), as well as many previous works with general function approximation in the context of single-agent RL (Jiang et al., 2017; Jin et al., 2021a; Dann et al., 2021; Du et al., 2021).
Revised: (2021), as well as many previous works with general function approximation in the context of single-agent RL (Jiang et al., 2017; Jin et al., 2021a; Dann et al., 2021; Du et al., 2021).

Original: It is an interesting future research topic to identify cases where efficient sampling is possible.
Revised: It is an interesting future research topic to identify cases where efficient sampling is possible.

Original: Moreover, we do not take credit for the asymmetrical structure in our algorithmic framework.
Revised: Moreover, we do not take credit for the asymmetrical structure in our algorithmic framework.

Original: The main contribution here is to extend the posterior sampling algorithm to MGs under the self-play framework.
Revised: The main contribution here is to extend the posterior sampling algorithm to MGs under the self-play framework.

Original: 1. R ELATED W ORKS
Revised: 1. R ELATED W ORKS

Original: Due to the space limitation, a detailed literature review is deferred to Appendix A.
Revised: Due to the space limitation, a detailed literature review is deferred to Appendix A.

Original:  P ROBLEM F ORMULATION
Revised:  P ROBLEM F ORMULATION

Original: Markov Games (MGs) generalize the standard Markov Decision Processes to the multi-agent setting.

Original: In this work, the episodic two-player zeros-sum MG is considered, which can be formally denoted as MG ( H, X , A , B , P , r ) .
Revised: We consider the episodic two-player zero-sum Markov Game (MG), which can be formally denoted as MG ( H, X , A , B , P , r ) .

Original: Here H denotes the length of each episode, X is the state space, A and B are the action spaces of two players (referred to as the max-player and the min-player), respectively,P h (
Revised: Here H denotes the length of each episode, X is the state space, A and B are the action spaces of two players (referred to as the max-player and the min-player), respectively,

Original: | x, a b ) is the transition measure of the next state from the current state x with two actions 
Revised: P h ( ·| x, a, b ) is the transition measure of the next state from the current state x with two actions ( a, b ) taken at step h , and r h ( x, a, b ) is the corresponding reward received with actions ( a, b ) taken for state x at step h .

Original: Specifically, in this MG, each episode t starts from an initial state x 1 t .
Revised: Specifically, in this MG, each episode t starts from an initial state x 1 t .

Original: At each step h , two players observe the current state x ht , take actions ( a ht , b ht ) individually, and observe the next state x h +1 t ∼ P h ( ·| x ht , a ht , b ht ) .
Revised: At each step h , two players observe the current state x ht , take actions ( a ht , b ht ) individually, and observe the next state x h +1 t ∼ P h ( ·| x ht , a ht , b ht ) .

Original: The current episode ends after step H and then a new episode starts.
Revised: The current episode ends after step H and then a new episode starts.

Original: Without loss of generality, each episode is assumed to have a fixed initial state x 1 t = x 1 , which can be easily generalized to having x 1 t sampled from a fixed but unknown distribution.
Revised: Without loss of generality, each episode is assumed to have a fixed initial state x 1 t = x 1 , which can be easily generalized to having x 1 t sampled from a fixed but unknown distribution.

Original: Also, for the ease of presentation, the reward r h ( x, a, b ) is assumed to be deterministic and in the interval of [0 , 1] for any ( x, a, b ) in this paper, while the algorithm designs and theoretical results can also be applied for stochastic bounded rewards with slight modifications.
Revised: Also, for the ease of presentation, the reward r h ( x, a, b ) is assumed to be deterministic and in the interval of [0 , 1] for any ( x, a, b ) in this paper, while the algorithm designs and theoretical results can also be applied for stochastic bounded rewards with slight modifications.

Original: Policies and Value Functions.
Revised: Policies and Value Functions.

Original: With ∆ A denoting the probability simplex over the action space A , a
Revised: With ∆ A denoting the probability simplex over the action space A , a

Original: Markov policy µ of the max-player can be defined as µ := { µ h : X → ∆ A } h ∈ [ H ] .
Revised: Markov policy µ of the max-player can be defined as µ := { µ h : X → ∆ A } h ∈ [ H ] .

Original: Similarly, we can define a Markov policy ν :=
Revised: Similarly, we can define a Markov policy ν :=

Original: { ν h : X → ∆ B } h ∈
Revised: { ν h : X → ∆ B } h ∈

Original: [ H ] for the min-player.
Revised: [ H ] for the min-player.

Original: Given a policy pair ( µ, ν ) , the value function V µ,νh : X → R at step h is defined as
Revised: Given a policy pair ( µ, ν ) , the value function V µ,νh : X → R at step h is defined as

Original: [Equation]
Revised: [Equation]

Original: , and the Q-value function Q µ,νh: X × A × B → R as Q
Revised: and the Q-value function Q µ,νh

Original: [Equation]
Revised: [Equation]

Revised: X × A × B

Revised: → R as

Original: [Equation]
Revised: [Equation]

Original: (cid:21) , where the expectations are taken over the randomness of the environment and the policies.
Revised: where the expectations are taken over the randomness of the environment and the policies.

Original: For a clean presentation, we use the notation P h (with a slight abuse) so that [ P h V ]( x, a, b ) =
Revised: For a clean presentation, we use the notation P h (with a slight abuse) so that [ P h V ]( x, a, b ) =

Original: E x ′ ∼ P h ( ·| x,a,b ) V ( x ′ ) for any value function V .
Revised: E x ′ ∼ P h ( ·| x,a,b ) V ( x ′ ) for any value function V .

Original: Similarly, the notation D π is adopted so that [ D π Q ] ( x ) :=
Revised: Similarly, the notation D π is adopted so that [ D π Q ] ( x ) :=

Original: E ( a,b ) ∼ π ( · , ·| x ) Q ( x, a, b ) , for any policy pair π = ( µ, ν ) and action-value function
Revised: E ( a,b ) ∼ π [Equation] · , ·| x ) Q ( x, a, b ) , for any policy pair π = ( µ, ν ) and action-value function

Original: Q .
Revised: Q .

Original: With these notations, the Bellman equations are given by
Revised: With these notations, the Bellman equations are given by

Revised: [Equation]

Revised: =

Revised: [Equation]

Revised: h

Original: [Equation]
Revised: [Equation]

Revised: x ) .

Original: Best Response.
Revised: Best Response.

Original: For any policy of max-player µ , a corresponding best response for the min-player can be found, denoted as ν † ( µ ) , such that V µ,ν † ( µ ) h ( x ) = inf ν V µ,νh ( x ) for all ( x, h ) .
Revised: For any policy of max-player µ , a corresponding best response for the min-player can be found, denoted as ν † ( µ ) , such that V µ,ν † ( µ ) h ( x ) = inf ν V µ,νh ( x ) for all ( x, h ) .

Original: This value is the best favorable result for the min-player if the max-player announces that she will play strategy µ .
Revised: This value is the best favorable result for the min-player if the max-player announces that she will play strategy µ .

Original: Similarly, for a min-player policy ν , there exists a best response for the max-player, denoted as µ † ( ν ) , such that V µ † ,ν h ( x ) = sup µ V µ,νh ( x ) for all ( x, h ) .
Revised: Similarly, for a min-player policy ν , there exists a best response for the max-player, denoted as µ † ( ν ) , such that V µ † ,ν h ( x ) = sup µ V µ,νh ( x ) for all ( x, h ) .

Original: For simplicity, we denote V µ, † h ( x ) :=
Revised: For simplicity, we denote V µ, † h ( x ) :=

Original: V µ,ν † ( µ ) h ( x ) ,
Revised: V µ,ν † ( µ ) h ( x ) ,

Original: [Equation]
Revised: Q µ, † h ( x ) : = Q µ,ν † ( µ ) h ( x ) , V † ,νh ( x ) : = V µ † ( ν ) ,ν h ( x ) , and Q † ,νh ( x ) :

Original: [Equation]
Revised: =

Original: and Q [Equation] ( x ) [Equation]
Revised: Q µ † ( ν ) ,ν h ( x ) .

Original: Nash Equilibrium.
Revised: Nash Equilibrium.

Original: Moreover, there exists a set of Nash equilibrium (NE) policies ( µ ∗ , ν ∗ ) Filar &
Revised: Moreover, there exists a set of Nash equilibrium (NE) policies ( µ ∗ , ν ∗ ) Filar &

Original: Vrieze (2012) that are optimal against their best response such that
Revised: Vrieze (2012) that are optimal against their best response such that

Original: [Equation]
Revised: [Equation]

Original: for all ( x, h ) ∈
Revised: for all ( x, h ) ∈

Original: X ×
Revised: X ×

Original: [ H ] .
Revised: [ H ] .

Original: For this NE, the following famous minimax equation holds:
Revised: For this NE, the following famous minimax equation holds:

Original: [Equation]
Revised: [Equation]

Original: for all ( x, h ) ∈
Revised: for all ( x, h ) ∈

Original: X ×
Revised: X ×

Original: [ H ] .
Revised: [ H ] .

Original: For simplicity, we denote V ∗ h ( x ) :
Revised: For simplicity, we denote V ∗ h ( x ) :

Original: = V µ ∗ ,ν ∗ h ( x ) and Q ∗ h ( x ) :
Revised: = V µ ∗ ,ν ∗ h [Equation] Q ∗ [Equation]

Original: = Q µ ∗ ,ν ∗ h ( x ) .
Revised: = Q [Equation] x [Equation]

Original: Note that although there might exist multiple NE policies, the NE value function is unique for a zero-sum MG.
Revised: Note that although there might exist multiple NE policies, the NE value function is unique for a zero-sum MG.

Original: Performance metrics.
Revised: Performance metrics.

Original: A max-player’s policy µ is said to be ϵ -close to the NE if it satisfies
Revised: A max-player’s policy µ is said to be ϵ -close to the NE if it satisfies

Original: V ∗ ( x 1 ) − V µ, † ( x 1 ) <
Revised: V ∗ ( x 1 ) − V µ, † ( x 1 ) <

Original: ϵ .
Revised: ϵ .

Original: Note that we have V ∗ ( x 1 ) − V µ,ν ( x 1 ) ≤ V ∗ ( x 1 ) − V µ, † ( x 1 ) for all min-player’s policy ν as the best response is the strongest opponent for the max-player.
Revised: Note that we have V ∗ ( x 1 ) − V µ,ν ( x 1 ) ≤ V ∗ ( x 1 ) − V µ, † ( x 1 ) for all min-player’s policy ν as the best response is the strongest opponent for the max-player.

Original: The main goal of this paper is to find an ϵ -close policy for the max-player and her regret over T episodes can be defined as
Revised: The main goal of this paper is to find an ϵ -close policy for the max-player and her regret over T episodes can be defined as

Original: [Equation]
Revised: [Equation]

Original: where µ t is the policy adopted by the max-player for episode t .
Revised: where µ t is the policy adopted by the max-player for episode t .

Original: Note that we can switch the roles of two players to learn a policy ν that is ϵ -close to the NE for the min-player.
Revised: Note that we can switch the roles of two players to learn a policy ν that is ϵ -close to the NE for the min-player.

Original: 2. F UNCTION A PPROXIMATION
Revised: 2. F UNCTION A PPROXIMATION

Original: We aim to approximate the Q -value functions for the MGs considered in this work by a class of functions F = F 1 × · · · × F H where F h ⊂ ( X × A × B → R ) .
Revised: We aim to approximate the Q -value functions for the MGs considered in this work by a class of functions F = F 1 × · · · × F H where F h ⊂ ( X × A × B → R ) .

Original: For f ∈ F , a NE can be induced and the corresponding policy µ f of the max-player is defined for all ( x, h ) as
Revised: For f ∈ F , a NE can be induced and the corresponding policy µ f of the max-player is defined for all ( x, h ) as

Original: [Equation]
Revised: [Equation]

Original: The induced value function for all ( x, h ) is then given by
Revised: The induced value function for all ( x, h ) is then given by

Original: [Equation]
Revised: [Equation]

Original: Moreover, for a fixed max-player policy µ and a function f ∈ F , the induced value function of the best response of the min-player is defined for all ( x, h ) as
Revised: Moreover, for a fixed max-player policy µ and a function f ∈ F , the induced value function of the best response of the min-player is defined for all ( x, h ) as

Original: [Equation]
Revised: [Equation]

Original: As common in Perolat et al.
Revised: As common in Perolat et al.

Original: ; Jin et al.
Revised: ; Jin et al.

Original: (2021b)
Revised: (2021b)

Original: ; Huang et al.
Revised: ; Huang et al.

Original: (2021), two types of Bellman operators are defined as( T h f ) ( x, a, b ) :
Revised: (2021), two types of Bellman operators are defined as

Original: = (cid:2) r h + P h
Revised: [Equation]

Original: V
Revised: a

Original: f,h +1 (cid:3) ( x, a, b ) , and ( T µh f ) ( x, a, b ) :
Revised: [Equation] and ( T µh f [Equation]

Original: = (cid:104)
Revised: ( x, a, b ) := (cid:104)

Original: r h + P h V µf,h +1 (cid:105) ( x, a, b ) .
Revised: r h + P h V µf,h +1 (cid:105) ( x, a, b ) .

Original: The corresponding Bellman residual are denoted as
Revised: The corresponding Bellman residual are denoted as

Original: [Equation]
Revised: [Equation]

Original: Sometimes the state-action pair ( x, a, b ) may be replaced with a trajectory ζ = { ( x h ′ , a h ′ , b h ′ , r h ′ ) }
Revised: Sometimes the state-action pair ( x, a, b ) may be replaced with a trajectory ζ = { ( x h ′ , a h ′ , b h ′ , r h ′ ) }

Original: Hh ′ =1 , which indicates that the corresponding state-action pair at step h , i.e.
Revised: Hh ′ =1 , which indicates that the corresponding state-action pair at step h , i.e.

Original: ( x h , a h , b h ) , is taken as input.
Revised: ( x h , a h , b h ) , is taken as input.

Original: We make the following assumptions that are commonly adopted in the literature of MDPs and MGs with general function approximation (Wang et al., 2020; Jin et al., 2021a; Dann et al., 2021; Jin et al., 2021b;
Revised: We make the following assumptions that are commonly adopted in the literature of MDPs and MGs with general function approximation (Wang et al., 2020; Jin et al., 2021a; Dann et al., 2021; Jin et al., 2021b;

Original: Huang et al., 2021).
Revised: Huang et al., 2021).

Original: Assumption 1 (Realizability) .
Revised: Assumption 1 (Realizability) .

Original: For the Nash equilibrium, it holds that Q ∗ h ∈ F h , ∀ h ∈
Revised: For the Nash equilibrium, it holds that Q ∗ h ∈ F h , ∀ h ∈

Original: [ H ] .
Revised: [ H ] .

Original: Moreover, for any f ∈ F , it holds that Q µ f , † h ∈ F h , ∀ h ∈
Revised: Moreover, for any f ∈ F , it holds that Q µ f , † h ∈ F h , ∀ h ∈

Original: [ H ] .
Revised: [ H ] .

Original: Assumption 2 (Completeness) .
Revised: Assumption 2 (Completeness) .

Original: For any f, g ∈ F and the induced policy µ f , it holds that T µ f h g ∈
Revised: For any f, g ∈ F and the induced policy µ f , it holds that T µ f h g ∈

Original: F h , ∀ h ∈
Revised: F h , ∀ h ∈

Original: [ H ] .
Revised: [ H ] .

Original: Assumption 3 (Boundedness) .
Revised: Assumption 3 (Boundedness) .

Original: There exists β ≥ 1 s.t.
Revised: There exists β > 1 s.t.

Original: f h ( x, a, b ) ∈ [0 , β − 1] , ∀ ( f, h, x, a, b ) ∈
Revised: f h ( x, a, b ) ∈ [0 , β − 1] , ∀ ( f, h, x, a, b ) ∈

Original: F × [ H ] × X ×
Revised: F × [ H ] × X ×

Original: A × B .
Revised: A × B .

Original:  A LGORITHM
Revised:  A LGORITHM

Original: The proposed Conditional Posterior Sampling with Booster algorithm is presented in this section.
Revised: The proposed Conditional Posterior Sampling with Booster algorithm is presented in this section.

Original: 1: Input : function class: F , learning rate η , horizon T , prior parameter λ .

Original: 2: S

Original: 3. O VERVIEW
Revised: 3. O VERVIEW

Original: The existing algorithms with frequentist guarantee are confined to OFU-based algorithms.
Revised: The existing algorithms with frequentist guarantee are confined to OFU-based algorithms.

Original: Algorithmically, these algorithms typically maintain a confidence set C whose components are consistent with the Bellman equation so far.
Revised: Algorithmically, these algorithms typically maintain a confidence set C whose components are consistent with the Bellman equation so far.

Original: Then, an optimistic function f ∈ C is selected to approximate value function through some optimization subroutine Jin et al.
Revised: Then, an optimistic function f ∈ C is selected to approximate value function through some optimization subroutine Jin et al.

Original: (2021b)
Revised: (2021b)

Original: ; Huang et al.
Revised: ; Huang et al.

Original: In contrast, the posterior sampling algorithm starts with a prior p 0 ( · ) over the function class F andcollects trajectories to compute the likelihood function ; they together lead to a posterior distribution p ( · ) over F . Then, a function is sampled from the posterior distribution to approximate the target. In addition tothe difference in algorithm structure, the analysis techniques for the posterior sampling algorithm are also different, particularly due to the lack of explicit optimism from the planning step.
Revised: Although suitable for the worst case, it can be far too optimistic for average instances and is inferior to posterior sampling, as supported by many existing works in bandits, contextual bandits, and MDPs (Chapelle & Li, 2011; Osband et al., 2016).

Original: A frequentist theoretical guarantee of posterior sampling algorithms is lacking for a long time even in the context of contextual bandit.
Revised: However, a frequentist theoretical guarantee of posterior sampling algorithms is lacking for a long time even in the context of contextual bandit.

Original: Recently, Zhang (2021) and Dann et al.
Revised: Recently, Zhang (2021) and Dann et al.

Original: (2021) show that adding an extra optimistic term can lead to frequentistly optimal posterior sampling algorithm in contextual bandit and MDP, respectively.
Revised: (2021) show that adding an extra optimistic term can lead to frequentistly optimal posterior sampling algorithm in contextual bandit and MDP, respectively.

Original: However, in the MARL setting, the multi-agent nature leads to complicated statistical dependence across the players.
Revised: However, in the MARL setting, the multi-agent nature leads to complicated statistical dependence across the players.

Original: In particular, in addition to the environment, the agent will also be affected by other strategic agents.
Revised: In particular, in addition to the environment, the agent will also be affected by other strategic agents.

Original: Therefore, the situation is far more complicated even in the two-player case and the algorithms designed for MDPs cannot be directly extended to MGs.
Revised: Therefore, the situation is far more complicated even in the two-player case.

Original: To overcome this issue, inspired by Jin et al.
Revised: Inspired by Jin et al.

Original: (2021b)
Revised: (2021b)

Original: ; Huang et al.
Revised: ; Huang et al.

Original: (2021), we leverage the innovative asymmetric structure to pick the max-player and the min-player as the main agent and the booster agent, respectively, where the booster agent, as the name suggests, aims to assist the main agent’s learning.
Revised: (2021), we leverage the innovative asymmetric structure to pick the max-player and the min-player as the main agent and the booster agent, respectively, where the booster agent, as the name suggests, aims to assist the main agent’s learning.

Original: Our algorithm is summarized in Algorithm 1 where the main agent’s algorithm and the booster agent’s algorithm are given in Algorithms 2 and 3, respectively.
Revised: Our algorithm is summarized in Algorithm 1 where the main agent’s algorithm and the booster agent’s algorithm are given in Algorithms 2 and 3, respectively.

Original: 3. T HE M AIN A GENT
Revised: T HE M AIN A GENT

Original: The main agent’s goal is to learn a Nash policy for the max-player, i.e., µ ∗ .
Revised: The main agent’s goal is to learn a Nash policy for the max-player, i.e., µ ∗ .

Original: With function class F available, she aims to find a function f ∈ F to approximate the Nash Q -value function, i.e., Q ∗ , which can be used to solve the Nash policy via the minimax equation.
Revised: With function class F available, she aims to find a function f ∈ F to approximate the Nash Q -value function, i.e., Q ∗ , which can be used to solve the Nash policy via the minimax equation.

Original: The following optimistic prior and temporal difference error likelihood are carefully crafted to induce a desired posterior distribution over F , which is further used to sample a suitable function f .
Revised: The following optimistic prior and temporal difference error likelihood are carefully crafted to induce a desired posterior distribution over F , which is further used to sample a suitable function f .

Original: Optimistic prior.
Revised: Optimistic prior.

Original: The following prior p 0 ( · ) over the function class F is adopted for the main agent:
Revised: The following prior p 0 ( · ) over the function class F is adopted for the main agent:

Original: [Equation]
Revised: [Equation]

Original: where λ > 0 is a tuning parameter, and p h 0 ( · ) is a distribution over F h .
Revised: where λ > 0 is a tuning parameter, and p h 0 ( · ) is a distribution over F h .

Original: Note that other than the standard prior of p 0 ( f ) =
Revised: Note that other than the standard prior of p 0 ( f ) =

Original: (cid:81)
Revised: (cid:81)

Original: Hh =1 p h 

Revised: Hh =1 p h

Original: ( f h ) , an additional optimistic term, i.e., exp( λV f, 1 ( x 1 ) , is involved in the prior, which plays an important role of encouraging exploration for the main agent.
Revised: ( f h ) , an additional optimistic term, i.e., exp( λV f, 1 ( x 1 ) , is involved in the prior, which plays an important role of encouraging exploration for the main agent.

Original: This prior is referred as an optimistic one because it favors large values for the initial state.
Revised: This prior is referred as an optimistic one because it favors large values for the initial state.

Original: Also, technically, it compensates for one extra term arising in the value decomposition in Lemma 1 when the optimism is not inherently available as in OFU-based algorithms.
Revised: Also, technically, it compensates for one extra term arising in the value decomposition in Lemma 1 when the optimism is not inherently available as in OFU-based algorithms.

Original: Similar techniques are also adopted in the design of posterior sampling for MDPs (Dann et al., 2021) and contextual bandits (Zhang, 2021).
Revised: Similar techniques are also adopted in the design of posterior sampling for MDPs (Dann et al., 2021) and contextual bandits (Zhang, 2021).

Original: Furthermore, Zhang (2021) argues that in the context of contextual bandit, such an optimistic component is necessary to design optimal posterior sampling-based algorithms in the frequentist setting.
Revised: Furthermore, Zhang (2021) argues that in the context of contextual bandit, such an optimistic component is necessary to design optimal posterior sampling-based algorithms in the frequentist setting.

Original: Likelihood function for the main agent.
Revised: Likelihood function for the main agent.

Original: With the history up to the end of episode t denoted as
Revised: With the history up to the end of episode t denoted as

Original: S t = { x hs , a hs , b hs , r hs } s ∈
Revised: S t = { x hs , a hs , b hs , r hs } s ∈

Original: [ t ] ,h ∈ [ H ] , a likelihood function over S t is specified as
Revised: [ t ] ,h ∈ [ H ] , a likelihood function over S t is specified as

Original: [Equation]
Revised: [Equation]

Original: { L h ( · ) } Hh =1 is a collection of squared loss functions as
Revised: { L h ( · ) } Hh =1 is a collection of squared loss functions as

Original: [Equation]
Revised: [Equation]

Original: which is a proxy to the squared T h -Bellman error.
Revised: which is a proxy to the squared T h -Bellman error.

Original: Posterior distribution for the main agent.
Revised: Posterior distribution for the main agent.

Original: Given the prior distribution and the likelihood function, the posterior at the end of episode t can be naturally expressed as
Revised: Given the prior distribution and the likelihood function, the posterior at the end of episode t can be naturally expressed as

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: 1: Draw g ∼ p µ f ( ·|D ) where the posterior is given by Eqn.
Revised: The ”conditional” in the name of the algorithm comes from that the distribution of f h is conditioned on f h +1 . This allows us to introduce the denominator as as to handle the double-sampling issue as in Dann et al.

Original: equation [Table]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 3. T HE B OOSTER A GENT
Revised: 3. T HE B OOSTER A GENT

Original: As aforementioned, the main agent aims to learn the Nash policy.
Revised: As aforementioned, the main agent aims to learn the Nash policy.

Original: However, given the competing nature of MGs, this task is not feasible if her opponent is naive.
Revised: However, given the competing nature of MGs, this task is not feasible if her opponent is naive.

Original: Thus, inspired by Jin et al.
Revised: Thus, inspired by Jin et al.

Original: (2021b);
Revised: (2021b);

Original: Huang et al.
Revised: Huang et al.

Original: (2021), the second learning agent is set to be the booster agent.
Revised: (2021), the second learning agent is set to be the booster agent.

Original: As opposed to the main agent, the booster agent does not aim at the Nash policy.
Revised: As opposed to the main agent, the booster agent does not aim at the Nash policy.

Original: Instead her goal is to assist the main agent’s learning.
Revised: Instead her goal is to assist the main agent’s learning.

Original: Specifically, she examines the adopted policy of the main agent and finds a best response for it (since the best response is the strongest opponent).
Revised: Specifically, she examines the adopted policy of the main agent and finds a best response for it (since the best response is the strongest opponent).

Original: In this way, the underlying weakness of the main agent is exploited, which facilitates the learning of the NE.
Revised: In this way, the underlying weakness of the main agent is exploited, which facilitates the learning of the NE.

Original: To better illustrate the role of the booster agent, we consider the following decomposition of the regret:
Revised: To better illustrate the role of the booster agent, we consider the following decomposition of the regret:

Original: [Equation]
Revised: [Equation]

Original: The non-negative booster agent part is zero if we can find the best response to µ t exactly.
Revised: The non-negative booster agent part is zero if we can find the best response to µ t exactly.

Original: Motivated by this observation, the booster agent keeps learning to approximate the best response to the given max-player’s policy based on the historical trajectories.
Revised: Motivated by this observation, the booster agent keeps learning to approximate the best response to the given max-player’s policy based on the historical trajectories.

Original: Due to the different goal, the design philosophy of the booster agent is fundamental differentfrom that of the main agent. Especially, she takes a pessimistic (instead of optimistic) prior anda different format of the likelihood function.
Revised: Due to the different goal, we adopt different prior and likelihood functions for the booster agent.

Original: Pessimistic prior.
Revised: Optimistic prior.

Original: A pessimistic prior is adopted for the booster agent instead of an optimistic one, defined as
Revised: A different optimistic prior is adopted for the booster agent (min-player), defined as

Original: [Equation]
Revised: [Equation]

Original: Intuitively, the booster agent favors small values for the initial state.
Revised: Intuitively, the booster agent favors small values for the initial state.

Original: The motivation for such apessimistic prior will be clearer after the value decomposition lemma, i.e., Lemma 2, is presented.
Revised: The motivation for such a prior will be clearer after the value decomposition lemma, i.e., Lemma 2, is presented.

Original: Likelihood function for the booster agent.
Revised: Likelihood function for the booster agent.

Original: As the booster agent mainly focuses on approximating the best response policy to µ instead of finding NE, a different squared loss function is specified as:
Revised: As the booster agent mainly focuses on approximating the best response policy to µ instead of finding NE, a different squared loss function is specified as:

Original: [Equation]
Revised: [Equation]

Original: which can be viewed as a proxy to the squared T µh -Bellman error.
Revised: which can be viewed as a proxy to the squared T µh -Bellman error.

Original: Consequently, a corresponding likelihood function can be obtained by replacing L h inEqn. equation 3 with L hµ .
Revised: Consequently, a corresponding likelihood function can be obtained by replacing L h in equation 3 with L hµ .

Original: Posterior distribution for the booster agent.
Revised: Posterior distribution for the booster agent.

Original: With the prior and the likelihood function, the posterior distribution for the booster agent can be obtained as:
Revised: With the prior and the likelihood function, the posterior distribution for the booster agent can be obtained as:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Note that sometimes we also employ the notation q ( g h | g h +1 , µ, S t ) =
Revised: Note that sometimes we also employ the notation q ( g h | g h +1 , µ, S t ) =

Original: q µ ( g h | g h +1 , S t ) when we need to use the superscript h .
Revised: q µ ( g h | g h +1 , S t ) when we need to use the superscript h .

Original:  S KETCH OF THE M AIN I DEAS
Revised:  S KETCH OF THE M AIN I DEAS

Original: In this section, a sketch of the main ideas is provided for theproposed algorithm  andthe theoretical proof.
Revised: In this section, a sketch of the main ideas is provided for the algorithm design and theoretical proof.

Original: 4. V ALUE -D ECOMPOSITION L EMMAS
Revised: 4. V ALUE -D ECOMPOSITION L EMMAS

Original: It is well known that the immediate regret in one episode can be related to the Bellman residuals in the single-agent setting Jiang et al.
Revised: It is well known that the immediate regret in one episode can be related to the Bellman residuals in the single-agent setting Jiang et al.

Original: and this technique is well-adopted in the literature Jin et al.
Revised: and this technique is well-adopted in the literature Jin et al.

Original: (2021a)
Revised: (2021a)

Original: ; Dann et al.
Revised: ; Dann et al.

Original: ; Du et al. (2021).
Revised: ; Du et al. (2021).

Original: With the regret decomposed as inEqn. equation 5, the immediate regrets of the main agent part and the booster agent part can be related to the T -Bellman residuals and the T µ t -Bellman residuals, respectively, as we show below.
Revised: With the regret decomposed as in equation 5, the immediate regrets of the main agent part and the booster agent part can be related to the T -Bellman residuals and the T µ t -Bellman residuals, respectively, as we show below.

Original: Lemma 1 (Value decomposition for the main agent.) .
Revised: Lemma 1 (Value decomposition for the main agent.) .

Original: Let µ = µ f and ν be an arbitrary policy taken by the min-player.
Revised: Let µ = µ f and ν be an arbitrary policy taken by the min-player.

Original: It holds that
Revised: It holds that

Original: [Equation]
Revised: [Equation]

Original: Lemma 2 (Value decomposition for the booster agent.) .
Revised: Lemma 2 (Value decomposition for the booster agent.) .

Original: Suppose that µ = µ f is taken by the max-player and g is sampled from the posterior by the booster agent.
Revised: Suppose that µ = µ f is taken by the max-player and g is sampled from the posterior by the booster agent.

Original: Let ν be taken as in Algorithm 3.
Revised: Let ν be taken as in Algorithm 3.

Original: Then, it holds that
Revised: Then, it holds that

Original: [Equation]
Revised: [Equation]

Original: We remark that these two lemmas also account for the extra optimisticand pessimistic terms in the prior distributions.
Revised: We remark that these two lemmas also account for the extra optimistic terms in the prior distributions.

Original: The proofs of these two lemmas are deferred to Appendix.
Revised: The proofs of these two lemmas are deferred to Appendix.

Original: G.
Revised: G.

Original:  M ULTI -A GENT D ECOUPLING C OEFFICIENTS
Revised: 4. M ULTI -A GENT D ECOUPLING C OEFFICIENTS

Original: In the previous subsection, we convert the problem of bounding Reg( T ) to bounding the summation of Bellman residuals.
Revised: In the previous subsection, we convert the problem of bounding Reg( T ) to bounding the summation of Bellman residuals.

Original: However, the posterior distribution is more related to the squared Bellman residuals.
Revised: However, the posterior distribution is more related to the squared Bellman residuals.

Original: Therefore, we need some structural information to relate the growth of the cumulative
Revised: Therefore, we need some structural information to relate the growth of the cumulative

Original: Bellman residuals to the growth of the cumulative squared Bellman residuals.
Revised: Bellman residuals to the growth of the cumulative squared Bellman residuals.

Original: To this end, the multi-agent decoupling coefficient is introduced, which is an extension of the single-agent version in
Revised: To this end, the multi-agent decoupling coefficient is introduced, which is an extension of the single-agent version in

Original: Dann et al.
Revised: Dann et al.

Original: (2021), as follows.
Revised: (2021), as follows.

Original: Definition 1 (Multi-agent decoupling coefficient) .
Revised: Definition 1 (Multi-agent decoupling coefficient) .

Original: Given an MG ( H, X , A , B , P , r ) , a function class
Revised: Given an MG ( H, X , A , B , P , r ) , a function class

Original: F , a time horizon T , and a parameter µ > 0 , the multi-agent decoupling coefficient dc ( F , MG, T, µ ) is given by 
Revised: F , a time horizon T , and a parameter µ > 0 , the multi-agent decoupling coefficient dc ( F , MG, T, µ ) is given by the minimum integer K such that the following inequality holds

Original: [Equation]
Revised: [Equation]

Original: where π s is a policy pair ( µ f s , ν f s ,g s ) induced by ( f s , g s ) as in the Algorithms 2 and 3 .
Revised: where π s is a policy pair ( µ f s , ν f s ,g s ) induced by ( f s , g s ) as in the Algorithms 2 and 3 .

Original: The set of these distributions induced by f, g ∈ F is denoted as D F .
Revised: The set of these distributions induced by f, g ∈ F is denoted as D F .

Original: Equipped with the multi-agent decoupling coefficient, it remains to bound the cumulative squared Bellman residuals (cid:80) [Equation] − 
Revised: It remains to bound the cumulative squared Bellman residuals (cid:80) t − 1 s =1 (cid:2)

Original: [Equation]

Original: [Equation]

Original: E [Equation] E µ 
Revised: E π s E µ ft h

Original: [Equation]
Revised: [Equation]

Original: (cid:0) g t ; x h , a h , b h (cid:1)(cid:3) 2 by connecting it to the cumulative loss function L hµ ft ( g h , g h
Revised: g t ; x h , a h , b h (cid:1)(cid:3)by connecting it to log-likelihood function L hµ ft ( g h , g h

Original: +1 ;S t − 1 ) used in the posterior distributions. 4. M ORE I NTUITION
Revised: +1 ;

Original: The analysis starts with the decomposition in Eqn. equation 5 and handle each bart by the corresponding value decomposition lemma.
Revised: S t − 1 ) .

Original: Then, with the help of multi-agent decoupling coefficient, the techniques  in Dann et al.
Revised: Then, similar online aggregation techniques can be used as in Dann et al.

Original: (2021) can be applied with modifications.

Original: 4. C OMPLEXITY OF F
Revised: 4. C OMPLEXITY OF F

Original: For optimization-based algorithms, the complexity of the function class F is usually characterized through the cardinality |F| or the covering number Jiang et al.
Revised: For optimization-based algorithms, the complexity of the function class F is usually characterized through the cardinality |F| or the covering number Jiang et al.

Original: ; Wang et al.
Revised: ; Wang et al.

Original: (2020); Jin et al.
Revised: (2020); Jin et al.

Original: (2021a;b); Huang et al.
Revised: (2021a;b); Huang et al.

Original: On the other hand, the posterior sampling algorithm employs a prior distribution p 0 over F , which allows the algorithm to favor certain parts of it.
Revised: On the other hand, the posterior sampling algorithm employs a prior distribution p 0 over F , which allows the algorithm to favor certain parts of it.

Original: Accordingly, our theoretical result depends on the complexity of F through the prior preference which is characterized by the following quantity.
Revised: Accordingly, our theoretical result depends on the complexity of F through the prior preference which is characterized by the following quantity.

Original: Definition 2.
Revised: Definition 2.

Original: For a policy µ f , f ∈ F and for any function
Revised: For a policy µ f , f ∈ F and for any function

Original: g ′ ∈ F h +1 , we define
Revised: g ′ ∈ F h +1 , we define

Original: [Equation]
Revised: [Equation]

Original: containing the functions that have small T µ f h -Bellman error against g ′ for all state-action pairs.
Revised: containing the functions that have small T µ f h -Bellman error against g ′ for all state-action pairs.

Original: We then define
Revised: We then define

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Under Assumption 2, it is assumed that κ ( ϵ ) < ∞ , which is supported by the following two examples.
Revised: Under Assumption 2, it is assumed that κ ( ϵ ) < ∞ , which is supported by the following two examples.

Original: For the finite function class with completeness, with a uniform prior p h 0 ( f ) = 1 / |F h | , we have
Revised: For the finite function class with completeness, with a uniform prior p h 0 ( f ) = 1 / |F h | , we have

Original: [Equation]
Revised: [Equation]

Original: due to the Realizability assumption.
Revised: due to the Realizability assumption.

Original: For an infinite function class, by replacing |F| with its covering number, similar result can also be ontained.
Revised: For an infinite function class, by replacing |F| with its covering number, similar result can also be ontained.

Original: For a d -dimensional Lipschitz parametric models
Revised: For a d -dimensional Lipschitz parametric models

Original: F h = { g θ ∈ R
Revised: F h = { g θ ∈ R

Original: d : θ ∈ Ω h } where Ω h is compact, we can generally assume that
Revised: d : θ ∈ Ω h } where Ω h is compact, we can generally assume that

Original: [Equation]
Revised: [Equation]

Original: for some constants c 0 and c 1 depending on the prior and the Lipschitz constant (Dann et al., 2021).
Revised: for some constants c 0 and c 1 depending on the prior and the Lipschitz constant (Dann et al., 2021).

Original:  M AIN R ESULTS
Revised:  M AIN R ESULTS

Original: In this section, we state the main theoretical result of this paper and interpret it using several examples.
Revised: In this section, we state the main theoretical result of this paper and interpret it using several examples.

Original: 5. T HEORETICAL G UARANTEE
Revised: 5. T HEORETICAL G UARANTEE

Original: We now provide an upper bound for the overall regret.
Revised: We now provide an upper bound for the overall regret.

Original: Theorem 1 (Overall regret) .
Revised: Theorem 1 (Overall regret) .

Original: Let Assumptions 1, 2 and 3 hold.
Revised: Let Assumptions 1, 2 and 3 hold.

Original: If ηβ 2 ≤ 0 .
Revised: If ηβ 2 ≤ 0 .

Original: 5 and λβ 2 ≥ 1 hold, and let dc ( F , MG, T ) be an upper bound for the sup µ ≤ 1 dc ( F , MG, T, µ ) , and we further take λ = (cid:114)
Revised: 5 and λβ 2 ≥ 1 hold, and let dc ( F , MG, T ) be an upper bound for the sup µ ≤ 1 dc ( F , MG, T, µ ) , and we further take λ = (cid:114)

Original: Tκ ( βT 2 ) β 2 dc ( F ,MG,T ) , η = 14 β 2 , then, it holds that
Revised: [Equation] ) , η = 14 β 2 , then, it holds that

Original: [Equation]
Revised: [Equation]

Revised: We shall use a sufficiently small learning rate η the condition about λ is easily to satisfy for large T .

Original: Theorem 1 can be established once we can bound the main agent part and the booster agent part.
Revised: Theorem 1 can be established once we can bound the main agent part and the booster agent part.

Original: Theorem 2 (Bound of the main agent) .
Revised: Theorem 2 (Bound of the main agent) .

Original: With the same conditions as Theorem 1, it holds that
Revised: With the same conditions as Theorem 1, it holds that

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Original: The regret of the booster agent is induced by approximating the best response policy.
Revised: The regret of the booster agent is induced by approximating the best response policy.

Original: Theorem 3 (Bound of the booster agent) .
Revised: Theorem 3 (Bound of the booster agent) .

Original: With the same conditions as Theorem 1, it holds that
Revised: With the same conditions as Theorem 1, it holds that

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Original: The detailed proofs can be found in the appendix.
Revised: The detailed proofs can be found in the appendix.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, a self-play posterior sampling algorithm is proposed for two-player zero-sum Markov games with general function approximation, which is the first to the best of our knowledge.
Revised: In this paper, a self-play posterior sampling algorithm is proposed for two-player zero-sum Markov games with general function approximation, which is the first to the best of our knowledge.

Original: A new complexity measure, multi-agent decoupling coefficient , is introduced to characterize the complexity of function class.
Revised: A new complexity measure, multi-agent decoupling coefficient , is introduced to characterize the complexity of function class.

Original: Rigorous theoretical analysis showed that the proposed algorithm can achieve comparable regret bounds compared with other OFU-based algorithms for problems with low multiagent decoupling coefficient, which extends the results in the single-agent RL.
Revised: Rigorous theoretical analysis showed that the proposed algorithm can achieve comparable regret bounds compared with other OFU-based algorithms for problems with low multiagent decoupling coefficient, which extends the results in the single-agent RL.

Original: As existing algorithms with general function approximation are computationally inefficient in general, one important direction for future works is to design computationally tractable algorithms for MGs (and MDPs).
Revised: As existing algorithms with general function approximation are computationally inefficient in general, one important direction for future works is to design computationally tractable algorithms for MGs (and MDPs).

Original: Another interesting open question is how to extend the posterior sampling algorithms for general-sum Markov games.
Revised: Another interesting open question is how to extend the posterior sampling algorithms for general-sum Markov games.

Original: A R ELATED W ORK
Revised: A R ELATED W ORK

Original: There have been a lot of works focusing on designing provably efficient algorithms for zero-sum
Revised: There have been a lot of works focusing on designing provably efficient algorithms for zero-sum

Original: MGs.
Revised: MGs.

Original: For the tabular setting, Bai et al.
Revised: For the tabular setting, Bai et al.

Original: (2020); Bai & Jin (2020);
Revised: (2020); Bai & Jin (2020);

Original: Liu et al.
Revised: Liu et al.

Original: (2020) provide
Revised: (2020) provide

Original: O ( poly ( |X| , |A| , |B| , H ) · √ T ) regret guarantees for the proposed algorithms, where |X| is the number of states, |A| and |B| are the number of action spaces of two players, respectively, H is the episode length and T is the number of episodes.
Revised: O ( poly ( |X| , |A| , |B| , H ) · √ T ) regret guarantees for the proposed algorithms, where |X| is the number of states, |A| and |B| are the number of action spaces of two players, respectively, H is the episode length, and T is the number of episodes.

Original: Then, Xie et al.
Revised: Then, Xie et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: (2021) study two linear-type MGs and design algorithms with O ( poly ( d, H ) · √ T ) regret, where d is the dimension of the linear features.
Revised: (2021) study two linear-type MGs and design algorithms with O ( poly ( d, H ) · √ T ) regret, where d is the dimension of the linear features.

Original: Recently, Jin et al.
Revised: Recently, Jin et al.

Original: (2021b)
Revised: (2021b)

Original: ; Huang et al.
Revised: ; Huang et al.

Original: (2021) further propose efficient algorithms for zero-sum MGs with general function approximation.
Revised: (2021) further propose efficient algorithms for zero-sum MGs with general function approximation.

Original: Our work is also closely related to another line of work on posterior sampling algorithms.
Revised: Our work is also closely related to another line of work on posterior sampling algorithms.

Original: In the context of contextual bandit, due to the impressive empirical performance of Thompson Sampling, there has been significant efforts in developing its theoretical analysis, including Russo & Van Roy (2014) in the form of Bayesian regret and Kaufmann et al.
Revised: In the context of contextual bandit, due to the impressive empirical performance of Thompson Sampling, there has been significant efforts in developing its theoretical analysis, including Russo & Van Roy (2014) in the form of Bayesian regret and Kaufmann et al.

Original: (2012); Zhang (2021) in the frequentist setting.
Revised: (2012); Zhang (2021) in the frequentist setting.

Original: For the Markov Decision Process (MDP), the seminal work Osband & Van Roy (2014) considers the Bayesian regret and proposes a general posterior sampling RL method.
Revised: For the Markov Decision Process (MDP), the seminal work Osband & Van Roy (2014) considers the Bayesian regret and proposes a general posterior sampling RL method.

Original: The randomized least-squares value iteration (RLSVI) algorithm
Revised: The randomized least-squares value iteration (RLSVI) algorithm

Original: Osband et al.
Revised: Osband et al.

Original: (2016) is shown to admit frequentist regret bounds for tabular MDP Russo (2019);
Revised: (2016) is shown to admit frequentist regret bounds for tabular MDP Russo (2019);

Original: Agrawal et al.
Revised: Agrawal et al.

Original: Xiong et al.
Revised: Xiong et al.

Original: (2021) and linear
Revised: (2021) and linear

Original: MDP Zanette et al.
Revised: MDP Zanette et al.

Original: Beyond the linear setting, a recent work Dann et al.
Revised: Beyond the linear setting, a recent work Dann et al.

Original: (2021) proposes a conditional posterior sampling algorithm to solve the MDP with general function approximation.
Revised: (2021) proposes a conditional posterior sampling algorithm to solve the MDP with general function approximation.

Original: A recent posterior-sampling-type work by Jafarnia-Jahromi et al.
Revised: A recent posterior-sampling-type work by Jafarnia-Jahromi et al.

Original: (2021) considers the infinite-horizon zero-sum MGs with average-reward criterion in the tabular setting, with a focus on the Bayesian regret, whose analysis technique is fundamentally different from ours.
Revised: (2021) considers the infinite-horizon zero-sum MGs with average-reward criterion in the tabular setting, with a focus on the Bayesian regret, whose analysis technique is fundamental different from ours.

Original: To the best of our knowledge, there is no posterior sampling algorithm with a frequentist regret guarantee to date.
Revised: To the best of our knowledge, there is no posterior sampling algorithm with a frequentist regret guarantee to date.

Original: B E QUIVALENT A LGORITHMS
Revised: B E QUIVALENT A LGORITHMS

Original: We will consider a slightly more general posterior sampling algorithm with an extra parameter α ∈ (0 , 1] .
Revised: We will consider a slightly more general posterior sampling algorithm with an extra parameter α ∈ (0 , 1] .

Original: We recall that the posterior defined inEqn. equation 4 is
Revised: We recall that the posterior defined in equation 4 is

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Equivalently, we may consider the excess loss
Revised: Equivalently, we may consider the excess loss

Original: [Equation]
Revised: [Equation]

Original: where we employ the notation that ζ s
Revised: where we employ the notation that ζ s

Revised: = { [ x hs

Revised: [Equation]

Revised: a hs

Original: = { [ x hs , a hs , b hs , r hs ] } Hh =1 .
Revised: [Equation] b hs [Equation] r hs ] } Hh =1 .

Original: We then define the potential function as
Revised: We then define the potential function as

Original: [Equation]
Revised: [Equation]

Original: where α ∈ (0 , 1] is the extra parameter to facilitate the proof.
Revised: where α ∈ (0 , 1] is the extra parameter to facilitate the proof.

Original: We also define
Revised: We also define

Original: [Equation]
Revised: [Equation]

Original: Then [Equation] we obtain a generalized posterior distribution on F :
Revised: Then, we obtain a generalized posterior distribution on F :

Original: [Equation]
Revised: [Equation]

Original: where it is equivalent to the posterior given inEqn. equation 4 when α = 1 .
Revised: where it is equivalent to the posterior given in equation 4 when α = 1 .

Original: We then recall the posterior distribution of the booster agent defined inEqn. equation 7 is given byp
Revised: We then recall the posterior distribution of the booster agent defined in equation 7 is given by

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Similarly, we define the excess loss for the booster agent:
Revised: Similarly [Equation] we define the excess loss for the booster agent:

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: and use the following notation (with slight abuse of notation) for the potential function:
Revised: and use the following notation (with slight abuse of notation) for the potential function:

Original: [Equation]
Revised: [Equation]

Original: since the analyses for Algorithm 2 and Algorithm 3 are separate so the meaning of Φ ht ( · ) will be clear from the context.
Revised: since the analyses for Algorithm 2 and Algorithm 3 are separate so the meaning of Φ ht ( · ) will be clear from the context.

Original: Finally [Equation] we obtain a generalized posterior function for the booster agent:
Revised: Finally, we obtain a generalized posterior function for the booster agent:

Original: [Equation]
Revised: [Equation]

Original: The main motivation to use ∆ L h ( · ) ( ∆ L hµ ( · ) ) is that the variance will be cancelled during our theoretical analysis as it is equivalent to the case where we know the Bellman operator.
Revised: The main motivation to use ∆ L h ( · ) ( ∆ L hµ ( · ) ) is that the variance will be cancelled during our theoretical analysis as it is equivalent to the case where we know the Bellman operator.

Original: This is possible because the novel denominator term is introduced in the likelihood function as in Dann et al.
Revised: This is possible because the novel denominator term is introduced in the likelihood function as in Dann et al.

Original: C U SEFUL L EMMAS AND A DDITIONAL N OTATIONS
Revised: C U SEFUL L EMMAS AND A DDITIONAL N OTATIONS

Original: In this section, we provide several useful lemmas and additional notations that are useful later.
Revised: In this section, we provide several useful lemmas and additional notations that are useful later.

Original: We start with the following definitions.
Revised: We start with the following definitions.

Original: First, we further define a quantity similar to Definition 2, which will be used for the analysis of the main agent.
Revised: First, we further define a quantity similar to Definition 2, which will be used for the analysis of the main agent.

Original: Definition 3.
Revised: Definition 3.

Original: For any f ′ ∈ F h +1 , we define the set
Revised: For any f ′ ∈ F h +1 , we define the set

Original: [Equation]
Revised: [Equation]

Original: containing the functions that have small T h -Bellman error against f ′ for all state-action pairs.
Revised: containing the functions that have small T h -Bellman error against f ′ for all state-action pairs.

Original: We then define the quantity
Revised: We then define the quantity

Original: [Equation]
Revised: [Equation]

Original: which is the probability assigned by the prior to functions that approximately satisfy the Bellman equation w.r.t.
Revised: which is the probability assigned by the prior to functions that approximately satisfy the Bellman equation w.r.t.

Original: f for all state-action pair.
Revised: f for all state-action pair.

Original: Note that κ 1 ( ϵ ) ≤ κ ( ϵ ) because
Revised: Note that κ 1 ( ϵ ) ≤ κ ( ϵ ) because

Original: [Equation]
Revised: [Equation]

Original: Definition 4.
Revised: Definition 4.

Original: For α ∈ (0 , 1) , we also use the notations:
Revised: For α ∈ (0 , 1) , we also use the notations:

Original: [Equation]
Revised: [Equation]

Original: and κ h 1 (1 , ϵ ) =
Revised: and κ h 1 (1 , ϵ ) =

Original: lim α → [Equation] − κ h ( α, ϵ ) where it holds that
Revised: lim α → 1 − κ h ( α, ϵ ) where it holds that

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Similarly, we define
Revised: Similarly, we define

Original: [Equation]
Revised: [Equation]

Original: and κ hµ (1 , ϵ ) =
Revised: and κ hµ (1 , ϵ ) =

Original: lim α → 1 − κ hµ ( α, ϵ ) , Then, it holds that
Revised: lim α → 1 − κ hµ ( α, ϵ ) , Then, it holds that

Original: [Equation]
Revised: [Equation]

Original: andκ
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Lemma 3.
Revised: Lemma 3.

Original: For any fixed g ∈ F and max-player’s policy µ := µ f for some f ∈ F , we define a random variable for all s and h as follows:
Revised: For any fixed g ∈ F and max-player’s policy µ := µ f for some f ∈ F , we define a random variable for all s and h as follows:

Original: [Equation]
Revised: [Equation]

Revised: s (cid:1)(cid:1) .

Original: Then, for all h , we have
Revised: Then, for all h , we have

Original: [Equation]
Revised: [Equation]

Original: A special case is that f = g where we have
Revised: A special case is that f = g where we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: This lemma is from Zhang (2005) and is also proved in Dann et al.
Revised: This lemma is from Zhang (2005) and is also proved in Dann et al.

Original: Lemma 4.
Revised: Lemma 4.

Original: Let ν be a probability distribution.
Revised: Let ν be a probability distribution.

Original: Then, E ν f − H ( ν ) is minimized at ν ( x ) ∝ exp( − f ( x )) .
Revised: Then, E ν f − H ( ν ) is minimized at ν ( x ) ∝ exp( − f ( x )) .

Original: Proof.
Revised: Proof.

Original: This is a corollary of Gibbs variational principle whose proof can be found in Van Handel (2014), Lemma 4.10.
Revised: This is a corollary of Gibbs variational principle whose proof can be found in Van Handel (2014), Lemma 4.10.

Original: Using Lemma [Equation] we can obtain the following key lemma as used in Dann et al.
Revised: Using Lemma 4, we can obtain the following key lemma as used in Dann et al.

Original: Lemma 5.
Revised: Lemma 5.

Original: It holds that
Revised: It holds that

Original: [Equation]
Revised: [Equation]

Original: where we remark that the definitions of Φ ht ( · ) in two equations are different.
Revised: where we remark that the definitions of Φ ht ( · ) in two equations are different.

Original: In what follows, we derive a lower bound of LHS ofEqn. equation 14, and an upper bound of RHS of Eqn
Revised: In what follows, we derive a lower bound of LHS of equation 14, and an upper bound of RHS of equation 14 for the proof of Theorems 2 and 3.

Original: D P ROOF OF THE T HEOREM
Revised: D P ROOF OF THE T HEOREM

Original: In this section, we provide the proof for Theorem 2.
Revised: In this section, we provide the proof for Theorem 2.

Original: The proof provided in this section basically follows the same line of that of single-agent RL because essentially the algorithms employ the same properties of the problem as discussed in Section 4 and for the main agent, and the Bellman residuals is free of the min-player’s policy.
Revised: The proof provided in this section basically follows the same line of that of single-agent RL because essentially the algorithms employ the same properties of the problem as discussed in Section 4 and for the main agent, and the Bellman residuals is free of the min-player’s policy.

Original: Lemma 6.
Revised: Lemma 6.

Original: For all functions f ∈ F , we have
Revised: For all functions f ∈ F , we have

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We define the random variable
Revised: We define the random variable

Original: [Equation]
Revised: [Equation]

Original: Let E be conditioned on [ x h s , a hs , b h s ] .
Revised: Let E be conditioned on [ x h s , a hs , b h s ] .

Original: Then, the randomness is from the state transition and we have
Revised: Then, the randomness is from the state transition and we have

Original: [Equation]
Revised: [Equation]

Original: We also have
Revised: We also have

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: E [

Original: [Equation]
Revised: [Equation]

Original: Also note that Z ∈ [ − β, β − 1] and max Z − min Z ≤ β if it is conditioned on [ x hs , a hs , b hs ] , this implies that
Revised: Also note that Z ∈ [ − β, β − 1] and max Z − min Z ≤ β if it is conditioned on [ x hs , a hs , b hs ] , this implies that

Original: [Equation]
Revised: [Equation]

Original: Lemma 7.
Revised: Lemma 7.

Original: If the learning rate η is sufficiently small such that ηβ 2 ≤ 0 . 8 , then for all functions f ∈ F , we have
Revised: If the learning rate η is sufficiently small such that ηβ 2 ≤ 0 . 8 , then for all functions f ∈ F , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: With ηβ 2 ≤ 0 .
Revised: With ηβ 2 ≤ 0 .

Original: [Equation]
Revised: [Equation]

Original: for all f ∈ F
Revised: for all f ∈ F

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: This implies that
Revised: This implies that

Original: [Equation]
Revised: [Equation]

Original: where we use the fact
Revised: where we use the fact

Original: that ψ ( z ) =
Revised: that ψ ( z ) =

Original: ( e z − 1 − z )
Revised: ( e z − 1 − z )

Original: /z 2 is increasing in z and ψ (0 .
Revised: /z 2 is increasing in z and ψ (0 .

Original: 8) < 0 .
Revised: 8) < 0 .

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: where the first inequality is because ln z ≤ z − 1 and the last inequality is because Lemma 6 and ( 4 3 ηb 2 0 .
Revised: where the first inequality is because ln z ≤ z − 1 and the last inequality is because Lemma 6 and ( 4 3 ηb 2 0 .

Original: 67) ≤ 0 .
Revised: 67) ≤ 0 .

Original: Lemma 8.
Revised: Lemma 8.

Original: It holds that
Revised: It holds that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Consider any fixed f ∈ F .
Revised: Consider any fixed f ∈ F .

Original: For any ˜ f h ∈ F h that depends on S s − 1 , we obtain from Lemmathat
Revised: For any ˜ f h ∈ F h that depends on S s − 1 , we obtain from Lemmathat

Original: [Equation]
Revised: [Equation]

Original: We let
Revised: let

Original: W ht :
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: and recall that
Revised: and recall that

Original: [Equation]
Revised: [Equation]

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: where we use ln z ≤ z − 1 .
Revised: where we use ln z ≤ z − 1 .

Original: By W h 0 =
Revised: By W h 0 =

Original: [Equation]
Revised: [Equation]

Original: we know that
Revised: we know that

Original: [Equation]
Revised: [Equation]

Original: equivalently,
Revised: equivalently,

Original: [Equation]
Revised: [Equation]

Original: This implies that for any p ( · ) , we have
Revised: This implies that for any p ( · ) , we have

Original: [Equation]
Revised: [Equation]

Original: where we use the definition of the potential function in first equality.
Revised: where we use the definition of the potential function in first equality.

Original: Since p ( · ) is arbitrary, we can take f h ∈ F h ( ϵ, Q ∗ h +1 ) for all h ∈
Revised: Since p ( · ) is arbitrary, we can take f h ∈ F h ( ϵ, Q ∗ h +1 ) for all h ∈

Original: [ H ] .
Revised: [ H ] .

Original: We then have | f h ( x, a, b ) − Q ∗ h ( x, a, b ) | = | f h ( x, a, b ) − T Q ∗ h
Revised: We then have | f h ( x, a, b ) − Q ∗ h ( x, a, b ) | = | f h ( x, a, b ) − T Q ∗ h

Original: +

Revised: +

Original: ( x, a, b ) | ≤ ϵ
Revised: ( x, a, b ) | ≤ ϵ

Original: [Equation]
Revised: [Equation]

Original: for all ( x, a, b, h ) ∈
Revised: for all ( x, a, b, h ) ∈

Original: X ×
Revised: X ×

Original: A × B × [ H ] .
Revised: A × B × [ H ] .

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: where we use
Revised: where we use

Original: [Equation]
Revised: [Equation]

Original: where the first inequality is because of
Revised: where the first inequality is because of

Original: [Equation]
Revised: [Equation]

Original: By taking p ( f ) = p 0 ( f ) I ( f ∈ F ( ϵ )) /p 

Revised: By taking p ( f ) = p 0 ( f ) I ( f ∈ F ( ϵ )) /p

Original: ( F ( ϵ )) , with F ( ϵ ) = (cid:81)
Revised: ( F ( ϵ )) , with F ( ϵ ) = (cid:81)

Original: h F h ( ϵ, Q ∗ h +1 ) , we obtain the desired result
Revised: h F h ( ϵ, Q ∗ h +1 ) , we obtain the desired result

Original: [Equation]
Revised: [Equation]

Original: Lemma 9.
Revised: Lemma 9.

Original: It holds that
Revised: It holds that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: To show the first inequality
Revised: To show the first inequality

Original: [Equation]
Revised: [Equation]

Original: we just subtract all terms of RHS from LHS to see that it is a
Revised: we just subtract all terms of RHS from LHS to see that it is a

Original: KL-divergence which is non-negative.
Revised: KL-divergence which is non-negative.

Original: The second inequality is equivalent to
Revised: The second inequality is equivalent to

Original: [Equation]
Revised: [Equation]

Original: This follows from
Revised: This follows from

Original: [Equation]
Revised: [Equation]

Original:  number ) and
Revised: a odd number ) and

Original: [Equation]
Revised: [Equation]

Original: an even number ) which is a result of the non-negativity of mutual information
Revised: an even number ) which is a result of the non-negativity of mutual information

Original: [Equation]
Revised: [Equation]

Original: Lemma 10.
Revised: Lemma 10.

Original: It holds that
Revised: It holds that

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We use the definition of the potential function and apply Lemma 9.
Revised: We use the definition of the potential function and apply Lemma 9.

Original: The desired result follows from some calculations.
Revised: The desired result follows from some calculations.

Original: Lemma 11.
Revised: Lemma 11.

Original: If ηβ 2 ≤ 0 .
Revised: If ηβ 2 ≤ 0 .

Revised: [Equation]

Original: 4 , it holds that
Revised: it holds that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The bound of A comes from the non-negativity of KL-divergence and α ∈ (0 , 1] .
Revised: The bound of A comes from the non-negativity of KL-divergence and α ∈ (0 , 1] .

Original: To prove the lower bound of B h , we define
Revised: To prove the lower bound of B h , we define

Original: [Equation]
Revised: [Equation]

Original: Then, for all h ∈ [ H ]
Revised: Then, for all h ∈ [ H ]

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: according to Lemma 3.
Revised: according to Lemma 3.

Original: Then [Equation] by Lemma
Revised: Then, by Lemma

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: , where we use the fact that Lemma 4 implies that the inf is achieved at
Revised: , where we use the fact that Lemma 4 implies that the inf is achieved at

Original: [Equation]
Revised: [Equation]

Original: and the expectation is equal to− E p
Revised: and the expectation is equal to

Original: [Equation]
Revised: [Equation]

Original: where c = E [Equation] ∼ 
Revised: where c = E f h ∼ p

Original: [Equation]
Revised: [Equation]

Original: E f h + [Equation] ∼ p h 
Revised: E f h +1 ∼ p h +

Original: [Equation]
Revised: [Equation]

Original: exp (cid:16)(cid:80) t − 1 s =1 ξ hs (cid:0) f h , f h +1 , ζ s (cid:1)(cid:17) is the normalized constant.
Revised: exp (cid:16)(cid:80) t − 1 s =1 ξ hs (cid:0) f h , f h +1 , ζ s (cid:1)(cid:17) is the normalized constant.

Original: It then follows that
Revised: It then follows that

Original: [Equation]
Revised: [Equation]

Revised: t (cid:34)

Revised: [Equation]

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Revised: p

Revised: [Equation]

Revised: [Equation]

Original:  where we use the above result in the first inequality and use the convexity of − ln( · ) in the last inequality.
Revised: [Equation] [Equation] [Equation] where we use the above result in the first inequality and use the convexity of − ln( · ) in the last inequality.

Original: We then have
Revised: We then have

Original: [Equation]
Revised: [Equation]

Original: where we use the definition of ξ hs (cid:0) f h , f h
Revised: where we use the definition of ξ hs (cid:0) f h , f h

Original: +1 , ζ s (cid:1) in the first inequality and we use Lemma 7 in the last step.
Revised: +1 , ζ s (cid:1) in the first inequality and we use Lemma 7 in the last step.

Original: We now turn to the lower bound of C h .
Revised: We now turn to the lower bound of C h .

Original: We have
Revised: We have

Original: E 
Revised: E f ∼

Original: [Equation]
Revised: [Equation]

Original: where we use the fact that the inf is achieved at
Revised: where we use the fact that the inf is achieved at

Original: [Equation]
Revised: [Equation]

Original: We now consider a fixed f h ∈ F h ( ϵ, f h +1 ) .
Revised: We now consider a fixed f h ∈ F h ( ϵ, f h +1 ) .

Original: It holds that
Revised: It holds that

Original: [Equation]
Revised: [Equation]

Original: To show this, we recall the definition
Revised: To show this [Equation] we recall the definition

Original: [Equation]
Revised: [Equation]

Original: f h +1 ; ζ s (cid:1) =
Revised: h +1 ; ζ s (cid:1) =

Original: (cid:0) f h (cid:0) x hs , a hs , b hs (cid:1) − r hs − V f,h +1 (cid:0) x
Revised: (cid:0) f h (cid:0) x hs , a hs , b hs (cid:1) − r hs − V f,h +1 (cid:0) x

Original: h +1 s (cid:1)(cid:1)− (cid:0) T
Revised: h +1 s (cid:1)(cid:1)− (cid:0) T

Original: h f h +1 (cid:0) x hs , a hs , b hs (cid:1) − r hs − V f,h +1 (cid:0) x
Revised: h f h +1 (cid:0) x hs , a hs , b hs (cid:1) − r hs − V f,h +1 (cid:0) x

Original: h +1 s (cid:1)(cid:1) 2 ,
Revised: h +1 s (cid:1)(cid:1) 2 ,

Original: and we subtract and add T h f h +

Revised: and we subtract and add T h f h +

Original: ( x h s , a h s ) inside the first term to obtain ∆ L h ( f h , f h +1 , ζ s ) =
Revised: ( x h s , a h s ) inside the first term to obtain ∆ L h ( f h , f h +1 , ζ s ) =

Original: E h ( f, x hs , a hs ) 2 + 2 E h ( f, x hs , a hs )( T ∗ h f h +1 ( x hs , a hs ) − r hs −
Revised: E h ( f, x hs , a hs ) 2 + 2 E h ( f, x hs , a hs )( T ∗ h f h +1 ( x hs , a hs ) − r hs −

Original: f h +

Revised: f h +

Original: ( x h +1 s )) .
Revised: ( x h +1 s )) .

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Original: Thus, we have
Revised: Thus, we have

Original: [Equation]
Revised: [Equation]

Original: where we use the definition
Revised: where we use the definition

Original: [Equation]
Revised: [Equation]

Original: α

Original: [Equation]

Original: [Equation]

Original: We are ready to prove Theorem 2.
Revised: We are ready to prove Theorem 2.

Original: Proof of Theorem 
Revised: Proof of Theorem 2.

Original: [Equation]

Original: Let π t denote the distribution induced by µ t × ν t and define
Revised: Let π t denote the distribution induced by µ t × ν t and define

Original: [Equation]
Revised: [Equation]

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: For arbitrary ν t induced by µ f t and g t
Revised: For arbitrary ν t induced by µ f t and g t

Original: [Equation]
Revised: [Equation]

Original: according to the value-decomposition Lemma 1 we have
Revised: according to the value-decomposition Lemma 1 we have

Original: [Equation]
Revised: [Equation]

Original: where the first inequality also uses the definition of ∆ f 1 t ( x 1 ) ; the second inequality comes from
Revised: where the first inequality also uses the definition of ∆ f 1 t ( x 1 ) ; the second inequality comes from

Original: Lemma 10 and Lemma 11; the equality is because Lemma 5, and the last step comes from Lemma 8.
Revised: Lemma 10 and Lemma 11; the equality is because Lemma 5, and the last step comes from Lemma 8.

Original: Summing over t , we obtain that
Revised: Summing over t , we obtain that

Original: [Equation]
Revised: [Equation]

Original: ≤O
Revised: ≤

Original: [Equation]

Original: β

Original: [Equation]
Revised: [Equation]

Original: dc

Original: [Equation]

Original: F , MG

Original: [Equation]

Original: T

Original: [Equation]

Original: T

Original: [Equation]

Original: T

Original: [Equation]

Original: Here in the last step
Revised: Here in the last step

Original: [Equation]
Revised: [Equation]

Original: we first let α → 1 − and note that
Revised: we first let α → 1 − and note that

Original: [Equation]
Revised: [Equation]

Original: ln p h 0 (cid:0) F ( ϵ, Q ∗ h +1 (cid:1)
Revised: p h 0 (cid:0) F ( ϵ, Q ∗ h +1 (cid:1)

Original: +
Revised: +

Original: H (cid:88) h
Revised: H (cid:88) h

Original: = κ h 1 (1 , ϵ ) ≤ 2 κ ( ϵ ) .
Revised: = κ h 1 (1 , ϵ ) ≤ 2 κ ( ϵ ) .

Original: Then, we take ϵ = βT 2 , λ = (cid:113)
Revised: Then, we take ϵ = βT 2 , λ = (cid:113)

Original: Tκ ( β/T 2 ) β 2 dc ( F ,MG,T ) , and η = 14 β 2 .
Revised: Tκ ( β/T 2 ) β 2 dc ( F ,MG,T ) , and η = 14 β 2 .

Original: This concludes the proof.
Revised: This concludes the proof.

Original: E P ROOF OF THE T HEOREM
Revised: E P ROOF OF THE T HEOREM

Original: In this section, we provide a proof for Theorem 3.
Revised: In this section, we provide a proof for Theorem 3.

Original: Lemma 12.
Revised: Lemma 12.

Original: For any max-player’s policy µ and all functions g ∈ F , we have
Revised: For any max-player’s policy µ and all functions g ∈ F , we have

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof of this lemma only employs the Markov property of the transition and the range of function g ∈ F .
Revised: The proof of this lemma only employs the Markov property of the transition and the range of function g ∈ F .

Original: By replacing the notations in the proof of Lemma 6, we conclude the proof.
Revised: By replacing the notations in the proof of Lemma 6, we conclude the proof.

Original: Lemma 13.
Revised: Lemma 13.

Original: Letting ηβ 2 ≤ 0 .
Revised: Letting ηβ 2 ≤ 0 .

Original: 8 , then for all functions g ∈ F and any max-player’s policy µ , we have
Revised: 8 , then for all functions g ∈ F and any max-player’s policy µ , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof of this lemma only employs the range of function g ∈ F .
Revised: The proof of this lemma only employs the range of function g ∈ F .

Original: By replacing the notations in the proof of Lemma 7, we conclude the proof.
Revised: By replacing the notations in the proof of Lemma 7, we conclude the proof.

Original: Lemma 14.
Revised: Lemma 14.

Original: It holds that
Revised: It holds that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Consider any fixed g ∈ F .
Revised: Consider any fixed g ∈ F .

Original: For any ˜ g h ∈ F h that depends on S s − 1 and µ f s , and for any µ f we obtain from Lemma 7 that
Revised: For any ˜ g h ∈ F h that depends on S s − 1 and µ f s , and for any µ f we obtain from Lemma 7 that

Original: [Equation]
Revised: [Equation]

Original: We now fix some t .
Revised: We now fix some t .

Original: For all s ≤ t , we define
Revised: For all s ≤ t , we define

Original: [Equation]
Revised: [Equation]

Original: and recall that
Revised: and recall that

Original: [Equation]
Revised: [Equation]

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: where we use ln z ≤ z − 1 .
Revised: where we use ln z ≤ z − 1 .

Original: By W h 0 = 0 , we know that
Revised: By W h 0 = 0 , we know that

Original: [Equation]
Revised: [Equation]

Original: equivalently,
Revised: equivalently,

Original: [Equation]
Revised: [Equation]

Original: Note that t is arbitrary.
Revised: Note that t is arbitrary.

Original: This implies that for any p ( · ) and any t , we have
Revised: This implies that for any p ( · ) and any t , we have

Original: [Equation]
Revised: [Equation]

Original: Since p ( · ) is arbitrary [Equation] we can take g h ∈ F µh ( ϵ, Q µ, † h +1 ) for all h ∈ [ H ] .
Revised: Since p ( · ) is arbitrary, we can take g h ∈ F µh ( ϵ, Q µ, † h +1 ) for all h ∈ [ H ] .

Original: We need to show that g h admits a small T µh -Bellman-residual.
Revised: We need to show that g h admits a small T µh -Bellman-residual.

Original: We have
Revised: We have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Revised: , b

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: for all ( x, a, b, h ) ∈
Revised: for all ( x, a, b, h ) ∈

Original: X ×
Revised: X ×

Original: A × B × [ H ] .
Revised: A × B × [ H ] .

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: where we use
Revised: where we use

Original: [Equation]
Revised: [Equation]

Original: where we use the fact that
Revised: where we use the fact that

Original: [Equation]
Revised: [Equation]

Original: By taking p ( f ) = p 0 ( f ) I ( f ∈ F ( ϵ, µ f )) /p 

Revised: By taking p ( f ) = p 0 ( f ) I ( f ∈ F ( ϵ, µ f )) /p

Original: ( F ( ϵ, µ f )) , with F ( ϵ, µ f ) =
Revised: ( F ( ϵ, µ f )) , with F ( ϵ, µ f ) =

Original: (cid:81)
Revised: (cid:81)

Original: h F µ f h ( ϵ, Q µ f , † h +1 ) , we obtain the desired result.
Revised: h F µ f h ( ϵ, Q µ f , † h +1 ) [Equation] we obtain the desired result.

Original: Lemma 15.
Revised: Lemma 15.

Original: For any max-player’s policy µ that is induced by some f ∈ F , we have
Revised: For any max-player’s policy µ that is induced by some f ∈ F [Equation] we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof of this lemma only relies on the non-negativity of mutual information and KLdivergence.
Revised: The proof of this lemma only relies on the non-negativity of mutual information and KLdivergence.

Original: By replacing the notations in the proof of Lemma 9, we conclude the proof.
Revised: By replacing the notations in the proof of Lemma 9, we conclude the proof.

Original: Lemma 16.
Revised: Lemma [Equation]

Original: It holds that
Revised: It holds that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We use the definition of the potential function and apply Lemma 15 (note that it is valid for any µ f , f ∈ F )
Revised: We use the definition of the potential function and apply Lemma 15 (note that it is valid for any µ f , f ∈ F )

Original: [Equation]
Revised: [Equation]

Original: Lemma 17.
Revised: Lemma 17.

Original: If ηβ 2 ≤ 0 .
Revised: If ηβ 2 ≤ 0 .

Original: 4 , it holds that
Revised: 4 , it holds that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The bound of A ′ comes from the non-negativity of KL-divergence and α ∈ (0 , 1] .
Revised: The bound of A ′ comes from the non-negativity of KL-divergence and α ∈ (0 , 1] .

Original: To prove the lower bound of B ′ h , we define
Revised: To prove the lower bound of B ′ h , we define

Original: [Equation]
Revised: [Equation]

Original: where µ t is an arbitrary policy induced by some f t ∈ F .
Revised: where µ t is an arbitrary policy induced by some f t ∈ F .

Original: Then, for all h ∈ [ H ] [Equation] we have
Revised: Then, for all h ∈ [ H ] , we have

Original: [Equation]
Revised: [Equation]

Original: , according to Lemma 3.
Revised: according to Lemma 3.

Original: Then, by Lemma 4, we have
Revised: Then, by Lemma 4, we have

Original: [Equation]
Revised: [Equation]

Original: where the last step is from some simple calculations and the fact that Lemma 4 implies that the inf is achieved by p ( g h , g h +1 )
Revised: where the last step is from some simple calculations and the fact that Lemma 4 implies that the inf is achieved by p ( g h , g h +1 )

Original: ∝ p h 0 ( g h ) p h
Revised: ∝ p h 0 ( g h ) p h

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: g h
Revised: g h

Original: [Equation]
Revised: [Equation]

Revised: exp

Revised: [Equation]

Original: t − 1 s = 
Revised: t − [Equation] = [Equation]

Revised: hs

Original: [Equation]
Revised: [Equation]

Original: ξ hs
Revised: g h

Original: [Equation]
Revised: [Equation]

Original: g h , g h +1 , µ t , ζ s (cid:1) )
Revised: g h + [Equation] µ t [Equation]

Revised: ζ

Original: [Equation]
Revised: [Equation]

Original: This implies that
Revised: This implies that

Original: [Equation]
Revised: [Equation]

Original: where we use the convexity of − ln( · ) .
Revised: where we use the convexity of − ln( · ) .

Original: With this result
Revised: With this result

Original: [Equation]
Revised: [Equation]

Original: the definition of B ′ h and the definition of ξ hs (cid:0) g h , g h +1 , µ t , ζ s (cid:1) [Equation] we have
Revised: the definition of B ′ h and the definition of ξ hs (cid:0) g h , g h +1 , µ t , ζ s (cid:1) , we have

Original: [Equation]
Revised: [Equation]

Original: where we use Lemma 13 in the last step.
Revised: where we use Lemma 13 in the last step.

Original: We now turn to the lower bound of C h .
Revised: We now turn to the lower bound of C h .

Original: For any max-player’s policy µ , we have
Revised: For any max-player’s policy µ , we have

Original: [Equation]
Revised: [Equation]

Original: where we use the fact that the inf is achieved at
Revised: where we use the fact that the inf is achieved at

Original: [Equation]
Revised: [Equation]

Original: We now consider a fixed g h ∈ F µ h ( ϵ, g h +1 ) .
Revised: We now consider a fixed g h ∈ F µ h ( ϵ, g h +1 ) .

Original: Using the same arguments as in the proof of Lemma 11, it holds that
Revised: Using the same arguments as in the proof of Lemma [Equation] it holds that

Original: [Equation]
Revised: [Equation]

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Original: Thus
Revised: Thus

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where we use the definition
Revised: where we use the definition

Original: [Equation]
Revised: [Equation]

Original: We are ready to prove Theorem
Revised: We are ready to prove Theorem

Original: [Equation]
Revised: [Equation]

Original: Proof of Theorem 3.
Revised: Proof of Theorem 3.

Original: Let π t denote the distribution induced by µ t × ν t and define
Revised: Let π t denote the distribution induced by µ t × ν t and define

Original: [Equation]
Revised: [Equation]

Original: According to the value-decomposition Lemma
Revised: According to the value-decomposition Lemma

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Revised: [Equation]

Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: Summing over t , we obtain that
Revised: Summing over t , we obtain that

Original: [Equation]
Revised: [Equation]

Original: The last step is proved as follows.
Revised: The last step is proved as follows.

Original: We find an upper bound for E S t − 1 E f t ∼ ˆ p t (cid:80)
Revised: We find an upper bound for E S t − 1 E f t ∼ ˆ p t (cid:80)

Original: Tt =1 (cid:80)
Revised: Tt =1 (cid:80)

Original: Hh =1 κ hµ t ( α, ϵ ) .
Revised: Hh =1 κ hµ t ( α, ϵ ) .

Original: We note that for all µ t , κ hµ t ( α, ϵ ) is increasing w.r.t.
Revised: We note that for all µ t , κ hµ t ( α, ϵ ) is increasing w.r.t.

Original: α with the limit κ hµ t (1 , ϵ ) ≤ κ ( ϵ ) < ∞ .
Revised: α with the limit κ hµ t (1 , ϵ ) ≤ κ ( ϵ ) < ∞ .

Original: By monotone convergence theorem, we know that
Revised: By monotone convergence theorem, we know that

Original: [Equation]
Revised: [Equation]

Original: We also have
Revised: We also have

Original: [Equation]
Revised: [Equation]

Original: ϵ
Revised: κ µ

Original: [Equation]
Revised: [Equation]

Revised: ( ϵ ) ≤ Tκ ( ϵ ) .

Original: It follows that −
Revised: It follows that −

Revised: T (cid:88)

Original: [Equation]
Revised: [Equation]

Original: Now we first let α → 1 − 1 .
Revised: Now we first let α → 1 − 1 .

Original: Then, we take ϵ = βT 2 , λ = (cid:113) 
Revised: Then, we take ϵ = βT 2 , λ = (cid:113) [Equation] =

Original: [Equation]
Revised: [Equation]

Original: This concludes the proof.
Revised: This concludes the proof.

Original: F I NTERPRETATION OF T HEOREMF. B OUNDS FOR THE M ULTI -A GENT D ECOUPLING C OEFFICIENT
Revised: F I NTERPRETATION OF T HEOREMF. B OUNDS FOR THE M ULTI -A GENT D ECOUPLING C OEFFICIENT

Original: In this subsection, we provide several examples whose multi-agent decoupling coefficient is provably small.
Revised: In this subsection, we provide several examples whose multi-agent decoupling coefficient is provably small.

Original: The proof can be found in Appendix H.
Revised: The proof can be found in Appendix H.

Original: Linear MG.
Revised: Linear MG.

Original: The first example is the MG with linear function approximation Xie et al.
Revised: The first example is the MG with linear function approximation Xie et al.

Original: In this case, there exists a feature map ϕ
Revised: In this case, there exists a feature map ϕ

Original: ( x, a, b ) ∈ R d
Revised: ( x, a, b ) ∈ R d

Original: and it holds that r h ( x, a, b ) =
Revised: and it holds that r h ( x, a, b ) =

Original: ϕ
Revised: ϕ

Original: ( x, a, b ) ⊤ θ h and P h ( x ′ | x, a, b ) =
Revised: ( x, a, b ) ⊤ θ h and P h ( x ′ | x, a, b ) =

Original: ϕ
Revised: ϕ

Original: ( x, a, b ) ⊤ µ h ( x ′ ) for some unknown θ h ∈ R d and µ h ( · ) ∈ R d satisfying max { (cid:13)(cid:13)
Revised: ( x, a, b ) ⊤ µ h ( x ′ ) for some unknown θ h ∈ R d and µ h ( · ) ∈ R d satisfying max { (cid:13)(cid:13)

Original: θ h (cid:13)(cid:13) , ∥ µ h ∥} ≤ √ d .
Revised: θ h (cid:13)(cid:13) , ∥ µ h ∥} ≤ √ d .

Original: We have the following upper bound for the multi-agent decoupling coefficient.
Revised: We have the following upper bound for the multi-agent decoupling coefficient.

Original: Proposition 1 (Linear MG) .
Revised: Proposition 1 (Linear MG) .

Original: For a d-dimensional MG with F h = { ϕ h ( · , · , · ) ⊤ θ h : (cid:13)(cid:13) θ h (cid:13)(cid:13) ≤ ( H + 1 − h ) √ d } and
Revised: For a d-dimensional MG with F h = { ϕ h ( · , · , · ) ⊤ θ h : (cid:13)(cid:13) θ h (cid:13)(cid:13) ≤ ( H + 1 − h ) √ d } and

Original: ∥ ϕ
Revised: ∥ ϕ

Original: ( x, a, b ) ∥ ≤ 1 , ∀ ( x, a, b ) ∈
Revised: ( x, a, b ) ∥ ≤ 1 , ∀ ( x, a, b ) ∈

Original: X ×
Revised: X ×

Original: A × B , then for all µ ≤ 1 , it holds that
Revised: A × B , then for all µ ≤ 1 , it holds that

Original: [Equation]
Revised: [Equation]

Original: Note that Jin et al.
Revised: Note that Jin et al.

Original: (2021b) considers a more general setting of linear function approximation whose multi-agent decoupling coefficient is also provably small due to Proposition 3.
Revised: (2021b) considers a more general setting of linear function approximation whose multi-agent decoupling coefficient is also provably small due to Proposition 3.

Original: Also note that as a special case, tabular MG is a linear MG of dimension d = |X||A||B| .
Revised: Also note that as a special case, tabular MG is a linear MG of dimension d = |X||A||B| .

Original: Generalized Linear MG.
Revised: Generalized Linear MG.

Original: We then consider the generalized linear MG.
Revised: We then consider the generalized linear MG.

Original: In this case, we have ( f h − T µh f h +1 )
Revised: In this case, we have ( f h − T µh f h +1 )

Original: ( x, a, b ) =
Revised: ( x, a, b ) =

Original: σ ( ϕ ( x, a, b ) ⊤ θ h ) for any µ induced by some function in F and f ∈
Revised: σ ( ϕ ( x, a, b ) ⊤ θ h ) for any µ induced by some function in F and f ∈

Original: F where σ is differentiable and strictly increasing.
Revised: F where σ is differentiable and strictly increasing.

Original: We further assume that σ ′ ∈ ( c 1 , c 2 ) and max {∥ ϕ ( x, a, b ) ∥ , (cid:13)(cid:13)
Revised: We further assume that σ ′ ∈ ( c 1 , c 2 ) and max {∥ ϕ ( x, a, b ) ∥ , (cid:13)(cid:13)

Original: θ h (cid:13)(cid:13) } ≤ R for some c 1 , c 2 , R > 0 .
Revised: θ h (cid:13)(cid:13) } ≤ R for some c 1 , c 2 , R > 0 .

Original: Proposition 2 (Generalized Linear MG.) .
Revised: Proposition 2 (Generalized Linear MG.) .

Original: For a generalized linear MG, with F = { ( x, a, b ) → σ ( ϕ ( x, a, b ) ⊤ θ : ∥ θ ∥ ≤ H √ d } , then for all µ ≤ 1 , it holds that
Revised: For a generalized linear MG, with F = { ( x, a, b ) → σ ( ϕ ( x, a, b ) ⊤ θ : ∥ θ ∥ ≤ H √ d } , then for all µ ≤ 1 , it holds that

Original: [Equation]
Revised: [Equation]

Original: We can also derive an upper bound for the multi-agent decoupling coefficient through multi-agent
Revised: We can also derive an upper bound for the multi-agent decoupling coefficient through multi-agent

Original: Bellman Eluder dimension introduced in Jin et al.
Revised: Bellman Eluder dimension introduced in Jin et al.

Original: (2021b).
Revised: (2021b).

Revised: Definition 5 ( ϵ -independence between distributions) .

Revised: Let F be a function class defined on X , and π, π 1 , · · · , π n be probability measures over X .

Revised: We say π is ϵ -independent of { π 1 , · · · , π n } with respect to F if there exists f ∈ F

Original: Proposition 3 (Reduction to multi-agent Bellman{+such that (cid:112)(cid:80) ni =
 ( E π i [ f ]) 2 ≤ ϵ but | E π [ f ] | > ϵ . Definition 6 (Distributional+} Eluder dimension) .
Revised: {+such that (cid:112)(cid:80) ni =
 ( E π i [ f ]) 2 ≤ ϵ but | E π [ f ] | > ϵ . Definition 6 (Distributional+} Eluder dimension) .

Revised: Let F be a function class defined on X , and Π be a family of probability measures over X .

Revised: The distributional Eluder dimension dim DE ( F , Π , ϵ ) is the length of the longest sequence ρ 1 , · · · , ρ i − 1 ⊂ (cid:81) such that there exists ϵ ′ >

Revised: ϵ where ρ i is ϵ ′ -independent of { ρ 1 , · · · , ρ i − 1 } for all i ∈ [ n ] .

Revised: Definition 7 (multi-agent Bellman Eluder dimension) .

Revised: Let H F be teh function classes of Bellman residuals where H F ,h :=

Revised: { f h −T µ g h f h

Revised: +1 | f, g ∈ F} .

Revised: Then, the Bellman-Eluder dimension is defined as

Revised: [Equation]

Revised: Proposition 3 (Reduction to multi-agent Bellman Eluder dimension) .

Original: Let Π F = D F be the set of probability measures over X × A × B at each step h obtained by following ( µ f , ν f,g ) for some f, g ∈ F .
Revised: Let Π F = D F be the set of probability measures over X × A × B at each step h obtained by following ( µ f , ν f,g ) for some f, g ∈ F .

Original: If for all ϵ > 

Revised: If for all ϵ >

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: then for all µ ≤ 1 , the multi-agent decoupling coefficient satisfies:
Revised: then for all µ ≤ 1 , the multi-agent decoupling coefficient satisfies:

Original: [Equation]
Revised: [Equation]

Original: Similar to the single-agent case, the multi-agent decoupling coefficient exhibits an additional factor of H due to the formulation of summation over all steps instead of maximum as in the multi-agent
Revised: Similar to the single-agent case, the multi-agent decoupling coefficient exhibits an additional factor of H due to the formulation of summation over all steps instead of maximum as in the multi-agent

Original: Bellman Eluder dimension case.
Revised: Bellman Eluder dimension case.

Original: This formulation can offer advantages when the complexity of the function class varies with time steps h .
Revised: This formulation can offer advantages when the complexity of the function class varies with time steps h .

Original: F. I NTERPRETATION OF T HEOREM
Revised: F. I NTERPRETATION OF T HEOREM

Original: We now illustrate Theorem 1 by concrete examples.
Revised: We now illustrate Theorem 1 by concrete examples.

Original: The first example is for the finite function classes.
Revised: The first example is for the finite function classes.

Original: Corollary 4 (Finite function classes with completeness) .
Revised: Corollary 4 (Finite function classes with completeness) .

Original: Let F be a finite function class satisfying
Revised: Let F be a finite function class satisfying

Original: Assumptions 1, 2 and 3 with β = 2 .
Revised: Assumptions 1, 2 and 3 with β = 2 .

Original: Assume that the prior is uniform p h 0 ( f ) = 1 / |F h | , and |F| =
Revised: Assume that the prior is uniform p h 0 ( f ) = 1 / |F h | , and |F| =

Original: (cid:81)
Revised: (cid:81)

Original: Hh =1 |F h | .
Revised: Hh =1 |F h | .

Original: With η = 0 .
Revised: With η = 0 .

Original: 1 and λ = (cid:113)
Revised: 1 and λ = (cid:113)

Original: [Equation]
Revised: [Equation]

Original:  F ,MG [Equation] T ) , we have
Revised: ln |F| dc ( F ,MG,T ) , we have

Original: [Equation]
Revised: [Equation]

Original: Note that it is straightforward to generalize this result to the infinite function classes by replacing the cardinality |F| with its covering number N ∞ ( F , ϵ ) with an appropriate choice of ϵ .
Revised: Note that it is straightforward to generalize this result to the infinite function classes by replacing the cardinality |F| with its covering number N ∞ ( F , ϵ ) with an appropriate choice of ϵ .

Original: We then illustrate Theorem 1 by considering the MGs with linear function approximation.
Revised: We then illustrate Theorem 1 by considering the MGs with linear function approximation.

Original: Corollary 5 (Linear MG) .
Revised: Corollary 5 (Linear MG) .

Original: For the linear MG, if we assume that the prior is uniform, we have κ ( ϵ ) =
Revised: For the linear MG, if we assume that the prior is uniform, we have κ ( ϵ ) =

Original: O ( Hd ln(1 /ϵ )) .
Revised: O ( Hd ln(1 /ϵ )) .

Original: With η = 0 .
Revised: With η = 0 .

Original: 4 H 2 and λ = (cid:113)
Revised: 4 H 2 and λ = (cid:113)

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: dH 3 (

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: )) , we have
Revised: ) , we have

Original: [Equation]
Revised: [Equation]

Original: Compared with Xie et al.
Revised: Compared with Xie et al.

Original: (2020), our algorithm improves the regret bound for linear MGs by a factor of √ d .
Revised: (2020), our algorithm improves the regret bound for linear MGs by a factor of √ d .

Original: Compared to the recent works on MGs (Jin et al., 2021b; Huang et al., 2021), our posterior sampling algorithm achieves comparable regret bound to that of algorithms based on the
Revised: Compared to the recent works on MGs (Jin et al., 2021b; Huang et al., 2021), our posterior sampling algorithm achieves comparable regret bound to that of algorithms based on the

Original: OFU principle in the two-player zero-sum MG.
Revised: OFU principle in the two-player zero-sum MG.

Original: G P ROOF OF THE V ALUE -D ECOMPOSITION L EMMA
Revised: G P ROOF OF THE V ALUE -D ECOMPOSITION L EMMA

Original: Proof of Lemma 1.
Revised: Proof of Lemma 1.

Original: Let µ = µ f and ν be an arbitrary policy taken by the min-player.
Revised: Let µ = µ f and ν be an arbitrary policy taken by the min-player.

Original: [Equation]
Revised: [Equation]

Original: where the first equality comes from the value-decomposition Theorem Jiang et al.
Revised: where the first equality comes from the value-decomposition Theorem Jiang et al.

Original: (2017) (can be verified easily by telescope sum and V H +1 = 0 ); the second equality is because of the definition of µ = µ f,h [Equation] x ) = argmax 
Revised: (2017) (can be verified easily by telescope sum and V H +1 = 0 ); the second equality is because of the definition of µ = µ f,h ( x ) = argmax µ ∈ ∆

Original: [Equation]

Original: A min ν ∈ ∆ B µ ⊤ f h ( x, · , · ) ν ; the inequality comes from the fact that µ = µ f and ν may not be arg min ν ′ D
Revised: A min ν ∈ ∆ B µ ⊤ f h ( x, · , · ) ν ; the inequality comes from the fact that µ = µ f and ν may not be arg min ν ′ D

Original: µ,ν ′
Revised: µ,ν ′

Original: f ( x h ) .
Revised: f ( x h ) .

Original: This decomposition accounts for the use of an optimistic prior in Algorithm 2.
Revised: This decomposition accounts for the use of an optimistic prior in Algorithm 2.

Original: Proof of Lemma 2.
Revised: Proof of Lemma 2.

Original: Suppose that µ = µ f is taken by the max-player and g is sampled from the posterior by the booster agent.
Revised: Suppose that µ = µ f is taken by the max-player and g is sampled from the posterior by the booster agent.

Original: Let ν be given by ν = arg min ν
Revised: Let ν be given by ν = arg min ν

Original: ′ V µh ( x ) for all ( x, h ) .
Revised: ′ V µh ( x ) for all ( x, h ) .

Original: Then, we have:
Revised: Then, we have:

Revised: V µ, † 1 ( x 1 ) − V µ,ν

Original: [Equation]
Revised: [Equation]

Original: This decomposition accounts for the use of a pessimistic prior in Algorithm 3.
Revised: x [Equation]

Original: H P ROOF OF THE D ECOUPLING C OEFFICIENT B OUNDS
Revised: H P ROOF OF THE D ECOUPLING C OEFFICIENT B OUNDS

Original: In this section, we provide proofs for the decoupling coefficient bounds.
Revised: In this section, we provide proofs for the decoupling coefficient bounds.

Original: We need the following lemma.
Revised: We need the following lemma.

Original: Lemma 18 (Elliptical Potential Lemma, Lemma 10 of Xie et al.
Revised: Lemma 18 (Elliptical Potential Lemma, Lemma 10 of Xie et al.

Original: Suppose { ϕ t } t ≥ 0 is a sequence in R d satisfying ∥ ϕ t ∥ ≤ 1 .
Revised: Suppose { ϕ t } t ≥ 0 is a sequence in R d satisfying ∥ ϕ t ∥ ≤ 1 .

Original: Let Λ 

Revised: Let Λ

Original: ∈ R d × d be a positive definite matrix, and Λ t = Λ 0 + (cid:80) ti =
Revised: ∈ R d × d be a positive definite matrix, and Λ t = Λ 0 + (cid:80) ti =

Original: 1 ϕ i ϕ
Revised: 1 ϕ i ϕ

Original: ⊤ i .
Revised: ⊤ i .

Original: If the smallest eigenvalues of Λ 0 is lower bounded by 1 , then
Revised: If the smallest eigenvalues of Λ 0 is lower bounded by 1 , then

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Proof of Proposition 1.
Revised: Proof of Proposition 1.

Original: We first note that the completeness assumption is satisfied in linear MG case whose proof can be found in Huang et al.
Revised: We first note that the completeness assumption is satisfied in linear MG case whose proof can be found in Huang et al.

Original: Now we consider two arbitrary θ h , θ h +1 whose norms are bounded by H √ d and f ∈ F .
Revised: Now we consider two arbitrary θ h , θ h +1 whose norms are bounded by H √ d and f ∈ F .

Original: We also define a function g ∈ F s.t.
Revised: We also define a function g ∈ F s.t.

Original: g h
Revised: g h

Original: = g ( θ h ) and g h
Revised: = g ( θ h ) and g h

Original: +1 =
Revised: +1 =

Original: g ( θ h +1 ) .
Revised: g ( θ h +1 ) .

Original: By Assumption 2, we can find some θ h ( f ) ∈ R d with (cid:13)(cid:13)
Revised: By Assumption 2, we can find some θ h ( f ) ∈ R d with (cid:13)(cid:13)

Original: θ h ( f ) (cid:13)(cid:13)
Revised: θ h ( f ) (cid:13)(cid:13)

Original: ≤ H √ d
Revised: ≤ H √ d

Original: s.t.
Revised: s.t.

Original: T µ f h ( ϕ ( x, a, b ) ⊤ θ h +1 )
Revised: T µ f h ( ϕ ( x, a, b ) ⊤ θ h +1 )

Original: =
Revised: =

Original: ϕ
Revised: ϕ

Original: ( x, a, b ) ⊤ θ h ( f ) .
Revised: ( x, a, b ) ⊤ θ h ( f ) .

Original: Therefore, we have
Revised: Therefore, we have

Revised: [Equation]

Revised: =

Revised: [Equation]

Revised: , a, b

Original: [Equation]
Revised: [Equation]

Revised: w

Revised: [Equation]

Revised: f, g

Revised: [Equation]

Original: where w h ( f, g ) ∈
Revised: where w h ( f, g ) ∈

Original: R
Revised: R

Original: d satisfies [Equation]
Revised: d satisfies (cid:13)(cid:13)

Original:  ≤ 2 H √ d .
Revised: w h ( f, g ) (cid:13)(cid:13) ≤ 2 H √ d .

Original: We denote ϕ hs =
Revised: We denote ϕ hs =

Original: E π s [ ϕ ( x h , a h , b h )] and Φ ht = λI
Revised: E π s [ ϕ ( x h , a h , b h )] and Φ ht = λI

Original: + (cid:80) ts =1 ϕ ( x h , a h , b h ) ϕ
Revised: + (cid:80) ts =1 ϕ ( x h , a h , b h ) ϕ

Original: ( x h , a h , b h ) ⊤ where λ ≥ 1 is a tuning parameter.
Revised: ( x h , a h , b h ) ⊤ where λ ≥ 1 is a tuning parameter.

Original: Then, we have
Revised: Then, we have

Original: E π t 
Revised: E [Equation] = w h ( f t , g t ) ⊤ ϕ ht

Original: [Equation]
Revised: [Equation]

Revised: µw

Original:  E µ ft h
Revised: [Equation] ( f t , g t ) ⊤ t − 1 (cid:88) s = E π s (cid:104) ϕ (cid:0) x h , a h , b h (cid:1) ϕ (cid:0) x h , a h

Original: [Equation]
Revised: [Equation]

Original: where the first inequality uses Jensen’s inequality and (cid:13)(cid:13) w h ( f t , g t )(cid:13)(cid:13) ≤ 2 H √ d and the second inequality comes from the fact ( a ⊤ b ) ≤ ( ∥ a ∥
Revised: b h (cid:1) ⊤ (cid:105) w h ( f t , g t ) ≤ w h ( f t , g t ) ⊤ ϕ h t − µw h ( f t , g t )

Original:  Φ ht − 1  ∥ {+a ∥ Φ h t − 
+}
Revised: ⊤ Φ h t − 1 w h ( f t , g t ) [Equation] [Equation] where the first inequality uses Jensen’s inequality and (cid:13)(cid:13) w h ( f t , g t ) (cid:13)(cid:13) ≤ 2 H √ d and the second inequality comes from the fact ( a ⊤ b ) ≤ ( ∥ {+a ∥ Φ h t − 
+}

Original:  b ∥ (Φ ht − 1 ) − 1 ) ≤ 12 ( ∥ a ∥ 2Φ ht − 1 + ∥ b ∥
Revised: ∥ b ∥ (Φ h t − 1 ) − 1 ) ≤ 12 ( ∥ a ∥ 2Φ ht − 1 + ∥ b ∥

Original: 2(Φ ht − 1 ) − 1 ) .
Revised: 2(Φ ht − 1 ) − 1 ) .

Original: Summing over t ∈ [ T ] and h ∈
Revised: Summing over t ∈ [ T ] and h ∈

Original: [ H ] , we have
Revised: [ H ] , we have

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: where we use the Elliptical Potential lemma in the first inequality and the second inequality uses

Original: [Equation]
Revised: [Equation]

Original: where we use the Elliptical Potential lemma in the first inequality and the second inequality uses
Revised: and, trace (Φ ht ) ≤ λd + T

Original: [Equation]
Revised: [Equation]

Original: By setting λ = min { 1 , 1 µ 2 H 2 T } , we conclude the proof.
Revised: By setting λ = min { 1 , 1 µ 2 H 2 T } , we conclude the proof.

Original: Proof of Proposition 2.
Revised: Proof of Proposition 2.

Original: We assume that c 1 ≤ 1 ≤ c 2 .
Revised: We assume that c 1 ≤ 1 ≤ c 2 .

Original: Otherwise, we can scale the feature maps and the link function accordingly.
Revised: Otherwise, we can scale the feature maps and the link function accordingly.

Original: By similar arguments with the completeness assumption as in the proof of Proposition 1, we have
Revised: By similar arguments with the completeness assumption as in the proof of Proposition 1, we have

Original: [Equation]
Revised: [Equation]

Original: By the Lipschitz property, we have
Revised: By the Lipschitz property, we have

Original: [Equation]
Revised: [Equation]

Original: for some w ( f, g ) ∈ R d satisfying w ( f, g ) ≤ 2 H √ d .
Revised: for some w ( f, g ) ∈ R d satisfying w ( f, g ) ≤ 2 H √ d .

Original: We denote ϕ sh
Revised: We denote ϕ sh

Original: =
Revised: =

Original: E π s [ ϕ ( x h , a h , b h )] and Φ ht = λI
Revised: E π s [ ϕ ( x h , a h , b h )] and Φ ht = λI

Original: + (cid:80) ts =1 ϕ ( x h , a h , b h ) ϕ
Revised: + (cid:80) ts =1 ϕ ( x h , a h , b h ) ϕ

Original: ( x h , a h , b h ) ⊤ where λ ≥ 1 is a tuning parameter.
Revised: ( x h , a h , b h ) ⊤ where λ ≥ 1 is a tuning parameter.

Original: Then, we have
Revised: Then, we have

Revised: E

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: t (cid:2) E µ ft h (cid:0)

Original: [Equation]
Revised: [Equation]

Original: Summing over t ∈ [ T ] and h ∈ [ H ] , we have
Revised: ; x ht , a [Equation] µ

Original: [Equation]
Revised: [Equation]

Original: Setting λ = min { 1 ,
Revised: (cid:0)

Original: [Equation]
Revised: [Equation]

Original: concludes the proof. In what follows
Revised: Summing over t ∈ [ T ] and h ∈ [ H ] , we have

Original: [Equation]
Revised: [Equation]

Revised: Setting λ = min { 1 , 1 µ 2 c 21 H 2 T } concludes the proof.

Original:  we prove the reduction of Bellman-Eluder dimension to the decoupling coefficient.
Revised: In what follows, we prove the reduction of Bellman-Eluder dimension to the decoupling coefficient.

Original: The proof is almost the same as that of Dann et al.
Revised: The proof is almost the same as that of Dann et al.

Original: (2021) with minor modification.
Revised: (2021) with minor modification.

Original: We start with the following lemma from Dann et al.
Revised: We start with the following lemma from Dann et al.

Original: Lemma 19.
Revised: Lemma 19.

Original: Let π 1 , · · · , π t − 1 denote the measures over X × A × B obtained by following the policy induced by ( f s , g s ) t − 1 s =1 ∈ F × F at stage h and { d 1 , · · · , d M } be the set of unique measures in decreasing order of occurrences and let ( N i )
Revised: Let π 1 , · · · , π t − 1 denote the measures over X × A × B obtained by following the policy induced by ( f s , g s ) t − 1 s =1 ∈ F × F at stage h and { d 1 , · · · , d M } be the set of unique measures in decreasing order of occurrences and let ( N i )

Original: Mi
Revised: Mi

Original: =1 be the number of times a measure appears in the sequence.
Revised: =1 be the number of times a measure appears in the sequence.

Original: If the ϵ -BE dimension for the distribution set D F and the function classes:
Revised: If the ϵ -BE dimension for the distribution set D F and the function classes:

Original: [Equation]
Revised: [Equation]

Original: is E and |E x,a,b ∼ π t E µ ft h ( g t ; x, a, b ) | > ϵ .
Revised: is E and |E x,a,b ∼ π t E µ ft h ( g t ; x, a, b ) | > ϵ .

Original: Then, it holds that
Revised: Then, it holds that

Original: [Equation]
Revised: [Equation]

Original: ft h ( g [Equation] [Equation] [Equation]
Revised: where w ht = (cid:40) N i if π t = d i ∧ i ∈ [

Original: , b

Original: [Equation]
Revised: [Equation]

Original: (cid:105)

Original: [Equation]
Revised: [Equation]

Original: (cid:0) E

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: (cid:2)
Revised: (cid:80)

Original: [Equation]
Revised: [Equation]

Original: ( g [Equation] ; x, a, b ) (cid:3)(cid:1)where w ht
Revised: otherwise.

Original: = (cid:40) N i if π t = d i ∧ i ∈
Revised: [Equation]

Original: [ E − 1] (cid:108) (cid:80) Mi = E N i E (cid:109) otherwise.
Revised: [Equation]

Original: Proof of Proposition 3.
Revised: Proof of Proposition 3.

Original: Denote ϵ ht,s =
Revised: Denote ϵ ht,s =

Original: E x hs ,a hs ,b hs E µ ft h ( g t ; x, a, b ) , the LHS is upper bounded by
Revised: E x hs ,a hs ,b hs E µ ft h ( g t ; x, a, b ) , the LHS is upper bounded by

Original: [Equation]
Revised: [Equation]

Original: For each h ∈ [ H ] , the RHS is bounded by
Revised: For each h ∈ [ H ] , the RHS is bounded by

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is because
Revised: where the last inequality is because

Original: [Equation]
Revised: [Equation]

Original: which is due to
Revised: which is due to

Original: [Equation]
Revised: [Equation]

Original: Every time a measure π t appears in the E − 1 most common measures, one of N i increases.
Revised: Every time a measure π t appears in the E − 1 most common measures, one of N i increases.

Original: Otherwise, (cid:80) i ≥ E N i increases by 1 .
Revised: Otherwise, (cid:80) i ≥ E N i increases by 1 .

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]