Original: C AN R EINFORCEMENT L EARNING E FFICIENTLY F IND S TACKELBERG -N ASH E QUILIBRIA IN G ENERAL -S UM M ARKOV G AMES WITH M YOPIC F OLLOWERS ?
Revised: C AN R EINFORCEMENT L EARNING E FFICIENTLY F IND S TACKELBERG -N ASH E QUILIBRIA IN G ENERAL -S UM M ARKOV G AMES WITH M YOPIC F OLLOWERS ?

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We study multi-player general-sum Markov games with one of the players designated as the leader and the rest regarded as the followers.
Revised: We study multi-player general-sum Markov games with one of the players designated as the leader and the rest regarded as the followers.

Original: In particular, we focus on the class of games where the followers are myopic, i e., the followers aim to maximize the instantaneous rewards.
Revised: In particular, we focus on the class of games where the followers are myopic, i.e., the followers aim to maximize the instantaneous rewards.

Original: For such a game, our goal is to ﬁnd the
Revised: For such a game, our goal is to ﬁnd the

Original: Stackelberg-Nash equilibrium (SNE), which is a policy pair ( π ∗ , ν ∗ ) such that
Revised: Stackelberg-Nash equilibrium (SNE), which is a policy pair ( π ∗ , ν ∗ ) such that

Original: (i) π ∗ is the optimal policy for the leader when the followers always play their best response and (ii) ν ∗ is the best response policy of the followers, which is a Nash equilibrium of the followers’ game induced by π ∗ .
Revised: (i) π ∗ is the optimal policy for the leader when the followers always play their best response, and (ii) ν ∗ is the best response policy of the followers, which is a Nash equilibrium of the followers’ game induced by π ∗ .

Original: We develop sample efﬁcient reinforcement learning (RL) algorithms for solving SNE under both the online and ofﬂine settings.
Revised: We develop sample efﬁcient reinforcement learning (RL) algorithms for solving SNE under both the online and ofﬂine settings.

Original: Respectively, our algorithms are optimistic and pessimistic variants of least-squares value iteration and are readily able to incorporate function approximation tools for handling large state spaces.
Revised: Respectively, our algorithms are optimistic and pessimistic variants of least-squares value iteration and are readily able to incorporate function approximation tools for handling large state spaces.

Original: Furthermore, for the case with linear function approximation, we prove that our algorithms achieve sublinear regret and suboptimality under online and ofﬂine setups respectively.
Revised: Furthermore, for the case with linear function approximation, we prove that our algorithms achieve sublinear regret and suboptimality under online and ofﬂine setups respectively.

Original: To our best knowledge, we establish the ﬁrst provably efﬁcient RL algorithms for solving
Revised: To our best knowledge, we establish the ﬁrst provably efﬁcient RL algorithms for solving

Original: SNE in general-sum Markov games with myopic followers.
Revised: SNE in general-sum Markov games with myopic followers.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Reinforcement learning (RL) has achieved striking empirical successes in solving complicated realworld sequential decision-making problems (Mnih et al., 2015; Duan et al., 2016; Silver et al., 2016; 2017; 2018; Agostinelli et al., 2019; Akkaya et al., 2019).
Revised: Reinforcement learning (RL) has achieved striking empirical successes in solving complicated realworld sequential decision-making problems (Mnih et al., 2015; Duan et al., 2016; Silver et al., 2016; 2017; 2018; Agostinelli et al., 2019; Akkaya et al., 2019).

Original: Motivated by these successes, multi-agent extensions of RL algorithms recently have gained great popularity in decision-making problems involving multiple interacting agents (Busoniu et al., 2008; Hernandez-Leal et al., 2018; 2019; OroojlooyJadid & Hajinezhad, 2019; Zhang et al., 2019).
Revised: Motivated by these successes, multi-agent extensions of RL algorithms recently have gained great popularity in decision-making problems involving multiple interacting agents (Busoniu et al., 2008; Hernandez-Leal et al., 2018; 2019; OroojlooyJadid & Hajinezhad, 2019; Zhang et al., 2019).

Original: Multi-agent RL is often modeled as a Markov game (Littman, 1994) where, at each time step, each player (agent) takes an action simultaneously at each state of the environment, observe her own immediate reward, and the environment evolves into a next state.
Revised: Multi-agent RL is often modeled as a Markov game (Littman, 1994) where, at each time step, each player (agent) takes an action simultaneously at each state of the environment, observe her own immediate reward, and the environment evolves into a next state.

Original: Here both the reward of each player and the state transition depend on the actions of all players.
Revised: Here both the reward of each player and the state transition depend on the actions of all players.

Original: From the perspective of each player, her goal is to ﬁnd a policy that maximizes her expected total reward in the presence of other agents.
Revised: From the perspective of each player, her goal is to ﬁnd a policy that maximizes her expected total reward in the presence of other agents.

Original: In Markov games, depending on the structure of the reward functions, the relationship among the players can be either collaborative, where each player has the same reward function, or competitive, where the sum of the reward function is equal to zero, or mixed, which corresponds to a generalsum game.
Revised: In Markov games, depending on the structure of the reward functions, the relationship among the players can be either collaborative, where each player has the same reward function, or competitive, where the sum of the reward function is equal to zero, or mixed, which corresponds to a generalsum game.

Original: While most of the existing theoretical results focus on the collaborative or two-player competitive settings, the mixed setting is oftentimes more pertinent to real-world multi-agent applications.
Revised: While most of the existing theoretical results focus on the collaborative or two-player competitive settings, the mixed setting is oftentimes more pertinent to real-world multi-agent applications.

Original: Moreover, in addition to having diverse reward functions, the players might also have asymmetric roles in the Markov game — the players might be divided into leaders and followers, where the leaders’ joint policy determines a general-sum game for the followers.
Revised: Moreover, in addition to having diverse reward functions, the players might also have asymmetric roles in the Markov game — the players might be divided into leaders and followers, where the leaders’ joint policy determines a general-sum game for the followers.

Original: Games with such a leaderfollower structure is popular in applications such as mechanism design (Conitzer & Sandholm, 2002;
Revised: Games with such a leaderfollower structure is popular in applications such as mechanism design (Conitzer & Sandholm, 2002;

Original: Roughgarden, 2004; Garg & Narahari, 2005; Kang & Wu, 2014), security games (Tambe, 2011; Korzhyk et al., 2011), incentive design (Zheng et al., 1984; Ratliff et al., 2014; Chen et al., 2016; Ratliff & Fiez, 2020), and model-based RL (Rajeswaran et al., 2020).
Revised: Roughgarden, 2004; Garg & Narahari, 2005; Kang & Wu, 2014), security games (Tambe, 2011; Korzhyk et al., 2011), incentive design (Zheng et al., 1984; Ratliff et al., 2014; Chen et al., 2016; Ratliff & Fiez, 2020), and model-based RL (Rajeswaran et al., 2020).

Original: Consider a simpliﬁed economic sys- tem that consists of a government and a group of companies, where the companies purchase or sell goods, and the government collects taxes from transactions.
Revised: Consider a simpliﬁed economic system that consists of a government and a group of companies, where the companies purchase or sell goods, and the government collects taxes from transactions.

Original: Such a problem can be viewed as a multi-player general-sum game, where the government serves as the leader and the companies are followers (Zheng et al., 2020).
Revised: Such a problem can be viewed as a multi-player general-sum game, where the government serves as the leader and the companies are followers (Zheng et al., 2020).

Original: In particular, when the government sets a tax rate, the companies form a general-sum game themselves, whose reward functions depend on the tax rate.
Revised: In particular, when the government sets a tax rate, the companies form a general-sum game themselves, whose reward functions depend on the tax rate.

Original: Each company aims to maximize their own revenue, and thus ideally they achieve a Nash equilibrium (NE) of the induced game.
Revised: Each company aims to maximize their own revenue, and thus ideally they achieve a Nash equilibrium (NE) of the induced game.

Original: Whereas the goal of the government might be achieving the social welfare, which can be measured via certain fairness metrics computed by the revenues of the companies.
Revised: Whereas the goal of the government might be achieving the social welfare, which can be measured via certain fairness metrics computed by the revenues of the companies.

Original: In multi-player Markov games with such a leader-follower structure, the desired solution concept is the Stackelberg-Nash equilibrium (SNE) (Bas¸ar & Olsder, 1998).
Revised: In multi-player Markov games with such a leader-follower structure, the desired solution concept is the Stackelberg-Nash equilibrium (SNE) (Bas¸ar & Olsder, 1998).

Original: In the setting where there is a single leader, SNE corresponds to a pair of leader’s policy π ∗ and followers’ joint policy ν ∗ that satisﬁes the following two properties: (i) when the leader adopts π ∗ , ν ∗ is the best-response policy of the followers, i.e., ν ∗ is a Nash equilibrium of the followers’ subgame induced by π ∗ ; and (ii) π ∗ is the optimal policy of the leader assuming the followers always adopt the best response.
Revised: In the setting where there is a single leader, SNE corresponds to a pair of leader’s policy π ∗ and followers’ joint policy ν ∗ that satisﬁes the following two properties: (i) when the leader adopts π ∗ , ν ∗ is the best-response policy of the followers, i.e., ν ∗ is a Nash equilibrium of the followers’ subgame induced by π ∗ ; and (ii) π ∗ is the optimal policy of the leader assuming the followers always adopt the best response.

Original: We are interested in ﬁnding an SNE in a multi-player Markov game when the reward functions and
Revised: We are interested in ﬁnding an SNE in a multi-player Markov game when the reward functions and

Original: Markov transition kernel are unknown.
Revised: Markov transition kernel are unknown.

Original: In particular, we focus on the setting with a single leader and multiple myopic followers.
Revised: In particular, we focus on the setting with a single leader and multiple myopic followers.

Original: That is, the followers at any step of the game do not take into account the future rewards, but only the rewards in the current step.
Revised: That is, the followers at any step of the game do not take into account the future rewards, but only the rewards in the current step.

Original: The formal deﬁnition of myopic followers is given in § 2.3.
Revised: The formal deﬁnition of myopic followers is given in § 2.3.

Original: This setting is a natural formalization of many real-world problems such as marketing and supply chain management.
Revised: This setting is a natural formalization of many real-world problems such as marketing and supply chain management.

Original: For example, in a market, the leader is an established ﬁrm and the followers are entrants.
Revised: For example, in a market, the leader is an established ﬁrm and the followers are entrants.

Original: The entrants are not sure whether the ﬁrm is going to exist in the future, so they might just want to maximize instantaneous rewards.
Revised: The entrants are not sure whether the ﬁrm is going to exist in the future, so they might just want to maximize instantaneous rewards.

Original: See Li & Sethi (2017);
Revised: See Li & Sethi (2017);

Original: Ka ´ nska & Wiszniewska-Matyszkiel (2021) and references therein for more examples.
Revised: Ka ´ nska & Wiszniewska-Matyszkiel (2021) and references therein for more examples.

Original: For such a game, we are interested in the following question:
Revised: For such a game, we are interested in the following question:

Original: Can we develop sample efﬁcient reinforcement learning methods that provably ﬁnd Stackelberg-Nash equilibria in general-sum Markov games with myopic followers?
Revised: Can we develop sample efﬁcient reinforcement learning methods that provably ﬁnd Stackelberg-Nash equilibria in general-sum Markov games with myopic followers?

Original: To this end, we consider both online and ofﬂine RL settings, where in the former, we learn the
Revised: To this end, we consider both online and ofﬂine RL settings, where in the former, we learn the

Original: SNE in a trial-an-error fashion by interacting with the environment and generating data, and in the latter, we learn the SNE from a given dataset that is collected a priori.
Revised: SNE in a trial-an-error fashion by interacting with the environment and generating data, and in the latter, we learn the SNE from a given dataset that is collected a priori.

Original: For the online setting, as the transition model is unknown, to achieve sample efﬁciency, the equilibrium-ﬁnding algorithm also needs to take the exploration-exploitation tradeoff into consideration.
Revised: For the online setting, as the transition model is unknown, to achieve sample efﬁciency, the equilibrium-ﬁnding algorithm also needs to take the exploration-exploitation tradeoff into consideration.

Original: Although the similar challenge has been studied in zero-sum Markov game, it seems unclear how to incorporate popular exploration mechanisms such as optimism in the face of uncertainty (Sutton & Barto, 2018) into
Revised: Although the similar challenge has been studied in zero-sum Markov game, it seems unclear how to incorporate popular exploration mechanisms such as optimism in the face of uncertainty (Sutton & Barto, 2018) into

Original: SNE ﬁnding.
Revised: SNE ﬁnding.

Original: Meanwhile, under the ofﬂine setting, as the RL agent has no control of data collection, it is ideal to design an RL algorithm with theoretical guarantees for an arbitrary dataset that might not be sufﬁciently explorative.
Revised: Meanwhile, under the ofﬂine setting, as the RL agent has no control of data collection, it is ideal to design an RL algorithm with theoretical guarantees for an arbitrary dataset that might not be sufﬁciently explorative.

Original: Our contributions
Revised: Our contributions

Original: Our contributions are three-fold.
Revised: Our contributions are three-fold.

Original: First, for the episodic general-sum Markov game with myopic followers, under the online and ofﬂine settings respectively, we propose optimistic and pessimistic variants of the least-squares value iteration (LSVI) algorithm.
Revised: First, for the episodic general-sum Markov game with myopic followers, under the online and ofﬂine settings respectively, we propose optimistic and pessimistic variants of the least-squares value iteration (LSVI) algorithm.

Original: In particular, in a version of LSVI, we estimate the optimal action-value function of the leader via least-squares regression and construct an estimate of the SNE by solving the SNE of the multi-matrix game for each state, whose payoff matrices are given by the leader’s estimated action-value function and the followers’ reward functions.
Revised: In particular, in a version of LSVI, we estimate the optimal action-value function of the leader via least-squares regression and construct an estimate of the SNE by solving the SNE of the multi-matrix game for each state, whose payoff matrices are given by the leader’s estimated action-value function and the followers’ reward functions.

Original: Moreover, we add a UCB exploration bonus to the least-squares solution to achieve optimism in the online setting.
Revised: Moreover, we add a UCB exploration bonus to the least-squares solution to achieve optimism in the online setting.

Original: Whereas in the ofﬂine setting, pessimism is achieved by subtracting a penalty function constructed using the ofﬂine data, which is equal to the negative bonus function.
Revised: Whereas in the ofﬂine setting, pessimism is achieved by subtracting a penalty function constructed using the ofﬂine data, which is equal to the negative bonus function.

Original: Moreover, these algorithms are readily able to incorporate function approximators and we showcase the version with linear function approximation.
Revised: Moreover, these algorithms are readily able to incorporate function approximators and we showcase the version with linear function approximation.

Original: Second, under the online setting, we prove that our optimistic LSVI algorithm achieves a sublinear (cid:101)
Revised: Second, under the online setting, we prove that our optimistic LSVI algorithm achieves a sublinear (cid:101)

Original: O ( H 2 √ d 3 K ) regret, where K is the number of episodes, H is the horizon, d is the dimension of the feature mapping, and (cid:101) O ( · ) omits logarithmic terms.
Revised: O ( H 2 √ d 3 K ) regret, where K is the number of episodes, H is the horizon, d is the dimension of the feature mapping, and (cid:101) O ( · ) omits logarithmic terms.

Original: Finally, under the ofﬂine setting, we establish an upper bound on the suboptimality of the proposed algorithm for an arbitrary dataset with K trajectories.
Revised: Finally, under the ofﬂine setting, we establish an upper bound on the suboptimal- ity of the proposed algorithm for an arbitrary dataset with K trajectories.

Original: Our upper bound yields a sublinear (cid:101) O
Revised: Our upper bound yields a sublinear (cid:101) O

Original: ( H 2 (cid:112) d 3 /K ) rate as long as the dataset has sufﬁcient coverage over the trajectory induced by the desired SNE.
Revised: ( H 2 (cid:112) d 3 /K ) rate as long as the dataset has sufﬁcient coverage over the trajectory induced by the desired SNE.

Original: Related work.
Revised: Related work.

Original: See Appendix A for details.
Revised: See Appendix A for details.

Original: Notation.
Revised: Notation.

Original: We denote by (cid:107) · (cid:107) 2 the (cid:96) 2 -norm of a vector or the spectral norm of a matrix.
Revised: We denote by (cid:107) · (cid:107) 2 the (cid:96) 2 -norm of a vector or the spectral norm of a matrix.

Original: We also let (cid:107) · (cid:107) op denote the matrix operator norm.
Revised: We also let (cid:107) · (cid:107) op denote the matrix operator norm.

Original: Furthermore, for a positive deﬁnite matrix A , we denote by (cid:107) x (cid:107)
Revised: Furthermore, for a positive deﬁnite matrix A , we denote by (cid:107) x (cid:107)

Original: A the weighted norm √ x (cid:62) Ax of a vector x .
Revised: A the weighted norm √ x (cid:62) Ax of a vector x .

Original: Also, we denote by ∆( A ) the set of probability distributions on a set A .
Revised: Also, we denote by ∆( A ) the set of probability distributions on a set A .

Original: For some positive integer K , [ K ] denotes the index set { 1 , 2 , · · · , K } .
Revised: For some positive integer K , [ K ] denotes the index set { 1 , 2 , · · · , K } .

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: In this section, we introduce the formulation of the general-sum simultaneous-move Markov games,
Revised: In this section, we introduce the formulation of the general-sum simultaneous-move Markov games,

Original: Stackelberg-Nash equilibrium, and the linear structure we use in this paper.
Revised: Stackelberg-Nash equilibrium, and the linear structure we use in this paper.

Original: 2. G ENERAL -S UM S IMULTANEOUS -M OVE M ARKOV G AMES
Revised: 2. G ENERAL -S UM S IMULTANEOUS -M OVE M ARKOV G AMES

Original: In this setting, two levels of hierarchy in decision making are considered: one leader l and N followers { f i } i ∈ [ N ] .
Revised: In this setting, two levels of hierarchy in decision making are considered: one leader l and N followers { f i } i ∈ [ N ] .

Original: Speciﬁcally, we deﬁne an episodic version of general-sum simultaneousmoves Markov game by the tuple ( S , A l , A f = {A f i } i ∈
Revised: Speciﬁcally, we deﬁne an episodic version of general-sum simultaneousmoves Markov game by the tuple ( S , A l , A f = {A f i } i ∈

Original: [ N ] , H, r l , r f = { r f i } i ∈ [ N ] , P ) , where S is the state space, A l and A f are the sets of actions of the leader and the followers respectively, H is the number of steps in each episode, r l = { r l,h : S ×
Revised: [ N ] , H, r l , r f = { r f i } i ∈ [ N ] , P ) , where S is the state space, A l and A f are the sets of actions of the leader and the followers respectively, H is the number of steps in each episode, r l = { r l,h : S ×

Original: A l × A f → [ − 1 , 1] } Hh =1 and r f i = { r f i ,h : S ×
Revised: A l × A f → [ − 1 , 1] } Hh =1 and r f i = { r f i ,h : S ×

Original: A l × A f → [ − 1 , 1] } Hh =1 are reward functions of the leader and the followers respectively, and P =
Revised: A l × A f → [ − 1 , 1] } Hh =1 are reward functions of the leader and the followers respectively, and P =

Original: {P h : S ×
Revised: {P h : S ×

Original: A l × A f × S → [0 , 1] } Hh =1 is a collection of transition kernels.
Revised: A l × A f × S → [0 , 1] } Hh =1 is a collection of transition kernels.

Original: Here A l × A f
Revised: Here A l × A f

Original: =
Revised: =

Original: A l × A f 1 × · · · × A f N .
Revised: A l × A f 1 × · · · × A f N .

Original: Throughout this paper, we also let (cid:63) be some element in { l, f 1 , · · · , f N } .
Revised: Throughout this paper, we also let (cid:63) be some element in { l, f 1 , · · · , f N } .

Original: Moreover, for any ( h, x, a ) ∈ [ H ] × S ×
Revised: Moreover, for any ( h, x, a ) ∈ [ H ] × S ×

Original: A l and b = { b i ∈
Revised: A l and b = { b i ∈

Original: A f i } i ∈ [ N ] , we use the shorthands r (cid:63),h ( x, a, b ) =
Revised: A f i } i ∈ [ N ] , we use the shorthands r (cid:63),h ( x, a, b ) =

Original: r (cid:63),h ( x, a, b 1 , · · · , b N ) and P h ( · | x, a, b ) =
Revised: r (cid:63),h ( x, a, b 1 , · · · , b N ) and P h ( · | x, a, b ) =

Original: P h ( · | x, a, b 1 , · · · , b N ) .
Revised: P h ( · | x, a, b 1 , · · · , b N ) .

Original: Policy and Value Function.
Revised: Policy and Value Function.

Original: A stochastic policy π = { π h : S → ∆( A l ) } Hh =1 of the leader is a set of probability distributions over actions given the state.
Revised: A stochastic policy π = { π h : S → ∆( A l ) } Hh =1 of the leader is a set of probability distributions over actions given the state.

Original: Meanwhile, a stochastic joint policy of the followers is deﬁned by ν = { ν f i } i ∈
Revised: Meanwhile, a stochastic joint policy of the followers is deﬁned by ν = { ν f i } i ∈

Original: [ N ] , where ν f i = { ν f i ,h : S → ∆( A f i ) }
Revised: [ N ] , where ν f i = { ν f i ,h : S → ∆( A f i ) }

Original: Hh =1 .
Revised: Hh =1 .

Original: We use the notation π h ( a | x ) and ν f i ,h ( b i | x ) to denote the probability of taking action a ∈
Revised: We use the notation π h ( a | x ) and ν f i ,h ( b i | x ) to denote the probability of taking action a ∈

Original: A l or b i ∈
Revised: A l or b i ∈

Original: A f i for state x at step h under policy π, ν f i respectively.
Revised: A f i for state x at step h under policy π, ν f i respectively.

Original: Throughout this paper, for any ν = { ν f i } i ∈
Revised: Throughout this paper, for any ν = { ν f i } i ∈

Original: [ N ] and b = { b i } i ∈
Revised: [ N ] and b = { b i } i ∈

Original: [ N ] , we use the shorthand ν h ( b | x ) =
Revised: [ N ] , we use the shorthand ν h ( b | x ) =

Original: ν f 1 ,h ( b 1 | x ) × · · · × ν f N ,h ( b N | x ) .
Revised: ν f 1 ,h ( b 1 | x ) × · · · × ν f N ,h ( b N | x ) .

Original: Given policies ( π, ν = { ν f i } i ∈
Revised: Given policies ( π, ν = { ν f i } i ∈

Original: [ N ] ) , the action-value (Q) and state-value (V) functions for the leader and followers are deﬁned by
Revised: [ N ] ) , the action-value (Q) and state-value (V) functions for the leader and followers are deﬁned by

Original: [Equation]
Revised: [Equation]

Original: where the expectation E π,ν,h,x,a,b is taken over state-action pairs induced by the policies ( π, ν
Revised: where the expectation E π,ν,h,x,a,b is taken over state-action pairs induced by the policies ( π, ν

Original: = { ν f i } i ∈
Revised: = { ν f i } i ∈

Original: [ N ] ) and the transition probability, when initializing the process with the triplet ( s, a, b = { b i } i ∈ [ N ] ) at step h .
Revised: [ N ] ) and the transition probability, when initializing the process with the triplet ( s, a, b = { b i } i ∈ [ N ] ) at step h .

Original: For notational simplicity, when h, x, a, b are clear from the context, we omit h, x, a, b from E π,ν,h,x,a,b .
Revised: For notational simplicity, when h, x, a, b are clear from the context, we omit h, x, a, b from E π,ν,h,x,a,b .

Original: By the deﬁnition in (2.1), we have the Bellman equation
Revised: By the deﬁnition in (2.1), we have the Bellman equation

Original: [Equation]
Revised: [Equation]

Original: where π h × ν h represents π h × ν f 1 ,h × · · · × ν f N ,h .
Revised: where π h × ν h represents π h × ν f 1 ,h × · · · × ν f N ,h .

Original: Here P h is the operator which is deﬁned by
Revised: Here P h is the operator which is deﬁned by

Original: [Equation]
Revised: [Equation]

Original: for any function f : S → R and ( x, a, b ) ∈ S
Revised: for any function f : S → R and ( x, a, b ) ∈ S

Original: ×
Revised: ×

Original: A l × A f .
Revised: A l × A f .

Original: 2. S TACKELBERG -N ASH E QUILIBRIUM
Revised: 2. S TACKELBERG -N ASH E QUILIBRIUM

Original: Given a leader policy π , a Nash equilibrium (Nash, 2016) of the followers is a joint policy ν ∗ =
Revised: Given a leader policy π , a Nash equilibrium (Nash, 2016) of the followers is a joint policy ν ∗ =

Original: { ν ∗ f i } i ∈ [ N ] , such that for any x ∈ S and ( i, h ) ∈
Revised: { ν ∗ f i } i ∈ [ N ] , such that for any x ∈ S and ( i, h ) ∈

Original: [ N ] × [ H ]
Revised: [ N ] × [ H ]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Here − i represents all indices in [ N ] except i .
Revised: Here − i represents all indices in [ N ] except i .

Original: For each leader policy π , we denote the set of bestresponse policies of the followers by BR ( π ) , which is deﬁned by
Revised: For each leader policy π , we denote the set of bestresponse policies of the followers by BR ( π ) , which is deﬁned by

Original: BR ( π ) =
Revised: BR ( π ) =

Original: { ν | ν is the NE of the followers given the leader policy π } .
Revised: { ν | ν is the NE of the followers given the leader policy π } .

Revised: [Equation]

Original: Given the best-response set BR ( π ) , we denote ν ∗ ( π ) the best-case responses, which break ties in favor of the leader.
Revised: Given the best-response set BR ( π ) , we denote ν ∗ ( π ) the best-case responses, which break ties in favor of the leader.

Original: This is also known as optimistic tie-breaking (Breton et al., 1988; Bucarey et al., 2019a).
Revised: This is also known as optimistic tie-breaking (Breton et al., 1988; Bucarey et al., 2019a).

Original: We will discuss pessimistic tie-breaking (Conitzer & Sandholm, 2006) in § I. Speciﬁcally, we deﬁne ν ∗ ( π ) by
Revised: We will discuss pessimistic tie-breaking (Conitzer & Sandholm, 2006) in § I. Speciﬁcally, we deﬁne ν ∗ ( π ) by

Original: [Equation]
Revised: [Equation]

Original: The Stackelberg-Nash equilibrium for the leader is the “best response to the best response”.
Revised: The Stackelberg-Nash equilibrium for the leader is the “best response to the best response”.

Original: In other words, we want to ﬁnd a leader’s policy π that maximizes her value function under the assumption that the followers always adopt ν ∗ ( π ) , i.e.,
Revised: In other words, we want to ﬁnd a leader’s policy π that maximizes her value function under the assumption that the followers always adopt ν ∗ ( π ) , i.e.,

Original: [Equation]
Revised: [Equation]

Original: A Stackelberg-Nash equilibrium of the general-sum game is a policy pair ( π ∗ , ν ∗ =
Revised: A Stackelberg-Nash equilibrium of the general-sum game is a policy pair ( π ∗ , ν ∗ =

Original: { ν ∗ f i } i ∈ [ N ] ) such that ν ∗ ∈
Revised: { ν ∗ f i } i ∈ [ N ] ) such that ν ∗ ∈

Original: ν ∗ ( π ∗ ) and π ∗ ∈ SNE l .
Revised: ν ∗ ( π ∗ ) and π ∗ ∈ SNE l .

Original: Our goal is to ﬁnd the Stackelberg equilibrium: the leader’s optimal strategy, under the assumption that the followers always play their best response (Nash equilibrium) to the leader.
Revised: Our goal is to ﬁnd the Stackelberg equilibrium: the leader’s optimal strategy, under the assumption that the followers always play their best response (Nash equilibrium) to the leader.

Original: Equivalently, we need to solve the following optimization problem:
Revised: Equivalently, we need to solve the following optimization problem:

Original: [Equation]
Revised: [Equation]

Original: We study this challenging bilevel optimization problem in both the online setting (Section 3) and the ofﬂine setting (Section 4) respectively.
Revised: We study this challenging bilevel optimization problem in both the online setting (Section 3) and the ofﬂine setting (Section 4) respectively.

Original:  L INEAR M ARKOV G AMES WITH M YOPIC F OLLOWERS
Revised: 2. L INEAR M ARKOV G AMES WITH M YOPIC F OLLOWERS

Original: Linear Markov Games.
Revised: Linear Markov Games.

Original: We study the linear Markov games (Xie et al., 2020), where the transition dynamics are linear in a feature map.
Revised: We study the linear Markov games (Xie et al., 2020), where the transition dynamics are linear in a feature map.

Original: Assumption 2.1 (Linear Markov Games) .
Revised: Assumption 2.1 (Linear Markov Games) .

Original: Markov game ( S , A l , A f = {A f i } i ∈
Revised: Markov game ( S , A l , A f = {A f i } i ∈

Original: [ N ] , H, r l , r f = { r f i } i ∈ [ N ] , P ) is a linear Markov game if there exists a feature map φ : S ×
Revised: [ N ] , H, r l , r f = { r f i } i ∈ [ N ] , P ) is a linear Markov game if there exists a feature map φ : S ×

Original: A l × A f
Revised: A l × A f

Original: → R d such that
Revised: → R d such that

Original: [Equation]
Revised: [Equation]

Original: for any ( x, a, b ) ∈ S×A l ×A f and h ∈
Revised: for any ( x, a, b ) ∈ S×A l ×A f and h ∈

Original: [ H ] .
Revised: [ H ] .

Original: Here µ h =
Revised: Here µ h =

Original: ( µ (1) h , µ (2) h , · · · , µ ( d ) h ) are d unknown signed measures over S .
Revised: ( µ (1) h , µ (2) h , · · · , µ ( d ) h ) are d unknown signed measures over S .

Original: Without loss of generality, we assume that (cid:107) φ ( · , · , · ) (cid:107) 2 ≤ 1 and (cid:107) µ h ( S ) (cid:107) ≤ √ d for all h ∈
Revised: Without loss of generality, we assume that (cid:107) φ ( · , · , · ) (cid:107) 2 ≤ 1 and (cid:107) µ h ( S ) (cid:107) ≤ √ d for all h ∈

Original: [ H ] .
Revised: [ H ] .

Original: The linear Markov game above is an extension of linear MDP studied in Jin et al.
Revised: The linear Markov game above is an extension of linear MDP studied in Jin et al.

Original: (2020b) for the single-agent RL.
Revised: (2020b) for the single-agent RL.

Original: Speciﬁcally, when the followers play ﬁxed and known policies, the linear Markov games reduce to the linear MDP.
Revised: Speciﬁcally, when the followers play ﬁxed and known policies, the linear Markov games reduce to the linear MDP.

Original: We also remark that Chen et al.
Revised: We also remark that Chen et al.

Original: (2021b) recently study another variant of linear Markov games.
Revised: (2021b) recently study another variant of linear Markov games.

Original: These two variants are incomparable in the sense that one does not imply the other.
Revised: These two variants are incomparable in the sense that one does not imply the other.

Original: Myopic Followers.
Revised: Myopic Followers.

Original: Throughout this paper, we make the following assumption.
Revised: Throughout this paper, we make the following assumption.

Original: Assumption 2.2 (Myopic followers) .
Revised: Assumption 2.2 (Myopic followers) .

Original: We assume that the followers are myopic.
Revised: We assume that the followers are myopic.

Original: Speciﬁcally, the followers at any step of the game do not consider the future rewards, but only the instantaneous rewards.
Revised: Speciﬁcally, the followers at any step of the game do not consider the future rewards, but only the instantaneous rewards.

Original: Formally, given a leader’s policy π , the NE of the myopic followers is deﬁned by the joint policy ν ∗ =
Revised: Formally, given a leader’s policy π , the NE of the myopic followers is deﬁned by the joint policy ν ∗ =

Original: { ν ∗ f i } i ∈ [ N ] , such that for any x ∈ S and ( i, h ) ∈
Revised: { ν ∗ f i } i ∈ [ N ] , such that for any x ∈ S and ( i, h ) ∈

Original: [ N ] × [ H ]
Revised: [ N ] × [ H ]

Original: [Equation]
Revised: [Equation]

Original: In other words, at each state x , the followers play a normal form game where the payoff matrices are determined only by the immediate reward functions and the leader’s policy.
Revised: In other words, at each state x , the followers play a normal form game where the payoff matrices are determined only by the immediate reward functions and the leader’s policy.

Original: Then, with slight abuse of notation, the best response set of the leader’s policy π is
Revised: Then, with slight abuse of notation, the best response set of the leader’s policy π is

Original: BR ( π ) =
Revised: [Equation]

Original: { ν | ν is the NE of the myopic followers given the leader policy π } .
Revised: myopic followers [Equation] leader policy [Equation]

Original: [Equation]
Revised: [Equation]

Original: And the best-case response ν ∗ ( π ) and Stackelberg-Nash equilibria SNE l follow the deﬁnitions in (2.6) and (2.7).
Revised: And the best-case response ν ∗ ( π ) and Stackelberg-Nash equilibria SNE l follow the deﬁnitions in (2.6) and (2.7).

Original: Leader-Controller Linear Markov Games.
Revised: Leader-Controller Linear Markov Games.

Original: A special case of the Markov games with myopic followers is leader-controller Markov games (Filar & Vrieze, 2012; Bucarey et al., 2019a), where the future state only depends on the current state and the leader’s action.
Revised: A special case of the Markov games with myopic followers is leader-controller Markov games (Filar & Vrieze, 2012; Bucarey et al., 2019a), where the future state only depends on the current state and the leader’s action.

Original: Such a setting is also well-motivated.
Revised: Such a setting is also well-motivated.

Original: One can consider the game where the leader is the government that dictates prices and the followers are companies.
Revised: One can consider the game where the leader is the government that dictates prices and the followers are companies.

Original: This is a leader-controller Markov game because the future state (price) is determined by the current state (price) and the leader (government).
Revised: This is a leader-controller Markov game because the future state (price) is determined by the current state (price) and the leader (government).

Original: Formally, it holds that
Revised: Formally, it holds that

Original: P h ( · | x, a, b ) =
Revised: P h ( · | x, a, b ) =

Original: P h ( · | x, a ) for any ( x, a, b ) ∈ S ×
Revised: P h ( · | x, a ) for any ( x, a, b ) ∈ S ×

Original: A l × A f and h ∈
Revised: A l × A f and h ∈

Original: [ H ] .
Revised: [ H ] .

Original: Hence, with slight abuse of notation, it is naturally to make the following assumption.
Revised: Hence, with slight abuse of notation, it is naturally to make the following assumption.

Original: Assumption 2.3 (Leader-Controller Linear Markov Games) .
Revised: Assumption 2.3 (Leader-Controller Linear Markov Games) .

Original: Markov game ( S , A l , A f = {A f i } i ∈
Revised: Markov game ( S , A l , A f = {A f i } i ∈

Original: [ N ] , H, r l , r f = { r f i } i ∈ [ N ] , P ) is a leader-controller linear Markov games if we assume the existence of a feature φ : S ×
Revised: [ N ] , H, r l , r f = { r f i } i ∈ [ N ] , P ) is a leader-controller linear Markov games if we assume the existence of a feature φ : S ×

Original: A l → R d such that
Revised: A l → R d such that

Original: [Equation]
Revised: [Equation]

Original:  for any ( s, a, b ) ∈
Revised: .11) for any ( s, a, b ) ∈

Original: A × A l × A f , where (cid:107) φ ( · , · ) (cid:107) 2 ≤ 1 and (cid:107) µ h ( S ) (cid:107) ≤ √ d for all h ∈
Revised: A × A l × A f , where (cid:107) φ ( · , · ) (cid:107) 2 ≤ 1 and (cid:107) µ h ( S ) (cid:107) ≤ √ d for all h ∈

Original: [ H ] .
Revised: [ H ] .

Original: Notably, Markov games with myopic followers subsume leader-controller Markov games as a special case.
Revised: Notably, Markov games with myopic followers subsume leader-controller Markov games as a special case.

Original: Speciﬁcally, since the followers’ policies cannot affect the following state, then the NE deﬁned in (2.4) is the same as (2.9), which further implies that the best-response set deﬁned in (2.5) is the same as (2.10).
Revised: Speciﬁcally, since the followers’ policies cannot affect the following state, then the NE deﬁned in (2.4) is the same as (2.9), which further implies that the best-response set deﬁned in (2.5) is the same as (2.10).

Original:  M AIN R ESULTS FOR THE O NLINE S ETTING
Revised:  M AIN R ESULTS FOR THE O NLINE S ETTING

Original: In this section, we study the online setting, where a central controller controls one leader l and
Revised: In this section, we study the online setting, where a central controller controls one leader l and

Original: N followers { f i } i ∈ [ N ] .
Revised: N followers { f i } i ∈ [ N ] .

Original: Our goal is to learn a Stackelberg-Nash equilibrium.
Revised: Our goal is to learn a Stackelberg-Nash equilibrium.

Original: In what follows, we formally describe the setup and learning objectives, and then present our algorithm and provide theoretic guarantees.
Revised: In what follows, we formally describe the setup and learning objectives, and then present our algorithm and provide theoretic guarantees.

Original: 3. S ETUP AND L EARNING O BJECTIVE
Revised: 3. S ETUP AND L EARNING O BJECTIVE

Original: We consider the setting where the reward functions r l and r f = { r f i } i ∈ [ N ] are revealed to the learner before the game.
Revised: We consider the setting where the reward functions r l and r f = { r f i } i ∈ [ N ] are revealed to the learner before the game.

Original: This is reasonable since in practice the reward functions are usually artiﬁcially designed.
Revised: This is reasonable since in practice the reward functions are usually artiﬁcially designed.

Original: Moreover, we focus on the episodic setting.
Revised: Moreover, we focus on the episodic setting.

Original: Speciﬁcally, a Markov game is played for K episodes, each of which consists of H timesteps.
Revised: Speciﬁcally, a Markov game is played for K episodes, each of which consists of H timesteps.

Original: At the beginning of the k -th episode, the leader and followers determine their policies ( π k , ν k
Revised: At the beginning of the k -th episode, the leader and followers determine their policies ( π k , ν k

Original: = { ν kf i } i ∈
Revised: = { ν kf i } i ∈

Original: [ N ] ) , and a ﬁxed initial state x k 1 = x 1 is chosen.
Revised: [ N ] ) , and a ﬁxed initial state x k 1 = x 1 is chosen.

Original: Here we assume the ﬁxed initial state just for ease of presentation, and our subsequent results can be generalized to the setting where x k 1 is picked from a ﬁxed distribution.
Revised: Here we assume the ﬁxed initial state just for ease of presentation, and our subsequent results can be generalized to the setting where x k 1 is picked from a ﬁxed distribution.

Original: Then the game proceeds as follows.
Revised: Then the game proceeds as follows.

Original: At each step h ∈
Revised: At each step h ∈

Original: [ H ] , the leader and the followers observe state x kh ∈ S and pick their own actions a kh ∼ π kh ( · | x kh ) and
Revised: [ H ] , the leader and the followers observe state x kh ∈ S and pick their own actions a kh ∼ π kh ( · | x kh ) and

Original: b kh = { b ki,h ∼ ν kf
Revised: b kh = { b ki,h ∼ ν kf

Original: i ,h ( · | x kh ) } i ∈ [ N ] .
Revised: i ,h ( · | x kh ) } i ∈ [ N ] .

Original: Subsequently, the environment transitions to the next state x kh +1 ∼ P h ( · | x kh , a kh , b kh ) .
Revised: Subsequently, the environment transitions to the next state x kh +1 ∼ P h ( · | x kh , a kh , b kh ) .

Original: Each episode terminates after H timesteps.
Revised: Each episode terminates after H timesteps.

Original: Learning Objective.
Revised: Learning Objective.

Original: Let ( π k , ν k
Revised: Let ( π k , ν k

Original: = { ν kf i } i ∈
Revised: = { ν kf i } i ∈

Original: [ N ] ) denote the policies executed by the algorithm in the k -th episode.
Revised: [ N ] ) denote the policies executed by the algorithm in the k -th episode.

Original: By the deﬁnition of the bilevel optimization problem in (2.8), we expect that ν k ∈ BR ( π k ) and that V π ∗ ,ν ∗ l, 
Revised: By the deﬁnition of the bilevel optimization problem in (2.8), we expect that ν k ∈ BR ( π k ) and that V π ∗ ,ν ∗ l, [Equation] l

Original: [Equation]
Revised: [Equation]

Original: ( x k 1 ) is small for any k ∈ [ K ] .
Revised: ( x k 1 ) is small for any k ∈ [ K ] .

Original: Hence, we evaluate the suboptimality of our algorithm by the regret, which is deﬁned as
Revised: Hence, we evaluate the suboptimality of our algorithm by the regret, which is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: The goal is to design algorithms with regret that is sublinear in K , and polynomial in d, H .
Revised: The goal is to design algorithms with regret that is sublinear in K , and polynomial in d, H .

Original: Here K is the number of episodes, d is the dimension of the feature map φ , and H is the episode horizon.
Revised: Here K is the number of episodes, d is the dimension of the feature map φ , and H is the episode horizon.

Original: 3. A LGORITHM
Revised: 3. A LGORITHM

Original: We now present our algorithm, Optimistic Value Iteration to Find Stackelberg-Nash Equilibrium (OVI-SNE), which is given in Algorithm 1. 
Revised: We now present our algorithm, Optimistic Value Iteration to Find Stackelberg-Nash Equilibrium (OVI-SNE), which is given in Algorithm 1. [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table] [Table] [Table]
Revised: · , H do 13:

Original: [Table] [Table] [Table] [Table]
Revised: Sample a kh ∼ π kh ( · | x kh ) , b k 1 ,h ∼ ν kf 1 ,h ( · | x kh ) , · · · , b kN,h ∼ ν kf N ,h ( · | x kh ) .

Original: [Table] [Table] [Table] [Table]
Revised: Leader takes action a kh ; Followers take actions b kh = { b ki,h i ∈ [ N ] .

Original: [Table] [Table] [Table] [Table]
Revised: Observe next state x kh +1 .

Original: , H do 13: Observe 16: end  17: end for 
Revised: 16: end for 17: end for the input x , let

Original:  Stackelberg-Nash  payoff 
Revised: | , { ν kf i ,h ( · | x ) } i [ N ] be the Stackelberg-Nash equilibrium for the matrix game with payoff matrices ( f i ,h ( x, · , · ) i

Original: At a high level, in each episode, our algorithm ﬁrst construct the policies for all players through backward induction with respect to the timestep h
Revised: At a high level, in each episode, our algorithm ﬁrst construct the policies for all players through backward induction with respect to the timestep h

Original: (Line 4-11), and then execute the policies to play the game (Line 12-16).
Revised: (Line 4-11), and then execute the policies to play the game (Line 12-16).

Original: In detail, at h -th step of k -th episode, OVI-SNE estimates leader’s Q-function based on the ( k − 1) historical trajectories.
Revised: In detail, at h -th step of k -th episode, OVI-SNE estimates leader’s Q-function based on the ( k − 1) historical trajectories.

Original: Inspired by previous optimistic least square value iteration (LSVI) algorithms (Jin et al., 2020b), for any h ∈
Revised: Inspired by previous optimistic least square value iteration (LSVI) algorithms (Jin et al., 2020b), for any h ∈

Original: [ H ] , we estimate the linear coefﬁcients by solving the following ridge regression problem:[Equation] +
Revised: [ H ] , we estimate the linear coefﬁcients by solving the following ridge regression problem:

Original: [Equation]
Revised: [Equation]

Original: τh
Revised: τh

Original: [Equation]
Revised: [Equation]

Original: a τh [Equation] b
Revised: , where V kh +1 (

Original: [Equation]
Revised: [Equation]

Original: (cid:62) w

Original: [Equation]

Original: (cid:107)
Revised: = (cid:104)

Original: [Equation]
Revised: [Equation]

Original: , where V kh +1 (

Original: [Equation]

Original: =

Original: [Equation]

Original: [Equation]

Original: π
Revised: π

Original: [Equation]
Revised: [Equation]

Original: +

Original: [Equation]

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original: By solving the ridge regression problem in (3.2), we have
Revised: By solving the ridge regression problem in (3.2), we have

Original: [Equation]
Revised: [Equation]

Original: where Λ k h
Revised: where Λ k h

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: To encourage exploration, we additionally adds a bonus function to estimate the leader’s Q-function:
Revised: To encourage exploration, we additionally adds a bonus function to estimate the leader’s Q-function:

Original: [Equation]
Revised: [Equation]

Original: where Γ kh ( · , · , ·) = β ·
Revised: where Γ kh ( · , · , ·

Original: [Equation]
Revised: [Equation]

Original: φ ( ·

Original: [Equation]

Original: [Equation]

Original: Here Γ kh : S ×
Revised: Here Γ kh : S ×

Original: A l → R is a bonus function and β > 0 is a parameter which will be speciﬁed later.
Revised: A l → R is a bonus function and β > 0 is a parameter which will be speciﬁed later.

Original: This form of bonus function is common in the literature of linear bandits (Lattimore & Szepesv´ari, 2020) and linear MDPs (Jin et al., 2020b).
Revised: This form of bonus function is common in the literature of linear bandits (Lattimore & Szepesv´ari, 2020) and linear MDPs (Jin et al., 2020b).

Original: Then, we construct policies for the leader and followers by the subroutine (cid:15) -SNE (Algorithm 2).
Revised: Then, we construct policies for the leader and followers by the subroutine (cid:15) -SNE (Algorithm 2).

Original: Speciﬁcally, let Q kh be the class of functions Q : S ×
Revised: Speciﬁcally, let Q kh be the class of functions Q : S ×

Original: A l × A f → R that takes form
Revised: A l × A f → R that takes form

Original: [Equation]
Revised: [Equation]

Original: where the parameters ( w, Λ) ∈ R d × R d × d satisfy (cid:107) w (cid:107) ≤ H √ dk and λ min (Λ) ≥ 1 .
Revised: where the parameters ( w, Λ) ∈ R d × R d × d satisfy (cid:107) w (cid:107) ≤ H √ dk and λ min (Λ) ≥ 1 .

Original: Moreover, let
Revised: Moreover, let

Original: Q kh,(cid:15) be a ﬁxed (cid:15) -covering of Q kh with respect to the (cid:96) ∞ norm.
Revised: Q kh,(cid:15) be a ﬁxed (cid:15) -covering of Q kh with respect to the (cid:96) ∞ norm.

Original: By Lemma D.10, we have Q kh ∈ Q kh , which allows us to pick a (cid:101) Q ∈ Q kh,(cid:15) such that (cid:107) (cid:101) Q − Q kh (cid:107) ∞ ≤ (cid:15) and calculate policies by
Revised: By Lemma D.10, we have Q kh ∈ Q kh , which allows us to pick a (cid:101) Q ∈ Q kh,(cid:15) such that (cid:107) (cid:101) Q − Q kh (cid:107) ∞ ≤ (cid:15) and calculate policies by

Original: [Equation]
Revised: [Equation]

Original: When there is only one follower, (3.6) requires ﬁnding a Stackelberg equilibrium in the matrix game.
Revised: When there is only one follower, (3.6) requires ﬁnding a Stackelberg equilibrium in the matrix game.

Original: Such a problem can be transformed to a linear programming (LP) problem (Conitzer & Sandholm, 2006; Von Stengel & Zamir, 2010), and thus can be solved efﬁciently.
Revised: Such a problem can be transformed to a linear programming (LP) problem (Conitzer & Sandholm, 2006; Von Stengel & Zamir, 2010), and thus can be solved efﬁciently.

Original: For the multi-follower case (i.e., N ≥ 2 ), however, solving such a matrix game in general is hard (Conitzer & Sandholm, 2006;
Revised: For the multi-follower case (i.e., N ≥ 2 ), however, solving such a matrix game in general is hard (Conitzer & Sandholm, 2006;

Original: Basilico et al., 2017a;b; Coniglio et al., 2020).
Revised: Basilico et al., 2017a;b; Coniglio et al., 2020).

Original: Given this computational hardness, we focus on the sample complexity and explicitly assume access to the following computational oracle:
Revised: Given this computational hardness, we focus on the sample complexity and explicitly assume access to the following computational oracle:

Original: Assumption 3.1.
Revised: Assumption 3.1.

Original: We assume access to an oracle that implements Line 3 of Algorithm 2.
Revised: We assume access to an oracle that implements Line 3 of Algorithm 2.

Original: Finally, the leader and the followers play the game according to the obtained policies.
Revised: Finally, the leader and the followers play the game according to the obtained policies.

Original: Remark 3.2.
Revised: Remark 3.2.

Original: Due to some technical challenge, the subroutine (cid:15) -SNE is necessary.
Revised: Due to some technical challenge, the subroutine (cid:15) -SNE is necessary.

Original: See § B.1 for more explanations.
Revised: See § B.1 for more explanations.

Original: 3. T HEORETICAL R ESULTS
Revised: 3. T HEORETICAL R ESULTS

Original: Our main theoretical result is the following bound on the regret incurred by Algorithm 1.
Revised: Our main theoretical result is the following bound on the regret incurred by Algorithm 1.

Original: Recall that the regret is deﬁned in (3.1) and T =
Revised: Recall that the regret is deﬁned in (3.1) and T =

Original: KH is the total number of timesteps.
Revised: KH is the total number of timesteps.

Original: Theorem 3.3.
Revised: Theorem 3.3.

Original: Under Assumptions 2.1, 2.2, and 3.1, there exists an absolute constant C > 0 such that, for any ﬁxed p ∈ (0 , 1) , by setting β = C · dH √ ι with ι = log(2 dT/p ) in Line 7 of Algorithmand (cid:15)
Revised: Under Assumptions 2.1, 2.2, and 3.1, there exists an absolute constant C > 0 such that, for any ﬁxed p ∈ (0 , 1) , by setting β = C · dH √ ι with ι = log(2 dT/p ) in Line 7 of Algorithmand (cid:15)

Original: = 1 KH in Algorithm 2, then we have ν k ∈ BR ( π k ) for any k ∈
Revised: = 1 KH in Algorithm 2, then we have ν k ∈ BR ( π k ) for any k ∈

Original: [ K ] .
Revised: [ K ] .

Original: Meanwhile, with probability at least 1 − p , the regret incurred by OVI-SNE satisﬁes that
Revised: Meanwhile, with probability at least 1 − p , the regret incurred by OVI-SNE satisﬁes that

Original: [Equation]
Revised: [Equation]

Original: The proof is deferred to Appendix D. We will discuss the Stackelberg equilibria learning, optimality of the bound, and unknown reward setting in Appendix B.2.
Revised: The proof is deferred to Appendix D. We will discuss the Stackelberg equilibria learning, optimality of the bound, and unknown reward setting in Appendix B.2.

Original:  M AIN R ESULTS FOR THE O FFLINE S ETTING
Revised:  M AIN R ESULTS FOR THE O FFLINE S ETTING

Original: In this section, we study the ofﬂine setting, where the central controller aims to ﬁnd a StackelbergNash equilibrium by an ofﬂine dataset.
Revised: In this section, we study the ofﬂine setting, where the central controller aims to ﬁnd a StackelbergNash equilibrium by an ofﬂine dataset.

Original: Below we describe the setup and learning objective, followed by our algorithm and theoretical results.
Revised: Below we describe the setup and learning objective, followed by our algorithm and theoretical results.

Original: 4. S ETUP AND L EARNING O BJECTIVE
Revised: 4. S ETUP AND L EARNING O BJECTIVE

Original: We study the ofﬂine setting, where the learner has access to the reward functions ( r l , r f = { r f i } Ni =1 ) and a dataset D = { ( x τh , a τh , b τh = { b τi,h } Ni =1 ) } K,Hτ,h =1 , which is collected a priori by some experimenter.
Revised: We study the ofﬂine setting, where the learner has access to the reward functions ( r l , r f = { r f i } Ni =1 ) and a dataset D = { ( x τh , a τh , b τh = { b τi,h } Ni =1 ) } K,Hτ,h =1 , which is collected a priori by some experimenter.

Original: Then we make a minimal assumption for the ofﬂine dataset.
Revised: Then we make a minimal assumption for the ofﬂine dataset.

Original: Assumption 4.1 (Compliance of Dataset) .
Revised: Assumption 4.1 (Compliance of Dataset) .

Original: We assume that the dataset D is compliant with the underlying Markov game ( S , A l , A f , H, r l , r f , P ) , that is, for any x (cid:48) ∈ S at step h ∈
Revised: We assume that the dataset D is compliant with the underlying Markov game ( S , A l , A f , H, r l , r f , P ) , that is, for any x (cid:48) ∈ S at step h ∈

Original: [ H ] of each trajectory τ ∈ [ K ] ,
Revised: [ H ] of each trajectory τ ∈ [ K ] ,

Original: [Equation]
Revised: [Equation]

Original: Here the probability on the left-hand side is with respect to the joint distribution over dataset D and the probability on the right-hand side is with respect to the underlying Markov game.
Revised: Here the probability on the left-hand side is with respect to the joint distribution over dataset D and the probability on the right-hand side is with respect to the underlying Markov game.

Original: Assumption 4.1 is adopted from Jin et al.
Revised: Assumption 4.1 is adopted from Jin et al.

Original: (2020c), which indicates the Markov property of the dataset
Revised: (2020c), which indicates the Markov property of the dataset

Original: D and that x τh +1 is generated by the underlying Markov game conditioned on ( x τh , a τh , b τh ) .
Revised: D and that x τh +1 is generated by the underlying Markov game conditioned on ( x τh , a τh , b τh ) .

Original: As a special case, Assumption 4.1 holds when the experimenter follows ﬁxed behavior policies.
Revised: As a special case, Assumption 4.1 holds when the experimenter follows ﬁxed behavior policies.

Original: More generally, Assumption 4.1 allows the experimenter to choose actions a τh and b τh arbitrarily, even in an adaptive or adversarial manner.
Revised: More generally, Assumption 4.1 allows the experimenter to choose actions a τh and b τh arbitrarily, even in an adaptive or adversarial manner.

Original: In particular, we can assume that a τh and b τh are interdependent across each trajectory τ ∈ [ K ] .
Revised: In particular, we can assume that a τh and b τh are interdependent across each trajectory τ ∈ [ K ] .

Original: For instance, the experimenter can sequentially improve the behavior policy using any online algorithm for Markov games.
Revised: For instance, the experimenter can sequentially improve the behavior policy using any online algorithm for Markov games.

Original: Learning Objective.
Revised: Learning Objective.

Original: Similar to the online setting, we deﬁne the following performance metric
Revised: Similar to the online setting, we deﬁne the following performance metric

Original: [Equation]
Revised: [Equation]

Original: which evaluates the suboptimality of policies ( π,
Revised: which evaluates the suboptimality of policies ( π,

Original: ν = { ν f i } Ni =1 ) given the initial state x ∈ S .
Revised: ν = { ν f i } Ni =1 ) given the initial state x ∈ S .

Original: 4. A LGORITHM
Revised: 4. A LGORITHM

Original: As is known to us, the key challenge of online setting is the the tradeoff between exploration and exploration.
Revised: As is known to us, the key challenge of online setting is the the tradeoff between exploration and exploration.

Original: In the online setting.
Revised: In the online setting.

Original: by following the “optimism in the face of uncertainty” principle (Sutton & Barto, 2018), we use bonus functions to incentivize exploration and thus achieve sampleefﬁcient.
Revised: by following the “optimism in the face of uncertainty” principle (Sutton & Barto, 2018), we use bonus functions to incentivize exploration and thus achieve sampleefﬁcient.

Original: This intrinsic challenge of online setting disappears in the ofﬂine setting because we do not need exploration any more.
Revised: This intrinsic challenge of online setting disappears in the ofﬂine setting because we do not need exploration any more.

Original: But another challenge arises: we only have access to the limited data.
Revised: But another challenge arises: we only have access to the limited data.

Original: To tackle this challenge, we need add some penalty functions to achieve robustness against the uncertainty due to the ﬁnite data.
Revised: To tackle this challenge, we need add some penalty functions to achieve robustness against the uncertainty due to the ﬁnite data.

Original: This is also known as pessimism (Yu et al., 2020; Jin et al., 2020c;
Revised: This is also known as pessimism (Yu et al., 2020; Jin et al., 2020c;

Original: Liu et al., 2020b; Buckman et al., 2020; Kidambi et al., 2020; Kumar et al., 2020; Rashidinejad et al., 2021).
Revised: Liu et al., 2020b; Buckman et al., 2020; Kidambi et al., 2020; Kumar et al., 2020; Rashidinejad et al., 2021).

Original: Here we simply ﬂip the sign of bonus functions deﬁned in (3.4) to serve as penalty functions.
Revised: Here we simply ﬂip the sign of bonus functions deﬁned in (3.4) to serve as penalty functions.

Original: See Algorithm 3 for details.
Revised: See Algorithm 3 for details.

Original: 4. T HEORETICAL R ESULTS
Revised: 4. T HEORETICAL R ESULTS

Original: Suppose that ( (cid:98) π
Revised: Suppose that ( (cid:98) π

Original: = { (cid:98) π h
Revised: = { (cid:98) π h

Original: Hh =1 , (cid:98) ν
Revised: Hh =1 , (cid:98) ν

Original: = { (cid:98) ν
Revised: = { (cid:98) ν

Original: f i = { ν f i ,h } Hh =1 } Ni =1 ) are the output policies of Algorithm 3.
Revised: f i = { ν f i ,h } Hh =1 } Ni =1 ) are the output policies of Algorithm 3.

Original: Then we evaluate the performance of ( (cid:98) π, (cid:98) ν ) by establishing an upper bound for the optimality gap deﬁned in (4.1).
Revised: Then we evaluate the performance of ( (cid:98) π, (cid:98) ν ) by establishing an upper bound for the optimality gap deﬁned in (4.1).

Original: Theorem 4.2.
Revised: Theorem 4.2.

Original: Under Assumptions 2.1, 2.2, 3.1, and 4.1, there exists an absolute constant C >such that, for any ﬁxed p ∈ (0 , 1) , by setting β (cid:48)
Revised: Under Assumptions 2.1, 2.2, 3.1, and 4.1, there exists an absolute constant C >such that, for any ﬁxed p ∈ (0 , 1) , by setting β (cid:48)

Original: =
Revised: =

Original: C · dH (cid:112) log(2 dHK/p ) in Line 6 of Algorithmand (cid:15)
Revised: C · dH (cid:112) log(2 dHK/p ) in Line 6 of Algorithmand (cid:15)

Original: = dKH in Algorithm 2, then it holds that (cid:98) ν ∈ BR ( (cid:98) π ) .
Revised: = dKH in Algorithm 2, then it holds that (cid:98) ν ∈ BR ( (cid:98) π ) .

Original: Meanwhile, with probability at least 1 − p , we have
Revised: Meanwhile, with probability at least 1 − p , we have

Original: SubOpt

Original: [Equation]
Revised: [Equation]

Original: where E π ∗ ,ν ∗ ,x is taken with respect to the trajectory incurred by ( π ∗ , ν ∗ ) in the underlying Markov game when initializing the progress at x .
Revised: where E π ∗ ,ν ∗ ,x is taken with respect to the trajectory incurred by ( π ∗ , ν ∗ ) in the underlying Markov game when initializing the progress at x .

Original: Here Λ h is deﬁned in Line 4 of Algorithm 3.
Revised: Here Λ h is deﬁned in Line 4 of Algorithm 3.

Original: The proof of Theorem 4.2 is deferred to § G. To illustrate our theory more, we will provide more comments on Theorem 4.2 in § C.2.
Revised: The proof of Theorem 4.2 is deferred to § G. To illustrate our theory more, we will provide more comments on Theorem 4.2 in § C.2.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we investigate the question of can we sample efﬁciently ﬁnd SNE in general-sum
Revised: In this paper, we investigate the question of can we sample efﬁciently ﬁnd SNE in general-sum

Original: Markov games with myopic followers and linear function approximation.
Revised: Markov games with myopic followers and linear function approximation.

Original: To the best of our knowledge, in both online and ofﬂine settings, we develop the ﬁrst sample efﬁcient reinforcement learning algorithms for solving SNE.
Revised: To the best of our knowledge, in both online and ofﬂine settings, we develop the ﬁrst sample efﬁcient reinforcement learning algorithms for solving SNE.

Original: We believe our work opens up many interesting directions for future work.
Revised: We believe our work opens up many interesting directions for future work.

Original: For example, we can ask the following questions:
Revised: For example, we can ask the following questions:

Original: Can we ﬁnd SNE in general-sum Markov games without the myopic followers assumption?
Revised: Can we ﬁnd SNE in general-sum Markov games without the myopic followers assumption?

Original: Can we design more computationally efﬁcient algorithms for solving SNE in general-sum Markov games?
Revised: Can we design more computationally efﬁcient algorithms for solving SNE in general-sum Markov games?

Original: Can we ﬁnd SNE in general-sum Markov games with general function approximation?
Revised: Can we ﬁnd SNE in general-sum Markov games with general function approximation?

Original: A R ELATED W ORK
Revised: A R ELATED W ORK

Original: In the sequel, we discuss the related works
Revised: In the sequel, we discuss the related works.

Original: . RL for solving NE in Markov games
Revised: RL for solving NE in Markov games

Original: Our work adds to the vast body of existing literature on
Revised: Our work adds to the vast body of existing literature on

Original: RL for ﬁnding Nash equilibria in Markov games.
Revised: RL for ﬁnding Nash equilibria in Markov games.

Original: In particular, there is a line of works that generalizes single-agent RL algorithms to Markov games under either the generative model (Azar et al., 2013) or ofﬂine settings with well-explored datasets (Littman, 2001; Greenwald et al., 2003; Hu &
Revised: In particular, there is a line of works that generalizes single-agent RL algorithms to Markov games under either the generative model (Azar et al., 2013) or ofﬂine settings with well-explored datasets (Littman, 2001; Greenwald et al., 2003; Hu &

Original: Wellman, 2003; Lagoudakis & Parr, 2012; Hansen et al., 2013; Perolat et al., 2015; Jia et al., 2019;
Revised: Wellman, 2003; Lagoudakis & Parr, 2012; Hansen et al., 2013; Perolat et al., 2015; Jia et al., 2019;

Original: Sidford et al., 2020; Cui & Yang,2020;
Revised: Sidford et al., 2020; Cui & Yang,

Original:  Fan et al., 2020; Daskalakis et al., 2021; Zhao et al., 2021).
Revised: ; Fan et al., 2020; Daskalakis et al., 2021; Zhao et al., 2021).

Original: These works all aim to ﬁnd the Nash equilibrium and their algorithms are generalizations of singleagent RL algorithms.
Revised: These works all aim to ﬁnd the Nash equilibrium and their algorithms are generalizations of singleagent RL algorithms.

Original: In particular, Littman (2001; 1994); Greenwald et al. (2003); Hu & Wellman (2003) generalize Q-learning (Watkins & Dayan, 1992) to Markov games and establish asymptotic convergence guarantees.
Revised: In particular, Littman (2001; 1994); Greenwald et al. (2003); Hu & Wellman (2003) generalize Q-learning (Watkins & Dayan, 1992) to Markov games and establish asymptotic convergence guarantees.

Original: Jia et al.
Revised: Jia et al.

Original: ; Sidford et al.
Revised: ; Sidford et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: (2020a)
Revised: (2020a)

Original: ; Cui & Yang (2020) propose variants of Q-learning or value iteration (Shapley, 1953) algorithms under the generative model setting.
Revised: ; Cui & Yang (2020) propose variants of Q-learning or value iteration (Shapley, 1953) algorithms under the generative model setting.

Original: Moreover, Perolat et al.
Revised: Moreover, Perolat et al.

Original: ; Fan et al.
Revised: ; Fan et al.

Original: (2020) study the sample efﬁciency of ﬁtted value iteration (Munos & Szepesv´ari, 2008) for zero-sum Markov games under the ofﬂine setting.
Revised: (2020) study the sample efﬁciency of ﬁtted value iteration (Munos & Szepesv´ari, 2008) for zero-sum Markov games under the ofﬂine setting.

Original: They assume the behavior policy is explorative in the sense that the concentrability coefﬁcients (Munos & Szepesv ´ ari, 2008) are uniformly bounded.
Revised: They assume the behavior policy is explorative in the sense that the concentrability coefﬁcients (Munos & Szepesv ´ ari, 2008) are uniformly bounded.

Original: Under similar assumptions, Daskalakis et al.
Revised: Under similar assumptions, Daskalakis et al.

Original: ; Zhao et al.
Revised: ; Zhao et al.

Original: (2021) study the sample complexity of policy gradient (Sutton et al., 1999) under the well-explored ofﬂine setting.
Revised: (2021) study the sample complexity of policy gradient (Sutton et al., 1999) under the well-explored ofﬂine setting.

Original: Moreover, under the online setting, there is a recent line of research that proposes provably efﬁcient RL algorithms for zero-sum Markov games.
Revised: Moreover, under the online setting, there is a recent line of research that proposes provably efﬁcient RL algorithms for zero-sum Markov games.

Original: See, e.g., Wei et al.
Revised: See, e.g., Wei et al.

Original: ; Bai et al.
Revised: ; Bai et al.

Original: (2020); Bai & Jin (2020);
Revised: (2020); Bai & Jin (2020);

Original: Liu et al.
Revised: Liu et al.

Original: (2020a)
Revised: (2020a)

Original: ; Tian et al.
Revised: ; Tian et al.

Original: Xie et al.
Revised: Xie et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: (2021b)
Revised: (2021b)

Original: and the references therein.
Revised: and the references therein.

Original: These works propose optimism-based algorithms and establish sublinear regret guarantees for ﬁnding NE.
Revised: These works propose optimism-based algorithms and establish sublinear regret guarantees for ﬁnding NE.

Original: Among these works, our work is particularly related to Xie et al.
Revised: Among these works, our work is particularly related to Xie et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: (2021b), whose algorithms also incorporate the linear function approximation.
Revised: (2021b), whose algorithms also incorporate the linear function approximation.

Original: Compared with these aforementioned works, we focus on solving the Stackelberg-Nash equilibrium, which involves a bilevel structure and is fundamentally different from the Nash equilibrium.
Revised: Compared with these aforementioned works, we focus on solving the Stackelberg-Nash equilibrium, which involves a bilevel structure and is fundamentally different from the Nash equilibrium.

Original: Thus, our work is not directly comparable.
Revised: Thus, our work is not directly comparable.

Original: Learning Stackelberg games
Revised: Learning Stackelberg games

Original: As for solving Stackelberg-Nash equilibrium, most of the existing results focus on the normal form game, which is equivalent to our Markov game with H = 1 .
Revised: As for solving Stackelberg-Nash equilibrium, most of the existing results focus on the normal form game, which is equivalent to our Markov game with H = 1 .

Original: Letchford et al.
Revised: Letchford et al.

Original: ; Blum et al.
Revised: ; Blum et al.

Original: ; Peng et al.
Revised: ; Peng et al.

Original: (2019) study learning Stackelberg equilibrium with a best response oracle.
Revised: (2019) study learning Stackelberg equilibrium with a best response oracle.

Original: In addition, Fiez et al.
Revised: In addition, Fiez et al.

Original: (2019) study the local convergence of ﬁrstorder methods for ﬁnding Stackelberg equilibria in general-sum games with differentiable reward functions, and Ghadimi & Wang (2018);
Revised: (2019) study the local convergence of ﬁrstorder methods for ﬁnding Stackelberg equilibria in general-sum games with differentiable reward functions, and Ghadimi & Wang (2018);

Original: Chen et al.
Revised: Chen et al.

Original: (2021a)
Revised: (2021a)

Original: ; Hong et al.
Revised: ; Hong et al.

Original: (2020) analyze the global convergence of ﬁrst-order methods for achieving global optimality of bilevel optimization.
Revised: (2020) analyze the global convergence of ﬁrst-order methods for achieving global optimality of bilevel optimization.

Original: A more related work is Bai et al.
Revised: A more related work is Bai et al.

Original: (2021), which studies the matrix Stackelberg game with bandit feedback.
Revised: (2021), which studies the matrix Stackelberg game with bandit feedback.

Original: This work also studies an RL extension where the leader has a ﬁnite action set and the follower is faced with an MDP speciﬁed by the leader’s action.
Revised: This work also studies an RL extension where the leader has a ﬁnite action set and the follower is faced with an MDP speciﬁed by the leader’s action.

Original: In comparison, we assume the leader knows the reward functions and the main challenge lies in the unknown transitions.
Revised: In comparison, we assume the leader knows the reward functions and the main challenge lies in the unknown transitions.

Original: Thus, our setting is different from that in Bai et al.
Revised: Thus, our setting is different from that in Bai et al.

Original: Furthermore, a more relevant work is (Bucarey et al., 2019b), which establishes the Bellman equation and value iteration algorithm for solving SNE in
Revised: Furthermore, a more relevant work is (Bucarey et al., 2019b), which establishes the Bellman equation and value iteration algorithm for solving SNE in

Original: Markov games.
Revised: Markov games.

Original: In comparison, we establish modiﬁcations of least-squares value iteration that are tailored to online and ofﬂine settings.
Revised: In comparison, we establish modiﬁcations of least-squares value iteration that are tailored to online and ofﬂine settings.

Original: Learning general-sum Markov games
Revised: Learning general-sum Markov games

Original: Liu et al.
Revised: Liu et al.

Original: (2020a)
Revised: (2020a)

Original: present the ﬁrst result of ﬁnding correlated equilibrium (CE) and coarse correlated equilibrium (CCE) in general-sum Markov games.
Revised: present the ﬁrst result of ﬁnding correlated equilibrium (CE) and coarse correlated equilibrium (CCE) in general-sum Markov games.

Original: However, their centralized algorithms suffer from the curse of many agents, that is, their sample complexity scales exponentially in the number of agents.
Revised: However, their centralized algorithms suffer from the curse of many agents, that is, their sample complexity scales exponentially in the number of agents.

Original: Recently, Song et al.
Revised: Recently, Song et al.

Original: ; Mao & Bas¸ar (2021); Jin et al.
Revised: ; Mao & Bas¸ar (2021); Jin et al.

Original: (2021) successfully escape the curse of many agents thanks to the decentralized structure of V-learning algorithm (Bai et al., 2020).
Revised: (2021) successfully escape the curse of many agents thanks to the decentralized structure of V-learning algorithm (Bai et al., 2020).

Original: Related single-agent RL methods Broadly speaking, our work is also related to the recent line of works that achieve sample efﬁciency in single-agent RL under the online setting.
Revised: Related single-agent RL methods Broadly speaking, our work is also related to the recent line of works that achieve sample efﬁciency in single-agent RL under the online setting.

Original: See, e.g., (Azar et al., 2017; Jin et al., 2018; Yang & Wang, 2019; Zanette & Brunskill, 2019; Jin et al., 2020b;
Revised: See, e.g., (Azar et al., 2017; Jin et al., 2018; Yang & Wang, 2019; Zanette & Brunskill, 2019; Jin et al., 2020b;

Original: Zhou et al., 2020; Ayoub et al., 2020; Yang & Wang, 2020; Zanette et al., 2020b;a; Zhang et al., 2020c;b;
Revised: Zhou et al., 2020; Ayoub et al., 2020; Yang & Wang, 2020; Zanette et al., 2020b;a; Zhang et al., 2020c;b;

Original: Agarwal et al., 2020) and the references therein.
Revised: Agarwal et al., 2020) and the references therein.

Original: In particular, following the optimism in the face of uncertainty principle, these works achieve near-optimal regret under either tabular or function approximation settings.
Revised: In particular, following the optimism in the face of uncertainty principle, these works achieve near-optimal regret under either tabular or function approximation settings.

Original: Meanwhile, for ofﬂine RL with an arbitrary dataset, various recent works propose to utilize pessimism for achieving robustness.
Revised: Meanwhile, for ofﬂine RL with an arbitrary dataset, various recent works propose to utilize pessimism for achieving robustness.

Original: See, e.g., (Yu et al., 2020;
Revised: See, e.g., (Yu et al., 2020;

Original: Kidambi et al., 2020; Kumar et al., 2020; Jin et al., 2020c;
Revised: Kidambi et al., 2020; Kumar et al., 2020; Jin et al., 2020c;

Original: Liu et al., 2020b; Buckman et al., 2020; Rashidinejad et al., 2021) and the references therein.
Revised: Liu et al., 2020b; Buckman et al., 2020; Rashidinejad et al., 2021) and the references therein.

Original: These aforementioned works all focus on the single-agent setting and we prove that optimism and pessimism also play an indispensable role in achieving sample efﬁciency in ﬁnding SNE.
Revised: These aforementioned works all focus on the single-agent setting and we prove that optimism and pessimism also play an indispensable role in achieving sample efﬁciency in ﬁnding SNE.

Original: B M ISSING P ARTS IN S ECTIONB. E XPLANATION OF (cid:15) - SUBROUTINE
Revised: B M ISSING P ARTS IN S ECTIONB. E XPLANATION OF (cid:15) - SUBROUTINE

Original: Now we explain the motivation for using the subroutine (cid:15) -SNE to construct policies instead of solving the matrix games with payoff matrices ( Q k h ( x, · , · ) , { r f i ,h ( x, · , · ) } i ∈ [ N ] ) directly.
Revised: Now we explain the motivation for using the subroutine (cid:15) -SNE to construct policies instead of solving the matrix games with payoff matrices ( Q k h ( x, · , · ) , { r f i ,h ( x, · , · ) } i ∈ [ N ] ) directly.

Original: By the deﬁnition of Q kh in (3.4), we know Q kh relies on the previous data via the estimated value function V kh + and feature maps { φ ( x τh , a τh , b τh ) } k − 1 τ =1 .
Revised: By the deﬁnition of Q kh in (3.4), we know Q kh relies on the previous data via the estimated value function V kh + and feature maps { φ ( x τh , a τh , b τh ) } k − 1 τ =1 .

Original: Similar to the analysis for linear MDPs (Jin et al., 2020b), we need to use a covering argument to establish uniform concentration bounds for all value V kh +1 .
Revised: Similar to the analysis for linear MDPs (Jin et al., 2020b), we need to use a covering argument to establish uniform concentration bounds for all value V kh +1 .

Original: Jin et al.
Revised: Jin et al.

Original: (2020b) directly constructs an (cid:15) -net for the value functions and establishes a polynomial log-covering number for this (cid:15) -net.
Revised: (2020b) directly constructs an (cid:15) -net for the value functions and establishes a polynomial log-covering number for this (cid:15) -net.

Original: This analysis, however, relies on that the policies executed by the players are greedy (deterministic), which is not valid for our setting.
Revised: This analysis, however, relies on that the policies executed by the players are greedy (deterministic), which is not valid for our setting.

Original: To overcome this technical issue, we construct an (cid:15) -net for Q-functions and solve an approximate matrix game.
Revised: To overcome this technical issue, we construct an (cid:15) -net for Q-functions and solve an approximate matrix game.

Original: Fortunately, by choosing a small enough (cid:15) , we can handle the errors caused by this approximation.
Revised: Fortunately, by choosing a small enough (cid:15) , we can handle the errors caused by this approximation.

Original: See § D for more details.
Revised: See § D for more details.

Original: Moreover, as shown in Xie et al.
Revised: Moreover, as shown in Xie et al.

Original: , this subroutine can be implemented efﬁciently without explicitly computing the exponentially large (cid:15) -net.
Revised: , this subroutine can be implemented efﬁciently without explicitly computing the exponentially large (cid:15) -net.

Original: B. M ORE D ISCUSSIONS ON T HOEREM 3. 
Revised: B. M ORE D ISCUSSIONS ON T HOEREM 3.

Original: Learning Stackelberg Equilibria.
Revised: Learning Stackelberg Equilibria.

Original: When there is only one follower, Stackelberg-Nash equilibrium reduces to the Stackelberg equilibrium (Simaan & Cruz, 1973; Conitzer & Sandholm, 2006;
Revised: When there is only one follower, Stackelberg-Nash equilibrium reduces to the Stackelberg equilibrium (Simaan & Cruz, 1973; Conitzer & Sandholm, 2006;

Original: Bai et al., 2021).
Revised: Bai et al., 2021).

Original: Thus, we partly answer the open problem in Bai et al.
Revised: Thus, we partly answer the open problem in Bai et al.

Original: (2021) on how to learn
Revised: (2021) on how to learn

Original: Stackelberg equilibria in general-sum Markov games (with myopic followers).
Revised: Stackelberg equilibria in general-sum Markov games (with myopic followers).

Original: Optimality of the Bound.
Revised: Optimality of the Bound.

Original: Assuming that the action of the follower won’t affect the transition kernel and reward function, the linear Markov games reduces to the linear MDP (Jin et al., 2020b).
Revised: Assuming that the action of the follower won’t affect the transition kernel and reward function, the linear Markov games reduces to the linear MDP (Jin et al., 2020b).

Original: Meanwhile, the lower bound established in Azar et al.
Revised: Meanwhile, the lower bound established in Azar et al.

Original: ; Jin et al.
Revised: ; Jin et al.

Original: (2018) for tabular MDPs and the lower bound established in Lattimore & Szepesv´ari (2020) for linear bandits directly imply a lower bound Ω( dH √ T ) for the linear MDPs, which further yields a lower bound Ω( dH √ T ) for our setting.
Revised: (2018) for tabular MDPs and the lower bound established in Lattimore & Szepesv´ari (2020) for linear bandits directly imply a lower bound Ω( dH √ T ) for the linear MDPs, which further yields a lower bound Ω( dH √ T ) for our setting.

Original: Ignoring the logarithmic factors, there is only a gap of √ dH between this lower bound and our upper bound.
Revised: Ignoring the logarithmic factors, there is only a gap of √ dH between this lower bound and our upper bound.

Original: We also point out that, by using the “Bernstein-type” bonus (Azar et al., 2017;
Revised: We also point out that, by using the “Bernstein-type” bonus (Azar et al., 2017;

Original: Jin et al., 2018;
Revised: Jin et al., 2018;

Original: Zhou et al., 2020), we can improve our upper bound by a factor of √ H .
Revised: Zhou et al., 2020), we can improve our upper bound by a factor of √ H .

Original: Here we do not apply this technique for the clarity of the analysis.
Revised: Here we do not apply this technique for the clarity of the analysis.

Original: Unknown Reward Setting.
Revised: Unknown Reward Setting.

Original: To relax the assumption that the reward is known, we consider the case where the reward functions are unknown.
Revised: To relax the assumption that the reward is known, we consider the case where the reward functions are unknown.

Original: At a high level, we ﬁrst conduct a reward-free exploration algorithm
Revised: At a high level, we ﬁrst conduct a reward-free exploration algorithm

Original: (Algorithm 4 in § E), a variant of Reward-Free RL-Explore algorithm in Jin et al.
Revised: (Algorithm 4 in § E), a variant of Reward-Free RL-Explore algorithm in Jin et al.

Original: (2020a), to obtain estimated reward functions { (cid:98) r l , (cid:98) r f 1 , · · · (cid:98) r f N } .
Revised: (2020a), to obtain estimated reward functions { (cid:98) r l , (cid:98) r f 1 , · · · (cid:98) r f N } .

Original: As asserted before, we can use
Revised: As asserted before, we can use

Original: Algorithm 1 to ﬁnd the SNE with respect to the known estimated reward functions { (cid:98) [Equation] f [Equation]
Revised: Algorithm 1 to ﬁnd the SNE with respect to the known estimated reward functions { (cid:98) r l , (cid:98) r f 1 , · · · (cid:98) r f N } .

Original: r f N } . Hence, we can obtain the approximate SNE if the value functions of estimated value functions are good approximation of the true value functions.
Revised: Hence, we can obtain the approximate SNE if the value functions of estimated value functions are good approximation of the true value functions.

Original: See § E for the detailed algorithm and theoretical guarantees.
Revised: See § E for the detailed algorithm and theoretical guarantees.

Original: C M ISSING P ARTS IN S ECTIONC. A LGORITHM
Revised: C M ISSING P ARTS IN S ECTION

Revised: C. A LGORITHM

Original: [Table]
Revised: [Table]

Original: ·| x ) ,b 1 ∼ (cid:98) ν f 1 ,h ( ·| x ) , ··· ,b N ∼ (cid:98) ν
Revised: [Table] [Table] [Table] | x ) ,b 1 ∼ (cid:98) ν f 1 ,h ( ·| x ) , ··· ,b N ∼ (cid:98) ν

Original: fN,h ( ·| x ) (cid:98) Q h ( x, a, b 1 , · · · , b N ) , ∀ x .
Revised: fN,h ( ·| x ) (cid:98) Q h ( x, a, b 1 , · · · , b N ) , ∀ x .

Original: 10: end for 11: Output: ( (cid:98) π
Revised: 10: end for 11: Output: ( (cid:98) π

Original: = { (cid:98) π h
Revised: = { (cid:98) π h

Original: Hh =1 , (cid:98) ν
Revised: Hh =1 , (cid:98) ν

Original: = { (cid:98) ν
Revised: = { (cid:98) ν

Original: f i = { ν f i ,h } Hh =1 } Ni =1 ) .
Revised: f i = { ν f i ,h } Hh =1 } Ni =1 ) .

Original: C. M ORE C OMMENTS ON T HEOREM 4. 
Revised: C. M ORE C OMMENTS ON T HEOREM 4.

Original: Minimal Assumption Requirement: Theorem 4.2 only relies on the compliance of the dataset with linear Markov games.
Revised: Minimal Assumption Requirement: Theorem 4.2 only relies on the compliance of the dataset with linear Markov games.

Original: Compared with existing literature on ofﬂine RL (Bertsekas & Tsitsiklis, 1996; Antos et al., 2007; 2008; Munos & Szepesv ´ ari, 2008; Farahmand et al., 2010; 2016;
Revised: Compared with existing literature on ofﬂine RL (Bertsekas & Tsitsiklis, 1996; Antos et al., 2007; 2008; Munos & Szepesv ´ ari, 2008; Farahmand et al., 2010; 2016;

Original: Scherrer et al., 2015; Liu et al., 2018; Chen & Jiang, 2019; Fan et al., 2020; Xie & Jiang, 2020), we impose no restrictions on the coverage of the dataset.
Revised: Scherrer et al., 2015; Liu et al., 2018; Chen & Jiang, 2019; Fan et al., 2020; Xie & Jiang, 2020), we impose no restrictions on the coverage of the dataset.

Original: Meanwhile, we need no assumption on the afﬁnity between ( (cid:98) π, (cid:98) ν ) and the behavior policies that induce the dataset, which is often employed as a regularizer (Fujimoto et al., 2019; Laroche et al., 2019;
Revised: Meanwhile, we need no assumption on the afﬁnity between ( (cid:98) π, (cid:98) ν ) and the behavior policies that induce the dataset, which is often employed as a regularizer (Fujimoto et al., 2019; Laroche et al., 2019;

Original: Jaques et al., 2019; Wu et al., 2019; Kumar et al., 2019; Wang et al., 2020; Siegel et al., 2020; Nair et al., 2020;
Revised: Jaques et al., 2019; Wu et al., 2019; Kumar et al., 2019; Wang et al., 2020; Siegel et al., 2020; Nair et al., 2020;

Original: Liu et al., 2020b).
Revised: Liu et al., 2020b).

Original: Dataset with Sufﬁcient Coverage:
Revised: Dataset with Sufﬁcient Coverage:

Original: In what follows, we specialize Theorem 4.2 to the setting where we assume the dataset with good “coverage”.
Revised: In what follows, we specialize Theorem 4.2 to the setting where we assume the dataset with good “coverage”.

Original: Note that Λ h is determined by the ofﬂine dataset D and acts as a ﬁxed matrix in the expectation, that is, the expectation in (4.2) is only taken with the trajectory induced by ( π ∗ , ν ∗ ) .
Revised: Note that Λ h is determined by the ofﬂine dataset D and acts as a ﬁxed matrix in the expectation, that is, the expectation in (4.2) is only taken with the trajectory induced by ( π ∗ , ν ∗ ) .

Original: As proofed in the following theorem, when the trajectory induced by ( π ∗ , ν ∗ ) is “covered” by the dataset D sufﬁciently well, we can establish that the suboptimality incurred by Algorithm 3 diminishes at rate of (cid:101)
Revised: As proofed in the following theorem, when the trajectory induced by ( π ∗ , ν ∗ ) is “covered” by the dataset D sufﬁciently well, we can establish that the suboptimality incurred by Algorithm 3 diminishes at rate of (cid:101)

Original: O
Revised: O

Original: (1 / √ K ) .
Revised: (1 / √ K ) .

Original: Corollary C.1.
Revised: Corollary C.1.

Original: Suppose it holds with probability at least 1 − p/ 2 that
Revised: Suppose it holds with probability at least 1 − p/ 2 that

Original: [Equation]
Revised: [Equation]

Original: for all ( x, h ) ∈ S × [ H ] .
Revised: for all ( x, h ) ∈ S × [ H ] .

Original: Here c > 0 is an absolute constant and E π ∗ ,ν ∗ ,x is taken with respect to the trajectory incurred by ( π ∗ , ν ∗ ) in the underlying Markov game when initializing the progress at x .
Revised: Here c > 0 is an absolute constant and E π ∗ ,ν ∗ ,x is taken with respect to the trajectory incurred by ( π ∗ , ν ∗ ) in the underlying Markov game when initializing the progress at x .

Original: Under Assumptions 2.1, 2.2, 3.1 and 4.1, there exists an absolute constant C > 0 such that, for any ﬁxed p ∈ (0 , 1) , by setting β (cid:48)
Revised: Under Assumptions 2.1, 2.2, 3.1 and 4.1, there exists an absolute constant C > 0 such that, for any ﬁxed p ∈ (0 , 1) , by setting β (cid:48)

Original: =
Revised: =

Original: C · dH (cid:112) log(4 dHK/p ) in Line 6 of Algorithm 3 and (cid:15)
Revised: C · dH (cid:112) log(4 dHK/p ) in Line 6 of Algorithm 3 and (cid:15)

Original: = dKH in Algorithm 2, then it holds with probability at least 1 − p that
Revised: = dKH in Algorithm 2, then it holds with probability at least 1 − p that

Original: [Equation]
Revised: [Equation]

Original: for all x ∈ S .
Revised: for all x ∈ S .

Original: Here ¯ C is another absolute constant that only depends on c and C .
Revised: Here ¯ C is another absolute constant that only depends on c and C .

Original: Proof.
Revised: Proof.

Original: See § H for a detailed proof.
Revised: See § H for a detailed proof.

Original: Note that, unlike the previous literature (Antos et al., 2007; Munos & Szepesv´ari, 2008; Farahmand et al., 2010; 2016; Scherrer et al., 2015; Liu et al., 2018;
Revised: Note that, unlike the previous literature (Antos et al., 2007; Munos & Szepesv´ari, 2008; Farahmand et al., 2010; 2016; Scherrer et al., 2015; Liu et al., 2018;

Original: Chen & Jiang, 2019; Fan et al., 2020; Xie & Jiang, 2020) which relies on the “uniform coverage” assumption, Corollary C.1 only assumes that the dataset has a good coverage of the trajectory incurred by the policies ( π ∗ , ν ∗ ) .
Revised: Chen & Jiang, 2019; Fan et al., 2020; Xie & Jiang, 2020) which relies on the “uniform coverage” assumption, Corollary C.1 only assumes that the dataset has a good coverage of the trajectory incurred by the policies ( π ∗ , ν ∗ ) .

Original: Optimality of the Bound: Fix x ∈ S .
Revised: Optimality of the Bound: Fix x ∈ S .

Original: Assuming r l = r f i for any i ∈
Revised: Assuming r l = r f i for any i ∈

Original: [ N ] , we know ( π ∗ , ν ∗ ) =
Revised: [ N ] , we know ( π ∗ , ν ∗ ) =

Original: argmax π,ν V π,νl, 1 ( x ) .
Revised: argmax π,ν V π,νl, 1 ( x ) .

Original: Then the information-theoretic lower bound for ofﬂine singleagent RL (e.g., Theorem 4.7 in Jin et al.
Revised: Then the information-theoretic lower bound for ofﬂine singleagent RL (e.g., Theorem 4.7 in Jin et al.

Original: (2020c)) can imply the information-theoretic lower bound Ω( (cid:80)
Revised: (2020c)) can imply the information-theoretic lower bound Ω( (cid:80)

Original: Hh =1 E π ∗ ,ν ∗ ,x [( φ ( s h , a h , b h ) (cid:62) (Λ h ) − 1 φ ( s h , a h , b h )) 1 / 2 ]) for our setting.
Revised: Hh =1 E π ∗ ,ν ∗ ,x [( φ ( s h , a h , b h ) (cid:62) (Λ h ) − 1 φ ( s h , a h , b h )) 1 / 2 ]) for our setting.

Original: In particular, our upper bound established in Theorem 4.2 matches this lower bound up to β (cid:48) and absolute constants and thus implies that our algorithm is nearly minimax optimal.
Revised: In particular, our upper bound established in Theorem 4.2 matches this lower bound up to β (cid:48) and absolute constants and thus implies that our algorithm is nearly minimax optimal.

Original: D P ROOF OF T HEOREM 3. 
Revised: D P ROOF OF T HEOREM 3.

Original: Proof of Theorem 3.3.
Revised: Proof of Theorem 3.3.

Original: By the myopic followers assumption, we have the following lemma.
Revised: By the myopic followers assumption, we have the following lemma.

Original: Lemma D.1.
Revised: Lemma D.1.

Original: For any k ∈ [ K ] , we have ν k ∈ BR ( π k ) .
Revised: For any k ∈ [ K ] , we have ν k ∈ BR ( π k ) .

Original: Here BR ( · ) is deﬁned in (2.10).
Revised: Here BR ( · ) is deﬁned in (2.10).

Original: Proof.
Revised: Proof.

Original: Combing the deﬁnition of ( π k , ν k ) in Line 9 of Algorithm 1 and the deﬁnition of the best response set in the Markov games with myopic followers in (2.10), we conclude the proof.
Revised: Combing the deﬁnition of ( π k , ν k ) in Line 9 of Algorithm 1 and the deﬁnition of the best response set in the Markov games with myopic followers in (2.10), we conclude the proof.

Original: Then we establish an upper bound for the regret deﬁned in (3.1).
Revised: Then we establish an upper bound for the regret deﬁned in (3.1).

Original: Recall the regret takes the following form
Revised: Recall the regret takes the following form

Original: [Equation]
Revised: [Equation]

Original: To facilitate our analysis, for any ( k, h ) ∈ [ K ] × [ H ] we deﬁne the model prediction error by
Revised: To facilitate our analysis, for any ( k, h ) ∈ [ K ] × [ H ] we deﬁne the model prediction error by

Original: [Equation]
Revised: [Equation]

Original: Moreover, for any ( k, h ) ∈ [ K ] × [ H ] , we deﬁne ζ 1 k,h and ζ 2 k,h as
Revised: Moreover, for any ( k, h ) ∈ [ K ] × [ H ] , we deﬁne ζ 1 k,h and ζ 2 k,h as

Original: [Equation]
Revised: [Equation]

Original: Recall that ( π k , ν k
Revised: Recall that ( π k , ν k

Original: = { ν kf i } i ∈
Revised: = { ν kf i } i ∈

Original: [ N ] ) are the policies executed by the leader and the followers in the k -th episode, which generate a trajectory { x kh , a kh , b kh = { b ki,h } i ∈
Revised: [ N ] ) are the policies executed by the leader and the followers in the k -th episode, which generate a trajectory { x kh , a kh , b kh = { b ki,h } i ∈

Original: [ N ] } h ∈ [ H ] .
Revised: [ N ] } h ∈ [ H ] .

Original: Thus, we know that ζ 1 k,h and ζ 2 k,h characterize the randomness of choosing actions a kh ∼ π kh ( · | x kh ) and b kh ∼ ν kh ( · | x kh ) and the randomness of drawing the next state x kh +1 ∼ P h ( · | x kh , a kh , b kh ) , respectively.
Revised: Thus, we know that ζ 1 k,h and ζ 2 k,h characterize the randomness of choosing actions a kh ∼ π kh ( · | x kh ) and b kh ∼ ν kh ( · | x kh ) and the randomness of drawing the next state x kh +1 ∼ P h ( · | x kh , a kh , b kh ) , respectively.

Original: To establish an upper bound for (D.1), we introduce the following lemma, which decomposes this term into three parts using the notations deﬁned above.
Revised: To establish an upper bound for (D.1), we introduce the following lemma, which decomposes this term into three parts using the notations deﬁned above.

Original: Lemma D.2 (Regret Decomposition) .
Revised: Lemma D.2 (Regret Decomposition) .

Original: We can decompose (D.1) as follows.
Revised: We can decompose (D.1) as follows.

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 1): Computational Error +
Revised: 1): Computational Error +

Original: [Equation]
Revised: [Equation]

Original: (cid:88) k
Revised: (cid:88) k

Original: =
Revised: =

Revised: [Equation]

Original: H (cid:88)
Revised: (cid:88)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:123)(cid:122) (cid:125)
Revised: (cid:125)

Original: [Equation]
Revised: [Equation]

Original: l.
Revised: l.

Original: [Equation]
Revised: [Equation]

Original: ): Statistical Error
Revised: ): Statistical Error

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original:  (cid:124) 
Revised: ) (cid:124) (cid:123)(cid:122) (cid:125) ( l. 3): Randomness , where

Original: [Equation]
Revised: [Equation]

Original: (cid:125) ( l.
Revised: × ν

Original: [Equation]
Revised: [Equation]

Original: ): Randomness , where
Revised: −

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Revised: ) × ν

Revised: [Equation]

Original: =
Revised: =

Revised: [Equation]

Revised: Q kh

Original: [Equation]
Revised: [Equation]

Revised: kh , · , · , · · · ,

Revised: [Equation]

Revised: [Equation]

Revised: kh ) × ν

Revised: [Equation]

Revised: [Equation]

Original: kh
Revised: kh

Original: [Equation]
Revised: [Equation]

Original: ×
Revised: ×

Original: [Equation]
Revised: [Equation]

Revised: ν

Revised: [Equation]

Revised: kh )

Revised: [Equation]

Original: ν kf
Revised: π kh

Original: [Equation]
Revised: [Equation]

Original: kh
Revised: kh

Original: [Equation]
Revised: [Equation]

Revised: kf

Revised: [Equation]

Revised: [Equation]

Revised: kh )

Revised: ×

Revised: [Equation]

Revised: ν kf

Revised: [Equation]

Revised: kh )

Revised: [Equation]

Revised: l

Revised: [Equation]

Original: f .
Revised: f .

Original: Proof.
Revised: Proof.

Original: See § D.1 for a detailed proof.
Revised: See § D.1 for a detailed proof.

Original: Remark D.3.
Revised: Remark D.3.

Original: Similar regret decomposition results also appear in the single-agent RL literature (Cai et al., 2020; Efroni et al., 2020; Yang et al., 2020), and they can be regarded as the special case of
Revised: Similar regret decomposition results also appear in the single-agent RL literature (Cai et al., 2020; Efroni et al., 2020; Yang et al., 2020), and they can be regarded as the special case of

Original: Lemma D.2.
Revised: Lemma D.2.

Original: Moreover, our regret decomposition lemma is independent of the myopic followers assumption, and thus, can be applied to more general settings.
Revised: Moreover, our regret decomposition lemma is independent of the myopic followers assumption, and thus, can be applied to more general settings.

Original: Lemma D.2 states that the regret has three sources: (i) computational error, which represents the convergence of the algorithm with the known model, (ii) statistical error, that is, the error caused by the inaccurate estimation of the model, and (iii) randomness, as aforementioned, which comes from executing random policies and interaction with random environment.
Revised: Lemma D.2 states that the regret has three sources: (i) computational error, which represents the convergence of the algorithm with the known model, (ii) statistical error, that is, the error caused by the inaccurate estimation of the model, and (iii) randomness, as aforementioned, which comes from executing random policies and interaction with random environment.

Original: Returning to the main proof, we only need to characterize these three types of errors, respectively.
Revised: Returning to the main proof, we only need to characterize these three types of errors, respectively.

Original: We ﬁrst characterize the computational error by the following lemma.
Revised: We ﬁrst characterize the computational error by the following lemma.

Original: Lemma D.4 (Optimization Error) .
Revised: Lemma D.4 (Optimization Error) .

Original: It holds that
Revised: It holds that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: See § D.2 for a detailed proof.
Revised: See § D.2 for a detailed proof.

Original: Then, we establish an upper bound for the statistical error.
Revised: Then, we establish an upper bound for the statistical error.

Original: Due to the uncertainty that arises from only observing limited data, the model prediction errors can be possibly large for the triple ( x, a, b ) that are less visited or even unseen.
Revised: Due to the uncertainty that arises from only observing limited data, the model prediction errors can be possibly large for the triple ( x, a, b ) that are less visited or even unseen.

Original: Fortunately, however, we have the following lemma which characterizes the model prediction errors deﬁned in (D.2).
Revised: Fortunately, however, we have the following lemma which characterizes the model prediction errors deﬁned in (D.2).

Original: Lemma D.5 (Optimism) .
Revised: Lemma D.5 (Optimism) .

Original: It holds with probability at least 1 − p/ 2 that
Revised: It holds with probability at least 1 − p/ 2 that

Original: [Equation]
Revised: [Equation]

Original: for any ( k, h ) ∈
Revised: for any ( k, h ) ∈

Original: [ K ] ×
Revised: [ K ] ×

Original: [ H ] and ( x, a, b ) ∈ S ×
Revised: [ H ] and ( x, a, b ) ∈ S ×

Original: A l × A f .
Revised: A l × A f .

Original: Proof.
Revised: Proof.

Original: See § D.3 a detailed proof.
Revised: See § D.3 a detailed proof.

Original: Lemma D.5 states that δ kh ( x, a, b ) ≤ 0 for any ( x, a, b ) ∈ S ×
Revised: Lemma D.5 states that δ kh ( x, a, b ) ≤ 0 for any ( x, a, b ) ∈ S ×

Original: A × A .
Revised: A × A .

Original: Combining the deﬁnition of model prediction error in (D.2), we obtain
Revised: Combining the deﬁnition of model prediction error in (D.2), we obtain

Original: [Equation]
Revised: [Equation]

Original: which further implies that the estimated Q-function Q k(cid:63),h is “optimistic in the face of uncertainty”.
Revised: which further implies that the estimated Q-function Q k(cid:63),h is “optimistic in the face of uncertainty”.

Original: Moreover, Lemma D.5 implies that − δ kh
Revised: Moreover, Lemma D.5 implies that − δ kh

Original: ( x, a, b ) ≤ 2 min { H, Γ kh
Revised: ( x, a, b ) ≤ 2 min { H, Γ kh

Original: ( x, a, b ) } .
Revised: ( x, a, b ) } .

Original: Thus we only need to establish an upper bound for 2 (cid:80)
Revised: Thus we only need to establish an upper bound for 2 (cid:80)

Original: Kk =1 (cid:80)
Revised: Kk =1 (cid:80)

Original: Hh =1 min { H, Γ kh
Revised: Hh =1 min { H, Γ kh

Original: ( x kh , a kh , b kh ) } , which is the total price paid for the optimism.
Revised: ( x kh , a kh , b kh ) } , which is the total price paid for the optimism.

Original: As shown in the following lemma, we can derive an upper bound for this term by the elliptical potential lemma (Abbasi-Yadkori et al., 2011).
Revised: As shown in the following lemma, we can derive an upper bound for this term by the elliptical potential lemma (Abbasi-Yadkori et al., 2011).

Original: Lemma D.6.
Revised: Lemma D.6.

Original: For the bonus function Γ kh deﬁned in Line 7 of Algorithm 1, it holds that
Revised: For the bonus function Γ kh deﬁned in Line 7 of Algorithm 1, it holds that

Original: [Equation]
Revised: [Equation]

Original: Here p ∈ (0 , 1) and ι = log(2 dT/p ) are deﬁned in Theorem 3.3.
Revised: Here p ∈ (0 , 1) and ι = log(2 dT/p ) are deﬁned in Theorem 3.3.

Original: Proof.
Revised: Proof.

Original: See § D.4 for a detailed proof.
Revised: See § D.4 for a detailed proof.

Original: It remains to analyze the randomness, which is the purpose of the following lemma.
Revised: It remains to analyze the randomness, which is the purpose of the following lemma.

Original: Lemma D.7.
Revised: Lemma D.7.

Original: For the ζ 1 k,h and ζ 2 k,h deﬁned in Lemma D.2 and any p ∈ (0 , 1) , it holds with probability at least 1 − p/ 2 that
Revised: For the ζ 1 k,h and ζ 2 k,h deﬁned in Lemma D.2 and any p ∈ (0 , 1) , it holds with probability at least 1 − p/ 2 that

Original: K (cid:88) k =
Revised: [Equation]

Original: H
Revised: k

Original: [Equation]
Revised: [Equation]

Original: h
Revised: h

Revised: ( ζ k h + ζ 2 k h ) ≤

Original: [Equation]
Revised: [Equation]

Original: 16 KH
Revised: 16 KH

Original: [Equation] p ) .
Revised: · log(4 /p ) .

Original: Proof.
Revised: Proof.

Original: See § D.5 for a detailed proof.
Revised: See § D.5 for a detailed proof.

Original: Putting above lemmas together, we obtain
Revised: Putting above lemmas together, we obtain

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − p , which concludes the proof of Theorem 3.3.
Revised: with probability at least 1 − p , which concludes the proof of Theorem 3.3.

Original: D. P ROOF OF L EMMA D. 
Revised: D. P ROOF OF L EMMA D.

Original: First, we establish a more general regret decomposition lemma, which immediately implies Lemma
Revised: First, we establish a more general regret decomposition lemma, which immediately implies Lemma

Original: D.2.
Revised: D.2.

Original: Lemma D.8 (General Decomposition for One Episode) .
Revised: Lemma D.8 (General Decomposition for One Episode) .

Original: Fix k ∈
Revised: Fix k ∈

Original: [ K ] .
Revised: [ K ] .

Original: Suppose ( π k , ν k
Revised: Suppose ( π k , ν k

Original: = { ν kf i } i ∈
Revised: = { ν kf i } i ∈

Original: [ N ] ) are the policies executed by the leader l and the followers { f i } i ∈ [ N ] in the k -th episode.
Revised: [ N ] ) are the policies executed by the leader l and the followers { f i } i ∈ [ N ] in the k -th episode.

Original: Moreover, suppose that Q k(cid:63),h and V k(cid:63),h = (cid:104) Q k(cid:63),h , π kh × ν kh (cid:105) are the estimated Q-function and value function for any (cid:63) ∈ { l, f 1 , · · · , f N } at h -th step of k -th episode.
Revised: Moreover, suppose that Q k(cid:63),h and V k(cid:63),h = (cid:104) Q k(cid:63),h , π kh × ν kh (cid:105) are the estimated Q-function and value function for any (cid:63) ∈ { l, f 1 , · · · , f N } at h -th step of k -th episode.

Original: Then, for any policies ( π, ν = { ν f i } i ∈
Revised: Then, for any policies ( π, ν = { ν f i } i ∈

Original: [ N ] ) and (cid:63) ∈ { l, f 1 , · · · , f N } , we have[Equation] Computational
Revised: [ N ] ) and (cid:63) ∈ { l, f 1 , · · · , f N } , we have

Original: [Equation]
Revised: [Equation]

Original: ,h ( x h , a h , b h )] − δ k(cid:63),h
Revised: ,h ( x h , a h , b h )] − δ k(cid:63),h

Original: ( x kh , a kh , b kh ) (cid:1) (cid:124) (cid:123)(cid:122) (cid:125)
Revised: ( x kh , a kh , b kh ) (cid:1) (cid:124) (cid:123)(cid:122) (cid:125)

Original: Statistical Error +
Revised: Statistical Error +

Original: H (cid:88) h
Revised: H (cid:88) h

Original: = ( ζ 1 (cid:63),k,h + ζ 2 (cid:63),k,h ) (cid:124) (cid:123)(cid:122) (cid:125)
Revised: = ( ζ 1 (cid:63),k,h + ζ 2 (cid:63),k,h ) (cid:124) (cid:123)(cid:122) (cid:125)

Original: Randomness , where (cid:104) Q k(cid:63),h , π kh × ν kh (cid:105) =
Revised: Randomness , where (cid:104) Q k(cid:63),h , π kh × ν kh (cid:105) =

Original: (cid:104) Q k(cid:63),h , π kh ×
Revised: (cid:104) Q k(cid:63),h , π kh ×

Original: ν kf 1 ,h × · · · × ν kf N ,h (cid:105)
Revised: ν kf 1 ,h × · · · × ν kf N ,h (cid:105)

Original: A l ×A f and (cid:104) Q kh ( x kh , · , · ) , π ∗ h ( · | x kh ) ×
Revised: A l ×A f and (cid:104) Q kh ( x kh , · , · ) , π ∗ h ( · | x kh ) ×

Original: ν ∗ h ( · | x kh ) − π kh ( · | x kh )
Revised: ν ∗ h ( · | x kh ) − π kh ( · | x kh )

Original: × ν kh
Revised: × ν kh

Original: ( · | x kh ) (cid:105) =
Revised: ( · | x kh ) (cid:105) =

Original: (cid:104) Q kh ( x kh , · , · , · · · , · ) , π ∗ h ( · | x kh ) ×
Revised: (cid:104) Q kh ( x kh , · , · , · · · , · ) , π ∗ h ( · | x kh ) ×

Original: ν ∗ f 1 ,h ( · | x kh ) × · · · ν ∗ f N ,h ( · | x kh )
Revised: ν ∗ f 1 ,h ( · | x kh ) × · · · ν ∗ f N ,h ( · | x kh )

Original: − π kh ( · | x kh ) ×
Revised: − π kh ( · | x kh ) ×

Original: ν kf 1 ,h ( · | x kh ) × · · · ν kf N ,h ( · | x kh ) (cid:105)
Revised: ν kf 1 ,h ( · | x kh ) × · · · ν kf N ,h ( · | x kh ) (cid:105)

Original: A l ×A f .
Revised: A l ×A f .

Original: Here δ k(cid:63),h is the model prediction error deﬁned by δ k(cid:63),h
Revised: Here δ k(cid:63),h is the model prediction error deﬁned by [Equation]

Original: [-= r (cid:63),h + P h V k(cid:63),h +
-]
Revised: [-= r (cid:63),h + P h V k(cid:63),h +
-][Equation] [Equation]

Original: − Q k(cid:63),h , (D.5) and ζ 1 (cid:63),k,h and ζ 2 (cid:63),k,h are deﬁned by
Revised: [Equation] , (D.5) and ζ 1 (cid:63),k,h and ζ 2 (cid:63),k,h are deﬁned by

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma D.8.
Revised: Proof of Lemma D.8.

Original: To facilitate our analysis, for any ν
Revised: To facilitate our analysis, for any ν

Original: = { ν f i } i ∈
Revised: = { ν f i } i ∈

Original: [ N ] and ( h, x ) ∈
Revised: [ N ] and ( h, x ) ∈

Original: [ H ] × S , we denote ν f 1 ,h ( · | x ) × · · · ν f N ,h ( · | x ) by ν h ( · | x ) .
Revised: [ H ] × S , we denote ν f 1 ,h ( · | x ) × · · · ν f N ,h ( · | x ) by ν h ( · | x ) .

Original: Moreover, we deﬁne two operators J h and J k,h respectively by
Revised: Moreover [Equation] we deﬁne two operators J h and J k,h respectively by

Original: [Equation]
Revised: [Equation]

Original: for any h ∈ [ H ] and any function f : S × A l × A f → R .
Revised: for any h ∈ [ H ] and any function f : S × A l × A f → R .

Original: Also, we deﬁne
Revised: Also

Original: [Equation]
Revised: [Equation]

Original: ( x, · , · )

Original: [Equation] h [Equation] · | x ) ×
Revised: we deﬁne

Original: [Equation]
Revised: [Equation]

Original:  · | [Equation] [Equation] ( · | x 
Revised: x, · , · ) , π h ( · | x ) × ν h ( · | x ) − π kh ( · | x )

Original: [Equation]
Revised: [Equation]

Original:  kh ( · | x ) (cid:105) (D.8) for any ( h, x ) ∈
Revised: ν kh ( · | x ) (cid:105) (D.8) for any ( h, x ) ∈

Original: [ H ] × S and (cid:63) ∈ { l, f 1 , · · · , f N } .
Revised: [ H ] × S and (cid:63) ∈ { l, f 1 , · · · , f N } .

Original: Under the above notations, we decompose the regret at the k -th episode into the following two terms, 
Revised: Under the above notations, we decompose the regret at the k -th episode into the following two terms, [Equation] [Equation] D.

Original: [Equation]
Revised: [Equation]

Original: Then we characterize these two terms respectively.
Revised: Then we characterize these two terms respectively.

Original: Term (i).
Revised: Term (i).

Original: By the Bellman equation in (2.2) and the deﬁnition of the operator J h in (D.7), we have
Revised: By the Bellman equation in (2.2) and the deﬁnition of the operator J h in (D.7), we have

Original: V π,ν(cid:63),h =
Revised: V π,ν(cid:63),h =

Original: J h Q π,ν(cid:63),h .
Revised: J h Q π,ν(cid:63),h .

Original: Similar, by the deﬁnition of V k(cid:63),h and the deﬁnition of the operator J k,h in (D.7), we have V k(cid:63),h
Revised: Similar, by the deﬁnition of V k(cid:63),h and the deﬁnition of the operator J k,h in (D.7), we have V k(cid:63),h

Original: =
Revised: =

Original: J k,h Q k(cid:63),h .
Revised: J k,h Q k(cid:63),h .

Original: Hence, for any h ∈ [ H ] , we have
Revised: Hence, for any h ∈ [ H ] , we have

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is obtained by the fact that J h is a linear operator and the deﬁnition of ξ k(cid:63),h in (D.8).
Revised: where the last inequality is obtained by the fact that J h is a linear operator and the deﬁnition of ξ k(cid:63),h in (D.8).

Original: Meanwhile, by the Bellman equation in (2.2) and the deﬁnition of the prediction error δ k(cid:63),h in (D.2), we obtain 
Revised: Meanwhile, by the Bellman equation in (2.2) and the deﬁnition of the prediction error δ k(cid:63),h in (D.2), we obtain [Equation]

Original: [Equation]
Revised: [Equation]

Original: Putting (D.10) and (D.11) together, we further obtain
Revised: Putting (D.10) and (D.11) together, we further obtain

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: for any h ∈ [ H ] and (cid:63) ∈ { l, f 1 , · · · , f N } .
Revised: for any h ∈ [ H ] and (cid:63) ∈ { l, f 1 , · · · , f N } .

Original: By recursively applying (D.12) for all h ∈
Revised: By recursively applying (D.12) for all h ∈

Original: [ H ] , we have
Revised: [ H ] , we have

Original: V π,
Revised: V π,

Original: [Equation]
Revised: [Equation]

Original: where the last equality follows from the fact that V π,
Revised: where the last equality follows from the fact that V π,

Original: ν(cid:63),H +1 =
Revised: ν(cid:63),H +1 =

Original: V π k ,
Revised: V π k ,ν k

Original: [Equation]
Revised: [Equation]

Original: H +1 = 0 .
Revised: ,H +1 = 0 .

Original: Thus, by utilizing the deﬁnition of ξ k(cid:63),h in (D.8), we further obtain
Revised: Thus, by utilizing the deﬁnition of ξ k(cid:63),h in (D.8), we further obtain

Original: V π,ν
Revised: V π,ν(cid:63), 1 ( x k [Equation]

Original: [Equation]
Revised: [Equation]

Original: for any k ∈ [ K ] and (cid:63) ∈ { l, f 1 , · · · , f N } .
Revised: for any k ∈ [ K ] and (cid:63) ∈ { l, f 1 , · · · , f N } .

Original: Term (ii).
Revised: Term (ii).

Original: Recall that we denote { b kf i ,h } i ∈
Revised: Recall that we denote { b kf i ,h } i ∈

Original: [ N ] by b kh for any h ∈
Revised: [ N ] by b kh for any h ∈

Original: [ H ] .
Revised: [ H ] .

Original: Then, for any h ∈ [ H ] and (cid:63) ∈ { l, f 1 , · · · , f N } , by the deﬁnition of model prediction error in (D.5), we have δ k (cid:63),h ( x kh , a kh , b kh ) =
Revised: Then, for any h ∈ [ H ] and (cid:63) ∈ { l, f 1 , · · · , f N } , by the deﬁnition of model prediction error in (D.5), we have δ k (cid:63),h ( x kh , a kh , b kh ) =

Original: r k(cid:63),h
Revised: r k(cid:63),h

Original: ( x kh , a kh , b kh )
Revised: ( x kh , a kh , b kh )

Original: + ( P h V k(cid:63),h +1 )
Revised: + ( P [Equation] k(cid:63)

Original: ( x kh , a kh , b kh ) − Q k(cid:63)

Original: [Equation]
Revised: [Equation]

Original: h ( x kh , a kh , b kh)
Revised: b kh

Original: [Equation]
Revised: [Equation]

Revised: Q

Original: [Equation]
Revised: [Equation]

Revised: a

Original: [Equation]
Revised: [Equation]

Original: , b kh )
Revised: , b kh )

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: +

Original: [Equation]
Revised: [Equation]

Original: + [
Revised: ( x kh , a kh , b kh ) −

Original: [Equation]
Revised: [Equation]

Original: h ( x kh , a kh , b kh ) − Q k(cid:63),h
Revised: k(cid:63),h ( x kh , a kh , b kh )] = ( P

Original: ( x kh , a kh , b kh )]
Revised: [Equation]

Original: = ( P h V k(cid:63),h +1 − P h V π k ,ν k (cid:63),h +1 ) ( x kh , a kh , b kh )
Revised: [Equation]

Original: + ( Q π k ,ν k (cid:63),h − Q k(cid:63),h )( x kh , a kh , b kh ) (D.15) where the last equation is obtained by the Bellman equation in (2.2).
Revised: ,ν [Equation] (cid:63), [Equation]

Revised: a

Revised: [Equation]

Revised: +

Revised: [Equation]

Revised: π

Revised: [Equation]

Revised: ν k (cid:63)

Revised: [Equation]

Revised: h

Revised: [Equation]

Revised: [Equation]

Revised: a

Revised: [Equation]

Revised: b

Revised: [Equation]

Revised: .15) where the last equation is obtained by the Bellman equation in (2.2).

Original: Thus, by (D.15), we have
Revised: Thus, by (D.15), we have

Original: V k(cid:63),h
Revised: V k(cid:63),h

Original: ( x kh )
Revised: ( x kh )

Original: − V π k ,ν k (cid:63),h ( x kh ) =
Revised: − V π k ,ν k (cid:63),h ( x kh ) =

Original: V k(cid:63),h
Revised: V k(cid:63),h

Original: ( x kh )
Revised: ( x kh )

Original: − V π k ,ν k (cid:63),h ( x kh )
Revised: − V π k ,ν k (cid:63),h ( x kh )

Original: + ( Q π k ,ν k (cid:63),h − Q k(cid:63),h )( x kh , a kh , b kh )
Revised: + ( Q π k ,ν k (cid:63),h − Q k(cid:63),h )( x kh , a kh , b kh )

Original: + ( P h V k
Revised: + ( P h V k

Original: (cid:63),h +1 − P
Revised: (cid:63),h +1 − P

Original: h V π k ,ν k (cid:63),h +1 )
Revised: h V π k ,ν k (cid:63),h +1 )

Original: ( x k h , a k h , b k h ) − δ k
Revised: ( x k h , a k h , b k h ) − δ k

Original: (cid:63),h ( x k h , a k h , b k h ) =
Revised: (cid:63),h ( x k h , a k h , b k h ) =

Original: V k(cid:63),h
Revised: V k(cid:63),h

Original: ( x kh )
Revised: ( x kh )

Original: − V π k ,ν k (cid:63),h ( x kh ) − ( Q k(cid:63),h − Q π k ,ν k (cid:63),h )( x kh , a kh , b kh )
Revised: − V π k ,ν k (cid:63),h ( x kh ) − ( Q k(cid:63),h − Q π k ,ν k (cid:63),h )( x kh , a kh , b kh )

Original: + (cid:0) P h
Revised: + (cid:0) P h

Original: ( V k(cid:63),h
Revised: ( V k(cid:63),h

Original: +

Revised: +

Original: − V π k ,ν k (cid:63),h +1 ) (cid:1) ( x kh , a kh , b kh ) − ( V k(cid:63),h
Revised: − V π k ,ν k (cid:63),h +1 ) (cid:1) ( x kh , a kh , b kh ) − ( V k(cid:63),h

Original: +

Revised: +

Original: − V π k ,ν k (cid:63),h +1 )
Revised: − V π k ,ν k (cid:63),h +1 )

Original: ( x kh )
Revised: ( x kh )

Revised: +

Original: [Equation]
Revised: [Equation]

Original: for any h ∈ [ H ] and (cid:63) ∈ { l, f 1 , · · · , f N } .
Revised: for any h ∈ [ H ] and (cid:63) ∈ { l, f 1 , · · · , f N } .

Original: By the deﬁnitions of ζ 1 (cid:63),k,h and ζ 2 (cid:63),k,h in (D.6), (D.16) can be written as
Revised: By the deﬁnitions of ζ 1 (cid:63),k,h and ζ 2 (cid:63),k,h in (D.6), (D.16) can be written as

Original: [Equation]
Revised: [Equation]

Original: For any (cid:63) ∈ { l, f 1 , · · · , f N } , recursively expanding (D.17) across h ∈
Revised: For any (cid:63) ∈ { l, f 1 , · · · , f N } , recursively expanding (D.17) across h ∈

Original: [ H ] yields
Revised: [ H ] yields

Original: [Equation]
Revised: [Equation]

Original: where the last equality follows from the fact that V k(cid:63)
Revised: where the last equality follows from the fact that V k(cid:63)

Original: [Equation]
Revised: [Equation]

Original: Plugging (D.14) and (D.18) into (D.9), we obtain
Revised: Plugging (D.14) and (D.18) into (D.9), we obtain

Original: [Equation]
Revised: [Equation]

Original: for any ( π, ν ) and (cid:63) ∈ { l, f 1 , · · · , f N } .
Revised: for any ( π, ν ) and (cid:63) ∈ { l, f 1 , · · · , f N } .

Original: Therefore [Equation] we conclude the proof of Lemma D.
Revised: Therefore, we conclude the proof of Lemma D.2.

Original: [Equation]

Original: Proof of Lemma D.2.
Revised: Proof of Lemma D.2.

Original: For any k ∈ [ K ] , applying Lemma D.8 with ( π, ν ) =
Revised: For any k ∈ [ K ] , applying Lemma D.8 with ( π, ν ) =

Original: ( π ∗ , ν ∗ ) , we obtain 
Revised: ( π ∗ , ν ∗ ) , we obtain V π

Original: [Equation]
Revised: [Equation]

Original:  Taking summation over k ∈ [ K ] , we decompose (D.1) as desired, which concludes the proof of
Revised: ν [Equation] Taking summation over k ∈ [ K ] , we decompose (D.1) as desired, which concludes the proof of

Original: Lemma D.2.
Revised: Lemma D.2.

Original: D. P ROOF OF L EMMA D. 
Revised: D. P ROOF OF L EMMA D.

Original: Proof of Lemma D.4.
Revised: Proof of Lemma D.4.

Original: By the myopic followers assumption, we have that, for the matrix game with payoff matrices ( (cid:101) Q ( x kh , · , · ) , { r kf i ,h ( x kh , · , · ) } i ∈ [ N ] ) , ν ∗ h ( · | x kh ) belongs to the best response set of π ∗ h ( · | x kh ) .
Revised: By the myopic followers assumption, we have that, for the matrix game with payoff matrices ( (cid:101) Q ( x kh , · , · ) , { r kf i ,h ( x kh , · , · ) } i ∈ [ N ] ) , ν ∗ h ( · | x kh ) belongs to the best response set of π ∗ h ( · | x kh ) .

Original: Moreover, we deﬁne (cid:101) ν ∗ h ( · | x kh )
Revised: Moreover, we deﬁne (cid:101) ν ∗ h ( · | x kh )

Original: as the policy belongs to the best response set of π ∗ h ( · | x kh ) and breaks ties in favor of the leader.
Revised: as the policy belongs to the best response set of π ∗ h ( · | x kh ) and breaks ties in favor of the leader.

Original: Recall that ( π kh ( · | x kh ) , ν kh ( · | x kh )
Revised: Recall that ( π kh ( · | x kh ) , ν kh ( · | x kh )

Original: = { ν kf i ,h ( · | x kh ) } i ∈
Revised: = { ν kf i ,h ( · | x kh ) } i ∈

Original: [ N ] ) is the Stackelberg-Nash equilibrium of the matrix game with payoff matrices ( (cid:101) Q ( x kh , · , · , · ) , { r kf i ,h ( x kh , · , · , · ) } i ∈ [ N ] ) , which implies that π kh ( · | x kh ) is the “best response to the best response”, which further implies that
Revised: [ N ] ) is the Stackelberg-Nash equilibrium of the matrix game with payoff matrices ( (cid:101) Q ( x kh , · , · , · ) , { r kf i ,h ( x kh , · , · , · ) } i ∈ [ N ] ) , which implies that π kh ( · | x kh ) is the “best response to the best response”, which further implies that

Original: [Equation]
Revised: [Equation]

Original: D.
Revised: D.

Original: [Equation]
Revised: [Equation]

Original: for any ( k, h ) ∈
Revised: for any ( k, h ) ∈

Original: [ K ] ×
Revised: [ K ] ×

Original: [ H ] .
Revised: [ H ] .

Original: Thus, for any ( k, h ) ∈
Revised: Thus, for any ( k, h ) ∈

Original: [ K ] × [ H ] , we have
Revised: [ K ] × [ H ] [Equation] we have

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality follows from the deﬁnition of (cid:101)
Revised: where the ﬁrst inequality follows from the deﬁnition of (cid:101)

Original: ν kh ( · | x kh ) and the last inequality uses (D.19) and the fact that (cid:107) Q kh − (cid:101) Q (cid:107) ∞ ≤ (cid:15) .
Revised: ν kh ( · | x kh ) and the last inequality uses (D.19) and the fact that (cid:107) Q kh − (cid:101) Q (cid:107) ∞ ≤ (cid:15) .

Original: By taking summation over ( k, h ) ∈ [ K ] × [ H ] , we conclude the proof of Lemma D.4.
Revised: By taking summation over ( k, h ) ∈ [ K ] × [ H ] , we conclude the proof of Lemma D.4.

Original: D. P ROOF OF L EMMA D. 
Revised: D. P ROOF OF L EMMA D.

Original: Proof of Lemma D.5.
Revised: Proof of Lemma D.5.

Original: Recall that the estimated Q-function Q kh deﬁned in Line 8 of Algorithmtakes the following form:
Revised: Recall that the estimated Q-function Q kh deﬁned in Line 8 of Algorithmtakes the following form:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Here Λ kh and Γ kh are deﬁned in Lines 5 and 7 of Algorithm 1, respectively.
Revised: Here Λ kh and Γ kh are deﬁned in Lines 5 and 7 of Algorithm 1, respectively.

Original: Meanwhile, by Assumption 2.1, we have
Revised: Meanwhile, by Assumption 2.1, we have

Original: [Equation]
Revised: [Equation]

Original: for any ( k, h, x, a, b ) ∈ [ K ] × [ H ] × S ×
Revised: for any ( k, h, x, a, b ) ∈ [ K ] × [ H ] × S ×

Original: A l × A f .
Revised: A l × A f .

Original: Here (cid:104) µ h , V kh +1 (cid:105) =
Revised: Here (cid:104) µ h , V kh +1 (cid:105) =

Original: (cid:82) S V kh +

Revised: (cid:82) S V kh +

Original: ( x (cid:48) )
Revised: ( x (cid:48) )

Original: d µ h ( x (cid:48) ) .
Revised: d µ h ( x (cid:48) ) .

Original: Together with the deﬁnition of Λ kh in Line 5 of Algorithm 1, we further obtain ( P
Revised: Together with the deﬁnition of Λ kh in Line 5 of Algorithm 1, we further obtain ( P

Original: [Equation]
Revised: [Equation]

Original: k − 1 (cid:88) τ
Revised: k − 1 (cid:88) τ

Original: = φ ( x τh , a τh , b τh ) φ ( x τh , a τh , b τh ) (cid:62) (cid:104) µ h , V kh +

Revised: = φ ( x τh , a τh , b τh ) φ ( x τh , a τh , b τh ) (cid:62) (cid:104) µ h , V kh +

Original: (cid:105)
Revised: (cid:105)

Original: + (cid:104)
Revised: + (cid:104)

Original: µ h , V kh +1 (cid:105) (cid:17) =
Revised: µ h , V kh +1 (cid:105) (cid:17) =

Original: φ ( x, a, b ) (cid:62) (Λ kh ) − 1 (cid:16) k − 1 (cid:88)
Revised: φ ( x, a, b ) (cid:62) (Λ kh ) − 1 (cid:16) k − 1 (cid:88)

Original: τ
Revised: τ

Original: = φ ( x τh , a τh , b τh ) · ( P h V kh +1 )
Revised: = φ ( x τh , a τh , b τh ) · ( P h V kh +1 )

Original: ( x τh , a τh , b τh )
Revised: ( x τh , a τh , b τh )

Original: + (cid:104) µ h , V kh +1 (cid:105) (cid:17) , (D.24) for any ( k, h, x, a, b ) ∈
Revised: + (cid:104) µ h , V kh +1 (cid:105) (cid:17) , (D.24) for any ( k, h, x, a, b ) ∈

Original: [ K ] ×
Revised: [ K ] ×

Original: [ H ] ×S ×A l ×A f .
Revised: [ H ] ×S ×A l ×A f .

Original: Here the last equality uses (D.23).
Revised: Here the last equality uses (D.23).

Original: Putting (D.22) and (D.24) together, we have φ ( x, a, b ) (cid:62) w kh − ( P h V kh +1 )
Revised: Putting (D.22) and (D.24) together, we have φ ( x, a, b ) (cid:62) w kh − ( P h V kh [Equation]

Original: ( x, a, b ) =
Revised: 1 )( x, a, b ) =

Original: φ ( x, a, b ) (cid:62) (Λ kh ) − 1 (cid:16) k − 1 (cid:88)
Revised: φ ( x, a, b ) (cid:62) (Λ [Equation] [Equation] 1 (cid:16) k − 1 (cid:88)

Original: τ
Revised: [Equation]

Original: = φ ( x τh , a τh , b τh ) · (cid:0) V 
Revised: = φ ( x τh , a τh , b τh ) · (cid:0) V kh

Original: [-kh +
-]
Revised: [-kh +
-][Equation]

Original:  ( x τh +1 ) − ( P h V kh +1 )
Revised: 1 ( x τh [Equation] 1 ) − ( P h V kh +1 )

Original: ( x τh , a τh , b τh ) (cid:1)(cid:17) (cid:124) (cid:123)(cid:122) (cid:125) (i) (D.25) − φ ( x, a, b ) (cid:62) (Λ kh ) − 1 (cid:104)
Revised: ( x τh , a τh , b τh ) (cid:1)(cid:17) (cid:124) (cid:123)(cid:122) (cid:125) (i) (D.25) − φ ( x, a, b ) (cid:62) (Λ kh ) − 1 (cid:104)

Original: µ h , V kh +1 (cid:105)
Revised: µ h , V kh +1 (cid:105)

Original: (cid:124) (cid:123)(cid:122) (cid:125) (ii) for any ( k, h, x, a, b ) ∈
Revised: (cid:124) (cid:123)(cid:122) (cid:125) (ii) for any ( k, h, x, a, b ) ∈

Original: [ K ] ×
Revised: [ K ] ×

Original: [ H ] ×S×A l ×A f .
Revised: [ H ] ×S×A l ×A f .

Original: Then we upper bound these two terms respectively.
Revised: Then we upper bound these two terms respectively.

Original: Term (i).
Revised: Term (i).

Original: By Cauchy-Schwarz inequality, we have
Revised: By Cauchy-Schwarz inequality, we have

Original: [Equation]
Revised: [Equation]

Original: D

Original: [Equation]

Original: for any ( k, h, x, a ) ∈ [ K ] × [ H ] × S ×
Revised: for any ( k, h, x, a ) ∈ [ K ] × [ H ] × S ×

Original: A l .
Revised: A l .

Original: Under the event E deﬁned in Lemma D.9, we further have
Revised: Under the event E deﬁned in Lemma D.9, we further have

Original: [Equation]
Revised: [Equation]

Original: for any ( k, h, x, a ) ∈ [ K ] × [ H ] × S ×
Revised: for any ( k, h, x, a ) ∈ [ K ] × [ H ] × S ×

Original: A l .
Revised: A l .

Original: Term (ii).
Revised: Term (ii).

Original: Similarly, by Cauchy-Schwarz inequality, we obtain
Revised: Similarly, by Cauchy-Schwarz inequality, we obtain

Original: [Equation]
Revised: [Equation]

Original: for any ( k, h, x, a, b ) ∈ [ K ] × [ H ] × S ×
Revised: for any ( k, h, x, a, b ) ∈ [ K ] × [ H ] × S ×

Original: A l × A f .
Revised: A l × A f .

Original: Here the second inequality follows from the fact that Λ kh (cid:23) I and the last inequality is obtained by
Revised: Here the second inequality follows from the fact that Λ kh (cid:23) I and the last inequality is obtained by

Original: [Equation]
Revised: [Equation]

Original: Here we use the fact that (cid:107) V kh +1 (cid:107) ∞
Revised: Here we use the fact that (cid:107) V kh +1 (cid:107) ∞

Original: ≤ H and Assumption 2.1, which assumes (cid:107) µ h ( S ) (cid:107) 2 ≤ √ d .
Revised: ≤ H and Assumption 2.1, which assumes (cid:107) µ h ( S ) (cid:107) 2 ≤ √ d .

Original: Plugging (D.27) and (D.28) into (D.25), we obtain that
Revised: Plugging (D.27) and (D.28) into (D.25), we obtain that

Original: [Equation]
Revised: [Equation]

Original: (D.29) for any ( k, h, x, a, b ) ∈
Revised: D.29) for any ( k, h, x, a, b ) ∈

Original: [ K ] ×
Revised: [ K ] ×

Original: [ H ] × S ×
Revised: [ H ] × S ×

Original: A l × A f under the event E .
Revised: A l × A f under the event E .

Original: Here C > 0 is a constant.
Revised: Here C > 0 is a constant.

Original: By setting
Revised: By setting

Original: [Equation]
Revised: [Equation]

Original: in Line 7 of Algorithm 1, (D.29) gives
Revised: in Line 7 of Algorithm 1, (D.29) gives

Original: [Equation]
Revised: [Equation]

Original: for any ( k, h, x, a, b ) ∈ [ K ] × [ H ] × S ×
Revised: for any ( k, h, x, a, b ) ∈ [ K ] × [ H ] × S ×

Original: A l × A f under the event E .
Revised: A l × A f under the event E .

Original: Meanwhile, by the truncation in Line 8 of Algorithm 1 and the fact that r l,h ∈ [ − 1 , 1] , we have Q kh ∈ [ − ( H − h + 1) , H − h + 1] , which further implies that
Revised: Meanwhile, by the truncation in Line 8 of Algorithm 1 and the fact that r l,h ∈ [ − 1 , 1] , we have Q kh ∈ [ − ( H − h + 1) , H − h + 1] , which further implies that

Original: [Equation]
Revised: [Equation]

Original: for any ( k, h ) ∈
Revised: for any ( k, h ) ∈

Original: [ K ] ×
Revised: [ K ] ×

Original: [ H ] .
Revised: [ H ] .

Original: Hence, by (D.31), we have
Revised: Hence, by (D.31), we have

Original: [Equation]
Revised: [Equation]

Original: for any ( k, h, x, a, b ) ∈ [ K ] × [ H ] × S ×
Revised: for any ( k, h, x, a, b ) ∈ [ K ] × [ H ] × S ×

Original: A l × A f under the event E , where the last inequality is obtained by (D.32).
Revised: A l × A f under the event E , where the last inequality is obtained by (D.32).

Original: Thus, for the model prediction error deﬁned in (D.2), we have
Revised: Thus, for the model prediction error deﬁned in (D.2), we have

Original: [Equation]
Revised: [Equation]

Original: x, a, b
Revised: x, a, b

Original: [Equation]
Revised: [Equation]

Revised: =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: D
Revised: (D

Original: [Equation]
Revised: [Equation]

Original: 34) for any ( k, h, x, a, b ) ∈ [ K ] ×
Revised: 34) for any ( k, h, x, a, b ) ∈ [ K ] ×

Original: [ H ] × S ×
Revised: [ H ] × S ×

Original: A l × A f under the event E .
Revised: A l × A f under the event E .

Original: Moreover, by the deﬁnition of the model prediction error, we have − δ kh
Revised: Moreover, by the deﬁnition of the model prediction error, we have − δ kh

Original: ( · , · , · ) ≤ 2 H .
Revised: ( · , · , · ) ≤ 2 H .

Original: Together with (D.34), we have
Revised: Together with (D.34), we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: , b)
Revised: , b

Original: [Equation]
Revised: [Equation]

Original: D
Revised: , a, b )

Original: [Equation]
Revised: [Equation]

Original: 35) for any ( k, h, x, a, b ) ∈ [ K ] ×
Revised: (D.35) for any ( k, h, x, a, b ) ∈ [ K ] ×

Original: [ H ] × S ×
Revised: [ H ] × S ×

Original: A l × A f under the event E .
Revised: A l × A f under the event E .

Original: On the other hand, by (3.4), we have
Revised: On the other hand, by (3.4), we have

Original: [Equation]
Revised: [Equation]

Original: ) for any ( k, h, x, a, b ) ∈ [ K ] × [ H ] ×S ×A l ×A f under the event E .
Revised: D [Equation] 36) for any ( k, h, x, a, b ) ∈ [ K ] × [ H ] ×S ×A l ×A f under the event E .

Original: Here the last inequality follows from (D.31) and the fact that V kh +1 ≤ H − h .
Revised: Here the last inequality follows from (D.31) and the fact that V kh +1 ≤ H − h .

Original: Combining (D.35) and (D.36), we conclude the proof of Lemma D.5.
Revised: Combining (D.35) and (D.36), we conclude the proof of Lemma D.5.

Original: Lemma D.9.
Revised: Lemma D.9.

Original: For any p ∈ (0 , 1] , the event E that, for any ( k, h ) ∈ [ K ] ×
Revised: For any p ∈ (0 , 1] , the event E that, for any ( k, h ) ∈ [ K ] ×

Original: [ H ] ,
Revised: [ H ] ,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: happens with probability at least 1 − p/ 2 , where C (cid:48) > 0 is an absolute constant.
Revised: happens with probability at least 1 − p/ 2 , where C (cid:48) > 0 is an absolute constant.

Original: Proof of Lemma D.9.
Revised: Proof of Lemma D.9.

Original: Fix ( k, h ) ∈
Revised: Fix ( k, h ) ∈

Original: [ K ] ×
Revised: [ K ] ×

Original: [ H ] .
Revised: [ H ] .

Original: By Lemma D.10, we have w kh +1 ≤ H √ dk , which implies that Q kh +1 ∈ Q kh +1 ,(cid:15) .
Revised: By Lemma D.10, we have w kh +1 ≤ H √ dk , which implies that Q kh +1 ∈ Q kh +1 ,(cid:15) .

Original: Here Q kh +1 ,(cid:15) is deﬁned in (3.5).
Revised: Here Q kh +1 ,(cid:15) is deﬁned in (3.5).

Original: Moreover, as shown in Algorithm 2, we ﬁnd a (cid:101) Q in the (cid:15) -net
Revised: Moreover, as shown in Algorithm 2, we ﬁnd a (cid:101) Q in the (cid:15) -net

Original: Q kh +1 ,(cid:15) such that (cid:107) Q kh +1 − (cid:101) Q (cid:107) ∞ ≤ (cid:15) .
Revised: Q kh +1 ,(cid:15) such that (cid:107) Q kh +1 − (cid:101) Q (cid:107) ∞ ≤ (cid:15) .

Original: For any x ∈ S , let ( (cid:101) π ( · | x ) , (cid:101) ν
Revised: For any x ∈ S , let ( (cid:101) π ( · | x ) , (cid:101) ν

Original: = { ν f i } Ni =1 ) be the Stackelberg-Nash equilibrium of the matrix game with payoff matrices ( (cid:101) Q ( x, · , · ) , { r f i ,h +1 ( x, · , · ) } Ni =1 ) .
Revised: = { ν f i } Ni =1 ) be the Stackelberg-Nash equilibrium of the matrix game with payoff matrices ( (cid:101) Q ( x, · , · ) , { r f i ,h +1 ( x, · , · ) } Ni =1 ) .

Original: Moreover, we deﬁne (cid:101) V( x
Revised: Moreover, we deﬁne (cid:101) V

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: E

Original: [Equation]
Revised: [Equation]

Original: ν ( ·| x ) [ (cid:101) Q ( x, a, b )] for any x ∈ S .
Revised: ν ( ·| x ) [ (cid:101) Q ( x, a, b )] for any x ∈ S .

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: By Lemma J.2 and a union bound argument, it holds for any (cid:101) Q ∈ Q kh +1 ,(cid:15) with probability at least 1 − p/ 2 that
Revised: By Lemma J.2 and a union bound argument, it holds for any (cid:101) Q ∈ Q kh +1 ,(cid:15) with probability at least 1 − p/ 2 that

Original: [Equation]
Revised: [Equation]

Original: where N (cid:15) is the covering number of Q h +1 ,(cid:15) .
Revised: where N (cid:15) is the covering number of Q h +1 ,(cid:15) .

Original: Meanwhile, by applying Lemma J.4 with L = H √ dk and λ = 1 , (D.38) gives that
Revised: Meanwhile, by applying Lemma J.4 with L = H √ dk and λ = 1 , (D.38) gives that

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − p/ 2 .
Revised: with probability at least 1 − p/ 2 .

Original: Here C (cid:48) is a constant.
Revised: Here C (cid:48) is a constant.

Original: Meanwhile, by the deﬁnition of V k h +1 in
Revised: Meanwhile, by the deﬁnition of V k h +1 in

Original: Line 10 of Algorithm 1, we have V
Revised: Line 10 of Algorithm 1, we have V

Original: kh +

Revised: kh +

Original: ( x ) =
Revised: ( x ) =

Original: E a ∼ (cid:98) π ( ·| x ) ,b ∼ (cid:98) ν ( ·| x ) [
Revised: E a ∼ (cid:98) π ( ·| x ) ,b ∼ (cid:98) ν ( ·| x ) [

Original: Q kh +

Revised: Q kh +

Original: ( x, a, b )] , which yields that
Revised: ( x, a, b )] , which yields that

Original: [Equation]
Revised: [Equation]

Original: for any x ∈ S , which further implies that
Revised: for any x ∈ S , which further implies that

Original: [Equation]
Revised: [Equation]

Original: where the last inequality follows from the fact that (cid:107) φ 
Revised: where the last inequality follows from the fact that (cid:107) φ ( ·

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: (cid:107)

Original: [Equation]
Revised: [Equation]

Original: kh ) − 1 ≤ (cid:107) φ ( · , · , · ) (cid:107) 2 ≤ 1 for any ( k, h ) ∈ [ K ] × [ H ] .
Revised: ) − 1 ≤ (cid:107) φ ( · , · , · ) (cid:107) 2 ≤ 1 for any ( k, h ) ∈ [ K ] × [ H ] .

Original: Plugging (D.39) and (D.40) into (D.37), together with the fact that (cid:15) = 1 /KH , we conclude the proof of Lemma D.9.
Revised: Plugging (D.39) and (D.40) into (D.37), together with the fact that (cid:15) = 1 /KH , we conclude the proof of Lemma D.9.

Original: Lemma D.10 (Bounded Weight of Value Functions) .
Revised: Lemma D.10 (Bounded Weight of Value Functions) .

Original: For all ( k, h ) ∈ [ K ] × [ H ] , the linear coefﬁcient w k h deﬁned in (3.3) satisﬁes (cid:107) w k h (cid:107) ≤ H √ kd .
Revised: For all ( k, h ) ∈ [ K ] × [ H ] , the linear coefﬁcient w k h deﬁned in (3.3) satisﬁes (cid:107) w k h (cid:107) ≤ H √ kd .

Original: Proof of Lemma D.10.
Revised: Proof of Lemma D.10.

Original: By the deﬁnition of w kh in (3.3) and triangle inequality, we have
Revised: By the deﬁnition of w kh in (3.3) and triangle inequality, we have

Original: [Equation]
Revised: [Equation]

Original: Together with the fact that | V kh ( · ) | ≤ H for any ( k, h ) ∈ [ K ] ×
Revised: Together with the fact that | V kh ( · ) | ≤ H for any ( k, h ) ∈ [ K ] ×

Original: [ H ] , (D.41) gives
Revised: [ H ] , (D.41) gives

Original: [Equation]
Revised: [Equation]

Original: where the second inequality uses Cauchy-Schwarz inequality and the last inequality follows from the fact that Λ kh (cid:23)
Revised: where the second inequality uses Cauchy-Schwarz inequality and the last inequality follows from the fact that Λ kh (cid:23)

Original: I for any ( k, h ) ∈
Revised: I for any ( k, h ) ∈

Original: [ K ] ×
Revised: [ K ] ×

Original: [ H ] .
Revised: [ H ] .

Original: Then, by Cauchy-Schwarz inequality, we obtain k − 1 (cid:88)
Revised: Then, by Cauchy-Schwarz inequality, we obtain k − 1 (cid:88)

Original: [Equation]
Revised: [Equation]

Original: (D
Revised: (D

Original: [Equation]
Revised: [Equation]

Original: Meanwhile, recall that Λ kh = (cid:80) k − 1 τ =1 φ ( x τh , a τh , b τh ) φ ( x τh , a τh , b τh ) (cid:62) + I , we have
Revised: Meanwhile, recall that Λ kh = (cid:80) k − 1 τ =1 φ ( x τh , a τh , b τh ) φ ( x τh , a τh , b τh ) (cid:62) + I , we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  Plugging (D.43) and (D.44) into (D.42), we conclude the proof of Lemma D.10.
Revised: (D.44) Plugging (D.43) and (D.44) into (D.42), we conclude the proof of Lemma D.10.

Original: D. P ROOF OF L EMMA D. 
Revised: D. P ROOF OF L EMMA D.

Original: Proof of Lemma D.6.
Revised: Proof of Lemma D.6.

Original: Recall the deﬁnition of Γ kh in Line 7 of Algorithm 1, we have
Revised: Recall the deﬁnition of Γ kh in Line 7 of Algorithm 1, we have

Original: K (cid:88) k
Revised: K (cid:88) k

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: Here the last inequality uses the fact that β = CdH (cid:112) log(2 dT/p ) , where C > 1 is a constant.
Revised: Here the last inequality uses the fact that β = CdH (cid:112) log(2 dT/p ) , where C > 1 is a constant.

Original: By
Revised: By

Original: Cauchy-Schwarz inequality
Revised: Cauchy-Schwarz inequality

Original: [Equation]
Revised: [Equation]

Original: we further obtain that
Revised: we further obtain that

Original: [Equation]
Revised: [Equation]

Original: where the last inequality follows from Lemma J.1.
Revised: where the last inequality follows from Lemma J.1.

Original: Moreover, Assumption 2.1 gives that
Revised: Moreover, Assumption 2.1 gives that

Original: [Equation]
Revised: [Equation]

Original: for any ( k, h, x, a, b ) ∈ [ K ] × [ H ] × S × A , which further implies that
Revised: for any ( k, h, x, a, b ) ∈ [ K ] × [ H ] × S × A , which further implies that

Original: [Equation]
Revised: [Equation]

Original: for any h ∈ [ H ] .
Revised: for any h ∈ [ H ] .

Original: Combining (D.47) and the fact that Λ 1 h
Revised: Combining (D.47) and the fact that Λ 1 h

Original: =
Revised: =

Original: I , we obtain
Revised: I , we obtain

Original: [Equation]
Revised: [Equation]

Original: Combining (D.45), (D.46), (D.47) and (D.48), it holds that
Revised: Combining (D.45), (D.46), (D.47) and (D.48), it holds that

Original: [Equation]
Revised: [Equation]

Original: where ι = log(2 dT/p )
Revised: where ι = log(2 dT/p )

Original: Therefore, we conclude the proof of Lemma D.6.
Revised: Therefore, we conclude the proof of Lemma D.6.

Original: D. P ROOF OF L EMMA D. 
Revised: D. P ROOF OF L EMMA D.

Original: Proof of Lemma D.7.
Revised: Proof of Lemma D.7.

Original: First, we show that { ζ 1 k,h , ζ 2 k,h } ( k,h ) ∈
Revised: First, we show that { ζ 1 k,h , ζ 2 k,h } ( k,h ) ∈

Original: [ K ] × [ H ] can be written as a bounded martingale difference with respect to a ﬁltration.
Revised: [ K ] × [ H ] can be written as a bounded martingale difference with respect to a ﬁltration.

Original: Similar to Cai et al.
Revised: Similar to Cai et al.

Original: (2020), we construct the following ﬁltration.
Revised: (2020), we construct the following ﬁltration.

Original: For any ( k, h ) ∈ [ K ] × [ H ] , we deﬁne σ -algebras F 1 k,h and F 2 k,h as follows:
Revised: For any ( k, h ) ∈ [ K ] × [ H ] , we deﬁne σ -algebras F 1 k,h and F 2 k,h as follows:

Original: [Equation]
Revised: [Equation]

Original: where x H +1 is a null state for any k ∈
Revised: where x H +1 is a null state for any k ∈

Original: [ K ] .
Revised: [ K ] .

Original: Here σ ( · ) denotes the σ -algebra generated by a ﬁnite set.
Revised: Here σ ( · ) denotes the σ -algebra generated by a ﬁnite set.

Original: Moreover, for any ( k, h, m ) ∈
Revised: Moreover, for any ( k, h, m ) ∈

Original: [ K ] × [ H ] × [2] , we deﬁne the timestep index t ( k, h, m ) as
Revised: [ K ] × [ H ] × [2] , we deﬁne the timestep index t ( k, h, m ) as

Original: [Equation]
Revised: [Equation]

Original: By the deﬁnitions of σ -algebras in (D.49), we have F mk,h ⊂ F m (cid:48) k (cid:48) ,h (cid:48) for any t ( k, h, m ) ≤ t ( k (cid:48) , m (cid:48) , h (cid:48) ) , which implies that the σ -algebra sequence {F mk,h } ( k,h,m ) ∈ 
Revised: By the deﬁnitions of σ -algebras in (D.49), we have F mk,h ⊂ F m (cid:48) k (cid:48) ,h (cid:48) for any t ( k, h, m ) ≤ t ( k (cid:48) , m (cid:48) , h [Equation] ) , which implies that the σ -algebra sequence {F mk,h } ( k,h,m ) ∈ [ K ] ×

Original: [K ] × [ H ] × [2] is a ﬁltration.
Revised: [ H ] × [2] is a ﬁltration.

Original: Moreover, by the deﬁnitions of { ζ 1 k,h , ζ 2 k,h } ( k,h ) ∈ [ K ] × [ H ] in (D.3), we have
Revised: Moreover, by the deﬁnitions of { ζ 1 k,h , ζ 2 k,h } ( k,h ) ∈ [ K ] × [ H ] in (D.3), we have

Original: [Equation]
Revised: [Equation]

Original: 51) for any ( k, h ) ∈
Revised: ) for any ( k, h ) ∈

Original: [ K ] ×
Revised: [ K ] ×

Original: [ H ] .
Revised: [ H ] .

Original: Here we identify F 2 k, 0 with F 2 k − 1 ,H for any k ≥ 2 and deﬁne F 1 , 0 , 2 be the empty set.
Revised: Here we identify F 2 k, 0 with F 2 k − 1 ,H for any k ≥ 2 and deﬁne F 1 , 0 , 2 be the empty set.

Original: Hence, we can deﬁne the martingale
Revised: Hence, we can deﬁne the martingale

Original: [Equation]
Revised: [Equation]

Original: Such a martingale is adaptive to the ﬁltration {F mk,h } ( k,h,m ) ∈
Revised: Such a martingale is adaptive to the ﬁltration {F mk,h } ( k,h,m ) ∈

Original: [ K ] × [ H ] × [2] .
Revised: [ K ] × [ H ] × [2] .

Original: In particular, we have
Revised: In particular, we have

Original: [Equation]
Revised: [Equation]

Original: Moreover, note the fact that V kh , Q kh , V π k ,ν k l,h 
Revised: Moreover, note the fact that V kh , Q kh , V π k ,ν k l,h , Q π k

Original: [Equation]
Revised: [Equation]

Original: ∈ [ − H, H ] , we further obtain | ζ mk,h | ≤ 2 H , for any ( k, h, m ) ∈
Revised: ∈ [ − H, H ] , we further obtain | ζ mk,h | ≤ 2 H , for any ( k, h, m ) ∈

Original: [ K ] × [ H ] × [2] .
Revised: [ K ] × [ H ] × [2] .

Original: Finally, by applying the Azuma-Hoeffding inequality to M 2 K,H deﬁned in (D.53), we have
Revised: Finally, by applying the Azuma-Hoeffding inequality to M 2 K,H deﬁned in (D.53), we have

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − p/ 2 , which concludes the proof of Lemma D.7.
Revised: with probability at least 1 − p/ 2 , which concludes the proof of Lemma D.7.

Original: E U NKNOWN R EWARD S ETTING
Revised: E U NKNOWN R EWARD S ETTING

Original: We focus on the tabular case for simplicity, and the extension to linear case is left as future work.
Revised: We focus on the tabular case for simplicity, and the extension to linear case is left as future work.

Original: We assume that S = |S| , A l = |A l | and A f = |A
Revised: We assume that S = |S| , A l = |A l | and A f = |A

Original: f | = |A
Revised: f | = |A

Original: f 1 × · · · × A f N | .
Revised: f 1 × · · · × A f N | .

Original: For simplicity, we use the shorthand V π,ν(cid:63)
Revised: For simplicity, we use the shorthand V π,ν(cid:63)

Original: =
Revised: =

Original: V π,ν(cid:63), 1 ( x 1 ) , where x 1 ∈ S is the ﬁxed initial state.
Revised: V π,ν(cid:63), 1 ( x 1 ) , where x 1 ∈ S is the ﬁxed initial state.

Original: E. A LGORITHM
Revised: E. A LGORITHM

Original: We present the pseudocode of Reward-Free Explore algorithm (Jin et al., 2020a) below.
Revised: We present the pseudocode of Reward-Free Explore algorithm (Jin et al., 2020a) below.

Revised: 1: Input: iteration number K 0 and K .

Original: 2: Let policy class Φ = ∅ .
Revised: 2: Let policy class Φ = ∅ .

Original: 3: for x h )  S ×
Revised: 3: for ( x, h ) ∈ S ×

Original: [ H ] do 5: Φ
Revised: [ H ] do 4: r h ( x , a

Original:  ← EULER
Revised: , b ) ← x

Original: 6: π
Revised: = x h h for all ( , a ∈ S ×

Original:  ← 
Revised: A l × A f × H ] . 5: Φ ( x,h ) ← EULER ( r, K 0 ) .6: π h ( · | x ) ← Uniform( A l ) and ν h ( · | x ) ← Uniform( A f ) for all ( π, ν ) ∈ Φ ( x,h ) .

Original: 7: Ψ ← Ψ ∪ Φ 
Revised: 7: Ψ ← Ψ ∪ Φ ( x,h ) .

Original: 8: end for 9: for k = 1 , · · · , K do 10:
Revised: 8: end for 9: for k = 1 , · · · , K do 10:

Original: Sample policy ( π, ν ) ∼ Uniform (Ψ) .
Revised: Sample policy ( π, ν ) ∼ Uniform (Ψ) .

Original: Play the game M using policy π and ν , and observe the trajectory { x kh , a kh , b kh } h ∈ [ H ] and rewards { r (cid:63),h ( x kh , a kh , b kh ) } h ∈
Revised: Play the game M using policy π and ν , and observe the trajectory { x kh , a kh , b kh } h ∈ [ H ] and rewards { r (cid:63),h ( x kh , a kh , b kh ) } h ∈

Original: [ H ] .
Revised: [ H ] .

Original: 12: end for 13: Calculate the empirical reward as
Revised: 12: end for 13: Calculate the empirical reward as

Original: [Equation]
Revised: [Equation]

Original: Lemma E.1.
Revised: Lemma E.1.

Original: Fix ε, p > 0 .
Revised: Fix ε, p > 0 .

Original: If we set K 0 ≥ Ω( H 7 S 4 A l /ε ) and K ≥ Ω( H 3 S 2 A l A f /ε 2 ) in Algorithm 4, then we have the empirical rewards { (cid:98) r l , (cid:98) r f 1 , · · · (cid:98) r f N } and corresponding value functions ( (cid:98) V l , (cid:98) V f 1 , · · · , (cid:98) V f N ) satisfying thatsup
Revised: If we set K 0 ≥ Ω( H 7 S 4 A l /ε ) and K ≥ Ω( H 3 S 2 A l A f /ε 2 ) in Algorithm 4, then we have the empirical rewards { (cid:98) r l , (cid:98) r f 1 , · · · (cid:98) r f N } and corresponding value functions ( (cid:98) V l , (cid:98) V f 1 , · · · , (cid:98) V f N ) satisfying that

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − p .
Revised: with probability at least 1 − p .

Original: Here Ω( · ) hides some logarithmic factors.
Revised: Here Ω( · ) hides some logarithmic factors.

Original: Proof.
Revised: Proof.

Original: This lemma is a simple extension of Lemma D.1 in Bai et al.
Revised: This lemma is a simple extension of Lemma D.1 in Bai et al.

Original: They focus on the
Revised: They focus on the

Original: MDP setting and we consider the more complex Markov games.
Revised: MDP setting and we consider the more complex Markov games.

Original: For completeness, we present the detailed proof in § E.2.
Revised: For completeness, we present the detailed proof in § E.2.

Original: Lemma E.1 states that we can obtain estimated reward functions and the associated value functions is an ε -approximation of the true value functions, which further implies that the SNE with respect to the estimated reward functions is a good approximation of the SNE in the original problem.
Revised: Lemma E.1 states that we can obtain estimated reward functions and the associated value functions is an ε -approximation of the true value functions, which further implies that the SNE with respect to the estimated reward functions is a good approximation of the SNE in the original problem.

Original: We also remark that if we consider the Markov games with only one follower and aim to ﬁnd the Stackelberg equilibria, we can provide a more reﬁned analysis.
Revised: We also remark that if we consider the Markov games with only one follower and aim to ﬁnd the Stackelberg equilibria, we can provide a more reﬁned analysis.

Original: See § F for more details.
Revised: See § F for more details.

Original: E. P ROOF OF L EMMA E. 
Revised: E. P ROOF OF L EMMA E.

Original: Before our proof, we present a useful lemma.
Revised: Before our proof, we present a useful lemma.

Original: Lemma E.2.
Revised: Lemma E.2.

Original: We deﬁne the set of δ -signiﬁcant states as
Revised: We deﬁne the set of δ -signiﬁcant states as

Original: [Equation]
Revised: [Equation]

Original: where P π,νh ( x ) is the probability of visiting x at h -th step under policies ( π, ν ) .
Revised: where P π,νh ( x ) is the probability of visiting x at h -th step under policies ( π, ν ) .

Original: Then, We have
Revised: Then, We have

Original: [Equation]
Revised: [Equation]

Original: for any s ∈ S δh .
Revised: for any s ∈ S δh .

Original: Here P πh ( x, a ) is the probability of visiting ( x, a ) at h -th step under policy π .
Revised: Here P πh ( x, a ) is the probability of visiting ( x, a ) at h -th step under policy π .

Original: Proof.
Revised: Proof.

Original: See the proof of Theorem 3.3 in Jin et al.
Revised: See the proof of Theorem 3.3 in Jin et al.

Original: (2020a) for more details.
Revised: (2020a) for more details.

Original: Now, we are ready to proof Lemma E.1.
Revised: Now, we are ready to proof Lemma E.1.

Original: Proof of Lemma E.1.
Revised: Proof of Lemma E.1.

Original: For any ( π, ν ) , we denote P π,νh ( x, a, b ) as the probability of visiting ( x, a, b ) at h -th step under policies ( π, ν ) .
Revised: For any ( π, ν ) , we denote P π,νh ( x, a, b ) as the probability of visiting ( x, a, b ) at h -th step under policies ( π, ν ) .

Original: Under this notion, by Lemma E.2 and the fact that all policies in Φ ( x,h ) are uniform at ( x, h ) , we have
Revised: Under this notion, by Lemma E.2 and the fact that all policies in Φ ( x,h ) are uniform at ( x, h ) , we have

Original: [Equation]
Revised: [Equation]

Original: where |A l | =
Revised: where |A l | =

Original: A l and A f = |A
Revised: A l and A f = |A

Original: f | = |A
Revised: f | = |A

Original: f 1 × · · · × A f N | .
Revised: f 1 × · · · × A f N | .

Original: Thus, for any δ -signiﬁcant ( x, h ) , we have
Revised: Thus, for any δ -signiﬁcant ( x, h ) , we have

Original: [Equation]
Revised: [Equation]

Original: Then the data obtained from Algorithm 4 is sampled i.i.d.
Revised: Then the data obtained from Algorithm 4 is sampled i.i.d.

Original: from some distribution ζ h , such that
Revised: from some distribution ζ h , such that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:124) (cid:123)(cid:122) (cid:125) (ii) .
Revised: (cid:125) (ii) .

Original: (E.3)
Revised: (E.3)

Original: Clearly, (i) ≤
Revised: Clearly, (i) ≤

Original: [Equation]
Revised: [Equation]

Original: where the second inequality uses the deﬁnition of δ -signiﬁcant set in (E.1) and the last inequality is implied by the fact that δ = ε/ 2 H 2 S .
Revised: where the second inequality uses the deﬁnition of δ -signiﬁcant set in (E.1) and the last inequality is implied by the fact that δ = ε/ 2 H 2 S .

Original: Meanwhile, we have
Revised: Meanwhile, we have

Original: [Equation]
Revised: [Equation]

Original: Note that P π,νh ( x, a, b )
Revised: Note that P π,νh ( x, a, b )

Original: =
Revised: =

Original: P π,νh ( x ) · π h ( a | x ) · ν h ( b | x ) , together with Cauchy-Schwarz inequality, we further have
Revised: P π,νh ( x ) · π h ( a | x ) · ν h ( b | x ) , together with Cauchy-Schwarz inequality, we further have

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Original: [Table] 
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation] 
Revised: [Equation] [Table] [Equation] [Table]

Revised: [Equation]

Original: [Table] 
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation] 
Revised: [Equation] [Table] [Equation] [Table]

Revised: [Equation]

Original: [Table] 
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where the last inequality follows from (E.2).
Revised: where the last inequality follows from (E.2).

Original: Meanwhile, by Hoeffding inequality and a union bound for the reward estimations we have
Revised: Meanwhile, by Hoeffding inequality and a union bound for the reward estimations we have

Original: [Equation]
Revised: [Equation]

Original: Choose δ = ε/ 2 H 2 S 
Revised: Choose δ = ε/ 2 H 2 S .

Original: [Equation]

Original: Together with (E.2)
Revised: Together with (E.2)

Original: [Equation]
Revised: [Equation]

Original: we have ζ h ( s, a, b ) ≥ ε/ 4 H 3 S 2 A l A f for any s ∈ S δh .
Revised: we have ζ h ( s, a, b ) ≥ ε/ 4 H 3 S 2 A l A f for any s ∈ S δh .

Original: Hence
Revised: Hence

Original: [Equation]
Revised: [Equation]

Original: we have K ≥ Ω( H 3 S 2 A l A f /ε ) ≥ Ω(1 / min s,a,b ζ h ( s, a, b )) .
Revised: we have K ≥ Ω( H 3 S 2 A l A f /ε ) ≥ Ω(1 / min s,a,b ζ h ( s, a, b )) .

Original: Applying multiplicative
Revised: Applying multiplicative

Original: Chernoff bound for the counter N h ( s, a, b ) ∼ Bin ( K, ζ h ( s, a, b )) , we have
Revised: Chernoff bound for the counter N h ( s, a, b ) ∼ Bin ( K, ζ h ( s, a, b )) , we have

Original: [Equation]
Revised: [Equation]

Original: Plugging (E.6), (E.7), and (E.7) into (E.5), we have
Revised: Plugging (E.6), (E.7), and (E.7) into (E.5), we have

Original: [Equation]
Revised: [Equation]

Original: where the last inequality follows form our choice that K ≥ Ω( H 3 S 2 A l A f /ε 2 ) .
Revised: where the last inequality follows form our choice that K ≥ Ω( H 3 S 2 A l A f /ε 2 ) .

Original: Combining (E.3), (E.4) and (E.9), we have | (cid:98) V π,ν(cid:63)
Revised: Combining (E.3), (E.4) and (E.9), we have | (cid:98) V π,ν(cid:63)

Original: − V π,ν(cid:63) | ≤ ε for any ( π, ν ) , which concludes the proof of Lemma E.1.
Revised: − V π,ν [Equation] | ≤ ε for any ( π, ν ) , which concludes the proof of Lemma E.1.

Original: F L EARNING S TACKELBERG E QUILIBRIA
Revised: F L EARNING S TACKELBERG E QUILIBRIA

Original: In this section, we analyze the sample-efﬁciency of learning Stackelberg equilibria in two-player tabular Markov games without the known reward assumption.
Revised: In this section, we analyze the sample-efﬁciency of learning Stackelberg equilibria in two-player tabular Markov games without the known reward assumption.

Original: For simplicity, we use the shorthands f = f 1 and V π,ν(cid:63)
Revised: For simplicity, we use the shorthands f = f 1 and V π,ν(cid:63)

Original: =
Revised: =

Original: V π,ν(cid:63), 1 ( x 1 ) , where x 1 ∈ S is the ﬁxed initial state.
Revised: V π,ν(cid:63), 1 ( x 1 ) , where x 1 ∈ S is the ﬁxed initial state.

Original: Meanwhile, for any ε > 0 , we deﬁne the ε -approximate value of best-case response by
Revised: Meanwhile, for any ε > 0 , we deﬁne the ε -approximate value of best-case response by

Original: [Equation]
Revised: [Equation]

Original: By the above deﬁnitions, we can immediately obtain that BR ( π ) ⊆
Revised: By the above deﬁnitions, we can immediately obtain that BR ( π ) ⊆

Original: BR ε ( π ) , which further implies
Revised: BR ε ( π ) , which further implies

Original: V πε ≥ V π,ν ∗ ( π ) l .
Revised: V πε ≥ V π,ν ∗ ( π ) l .

Original: Then we can deﬁne the gap
Revised: Then we can deﬁne the gap

Original: [Equation]
Revised: [Equation]

Original: F. A LGORITHM
Revised: F. A LGORITHM

Original: As stated before, we ﬁrst conduct a Reward-Free Explore algorithm (Algorithm 4) to obtain the estimated rewards ( (cid:98) r l , (cid:98) r f ) .
Revised: As stated before, we ﬁrst conduct a Reward-Free Explore algorithm (Algorithm 4) to obtain the estimated rewards ( (cid:98) r l , (cid:98) r f ) .

Original: We also deﬁne ( (cid:98) V l , (cid:98) V f ) as the corresponding value functions.
Revised: We also deﬁne ( (cid:98) V l , (cid:98) V f ) as the corresponding value functions.

Original: Then we use Algorithm 1 to solve the SNE with respect to the known reward functions ( (cid:98) r l , (cid:98) r f ) .
Revised: Then we use Algorithm 1 to solve the SNE with respect to the known reward functions ( (cid:98) r l , (cid:98) r f ) .

Original: Speciﬁcally, we consider the following optimization problem of ﬁnding approximation Stackelberg equilibria with respect to the empirical rewards ( (cid:98) r l , (cid:98) r f ) .
Revised: Speciﬁcally, we consider the following optimization problem of ﬁnding approximation Stackelberg equilibria with respect to the empirical rewards ( (cid:98) r l , (cid:98) r f ) .

Original: [Equation]
Revised: [Equation]

Original: Since ( (cid:98) r l , (cid:98) r f ) are known to us, we can use Algorithm 1 to obtain the solution ( (cid:98) π, (cid:98) ν
Revised: Since ( (cid:98) r l , (cid:98) r f ) are known to us, we can use Algorithm 1 to obtain the solution ( (cid:98) π, (cid:98) ν

Original: = ν ( (cid:98) π )) , which is our approximate solution.
Revised: = ν ( (cid:98) π )) , which is our approximate solution.

Original: See Algorithm 5 for more details.
Revised: See Algorithm 5 for more details.

Original: Algorithm
Revised: Algorithm

Original: Reward-Free Explore then Commit 1: Input: Accuracy coefﬁcient ε > 0 . 2: Run the Reward-Free Explore algorithm (Algorithm 4) with K 0 ≥ Ω( H 7 S 4 A l /ε ) and {+K ≥ Ω( H 3 S 
 A l A+}
Revised: Commit 1: Input: Accuracy coefﬁcient ε > 0 . 2: Run the Reward-Free Explore algorithm (Algorithm 4) with K 0 ≥ Ω( H 7 S 4 A l /ε ) and {+K ≥ Ω( H 3 S 
 A l A+}

Original: K ≥ Ω( H 3 S 2 A l A f /ε 2 ) , and obtain empirical rewards ( (cid:98) r l , (cid:98) r f ) .
Revised: f /ε 2 ) , and obtain empirical rewards ( (cid:98) r l , (cid:98) r f ) .

Original: 3: Use Algorithm 1 as an oracle to solve the problem deﬁned in (F.2) and obtain the solution ( (cid:98) π, (cid:98) ν
Revised: : Use Algorithm 1 as an oracle to solve the problem deﬁned in F.2) and obtain the solution ( (cid:98) π, (cid:98) ν

Original: = ν ( π )) .
Revised: = ν ( π )) .

Original:  Output:( (cid:98) π, (cid:98) ν ) .
Revised: 4: Output: (cid:98) π, (cid:98) ν ) .

Original: F. T HEORETICAL R ESULTS
Revised: F. T HEORETICAL R ESULTS

Original: The performance of Algorithm 5 is guaranteed by the following theorem.
Revised: The performance of Algorithm 5 is guaranteed by the following theorem.

Original: Theorem F.1.
Revised: Theorem F.1.

Original: Suppose Algorithm 5 outputs ( (cid:98) π, (cid:98) ν ) .
Revised: Suppose Algorithm 5 outputs ( (cid:98) π, (cid:98) ν ) .

Original: Then it holds with probability at least 1 − p that
Revised: Then it holds with probability at least 1 − p that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Similar analysis also appears in Bai et al.
Revised: Similar analysis also appears in Bai et al.

Original: As stated before, however, their setting is different with ours.
Revised: As stated before, however, their setting is different with ours.

Original: For completeness, we provide a detailed proof here.
Revised: For completeness, we provide a detailed proof here.

Original: First, we show that
Revised: First, we show that

Original: [Equation]
Revised: [Equation]

Original: By choosing a large absolute constant in K , together with Lemma E.1, it holds for any (cid:63) ∈ { l, f } that
Revised: By choosing a large absolute constant in K , together with Lemma E.1, it holds for any (cid:63) ∈ { l, f } that

Original: [Equation]
Revised: [Equation]

Original: Meanwhile, for the empirical rewards ( (cid:98) r l , (cid:98) r f ) , we deﬁne the best response of leader’s policy π as (cid:92)
Revised: Meanwhile, for the empirical rewards ( (cid:98) r l , (cid:98) r f ) , we deﬁne the best response of leader’s policy π as (cid:92)

Original: [Equation]
Revised: [Equation]

Original: Under this notation, for any ν ∈ (cid:99) BR 3 ε/ 4 ( π ), we have
Revised: notation any ν ∈ (cid:99) BR 3 ε/ 4 ( π ) we have

Original: V π,ν ∗ ( π ) f − V π,ν f
Revised: V π,ν ∗ ( π ) f − V π,ν f

Original: = ( V π,ν ∗ ( π ) f − (cid:98) V π,ν ∗ ( π ) f ) (cid:124)
Revised: = ( V π,ν ∗ ( π ) f − (cid:98) V π,ν ∗ ( π ) f ) (cid:124)

Original: (cid:123)(cid:122) (cid:125)
Revised: (cid:123)(cid:122) (cid:125)

Original: (i) + ( (cid:98) V π,ν ∗ ( π ) f − (cid:98)
Revised: (i) + ( (cid:98) V π,ν ∗ ( π ) f − (cid:98)

Original: V π, (cid:92) ν ∗ ( π ) f ) (cid:124)
Revised: V π, (cid:92) ν ∗ ( π ) f ) (cid:124)

Original: (cid:123)(cid:122) (cid:125) (ii)
Revised: (cid:123)(cid:122) (cid:125) (ii)

Original: + ( (cid:98) V π, (cid:92) ν ∗ ( π ) f − (cid:98)
Revised: + ( (cid:98) V π, (cid:92) ν ∗ ( π ) f − (cid:98)

Original: V π,νf ) (cid:124) (cid:123)(cid:122) (cid:125) (iii)
Revised: V π,νf ) (cid:124) (cid:123)(cid:122) (cid:125) (iii)

Original: + ( (cid:98) V π,νf i − V π,νf ) (cid:124) (cid:123)(cid:122) (cid:125) (iv) ≤ ε/ 8 + 0 + 3 ε/ 4 + ε/ 8 ≤ ε.
Revised: + ( (cid:98) V π,νf i − V π,νf ) (cid:124) (cid:123)(cid:122) (cid:125) (iv) ≤ ε/ 8 + 0 + 3 ε/ 4 + ε/ 8 ≤ ε.

Original: (F.5) where (i) ≤ ε/ 8 and (iv) ≤ ε/ 8 is implied by the uniform convergence in (F.4), (ii) ≤ 0 uses the deﬁnition of (cid:92) ν ∗ ( π ) , and (iii) ≤ 0 follows from the fact
Revised: (F.5) where (i) ≤ ε/ 8 and (iv) ≤ ε/ 8 is implied by the uniform convergence in (F.4), (ii) ≤ 0 uses the deﬁnition of (cid:92) ν ∗ ( π ) , and (iii) ≤ 0 follows from the fact

Original: that ν ∈ (cid:99) BR 3 ε/ 4 ( π ) .
Revised: that ν ∈ (cid:99) BR 3 ε/ 4 ( π ) .

Original: Similarly, for any ν ∈ BR ε/ 2 ( π ) , we can show that
Revised: Similarly, for any ν ∈ BR ε/ 2 ( π ) , we can show that

Original: [Equation]
Revised: [Equation]

Original: Combining (F.5) and (F.6), we obtain BR ε/ 2 ( π ) ⊆
Revised: Combining (F.5) and (F.6), we obtain BR ε/ 2 ( π ) ⊆

Original: (cid:99) BR 

Revised: (cid:99) BR

Original: ε/ 4 ( π ) ⊆
Revised: ε/ 4 ( π ) ⊆

Original: BR ε ( π ) as desired.
Revised: BR ε ( π ) as desired.

Original: Back to our proof, by the fact that (cid:98) π maximizes (cid:98) V
Revised: Back to our proof, by the fact that (cid:98) π maximizes (cid:98) V

Original: [Equation]
Revised: [Equation]

Original:  =
Revised: 4 =

Original: [Equation]
Revised: [Equation]

Original: ∈
Revised: ∈

Revised: [Equation]

Revised: BR

Original: [Equation]
Revised: [Equation]

Original: π
Revised: π

Original: [Equation]
Revised: [Equation]

Original: (cid:98) V l ( π, ν ) , we have
Revised: (cid:98) V l ( π, ν ) , we have

Original: [Equation]
Revised: [Equation]

Original: F

Original: [Equation]
Revised: [Equation]

Original: 7) for any π .
Revised: F.7) for any π .

Original: Here the last inequality uses the fact BR ε/ 2 ( π ) ⊆
Revised: Here the last inequality uses the fact BR ε/ 2 ( π ) ⊆

Original: (cid:99) BR 3 ε/ 4 ( π ) in (F.3).
Revised: (cid:99) BR 3 ε/ 4 ( π ) in (F.3).

Original: Together with the uniform convergence in (F.4), (F.7) yields
Revised: Together with the uniform convergence in (F.4), (F.7) yields

Original: [Equation]
Revised: [Equation]

Original: Meanwhile, by the fact (cid:99) BR 3 ε/ 4 ( π ) ⊆ BR ε ( π ) in (F.4), we have
Revised: Meanwhile, by the fact (cid:99) BR 3 ε/ 4 ( π ) ⊆ BR ε ( π ) in (F.4), we have

Original: [Equation]
Revised: [Equation]

Original: Combining (F.8) and (F.9), we have
Revised: Combining (F.8) and (F.9), we have

Original: [Equation]
Revised: [Equation]

Original: which implies that (cid:98) π ∈ Π ε .
Revised: which implies that (cid:98) π ∈ Π ε .

Original: Furthermore, (F.10) is equivalent to
Revised: Furthermore, (F.10) is equivalent to

Original: [Equation]
Revised: [Equation]

Original: where the equality uses the deﬁnition of gap ε in (F.1).
Revised: where the equality uses the deﬁnition of gap ε in (F.1).

Original: as desired.
Revised: as desired.

Original: Meanwhile, by the facts that (cid:98) ν ∈ (cid:99) BR 3 ε/ 4 ( (cid:98) π ) and (cid:99) BR 3 ε/ 4 ( (cid:98) π ) ⊆
Revised: Meanwhile, by the facts that (cid:98) ν ∈ (cid:99) BR 3 ε/ 4 ( (cid:98) π ) and (cid:99) BR 3 ε/ 4 ( (cid:98) π ) ⊆

Original: BR ε ( (cid:98) π ) , we have
Revised: BR ε ( (cid:98) π ) , we have

Original: [Equation]
Revised: [Equation]

Revised: −

Revised: [Equation]

Original: Therefore, we conclude the proof of Theorem F.1.
Revised: Therefore, we conclude the proof of Theorem F.1.

Original: G P ROOF OF T HEOREM 4. 
Revised: G P ROOF OF T HEOREM 4.

Original: To facilitate our analysis, we ﬁrst deﬁne the prediction error
Revised: To facilitate our analysis, we ﬁrst deﬁne the prediction error

Original: [Equation]
Revised: [Equation]

Original: for any h ∈ [ H ] .
Revised: for any h ∈ [ H ] .

Original: Then we show the proof of Theorem 4.2.
Revised: Then we show the proof of Theorem 4.2.

Original: Proof of Theorem 4.2.
Revised: Proof of Theorem 4.2.

Original: Similar to Lemma D.1, we have the following lemma.
Revised: Similar to Lemma D.1, we have the following lemma.

Original: Lemma G.1.
Revised: Lemma G.1.

Original: It holds that (cid:98) ν ∈ BR ( (cid:98) π ) .
Revised: It holds that (cid:98) ν ∈ BR ( (cid:98) π ) .

Original: Here BR ( · ) is deﬁned in (2.10).
Revised: Here BR ( · ) is deﬁned in (2.10).

Original: Proof.
Revised: Proof.

Original: This is implied by the deﬁnitions of ( (cid:98) π, (cid:98) ν ) and the assumption that the followers are myopic.
Revised: This is implied by the deﬁnitions of ( (cid:98) π, (cid:98) ν ) and the assumption that the followers are myopic.

Original: Recall that the deﬁnition of optimality gap deﬁned in (4.1) takes the following form
Revised: Recall that the deﬁnition of optimality gap deﬁned in (4.1) takes the following form

Original: [Equation]
Revised: [Equation]

Original: In that follows, we decompose it by the following lemma.
Revised: In that follows, we decompose it by the following lemma.

Original: Lemma G.2.
Revised: Lemma G.2.

Original: For the (cid:98) V 1 deﬁned in Line 10 of Algorithm 3 and any ( π, ν ) , it holds that
Revised: For the (cid:98) V 1 deﬁned in Line 10 of Algorithm 3 and any ( π, ν ) , it holds that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: This proof is the same as the proof of (D.14), and we omit it to avoid repetition.
Revised: This proof is the same as the proof of (D.14), and we omit it to avoid repetition.

Original: Applying Lemma G.2 with ( π, ν ) =
Revised: Applying Lemma G.2 with ( π, ν ) =

Original: ( π ∗ , ν ∗ ) , we have
Revised: ( π ∗ , ν ∗ ) , we have

Original: [Equation]
Revised: [Equation]

Original: Similarly, applying Lemma G.2 with ( π, ν ) =
Revised: Similarly, applying Lemma G.2 with ( π, ν ) =

Original: ( (cid:98) π, (cid:98) ν ) gives that
Revised: ( (cid:98) π, (cid:98) ν ) gives that

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Combining (G.3) and (G.4), we obtain
Revised: Combining (G.3) and (G.4), we obtain

Original: V π
Revised: V π

Original: [Equation]
Revised: [Equation]

Original: As stated in § D, these two terms characterize the optimization error and the statistical error, respectively.
Revised: As stated in § D [Equation] these two terms characterize the optimization error and the statistical error, respectively.

Original: Similar to Lemmas D.4 and D.5, we introduce the following two lemmas to analyze these two errors.
Revised: Similar to Lemmas D.4 and D. [Equation] we introduce the following two lemmas to analyze these two errors.

Original: Lemma G.3.
Revised: Lemma G.3.

Original: It holds that
Revised: It holds that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: This proof is similar to the proof of Lemma D.4, and we omit it to avoid repetition.
Revised: This proof is similar to the proof of Lemma D.4, and we omit it to avoid repetition.

Original: Lemma G.4.
Revised: Lemma G.4.

Original: It holds with probability at least 1 − p/ 2 that
Revised: It holds with probability at least 1 − p/ 2 that

Original: [Equation]
Revised: [Equation]

Original: for any h ∈ [ H ] and ( x, a, b ) ∈ S ×
Revised: for any h ∈ [ H ] and ( x, a, b ) ∈ S ×

Original: A l ×
Revised: A l ×

Original: A f .
Revised: A f .

Original: Proof.
Revised: Proof.

Original: See § G.1 for a detailed proof.
Revised: See § G.1 for a detailed proof.

Original: Combining (G.5) and Lemmas G.3 and G.4, we further obtain that
Revised: Combining (G.5) and Lemmas G.3 and G.4, we further obtain that

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is obtained by the deﬁnition of Γ h in Line 6 of Algorithm 3 and the fact that (cid:15) =
Revised: where the last inequality is obtained by the deﬁnition of Γ h in Line 6 of Algorithm 3 and the fact that (cid:15) =

Original: d/KH .
Revised: d/KH .

Original: Therefore, we conclude the proof of Theorem 4.2.
Revised: Therefore, we conclude the proof of Theorem 4.2.

Original: G. P ROOF OF L EMMA G. 
Revised: G. P ROOF OF L EMMA G.

Original: Proof of Lemma G.4.
Revised: Proof of Lemma G.4.

Original: Similar to (D.31), it holds with probability at least 1 − p/ 2 that
Revised: Similar to (D.31), it holds with probability at least 1 − p/ 2 that

Original: [Equation]
Revised: [Equation]

Original: for any h ∈ [ H ] .
Revised: for any h ∈ [ H ] .

Original: The only exception is that we use Lemma J.3 instead of the classical concentration lemma (Lemma J.2) for the self-normalized process.
Revised: The only exception is that we use Lemma J.3 instead of the classical concentration lemma (Lemma J.2) for the self-normalized process.

Original: Here we omit the detailed proof to avoid repetition.
Revised: Here we omit the detailed proof to avoid repetition.

Original: By (G.7) and the fact that (cid:98) V h
Revised: By (G.7) and the fact that (cid:98) V h

Original: +1 ( · ) ≤ H − h , we obtain
Revised: +1 ( · ) ≤ H − h , we obtain

Original: [Equation]
Revised: [Equation]

Original: Thus, we have (cid:98) Q h ≥ φ (cid:62) w h − Γ h , which further implies that
Revised: Thus, we have (cid:98) Q h ≥ φ (cid:62) w h − Γ h , which further implies that

Original: [Equation]
Revised: [Equation]

Original: where the last inequality uses (G.7).
Revised: where the last inequality uses (G.7).

Original: Meanwhile, it holds that
Revised: Meanwhile [Equation] it holds that

Original: [Equation]
Revised: [Equation]

Original: where the last inequality follows from (G.7).
Revised: where the last inequality follows from (G.7).

Original: Combining (G.9) and (G.10), we conclude the proof of Lemma G.4.
Revised: Combining (G.9) and (G.10), we conclude the proof of Lemma G.4.

Original: H P ROOF OF C OROLLARY C. 
Revised: H P ROOF OF C OROLLARY C.

Original: Proof of Corollary C.1.
Revised: Proof of Corollary C.1.

Original: The proof is an extension of Corollary 4.5 in Jin et al.
Revised: The proof is an extension of Corollary 4.5 in Jin et al.

Original: (2020c).
Revised: (2020c).

Original: For notational simplicity, we deﬁne
Revised: For notational simplicity, we deﬁne

Original: [Equation]
Revised: [Equation]

Original: for all x ∈ S and h ∈ [ H ] .
Revised: for all x ∈ S and h ∈ [ H ] .

Original: With this notation and Cauchy-Schwarz inequality [Equation] we have
Revised: With this notation and Cauchy-Schwarz inequality, we have

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: Plugging (H.1) into Theorem 4.2, together with the assumption that Λ h (cid:23) I + c · K · E π ∗ ,ν ∗ ,x [ φ ( s h , a h , b h ) φ ( s h , a h , b h ) (cid:62) ] with probability at least 1 − p/ 2 and a union bound argument, we further with probability at least 1 − p have
Revised: Plugging (H.1) into Theorem 4.2, together with the assumption that Λ h (cid:23) I + c · K · E π ∗ ,ν ∗ ,x [ φ ( s h , a h , b h ) φ ( s h , a h , b h ) (cid:62) ] with probability at least 1 − p/ 2 and a union bound argument, we further with probability at least 1 − p have

Original: [Equation]
Revised: [Equation]

Original: for all x ∈ S .
Revised: for all x ∈ S .

Original: Here { λ h,j ( x ) } dj
Revised: Here { λ h,j ( x ) } dj

Original: =1 are the eigenvalues of Σ h ( x ) .
Revised: =1 are the eigenvalues of Σ h ( x ) .

Original: Meanwhile, by Jensen’s inequality, we obtain
Revised: Meanwhile, by Jensen’s inequality, we obtain

Original: [Equation]
Revised: [Equation]

Original: where the last inequality follows from the fact that (cid:107) φ ( · , · , · ) (cid:107) 2 ≤ 1 .
Revised: where the last inequality follows from the fact that (cid:107) φ ( · , · , · ) (cid:107) 2 ≤ 1 .

Original: Combining (H.2) and (H.3), it holds with probability at least 1 − p that
Revised: Combining (H.2) and (H.3), it holds with probability at least 1 − p that

Original: [Equation]
Revised: [Equation]

Original: where ¯ C = 3 C/ √ c , which concludes the proof of Corollary C.1.
Revised: where ¯ C = 3 C/ √ c , which concludes the proof of Corollary C.1.

Original: I R ESULTS WITH P ESSIMISTIC T IE - BREAKING I. S TACKELBERG -N ASH E QUILIBRIA IN P ESSIMISTIC T IE - BREAKING S ETUP
Revised: I R ESULTS WITH P ESSIMISTIC T IE - BREAKING I. S TACKELBERG -N ASH E QUILIBRIA IN P ESSIMISTIC T IE - BREAKING S ETUP

Original: For any leader policy π , we can deﬁne
Revised: For any leader policy π , we can deﬁne

Original: [Equation]
Revised: [Equation]

Original: where BR ( π ) is the best-response set deﬁned in (2.10).
Revised: where BR ( π ) is the best-response set deﬁned in (2.10).

Original: That is, ν † ( π ) is the worst-case response in the set BR ( π ) .
Revised: That is, ν † ( π ) is the worst-case response in the set BR ( π ) .

Original: Then we deﬁne the Stackelberg-Nash equilibria by
Revised: Then we deﬁne the Stackelberg-Nash equilibria by

Original: [Equation]
Revised: [Equation]

Original: We point out that ﬁnding the Stackelberg-Nash equilibria in the pessimistic tie-breaking setting is harder.
Revised: We point out that ﬁnding the Stackelberg-Nash equilibria in the pessimistic tie-breaking setting is harder.

Original: Speciﬁcally, compared with optimistic tie-breaking setting (cf.
Revised: Speciﬁcally, compared with optimistic tie-breaking setting (cf.

Original: (2.8)), we need to solve a more complicated constrained max-min optimization problem:
Revised: (2.8)), we need to solve a more complicated constrained max-min optimization problem:

Original: [Equation]
Revised: [Equation]

Original: Under this more challenging setting, we focus on the leader-controller linear Markov games setting (Assumption 2.3).
Revised: Under this more challenging setting, we focus on the leader-controller linear Markov games setting (Assumption 2.3).

Original: Similar to Theorems 3.3 and 4.2, we can have the following two theorems in the online and ofﬂine settings.
Revised: Similar to Theorems 3.3 and 4.2, we can have the following two theorems in the online and ofﬂine settings.

Original: I. M AIN R ESULTS FOR THE O NLINE S ETTING
Revised: I. M AIN R ESULTS FOR THE O NLINE S ETTING

Original: Theorem I.1.
Revised: Theorem I.1.

Original: Under Assumptions 2.1, 2.3, and 3.1, there exists an absolute constant C > 0 such that, for any ﬁxed p ∈ (0 , 1) , by setting β = C · dH √ ι with ι = log(2 dT/p ) in Line 7 of Algorithmand (cid:15)
Revised: Under Assumptions 2.1, 2.3, and 3.1, there exists an absolute constant C > 0 such that, for any ﬁxed p ∈ (0 , 1) , by setting β = C · dH √ ι with ι = log(2 dT/p ) in Line 7 of Algorithmand (cid:15)

Original: = 1 KH in Algorithm 7, then have ν k
Revised: = 1 KH in Algorithm 7, then have ν k

Original: = ν † ( π k ) for any k ∈ [ K ] .
Revised: = ν † ( π k ) for any k ∈ [ K ] .

Original: Meanwhile, with probability at least 1 − p , the regret incurred by Algorithm 6 satisﬁes that
Revised: Meanwhile, with probability at least 1 − p , the regret incurred by Algorithm 6 satisﬁes that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: See § I.4 for a detailed proof.
Revised: See § I.4 for a detailed proof.

Original: Misspeciﬁcation.
Revised: Misspeciﬁcation.

Original: When the transitions do not ideally satisfy the leader-controller assumption, we can potentially consider cases that transitions satisfy, for instance,
Revised: When the transitions do not ideally satisfy the leader-controller assumption, we can potentially consider cases that transitions satisfy, for instance,

Original: |P
Revised: |P

Original: h ( · | x, a, b ) −P h ( · | x, a ) (cid:107) ∞ ≤ (cid:37) for any ( h, x, a, b ) ∈
Revised: h ( · | x, a, b ) −P h ( · | x, a ) (cid:107) ∞ ≤ (cid:37) for any ( h, x, a, b ) ∈

Original: [ H ] × S ×
Revised: [ H ] × S ×

Original: A l × A f , Here (cid:37) is the misspeciﬁcation error.
Revised: A l × A f , Here (cid:37) is the misspeciﬁcation error.

Original: We can still follow the above method to tackle the misspeciﬁed cases.
Revised: We can still follow the above method to tackle the misspeciﬁed cases.

Original: However, because of the misspeciﬁcation error cumulated during T steps, an extra term O ( (cid:37)T ) will appear in the ﬁnal result.
Revised: However, because of the misspeciﬁcation error cumulated during T steps, an extra term O ( (cid:37)T ) will appear in the ﬁnal result.

Original: In particular,
Revised: In particular,

Original: When (cid:37) is small, that is the Markov games have approximately leader-controller transitions, the extra term O ( (cid:37)T ) should be small, which further indicates that we can ﬁnd SNEs efﬁciently in some misspeciﬁed general-sum Markov games.
Revised: When (cid:37) is small, that is the Markov games have approximately leader-controller transitions, the extra term O ( (cid:37)T ) should be small, which further indicates that we can ﬁnd SNEs efﬁciently in some misspeciﬁed general-sum Markov games.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Sample [Equation]
Revised: a kh ∼ π kh ( · | x kh ) , b k 1 ,h ∼ ν kf 1 ,h ( · | x kh ) , · · · , b kN,h ∼ ν kf N ,h ( · | x kh ) .

Original: Leader takes action a kh ; Followers take actions b kh = { b ki,h } i ∈ [ N ] .
Revised: Leader takes action a kh ; Followers take actions b kh = { b ki,h } i ∈ [ N ] .

Original: Observe next state x kh +1 .
Revised: Observe next state x kh +1 .

Original: 16: end for 17: end for
Revised: 16: end for 17: end for

Original: 1: Input: Q kh , x , and parameter(cid:15) .
Revised: 1: Input kh , x , and parameter

Original: 2: Select(cid:101) Q from Q kh,(cid:15) satisfying(cid:107) (cid:101) Q − Q kh (cid:107) ∞ ≤ (cid:15) .
Revised: : Select Q from Q kh satisfying (cid:101) Q − Q kh (cid:107) ∞ ≤ (cid:15) .

Original:  For the input state x , let ( π kh ( · | x ) , { ν kf i ,h ( · | x ) } i ∈
Revised: 3: For the input state x , let ( π kh ( · | x ) , { ν kf i ,h ( · | x ) } i ∈

Original:  N ] ) be the Stackelberg-Nash equilibrium for the matrix game with payoff matrices ( (cid:101) Q ( x, · , · ) , { r f i ,h ( x, · , · ) } i ∈ [ N ] ) in the pessimistic tiebreaking setting.
Revised: [ N ] ) be the Stackelberg-Nash equilibrium for the matrix game with payoff matrices ( (cid:101) Q ( x, · , · ) , { r f i ,h ( x, · , · ) } i ∈ [ N ] ) in the pessimistic tiebreaking setting.

Original:  Output: ( π kh ( · | x ) , { ν kf i ,h ( · | x ) } i ∈
Revised: 4: Output: ( π kh ( · | x ) , { ν kf i ,h ( · | x ) } i ∈

Original: [ N ] ) .
Revised: [ N ] ) .

Original: I. M AIN R ESULTS FOR THE O FFLINE S ETTING
Revised: I. M AIN R ESULTS FOR THE O FFLINE S ETTING

Original: Theorem I.2.
Revised: Theorem I.2.

Original: Under Assumptions 2.1, 2.3, 3.1, and 4.1, there exists an absolute constant C >such that, for any ﬁxed p ∈ (0 , 1) , by setting β (cid:48)
Revised: Under Assumptions 2.1, 2.3, 3.1, and 4.1, there exists an absolute constant C >such that, for any ﬁxed p ∈ (0 , 1) , by setting β (cid:48)

Original: =
Revised: =

Original: C · dH (cid:112) log(2 dHK/p ) in Line 6 of Algorithmand (cid:15)
Revised: C · dH (cid:112) log(2 dHK/p ) in Line 6 of Algorithmand (cid:15)

Original: = dKH in Algorithm 7, then we have (cid:98) ν
Revised: = dKH in Algorithm 7, then we have (cid:98) ν

Original: = ν † ( (cid:98) π ) .
Revised: = ν † ( (cid:98) π ) .

Original: Meanwhile, with probability at least 1 − p , we have
Revised: Meanwhile, with probability at least 1 − p , we have

Original: SubOpt( (cid:98) π, (cid:98) ν, x ) = V
Revised: SubOpt

Original: [Equation]
Revised: [Equation]

Original: where E π ∗ ,x is taken with respect to the trajectory incurred by π ∗ in the underlying leader-controller
Revised: where E π ∗ ,x is taken with respect to the trajectory incurred by π ∗ in the underlying leader-controller

Original: Markov game when initializing the progress at x .
Revised: Markov game when initializing the progress at x .

Original: Here Λ h is deﬁned in Line 4 of Algorithm 8.
Revised: Here Λ h is deﬁned in Line 4 of Algorithm 8.

Original: Proof.
Revised: Proof.

Original: Combining the proofs of Theorems 4.2 and I.1, we can conclude the proof of Theorem I.2.
Revised: Combining the proofs of Theorems 4.2 and I.1, we can conclude the proof of Theorem I.2.

Original: To avoid repetition, we omit the detailed proof here.
Revised: To avoid repetition, we omit the detailed proof here.

Original: Optimality of the Bound:
Revised: Optimality of the Bound:

Original: Assuming the dummy followers, that is, the actions taken by the followers won’t affect the reward functions and transition kernels, the Markov games reduces to the linear MDP (Jin et al., 2020b).
Revised: Assuming the dummy followers, that is, the actions taken by the followers won’t affect the reward functions and transition kernels, the Markov games reduces to the linear MDP (Jin et al., 2020b).

Original: Together with the information-theoretic lower bound Ω( (cid:80) Hh =1 E π ∗ ,
Revised: Together with the information-theoretic lower bound Ω( (cid:80) Hh =1 E π ∗ ,

Original: x [( φ ( s h , a h ) (cid:62) (Λ h ) − 1 φ ( s h , a h )) 1 / 2 ]) established in Jin et al.
Revised: x [( φ ( s h , a h ) (cid:62) (Λ h ) − 1 φ ( s h , a h )) 1 / 2 ]) established in Jin et al.

Original: (2020c) for linear MDPs, we immediately obtain the same lower bound for our setting.
Revised: (2020c) for linear MDPs, we immediately obtain the same lower bound for our setting.

Original: In particular, our upper bound established in Theorem I.2 matches this lower bound up to β (cid:48) and absolute constants and thus implies that our algorithm is nearly minimax optimal.
Revised: In particular, our upper bound established in Theorem I.2 matches this lower bound up to β (cid:48) and absolute constants and thus implies that our algorithm is nearly minimax optimal.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ν fN,h ( ·| x ) (cid:98) Q h ( x, a, b 1 , · · · , b N ) , ∀ x .
Revised: ν fN,h ( ·| x ) (cid:98) Q h ( x, a, b 1 , · · · , b N ) , ∀ x .

Original: 10: end for 11: Output: ( (cid:98) π
Revised: 10: end for 11: Output: ( (cid:98) π

Original: = { (cid:98) π h
Revised: = { (cid:98) π h

Original: Hh =1 , (cid:98) ν
Revised: Hh =1 , (cid:98) ν

Original: = { (cid:98) ν
Revised: = { (cid:98) ν

Original: f i = { ν f i ,h } Hh =1 } Ni =1 ) .
Revised: f i = { ν f i ,h } Hh =1 } Ni =1 ) .

Original: I. P ROOF OF T HEOREM I. 
Revised: I. P ROOF OF T HEOREM I.

Original: Proof of Theorem I.1.
Revised: Proof of Theorem I.1.

Original: For leader-controller Markov games, we have a stronger version of Lemma
Revised: For leader-controller Markov games, we have a stronger version of Lemma

Original: D.1.
Revised: D.1.

Original: Lemma I.3.
Revised: Lemma I.3.

Original: For any k ∈ [ K ] , we have ν k
Revised: For any k ∈ [ K ] , we have ν k

Original: = ν † ( π k ) .
Revised: = ν † ( π k ) .

Original: Here ν † ( · ) is deﬁned in (I.1).
Revised: Here ν † ( · ) is deﬁned in (I.1).

Original: Proof.
Revised: Proof.

Original: Fix k ∈
Revised: Fix k ∈

Original: [ K ] , by the deﬁnition of the best response in (2.5), we have
Revised: [ K ] , by the deﬁnition of the best response in (2.5), we have

Original: BR ( π k ) = { ν = { ν f i } i ∈ [ N ] | ν is the NE of the followers given the leader policy π k } = { ν = { ν f i } i ∈
Revised: BR ( π k ) = { ν = { ν f i } i ∈ [ N ] | ν is the NE of the followers given the leader policy π k } = { ν = { ν f i } i ∈

Original: [ N ] | ν is the NE of { V π k ,ν f i ,h ( x ) } i ∈ [ N ] , ∀ h ∈
Revised: [ N ] | ν is the NE of { V π k ,ν f i ,h ( x ) } i ∈ [ N ] , ∀ h ∈

Original: [ H ] and x ∈ S} = { ν = { ν f i } i ∈
Revised: [ H ] and x ∈ S} = { ν = { ν f i } i ∈

Original: [ N ] | ν is the NE of { r π k ,ν f i ,h ( x ) } i ∈ [ N ] , ∀ h ∈
Revised: [ N ] | ν is the NE of { r π k ,ν f i ,h ( x ) } i ∈ [ N ] , ∀ h ∈

Original: [ H ] and x ∈ S} , (I.3) where r π k ,ν f i ,h ( x ) = (cid:104) r f i ,h ( x, · , · , · · · , · ) , π kh ( · | x ) × ν f 1 ,h ( · | x ) × · · · × ν f N ,h ( · | x ) (cid:105)
Revised: [ H ] and x ∈ S} , (I.3) where r π k ,ν f i ,h ( x ) = (cid:104) r f i ,h ( x, · , · , · · · , · ) , π kh ( · | x ) × ν f 1 ,h ( · | x ) × · · · × ν f N ,h ( · | x ) (cid:105)

Original: A l ×A f .
Revised: A l ×A f .

Original: Here the last inequality uses Bellman equality (2.2) and the leader-controller assumption.
Revised: Here the last inequality uses Bellman equality (2.2) and the leader-controller assumption.

Original: Moreover, by the deﬁnition of ν † ( π k ) deﬁned in (2.6), we have that
Revised: Moreover, by the deﬁnition of ν † ( π k ) deﬁned in (2.6), we have that

Original: [Equation]
Revised: [Equation]

Original: where r π k ,ν l,h ( x ) = (cid:104) r l,h ( x, · , · , · · · , · ) , π kh ( · | x ) × ν f 1 ,h ( · | x ) × · · · × ν f N ,h ( · | x ) (cid:105)
Revised: where r π k ,ν l,h ( x ) = (cid:104) r l,h ( x, · , · , · · · , · ) , π kh ( · | x ) × ν f 1 ,h ( · | x ) × · · · × ν f N ,h ( · | x ) (cid:105)

Original: A l ×A f .
Revised: A l ×A f .

Original: Here the last equality uses the single-controller assumption.
Revised: Here the last equality uses the single-controller assumption.

Original: Recall that, in the subroutine (cid:15) -SNE (Algorithm 2), we pick the function (cid:101) Q ∈ Q kh,(cid:15)
Revised: Recall that, in the subroutine (cid:15) -SNE (Algorithm 2), we pick the function (cid:101) Q ∈ Q kh,(cid:15)

Original: such that (cid:107) Q
Revised: such that (cid:107) Q

Original: kh
Revised: kh

Original: − (cid:101) Q (cid:107) ∞ ≤ (cid:15) and solve the matrix game deﬁned in (3.6).
Revised: − (cid:101) Q (cid:107) ∞ ≤ (cid:15) and solve the matrix game deﬁned in (3.6).

Original: Here Q kh,(cid:15) is the class of functions Q :
Revised: Here Q kh,(cid:15) is the class of functions Q :

Original: S ×
Revised: S ×

Original: A l × A f → R that takes form
Revised: A l × A f → R that takes form

Original: [Equation]
Revised: [Equation]

Original: where (cid:107) w (cid:107) 2 ≤ H √ dk and λ min (Λ) ≥ 1 .
Revised: where (cid:107) w (cid:107) 2 ≤ H √ dk and λ min (Λ) ≥ 1 .

Original: Thus, given the leader policy π k , the best response of the followers for the matrix game deﬁned in (3.6) takes the form
Revised: Thus, given the leader policy π k , the best response of the followers for the matrix game deﬁned in (3.6) takes the form

Original: BR (cid:48) ( π k ) =
Revised: BR (cid:48) ( π k ) =

Original: { ν | ν is the NE of {(cid:104) r f i ,h ( x, · , · ) , π kh ( · | x ) × ν h ( · | x ) (cid:105)} i ∈
Revised: { ν | ν is the NE of {(cid:104) r f i ,h ( x, · , · ) , π kh ( · | x ) × ν h ( · | x ) (cid:105)} i ∈

Original: [ N ] , ∀ h ∈
Revised: [ N ] , ∀ h ∈

Original: [ H ] and x ∈ S} = 
Revised: [ H ] and x ∈ S} = BR

Original: [Equation]
Revised: [Equation]

Original: where (cid:104)
Revised: where (cid:104)

Original: r
Revised: r

Original: f i ,h ( x, · , · ) , π kh ( · | x ) × ν h ( · | x ) (cid:105) is the shorthand of (cid:104) r
Revised: f i ,h ( x, · , · ) , π kh ( · | x ) × ν h ( · | x ) (cid:105) is the shorthand of (cid:104) r

Original: f
Revised: f

Original: i ,h ( x, · , · , · · · , · ) , π kh ( · | x ) × ν f 1 ,h ( · | x ) × · · · × ν f N ,h ( · | x ) (cid:105)
Revised: i ,h ( x, · , · , · · · , · ) , π kh ( · | x ) × ν f 1 ,h ( · | x ) × · · · × ν f N ,h ( · | x ) (cid:105)

Original: A l ×A f .
Revised: A l ×A f .

Original: Here the last equality uses (I.3).
Revised: Here the last equality uses (I.3).

Original: Similarly, by the deﬁnition of Q kh,(cid:15) in (I.5), we can obtain that
Revised: Similarly, by the deﬁnition of Q kh,(cid:15) in (I.5), we can obtain that

Original: [Equation]
Revised: [Equation]

Original: where (cid:104) r l,h ( x, · , · ) , π kh ( · | x ) × ν h ( · | x ) (cid:105) is the abbreviation of (cid:104) r
Revised: where (cid:104) r l,h ( x, · , · ) , π kh ( · | x ) × ν h ( · | x ) (cid:105) is the abbreviation of (cid:104) r

Original: f i ,h ( x, · , · , · · · , · ) , π kh ( · | x ) × ν f 1 ,h ( · | x ) ×· · ·× ν f N ,h ( · | x ) (cid:105)
Revised: f i ,h ( x, · , · , · · · , · ) , π kh ( · | x ) × ν f 1 ,h ( · | x ) ×· · ·× ν f N ,h ( · | x ) (cid:105)

Original: A l ×A f Together with (I.4) and (I.6), we have that, for the matrix game with payoff matrices ( (cid:101) Q ( x kh , · , · ) , { r kf i ,h ( x kh , · , · ) } i ∈ [ N ] ) , the policy ν kh ( · | x kh )
Revised: A l ×A f Together with (I.4) and (I.6), we have that, for the matrix game with payoff matrices ( (cid:101) Q ( x kh , · , · ) , { r kf i ,h ( x kh , · , · ) } i ∈ [ N ] ) , the policy ν kh ( · | x kh )

Original: =
Revised: =

Original: { ν kf i ,h ( · | x kh ) } i ∈
Revised: { ν kf i ,h ( · | x kh ) } i ∈

Original: [ N ] is also the best response of π kh ( · | x kh ) and breaks ties against favor of the leader.
Revised: [ N ] is also the best response of π kh ( · | x kh ) and breaks ties against favor of the leader.

Original: Therefore, we have ν k
Revised: Therefore, we have ν k

Original: = ν † ( π k ) for any k ∈ [ K ] , which concludes the proof of Lemma I.3.
Revised: = ν † ( π k ) for any k ∈ [ K ] , which concludes the proof of Lemma I.3.

Original: Then we only need to bound the quantity (cid:80)
Revised: Then we only need to bound the quantity (cid:80)

Revised: Kk =

Revised: [Equation]

Revised: ( x

Revised: [Equation]

Revised: 1 ) −

Original: [Equation]
Revised: [Equation]

Original: ( x
Revised: ( x

Original: [Equation]
Revised: [Equation]

Original: By Lemma D.2, we have
Revised: By Lemma D.2, we have

Original: Regret ( K ) =
Revised: Regret ( K ) =

Original: [Equation]
Revised: [Equation]

Original: (cid:88)k
Revised: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: +

Original: [Equation]
Revised: [Equation]

Original: +
Revised: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: By the same argument of Lemma I.
Revised: By the same argument of Lemma I.

Original: [Equation]
Revised: [Equation]

Original: we have that, for the matrix game with payoff matrices ( (cid:101) Q ( x kh , · , · ) , { r kf i ,h ( x kh , · , · ) } i ∈ [ N ] ) , ν ∗ h ( · | x kh ) belongs to the best response set of π ∗ h ( · | x kh ) and breaks ties against favor of the leader.
Revised: we have that, for the matrix game with payoff matrices ( (cid:101) Q ( x kh , · , · ) , { r kf i ,h ( x kh , · , · ) } i ∈ [ N ] ) , ν ∗ h ( · | x kh ) belongs to the best response set of π ∗ h ( · | x kh ) and breaks ties against favor of the leader.

Original: Recall that ( π 
Revised: Recall that ( π kh ( · | x kh

Original: [Equation]
Revised: [Equation]

Original:  is the Stackelberg-Nash equilibrium of the matrix game with payoff matrices ( (cid:101) Q ( x kh , · , · , · ) , { r kf i ,h ( x kh , · , · , · ) } i ∈ [ N ] ) in the pessimistic tie-breaking setting, which implies that π kh ( · | x kh ) is the “worst response to the best response”, which further implies that
Revised: ν [Equation] = { ν kf i ,h ( · | x kh ) } i ∈ [ N ] ) is the Stackelberg-Nash equilibrium of the matrix game with payoff matrices ( (cid:101) Q ( x kh , · , · , · ) , { r kf i ,h ( x kh , · , · , · ) } i ∈ [ N ] ) in the pessimistic tie-breaking setting, which implies that π kh ( · | x kh ) is the “worst response to the best response”, which further implies that

Original: [Equation]
Revised: [Equation]

Original: for any ( k, h ) ∈
Revised: for any ( k, h ) ∈

Original: [ K ] ×
Revised: [ K ] ×

Original: [ H ] .
Revised: [ H ] .

Original: Thus, for any ( k, h ) ∈
Revised: Thus, for any ( k, h ) ∈

Original: [ K ] × [ H ] , we have
Revised: [ K ] × [ H ] , we have

Original: [Equation]
Revised: [Equation]

Original: where the last inequality uses (D.19) and the fact that (cid:107) Q kh − (cid:101) Q (cid:107) ∞ ≤ (cid:15) .
Revised: where the last inequality uses (D.19) and the fact that (cid:107) Q kh − (cid:101) Q (cid:107) ∞ ≤ (cid:15) .

Original: By taking summation over ( k, h ) ∈ [ K ] × [ H ] , we bound the computational error as desired.
Revised: By taking summation over ( k, h ) ∈ [ K ] × [ H ] , we bound the computational error as desired.

Original: Moreover, we can characterize statistical error by Lemmas D.5 and D.6.
Revised: Moreover, we can characterize statistical error by Lemmas D.5 and D.6.

Original: The remaining randomness term can be bounded by
Revised: The remaining randomness term can be bounded by

Original: Lemma D.7.
Revised: Lemma D.7.

Original: Putting these together, we have Regret ( K ) ≤ O ( √ d 3 H 3 Tι 2 ) , which concludes the proof of Theorem I.1.
Revised: Putting these together, we have Regret ( K ) ≤ O ( √ d 3 H 3 Tι 2 ) , which concludes the proof of Theorem I.1.

Original: J S UPPORTING L EMMAS
Revised: J S UPPORTING L EMMAS

Original: Lemma J.1 (Elliptical Potential Lemma (Dani et al., 2008; Abbasi-Yadkori et al., 2011; Jin et al., 2020b; Cai et al., 2020)) .
Revised: Lemma J.1 (Elliptical Potential Lemma (Dani et al., 2008; Abbasi-Yadkori et al., 2011; Jin et al., 2020b; Cai et al., 2020)) .

Original: Let { φ t } ∞ t =1 be an R d -valued sequence.
Revised: Let { φ t } ∞ t =1 be an R d -valued sequence.

Original: Meanwhile, let Λ
Revised: Meanwhile, let Λ

Original: 0 ∈ R d × d be a positive-deﬁnite matrix and Λ t = Λ 0 + (cid:80) t − 1 j =1 φ j φ
Revised: 0 ∈ R d × d be a positive-deﬁnite matrix and Λ t = Λ 0 + (cid:80) t − 1 j =1 φ j φ

Original: [Equation]
Revised: [Equation]

Original: j .
Revised: j .

Original: It holds for any t ∈ Z + that
Revised: It holds for any t ∈ Z + that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: See Lemma 11 of Abbasi-Yadkori et al.
Revised: See Lemma 11 of Abbasi-Yadkori et al.

Original: (2011) for a detailed proof.
Revised: (2011) for a detailed proof.

Original: Lemma J.2 (Concentration of Self-Normalized Process (Abbasi-Yadkori et al., 2011)) .
Revised: Lemma J.2 (Concentration of Self-Normalized Process (Abbasi-Yadkori et al., 2011)) .

Original: Let { (cid:101) F t } ∞ t =0 be a ﬁltration and { η t } ∞ t =1 be an R -valued stochastic process such that η t is (cid:101) F t -measurable for any t ≥ 0 .
Revised: Let { (cid:101) F t } ∞ t =0 be a ﬁltration and { η t } ∞ t =1 be an R -valued stochastic process such that η t is (cid:101) F t -measurable for any t ≥ 0 .

Original: We also assume that, for any t ≥ 0 , conditioning on (cid:101) F t , η t is a zero-mean and σ -sub-Gaussian random variable, that is,
Revised: We also assume that, for any t ≥ 0 , conditioning on (cid:101) F t , η t is a zero-mean and σ -sub-Gaussian random variable, that is,

Original: [Equation]
Revised: [Equation]

Original: for any λ ∈ R .
Revised: for any λ ∈ R .

Original: Let { X t } ∞ t =1 be an R d -valued stochastic process such that X t is (cid:101) F t -measurable for any t ≥ 0 .
Revised: Let { X t } ∞ t =1 be an R d -valued stochastic process such that X t is (cid:101) F t -measurable for any t ≥ 0 .

Original: Also, let Y ∈
Revised: Also, let Y ∈

Original: R
Revised: R

Original: d × d be a deterministic and positive-deﬁnite matrix.
Revised: d × d be a deterministic and positive-deﬁnite matrix.

Original: For any t ≥ 0 , we deﬁne
Revised: For any t ≥ 0 , we deﬁne

Original: [Equation]
Revised: [Equation]

Original: For any δ > 0 and t ≥ 0 ., it holds with probability at least 1 − δ that
Revised: For any δ > 0 and t ≥ 0 ., it holds with probability at least 1 − δ that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: See Theorem 1 of Abbasi-Yadkori et al.
Revised: See Theorem 1 of Abbasi-Yadkori et al.

Original: (2011) for a detailed proof.
Revised: (2011) for a detailed proof.

Original: Lemma J.3.
Revised: Lemma J.3.

Original: For any ﬁxed h ∈ [ H ] , let V : S → [0 , H ] be any ﬁxed value function.
Revised: For any ﬁxed h ∈ [ H ] , let V : S → [0 , H ] be any ﬁxed value function.

Original: Under
Revised: Under

Original: Assumption 4.1, for any ﬁxed δ > 0 , we have
Revised: Assumption 4.1, for any ﬁxed δ > 0 , we have

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: See Lemma B.2 of Jin et al.
Revised: See Lemma B.2 of Jin et al.

Original: (2020c)
Revised: (2020c)

Original: for a detailed proof.
Revised: for a detailed proof.

Original: Lemma J.4 (Covering) .
Revised: Lemma J.4 (Covering) .

Original: Let Q h be the class of value functions Q : S ×
Revised: Let Q h be the class of value functions Q : S ×

Original: A l × A f → R that takes the form
Revised: A l × A f → R that takes the form

Original: [Equation]
Revised: [Equation]

Original: which are parameterized by ( w, Λ) ∈ R d × R d × d such that (cid:107) w (cid:107) ≤ L and λ min (Λ) ≥ λ .
Revised: which are parameterized by ( w, Λ) ∈ R d × R d × d such that (cid:107) w (cid:107) ≤ L and λ min (Λ) ≥ λ .

Original: We assume that β is ﬁxed and satisfy that β ∈ [0 , B ] , and the feature map φ : S ×
Revised: We assume that β is ﬁxed and satisfy that β ∈ [0 , B ] , and the feature map φ : S ×

Original: A → R d satisﬁes that (cid:107) φ ( · , · ) (cid:107) 2 ≤ 1 .
Revised: A → R d satisﬁes that (cid:107) φ ( · , · ) (cid:107) 2 ≤ 1 .

Original: We have that, for any L, B, (cid:15) > 0 , there exists an (cid:15) -covering of Q h with respect to the (cid:96) ∞ norm such that the covering number N (cid:15) satisﬁes
Revised: We have that, for any L, B, (cid:15) > 0 , there exists an (cid:15) -covering of Q h with respect to the (cid:96) ∞ norm such that the covering number N (cid:15) satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: See Jin et al.
Revised: See Jin et al.

Original: (2020b)
Revised: (2020b)

Original: for a detailed proof.
Revised: for a detailed proof.