Original: Learning in Congestion Games with Bandit Feedback
Revised: Learning in Congestion Games with Bandit Feedback

Original: Abstract
Revised: Abstract

Original: In this paper, we investigate Nash-regret minimization in congestion games, a class of games with benign theoretical structure and broad real-world applications.
Revised: In this paper, we investigate Nash-regret minimization in congestion games, a class of games with benign theoretical structure and broad real-world applications.

Original: We first propose a centralized algorithm based on the optimism in the face of uncertainty principle for congestion games with (semi-)bandit feedback, and obtain finite-sample guarantees.
Revised: We first propose a centralized algorithm based on the optimism in the face of uncertainty principle for congestion games with (semi-)bandit feedback, and obtain finite-sample guarantees.

Original: Then we propose a decentralized algorithm via a novel combination of the Frank-Wolfe method and G-optimal design.
Revised: Then we propose a decentralized algorithm via a novel combination of the Frank-Wolfe method and G-optimal design.

Original: By exploiting the structure of the congestion game, we show the sample complexity of both algorithms depends only polynomially on the number of players and the number of facilities, but not the size of the action set, which can be exponentially large in terms of the number of facilities.
Revised: By exploiting the structure of the congestion game, we show the sample complexity of both algorithms depends only polynomially on the number of players and the number of facilities, but not the size of the action set, which can be exponentially large in terms of the number of facilities.

Original: We further define a new problem class, Markov congestion games, which allows us to model the non-stationarity in congestion games.
Revised: We further define a new problem class, Markov congestion games, which allows us to model the non-stationarity in congestion games.

Original: We propose a centralized algorithm for Markov congestion games, whose sample complexity again has only polynomial dependence on all relevant problem parameters, but not the size of the action set.
Revised: We propose a centralized algorithm for Markov congestion games, whose sample complexity again has only polynomial dependence on all relevant problem parameters, but not the size of the action set.

Original:  Introduction
Revised:  Introduction

Original: Nash equilibrium (NE) is a widely adopted concept in game theory community, used to describe the behavior of multi-agent systems with selfish players [Roughgarden, 2010].
Revised: Nash equilibrium (NE) is a widely adopted concept in game theory community, used to describe the behavior of multi-agent systems with selfish players [Roughgarden, 2010].

Original: At the Nash equilibrium, no player has the incentive to change its own strategy unilaterally, which implies it is a steady state of the game dynamics.
Revised: At the Nash equilibrium, no player has the incentive to change its own strategy unilaterally, which implies it is a steady state of the game dynamics.

Original: For a general-sum game, computing the Nash equilibrium is PPAD-hard
Revised: For a general-sum game, computing the Nash equilibrium is PPAD-hard

Original: [Daskalakis, 2013] and the query complexity is exponential in the number of players [Rubinstein, 2016].
Revised: [Daskalakis, 2013] and the query complexity is exponential in the number of players [Rubinstein, 2016].

Original: To help address these issues, a natural approach is to consider games with special structures.
Revised: To help address these issues, a natural approach is to consider games with special structures.

Original: In this paper we focus on congestion games.
Revised: In this paper we focus on congestion games.

Original: Congestion games are general-sum games with facilities (resources) shared among players [Rosenthal, 1973].
Revised: Congestion games are general-sum games with facilities (resources) shared among players [Rosenthal, 1973].

Original: During the game, each player will decide what combination of facilities to utilize, and popular facilities will become congested, which results in a possibly higher cost on each user.
Revised: During the game, each player will decide what combination of facilities to utilize, and popular facilities will become congested, which results in a possibly higher cost on each user.

Original: One example of congestion game is the routing game
Revised: One example of congestion game is the routing game

Original: [Fotakis et al., 2002], where each player needs to travel from a given starting point to a destination point through some shared routes.
Revised: [Fotakis et al., 2002], where each player needs to travel from a given starting point to a destination point through some shared routes.

Original: These routes are represented as a traffic graph and the facilities are the edges.
Revised: These routes are represented as a traffic graph and the facilities are the edges.

Original: Each player will decide her path to go, and the more players use the same edge, the longer the edge travel time will be.
Revised: Each player will decide her path to go, and the more players use the same edge, the longer the edge travel time will be.

Original: Congestion games also have wide applications in electrical grids [Ibars et al., 2010], internet routing [Al-Kashoash
Revised: Congestion games also have wide applications in electrical grids [Ibars et al., 2010], internet routing [Al-Kashoash

Original: et al., 2017] and ∗ Equal contribution.
Revised: et al., 2017] and ∗ Equal contribution.

Original: rate allocation [Johari and Tsitsiklis, 2004].
Revised: rate allocation [Johari and Tsitsiklis, 2004].

Original: In many real-world scenarios, players can only have (semi-)bandit feedback, i.e., players know only the payoff of the facilities they choose.
Revised: In many real-world scenarios, players can only have (semi-)bandit feedback, i.e., players know only the payoff of the facilities they choose.

Original: This kind of learning under uncertainty has been widely studied in bandits and in reinforcement learning for the single-agent setting, while theoretical understanding for the multi-agent case is still largely missing.
Revised: This kind of learning under uncertainty has been widely studied in bandits and in reinforcement learning for the single-agent setting, while theoretical understanding for the multi-agent case is still largely missing.

Original: There are two types of algorithms in multi-agent systems, namely centralized algorithms and decentralized algorithms.
Revised: There are two types of algorithms in multi-agent systems, namely centralized algorithms and decentralized algorithms.

Original: For centralized algorithms, there exists a central authority that can control and receive feedback from all players in the game.
Revised: For centralized algorithms, there exists a central authority that can control and receive feedback from all players in the game.

Original: As we have global coordination, centralized algorithms usually have favorable performance.
Revised: As we have global coordination, centralized algorithms usually have favorable performance.

Original: On the other hand, such a central authority may not always be available in practice, and thus people turn to decentralized algorithms, i.e., each player makes decisions individually and can only observe her own feedback.
Revised: On the other hand, such a central authority may not always be available in practice, and thus people turn to decentralized algorithms, i.e., each player makes decisions individually and can only observe her own feedback.

Original: However, decentralized algorithms are vulnerable to nonstationarity because each player is making decisions in a nonstationary environment as others’ strategies are changing [Zhang et al., 2021a].
Revised: However, decentralized algorithms are vulnerable to nonstationarity because each player is making decisions in a nonstationary environment as others’ strategies are changing [Zhang et al., 2021a].

Original: In this paper, we will study both centralized and decentralized algorithms in congestion games with bandit feedback, and we will provide motivating scenarios for both algorithms in Section 1.2.
Revised: In this paper, we will study both centralized and decentralized algorithms in congestion games with bandit feedback, and we will provide motivating scenarios for both algorithms in Section 1.2.

Original: The main challenge in designing algorithms for m -player congestion games with bandit feedback is the curse of exponential action set, i.e., the number of actions can be exponential in the number of facilities F because every subset of facilities can be an action.
Revised: The main challenge in designing algorithms for m -player congestion games with bandit feedback is the curse of exponential action set, i.e., the number of actions can be exponential in the number of facilities F because every subset of facilities can be an action.

Original: As a result, an efficient algorithm should have sample complexity polynomial in m and F and has no dependence on the size of the action space.
Revised: As a result, an efficient algorithm should have sample complexity polynomial in m and F and has no dependence on the size of the action space.

Original: One closely related type of general-sum game is the potential game, in which each individual’s payoff changes, resulting from strategy modification, can be quantified by a common potential function.
Revised: One closely related type of general-sum game is the potential game, in which each individual’s payoff changes, resulting from strategy modification, can be quantified by a common potential function.

Original: It is well-known that all congestion games are potential games, and each potential game has an equivalent congestion game formulation [Monderer and Shapley, 1996].
Revised: It is well-known that all congestion games are potential games, and each potential game has an equivalent congestion game formulation [Monderer and Shapley, 1996].

Original: However, existing algorithms designed for potential games all have sample complexity scaling at least linearly in the number of actions [Leonardos et al., 2021, Ding et al., 2022], which is inefficient for congestion games.
Revised: However, existing algorithms designed for potential games all have sample complexity scaling at least linearly in the number of actions [Leonardos et al., 2021, Ding et al., 2022], which is inefficient for congestion games.

Original: This motivates the following question:
Revised: This motivates the following question:

Original: Can we
Revised: Can we

Original: design provably sample-efficient centralized and decentralized learning algorithms for congestion games with bandit feedback?
Revised: design provably sample-efficient centralized and decentralized learning algorithms for congestion games with bandit feedback?

Original: We provide an affirmative answer to this question.
Revised: We provide an affirmative answer to this question.

Original: To be precise, we use Nash-regret minimization (formally defined in Section 3) as our objective for learning in congestion games.
Revised: To be precise, we use Nash-regret minimization (formally defined in Section 3) as our objective for learning in congestion games.

Original: This regretlike objective commonly appears in the literature of online learning and reinforcement learning [Orabona, 2019, Ding et al., 2022, Liu et al., 2021], which focuses on finite-time analysis and accumulative rewards throughout the learning process instead of the asymptotic behavior.
Revised: This regretlike objective commonly appears in the literature of online learning and reinforcement learning [Orabona, 2019, Ding et al., 2022, Liu et al., 2021], which focuses on finite-time analysis and accumulative rewards throughout the learning process instead of the asymptotic behavior.

Original: In general, a sublinear Nash regret implies a best-iterate convergence, meaning that the algorithm has reached the approximate Nash equilibrium at least once, while the converse does not hold.
Revised: In general, a sublinear Nash regret implies a best-iterate convergence, meaning that the algorithm has reached the approximate Nash equilibrium at least once, while the converse does not hold.

Original: We highlight our contributions below and compare our results with previous algorithms in Table 1.
Revised: We highlight our contributions below and compare our results with previous algorithms in Table 1.

Original: Our algorithms are shaded and we prove sublinear Nash regrets for all of them.
Revised: Our algorithms are shaded and we prove sublinear Nash regrets for all of them.

Original: In Table 1, sample complexity refers to the number of samples required to reach best-iterate convergence to an ϵ -approximate Nash equilibrium and the results are obtained by standard online-to-batch conversion as in Section 3.1 of [Jin et al., 2018].
Revised: In Table 1, sample complexity refers to the number of samples required to reach best-iterate convergence to an ϵ -approximate Nash equilibrium and the results are obtained by standard online-to-batch conversion as in Section 3.1 of [Jin et al., 2018].

Original: 1.1 Main Novelties and Contributions
Revised: 1.1 Main Novelties and Contributions

Original: Centralized algorithm for congestion game.
Revised: Centralized algorithm for congestion game.

Original: We adapt the principle of optimism in the face of uncertainty in stochastic bandits to ensure sufficient exploration in congestion games.
Revised: We adapt the principle of optimism in the face of uncertainty in stochastic bandits to ensure sufficient exploration in congestion games.

Original: We begin with congestion games with semi-bandit feedback, in which each player can observe the reward of every facility in the action.
Revised: We begin with congestion games with semi-bandit feedback, in which each player can observe the reward of every facility in the action.

Original: Instead of estimating the action reward as in stochastic multi-armed bandits, we estimate the facility rewards directly, which removes the dependence on the size of action space .
Revised: Instead of estimating the action reward as in stochastic multi-armed bandits, we estimate the facility rewards directly, which removes the dependence on the size of action space .

Original: Furthermore, we consider congestion games with bandit feedback, in which each player can only observe the overall reward.
Revised: Furthermore, we consider congestion games with bandit feedback, in which each player can only observe the overall reward.

Original: In this setting, we borrow ideas from linear bandits to estimate the reward function and analyze the algorithm.
Revised: In this setting, we borrow ideas from linear bandits to estimate the reward function and analyze the algorithm.

Original: The algorithm is provably sample efficient in both cases.
Revised: The algorithm is provably sample efficient in both cases.

Original: Decentralized algorithm for congestion game.
Revised: Decentralized algorithm for congestion game.

Original: Our decentralized algorithm is a Frank-Wolfe method with exploration, in which each player only observes her own actions and rewards.
Revised: Our decentralized algorithm is a Frank-Wolfe method with exploration, in which each player only observes her own actions and rewards.

Original: To efficiently explore in the congestion game, we utilize G-optimal design allocation for bandit feedback and a specific distribution for semi-bandit feedback.
Revised: To efficiently explore in the congestion game, we utilize G-optimal design allocation for bandit feedback and a specific distribution for semi-bandit feedback.

Original: As a result, the sample complexity does not depend on the number of actions
Revised: As a result, the sample complexity does not depend on the number of actions

Original: In addition, the L 1 smoothness parameter of the potential function does not depend on the number of actions, which is exploited by the Frank-Wolfe method.
Revised: In addition, the L 1 smoothness parameter of the potential function does not depend on the number of actions, which is exploited by the Frank-Wolfe method.

Original: With the
Revised: With the

Original: [Table]
Revised: [Table]

Original: Frank-Wolfe with Exploration
Revised: Frank-Wolfe with Exploration

Original: I
Revised: I

Original: [Table]
Revised: [Table]

Original: Frank-Wolfe with Exploration II
Revised: Frank-Wolfe with Exploration II

Original: [Table]
Revised: [Table]

Original: help of these two specific algorithmic designs for congestion games, we give the first decentralized algorithm for both semi-bandit feedback and bandit feedback that has no dependence on the size of the action space in congestion games.
Revised: help of these two specific algorithmic designs for congestion games, we give the first decentralized algorithm for both semi-bandit feedback and bandit feedback that has no dependence on the size of the action space in congestion games.

Original: Centralized algorithm for independent Markov congestion game.
Revised: Centralized algorithm for independent Markov congestion game.

Original: We extend the formulation of congestion game into a Markov setting and propose the independent Markov congestion game (IMCG), in which each facility has its own internal state and state transition happens independently among all the facilities.
Revised: We extend the formulation of congestion game into a Markov setting and propose the independent Markov congestion game (IMCG), in which each facility has its own internal state and state transition happens independently among all the facilities.

Original: In Section 1.2, we give some examples that fit in this model.
Revised: In Section 1.2, we give some examples that fit in this model.

Original: By utilizing techniques from factored MDPs, we extend our centralized algorithms for congestion games to efficiently solve IMCGs, with both semi-bandit and bandit feedback.
Revised: By utilizing techniques from factored MDPs, we extend our centralized algorithms for congestion games to efficiently solve IMCGs, with both semi-bandit and bandit feedback.

Original: 1.2 Motivating Examples
Revised: 1.2 Motivating Examples

Original: We provide an exmple here to motivate our proposed models.
Revised: We provide an exmple here to motivate our proposed models.

Original: See Section 3 for the formal definition of (semi-)bandit feedback and (Markov) congestion games and Appendix A for additional examples.
Revised: See Section 3 for the formal definition of (semi-)bandit feedback and (Markov) congestion games and Appendix A for additional examples.

Original: Example 1 ( Routing Games) .
Revised: Example 1 ( Routing Games) .

Original: For a routing game, there are multiple players in a traffic graph travelling from starting points to destination points, and the facilities are the edges (roads
Revised: For a routing game, there are multiple players in a traffic graph travelling from starting points to destination points, and the facilities are the edges (roads

Original: The cost of each edge is the waiting time, which depends on the
Revised: The cost of each edge is the waiting time, which depends on the

Original: number of players
Revised: number of players

Original: using that edge
Revised: using that edge

Original: • Centralized algorithm for routing games: Imagine each player is using Google Maps to navigate.
Revised: • Centralized algorithm for routing games: Imagine each player is using Google Maps to navigate.

Original: Then Google Maps can serve as a center that knows the starting points and the destination points, as well as the real-time feedback of the waiting time on each edge of all the players.
Revised: Then Google Maps can serve as a center that knows the starting points and the destination points, as well as the real-time feedback of the waiting time on each edge of all the players.

Original: Google Maps itself also has the incentive to assign paths according to the Nash equilibrium strategy as then each player will find out that deviating from the navigation has no benefit and thus sticks to the app.
Revised: Google Maps itself also has the incentive to assign paths according to the Nash equilibrium strategy as then each player will find out that deviating from the navigation has no benefit and thus sticks to the app.

Original: • Decentralized algorithm for routing games: Consider the case where players are still using Google Maps but due to privacy concerns or limited bandwidth, they only use the offline version, which has access only to the information of each single user.
Revised: • Decentralized algorithm for routing games: Consider the case where players are still using Google Maps but due to privacy concerns or limited bandwidth, they only use the offline version, which has access only to the information of each single user.

Original: Then Google Maps needs to use decentralized algorithms so that it can still assign Nash equilibrium strategy to each user after repeated plays.
Revised: Then Google Maps needs to use decentralized algorithms so that it can still assign Nash equilibrium strategy to each user after repeated plays.

Original: • Markov routing games: For Markov routing games, the time cost on each edge will change between different timesteps, which is a more accurate model of the real-world.
Revised: • Markov routing games: For Markov routing games, the time cost on each edge will change between different timesteps, which is a more accurate model of the real-world.

Original: For instance, some roads are prone to car accidents, which will result in an increasing cost on the next timestep, and the chance of accidents also depends on the number of players using that edge currently.
Revised: For instance, some roads are prone to car accidents, which will result in an increasing cost on the next timestep, and the chance of accidents also depends on the number of players using that edge currently.

Original: This is modeled by the Markovian facility state transition in independent Markov congestion games.
Revised: This is modeled by the Markovian facility state transition in independent Markov congestion games.

Original:  Related Work
Revised:  Related Work

Original: Potential Games.
Revised: Potential Games.

Original: Potential games are general-sum games that admit a common potential function to quantify the changes in individual’s payoff [Monderer and Shapley, 1996].
Revised: Potential games are general-sum games that admit a common potential function to quantify the changes in individual’s payoff [Monderer and Shapley, 1996].

Original: Algorithmic game theory community has studied how different dynamics converge to the Nash equilibium, e.g., best response dynamics [Durand, 2018, Swenson et al., 2018] and no-regret dynamics [Heliou et al., 2017,
Revised: Algorithmic game theory community has studied how different dynamics converge to the Nash equilibium, e.g., best response dynamics [Durand, 2018, Swenson et al., 2018] and no-regret dynamics [Heliou et al., 2017,

Original: Cheung and Piliouras, 2020], while usually they provide only asymptotic convergence, with either full information setting or bandit feedback setting.
Revised: Cheung and Piliouras, 2020], while usually they provide only asymptotic convergence, with either full information setting or bandit feedback setting.

Original: Recently, reinforcement learning community studied Markov potential games with bandit feedback, which can be applied to standard potential games.
Revised: Recently, reinforcement learning community studied Markov potential games with bandit feedback, which can be applied to standard potential games.

Original: See the Markov Games part below for more details.
Revised: See the Markov Games part below for more details.

Original: Congestion Games.
Revised: Congestion Games.

Original: Congestion games are developed in the seminal work [Rosenthal, 1973], and later Monderer and Shapley [1996] builds a close connection between congestion games and potential games.
Revised: Congestion games are developed in the seminal work [Rosenthal, 1973], and later Monderer and Shapley [1996] builds a close connection between congestion games and potential games.

Original: Congestion games are divided into atomic and non-atomic congestion games depending on whether each player is separable.
Revised: Congestion games are divided into atomic and non-atomic congestion games depending on whether each player is separable.

Original: Many papers consider non-atomic congestion games with non-decreasing cost function, which implies a convex potential function [Roughgarden and Tardos, 2004].
Revised: Many papers consider non-atomic congestion games with non-decreasing cost function, which implies a convex potential function [Roughgarden and Tardos, 2004].

Original: We consider the more difficult atomic congestion game where the potential function can be non-convex.
Revised: We consider the more difficult atomic congestion game where the potential function can be non-convex.

Original: For online non-atomic case, [Krichene et al., 2015] considers partial information setting while they provide convergence in the sense of Cesaro means.
Revised: For online non-atomic case, [Krichene et al., 2015] considers partial information setting while they provide convergence in the sense of Cesaro means.

Original: [Kleinberg et al., 2009, Krichene et al., 2014] show that some no-regret online learning algorithms asymptotically converges to Nash equilibrium.
Revised: [Kleinberg et al., 2009, Krichene et al., 2014] show that some no-regret online learning algorithms asymptotically converges to Nash equilibrium.

Original: [Chen and Lu, 2015, 2016] are two closely related works that consider bandit feedback in atomic congestion games and provide non-asymptotic convergence.
Revised: [Chen and Lu, 2015, 2016] are two closely related works that consider bandit feedback in atomic congestion games and provide non-asymptotic convergence.

Original: However, they still assume a convex potential function and the sample complexity has exponential dependence on the number of facilities, which is far from ideal.
Revised: However, they still assume a convex potential function and the sample complexity has exponential dependence on the number of facilities, which is far from ideal.

Original: Markov Games.
Revised: Markov Games.

Original: Markov games are widely studied since the seminal work [Shapley, 1953].
Revised: Markov games are widely studied since the seminal work [Shapley, 1953].

Original: Recently, the topic has received much attention due to advances in reinforcement learning theory.
Revised: Recently, the topic has received much attention due to advances in reinforcement learning theory.

Original: Liu et al.
Revised: Liu et al.

Original: [2021] provides a centralized algorithm for learning the Nash equilibrium in general-sum Markov games, and [Jin et al., 2021a, Song et al., 2021] provide decentralized algorithms for learning the (coarse) correlated equilibrium.
Revised: [2021] provides a centralized algorithm for learning the Nash equilibrium in general-sum Markov games, and [Jin et al., 2021a, Song et al., 2021] provide decentralized algorithms for learning the (coarse) correlated equilibrium.

Original: One closely related line of research is on Markov potential games [Leonardos et al., 2021, Zhang et al., 2021b,
Revised: One closely related line of research is on Markov potential games [Leonardos et al., 2021, Zhang et al., 2021b,

Original: Fox et al., 2021, Cen et al., 2022, Ding et al., 2022].
Revised: Fox et al., 2021, Cen et al., 2022, Ding et al., 2022].

Original: However, applying their algorithms to congestion games leads to explicit dependence on the number of actions, which would be exponentially worse than our algorithms.
Revised: However, applying their algorithms to congestion games leads to explicit dependence on the number of actions, which would be exponentially worse than our algorithms.

Original: See Table 1 for comparisons.
Revised: See Table 1 for comparisons.

Original: Our independent Markov congestion game is motivated by the state-based potential games studied in
Revised: Our independent Markov congestion game is motivated by the state-based potential games studied in

Original: Marden [2012] and Macua et al.
Revised: Marden [2012] and Macua et al.

Original: [2018], and its transition kernel is closely related to the factored
Revised: [2018], and its transition kernel is closely related to the factored

Original: MDPs, for which single agent algorithms are studied in [Osband and Van Roy, 2014, Chen et al., 2020, Xu and Tewari, 2020, Tian et al., 2020, Rosenberg and Mansour, 2021].
Revised: MDPs, for which single agent algorithms are studied in [Osband and Van Roy, 2014, Chen et al., 2020, Xu and Tewari, 2020, Tian et al., 2020, Rosenberg and Mansour, 2021].

Original: Learning in Games.
Revised: Learning in Games.

Original: Different from our paper, learning in games in traditional literature of game theory mainly considers players’ asymptotic behavior
Revised: Different from our paper, learning in games in traditional literature of game theory mainly considers players’ asymptotic behavior

Original: [Leslie and Collins, 2005, Cominetti et al., 2010, Coucheney et al., 2015].
Revised: [Leslie and Collins, 2005, Cominetti et al., 2010, Coucheney et al., 2015].

Original: In early literature, Leslie [2004] investigates actor-critic learning and Q -learning algorithms in games with bandit feedback and their connection to best-response dynamics.
Revised: In early literature, Leslie [2004] investigates actor-critic learning and Q -learning algorithms in games with bandit feedback and their connection to best-response dynamics.

Original: Leslie and Collins [2005] proposes individual Q -learning algorithm and shows that it converges to the NE almost surely in two-player zero-sum game and Leslie and Collins [2006] studies learning the NE from the perspective of a fictitious play-like process.
Revised: Leslie and Collins [2005] proposes individual Q -learning algorithm and shows that it converges to the NE almost surely in two-player zero-sum game and Leslie and Collins [2006] studies learning the NE from the perspective of a fictitious play-like process.

Original: Later, Cominetti et al.
Revised: Later, Cominetti et al.

Original: [2010] considers payoff-based learning rules and shows convergence to NE in traffic games, while another payoff-based learning model for continuous games is developed in Bervoets et al.
Revised: [2010] considers payoff-based learning rules and shows convergence to NE in traffic games, while another payoff-based learning model for continuous games is developed in Bervoets et al.

Original: Coucheney et al.
Revised: Coucheney et al.

Original: [2015] derives a new penalty-regulated dynamics and proposes a corresponding learning algorithms that converges to NE in potential games with bandit feedback.
Revised: [2015] derives a new penalty-regulated dynamics and proposes a corresponding learning algorithms that converges to NE in potential games with bandit feedback.

Original: Bravo et al.
Revised: Bravo et al.

Original: [2018] proposes that in monotone games with bandit feedback, as long as all players are using some no-regret learning algorithm, the dynamics will converge to the NE.
Revised: [2018] proposes that in monotone games with bandit feedback, as long as all players are using some no-regret learning algorithm, the dynamics will converge to the NE.

Original: An improved analysis of the same algorithm is then proposed in Drusvyatskiy et al.
Revised: An improved analysis of the same algorithm is then proposed in Drusvyatskiy et al.

Original: In contrast, our learning objective focuses on finite-time cumulative rewards, which is more widely used in current multi-agent reinforcement learning literature [Ding et al., 2022, Liu et al., 2021].
Revised: In contrast, our learning objective focuses on finite-time cumulative rewards, which is more widely used in current multi-agent reinforcement learning literature [Ding et al., 2022, Liu et al., 2021].

Original:  Preliminaries
Revised:  Preliminaries

Original: General-sum Matrix Games.
Revised: General-sum Matrix Games.

Original: We consider the model of general-sum matrix games, defined by the tuple G =
Revised: We consider the model of general-sum matrix games, defined by the tuple G =

Original: ( {A i } mi =1 , R ) , where m is the number of players, A i is the action space of player i and
Revised: ( {A i } mi =1 , R ) , where m is the number of players, A i is the action space of player i and

Original: R ( ·| a ) is the reward distribution on [0 , r max ] m with mean r ( a ) .
Revised: R ( ·| a ) is the reward distribution on [0 , r max ] m with mean r ( a ) .

Original: Let A =
Revised: Let A =

Original: A 1 × · · · × A m be the whole action space and denote an element as a =
Revised: A 1 × · · · × A m be the whole action space and denote an element as a =

Original: ( a 1 , . .
Revised: ( a 1 , . .

Original: a m ) ∈ A .
Revised: a m ) ∈ A .

Original: After all players take actions a ∈ A , a reward vector is sampled r ∼ R ( ·| a ) and player i will receive reward r i ∈ [0 , r max ] with mean r i ( a ) .
Revised: After all players take actions a ∈ A , a reward vector is sampled r ∼ R ( ·| a ) and player i will receive reward r i ∈ [0 , r max ] with mean r i ( a ) .

Original: Each player’s objective is to maximize her own reward.
Revised: Each player’s objective is to maximize her own reward.

Original: A general policy π is defined as a vector in ∆( A ) , the probability simplex over the action space A .
Revised: A general policy π is defined as a vector in ∆( A ) , the probability simplex over the action space A .

Original: A product policy π = ( π 1 , . . .
Revised: A product policy π = ( π 1 , . . .

Original: , π m ) is defined as a tuple in ∆( A 1 ) × · · · × ∆( A m ) , in which a = ( a 1 , . .
Revised: , π m ) is defined as a tuple in ∆( A 1 ) × · · · × ∆( A m ) , in which a = ( a 1 , . .

Original: . , a m ) ∼ π represents a i i .
Revised: . , a m ) ∼ π represents a i i .

Original: i .
Revised: i .

Original: d . ∼ π i .
Revised: d . ∼ π i .

Original: The value of policy π for player i is V πi =
Revised: The value of policy π for player i is V πi =

Original: E a ∼ π
Revised: E a ∼ π

Original: [ r i ( a )] .
Revised: [ r i ( a )] .

Original: Nash Equilibrium and Nash Regret.
Revised: Nash Equilibrium and Nash Regret.

Original: Given a general policy π , let π −
Revised: Given a general policy π , let π −

Original: i be the marginal joint policy of players 1 , . .
Revised: i be the marginal joint policy of players 1 , . .

Original: , i − 1 , i + 1 , . . .
Revised: , i − 1 , i + 1 , . . .

Original: , m .
Revised: , m .

Original: Then, the best response of player i under policy π is π
Revised: Then, the best response of player i under policy π is π

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: and the corresponding value is V † ,π
Revised: and the corresponding value is V † ,π

Original: [Equation]
Revised: [Equation]

Original: i i :
Revised: i i :

Original: [Equation]
Revised: [Equation]

Original: Our goal is to find the approximate Nash equilibrium of the matrix game, which is defined below.
Revised: Our goal is to find the approximate Nash equilibrium of the matrix game, which is defined below.

Original: Definition 1.
Revised: Definition 1.

Original: A product policy π is an ϵ -approximate Nash equilibrium if max i ( V † ,π − i
Revised: A product policy π is an ϵ -approximate Nash equilibrium if max i ( V † ,π − i

Original: i − V πi ) ≤ ϵ .
Revised: i − V πi ) ≤ ϵ .

Original: An ϵ -approximate Nash equilibrium can be obtained by achieving a sublinear Nash regret, which is defined below.
Revised: An ϵ -approximate Nash equilibrium can be obtained by achieving a sublinear Nash regret, which is defined below.

Original: See Section 3 in Ding et al.
Revised: See Section 3 in Ding et al.

Original: [2022] for a more detailed discussion.
Revised: [2022] for a more detailed discussion.

Original: Definition 2.
Revised: Definition 2.

Original: With π k being the policy at k -th episode, the Nash regret after K episodes is define as
Revised: With π k being the policy at k -th episode, the Nash regret after K episodes is define as

Original: [Equation]
Revised: [Equation]

Original: Remark 1 .
Revised: Remark 1 .

Original: Here, if we replace max i ∈ [ m ] by (cid:80) mi =1 in the definition of Nash regret, the single-step
Revised: Here, if we replace max i ∈ [ m ] by (cid:80) mi =1 in the definition of Nash regret, the single-step

Original: Nash regret at episode k will become the Nikaido-Isoda (NI) function evaluated at π k , which is a popular objective for equilibrium computation [Nikaidô and Isoda, 1955, Raghunathan et al., 2019].
Revised: Nash regret at episode k will become the Nikaido-Isoda (NI) function evaluated at π k , which is a popular objective for equilibrium computation [Nikaidô and Isoda, 1955, Raghunathan et al., 2019].

Original: Replacing max i ∈ [ m ] by (cid:80) mi =1 will multiply our regret bounds by a factor of m , while our conclusion will not be affected.
Revised: Replacing max i ∈ [ m ] by (cid:80) mi =1 will multiply our regret bounds by a factor of m , while our conclusion will not be affected.

Original: Potential Games.
Revised: Potential Games.

Original: A potential game is a general-sum game such that there exists a potential function Φ : ∆( A ) →
Revised: A potential game is a general-sum game such that there exists a potential function Φ : ∆( A ) →

Original: [0 , Φ max ] such that for any player i ∈
Revised: [0 , Φ max ] such that for any player i ∈

Original: [ m ] and policies π i , π ′ i , π − i , it satisfies
Revised: [ m ] and policies π i , π ′ i , π − i , it satisfies

Original: [Equation]
Revised: [Equation]

Original: We can immediately see that a policy that maximizes the potential function is a Nash equilibrium.
Revised: We can immediately see that a policy that maximizes the potential function is a Nash equilibrium.

Original: Congestion Games.
Revised: Congestion Games.

Original: A congestion game is defined by G =
Revised: A congestion game is defined by G =

Original: ( F , {A i } mi =1 , (cid:8) R f (cid:9) f ∈F ) , where
Revised: ( F , {A i } mi =1 , (cid:8) R f (cid:9) f ∈F ) , where

Original: F =
Revised: F =

Original: [ F ] is called the facility set and R f ( ·| n ) ∈ [0 , 1] is the reward distribution for facility f with mean r f ( n ) , where n ∈ [ m ] .
Revised: [ F ] is called the facility set and R f ( ·| n ) ∈ [0 , 1] is the reward distribution for facility f with mean r f ( n ) , where n ∈ [ m ] .

Original: Each action a i ∈
Revised: Each action a i ∈

Original: A i is a subset of F (i.e., a i ⊆ F ).
Revised: A i is a subset of F (i.e., a i ⊆ F ).

Original: Suppose the joint action chosen by all the players is a ∈ A , then a random reward is sampled r f ∼ R f ( ·| n f ( a )) for each facility f , where n f ( a ) = (cid:80) mi =1 1 { f ∈ a i } is the number of players using facility f .
Revised: Suppose the joint action chosen by all the players is a ∈ A , then a random reward is sampled r f ∼ R f ( ·| n f ( a )) for each facility f , where n f ( a ) = (cid:80) mi =1 1 { f ∈ a i } is the number of players using facility f .

Original: The reward collected by player i is r i
Revised: The reward collected by player i is r i

Original: = (cid:80) f ∈ a i r f with mean r i ( a )
Revised: = (cid:80) f ∈ a i r f with mean r i ( a )

Original: = (cid:80) f ∈ a i r f ( n f ( a )) ∈ [0 , F ] .
Revised: = (cid:80) f ∈ a i r f ( n f ( a )) ∈ [0 , F ] .

Original: Connection to Potential Games [Monderer and Shapley, 1996].
Revised: Connection to Potential Games [Monderer and Shapley, 1996].

Original: As a special class of potential game, all congestion games have the potential function: Φ( a ) = (cid:80) f ∈F (cid:80) n f ( a ) i =1 r f ( i ) .
Revised: As a special class of potential game, all congestion games have the potential function: Φ( a ) = (cid:80) f ∈F (cid:80) n f ( a ) i =1 r f ( i ) .

Original: To see this, we can easily verify that Φ( a i , a − i ) − Φ( a ′
Revised: To see this, we can easily verify that Φ( a i , a − i ) − Φ( a ′

Original: i , a − i )
Revised: i , a − i )

Original: =
Revised: =

Original: r i ( a i , a − i ) − r i ( a ′
Revised: r i ( a i , a − i ) − r i ( a ′

Original: i , a − i ) holds.
Revised: i , a − i ) holds.

Original: Then, by defining Φ( π ) =
Revised: Then, by defining Φ( π ) =

Original: E a ∼ π [Φ( a )] , we can have Φ( π i , π − i ) − Φ( π ′ i , π − i ) =
Revised: E a ∼ π [Φ( a )] , we can have Φ( π i , π − i ) − Φ( π ′ i , π − i ) =

Original: V π
Revised: V π

Original: i ,π − i i −
Revised: i ,π − i i −

Original: V π ′
Revised: V π ′

Original: i ,π − i i .
Revised: i ,π − i i .

Original: Types of feedback.
Revised: Types of feedback.

Original: There are in general two types of reward feedback for the congestion games, semi-bandit feedback and bandit feedback, both of which are reasonable under different scenarios.
Revised: There are in general two types of reward feedback for the congestion games, semi-bandit feedback and bandit feedback, both of which are reasonable under different scenarios.

Original: In semi-bandit feedback, after taking the action, player i will receive reward information r f for each f ∈ a i ; in bandit feedback, after taking the action, player i will only receive the reward r i
Revised: In semi-bandit feedback, after taking the action, player i will receive reward information r f for each f ∈ a i ; in bandit feedback, after taking the action, player i will only receive the reward r i

Original: = (cid:80) f ∈ a i r f with no knowledge about each r f .
Revised: = (cid:80) f ∈ a i r f with no knowledge about each r f .

Original: In this paper, we will address both of them, with more focus on the bandit feedback, which can be directly generalized to semi-bandit feedback.
Revised: In this paper, we will address both of them, with more focus on the bandit feedback, which can be directly generalized to semi-bandit feedback.

Original:  Centralized Algorithms for Congestion Games
Revised:  Centralized Algorithms for Congestion Games

Original: In this section, we introduce two centralized algorithms for congestion games – one for the semibandit feedback and one for the bandit feedback.
Revised: In this section, we introduce two centralized algorithms for congestion games – one for the semibandit feedback and one for the bandit feedback.

Original: We will see that both of them can achieve sublinear
Revised: We will see that both of them can achieve sublinear

Original: Nash regret with polynomial dependence on both m and F .
Revised: Nash regret with polynomial dependence on both m and F .

Original: 4.1 Algorithm for Semi-bandit Feedback
Revised: 4.1 Algorithm for Semi-bandit Feedback

Original: Summarized in Algorithm 1, Nash upper confidence bound (Nash-UCB) for congestion games is developed based on optimism in the face of uncertainty.
Revised: Summarized in Algorithm 1, Nash upper confidence bound (Nash-UCB) for congestion games is developed based on optimism in the face of uncertainty.

Original: In particular, the algorithm estimates the reward matrices optimistically in line 4, computes its Nash equilibrium policy in line 5 and then follows this policy.
Revised: In particular, the algorithm estimates the reward matrices optimistically in line 4, computes its Nash equilibrium policy in line 5 and then follows this policy.

Original: For convenience, we define the empirical counter N k,f ( n ) =
Revised: For convenience, we define the empirical counter N k,f ( n ) =

Original: (cid:80) kk ′
Revised: (cid:80) kk ′

Original: =1 1 (cid:110) n f ( a k ′ )
Revised: =1 1 (cid:110) n f ( a k ′ )

Original: = n (cid:111) and ˜ ι = 2 log(4( m + 1) K/δ ) .
Revised: = n (cid:111) and ˜ ι = 2 log(4( m + 1) K/δ ) .

Original: Then, the reward estimator for f and the bonus term are defined as
Revised: Then, the reward estimator for f and the bonus term are defined as

Original: [Equation]
Revised: [Equation]

Original: n )
Revised: n )

Original: [Equation]
Revised: [Equation]

Original: where r k,f ∈ [0 , 1] is the random reward realization of r f ( n f ( a k )) .
Revised: where r k,f ∈ [0 , 1] is the random reward realization of r f ( n f ( a k )) .

Original: Naturally, the reward estimator for player i is ˆ r ki ( a ) =
Revised: Naturally, the reward estimator for player i is ˆ r ki ( a ) =

Original: (cid:80) f ∈ a i ˆ r k,f ( n f ( a )) .
Revised: (cid:80) f ∈ a i ˆ r k,f ( n f ( a )) .

Original: Algorithm
Revised: Algorithm

Original: Take action a k ∼ π k and observe reward f 7:
Revised: Take action a k ∼ π k and observe reward f 7:

Original: Update reward estimators and bonus term
Revised: Update reward estimators and bonus term

Original: b i
Revised: b i

Original: Algorithm 1 is motivated by the Nash-VI algorithm in [Liu et al., 2021] plus a deliberate utilization of the special reward structure in the congestion games.
Revised: Algorithm 1 is motivated by the Nash-VI algorithm in [Liu et al., 2021] plus a deliberate utilization of the special reward structure in the congestion games.

Original: Moreover, notice that a matrix game with reward functions Q k 1 ( · ) , . . .
Revised: Moreover, notice that a matrix game with reward functions Q k 1 ( · ) , . . .

Original: , Q km ( · ) forms a potential game (see Lemma 1).
Revised: , Q km ( · ) forms a potential game (see Lemma 1).

Original: As a result, in line 5, we can efficiently compute the ϵ -approximate Nash equilibrium π k for that matrix game by utilizing
Revised: As a result, in line 5, we can efficiently compute the ϵ -approximate Nash equilibrium π k for that matrix game by utilizing

Original: Algorithm 2, (see Lemma 2).
Revised: Algorithm 2, (see Lemma 2).

Original: It is a simple greedy algorithm such that in each round, it modifies one player’s policy whose modification can increase the potential function most.
Revised: It is a simple greedy algorithm such that in each round, it modifies one player’s policy whose modification can increase the potential function most.

Original: In addition, Algorithmalways outputs a deterministic product policy.
Revised: In addition, Algorithmalways outputs a deterministic product policy.

Original: Algorithm 2 ϵ -approximate Nash Equilibrium for Potential Games 1: Input: ϵ , accuracy parameter; full information potential game ( {A i } mi
Revised: Algorithm 2 ϵ -approximate Nash Equilibrium for Potential Games 1: Input: ϵ , accuracy parameter; full information potential game ( {A i } mi

Original: 1 , { r i } mi =1 ) such that r i ∈ [0 , r max ] for all i ∈ m
Revised: 1 , { r i } mi =1 ) such that r i ∈ [0 , r max ] for all i ∈ m

Original: 2: Initialize: π 1 = a 1 , arbitrary deterministic product policy 3: for round k = 1 , . .
Revised: 2: Initialize: π 1 = a 1 , arbitrary deterministic product policy 3: for round k = 1 , . .

Original: , (cid:6) mr max ϵ (cid:7) do 4: for player
Revised: , (cid:6) mr max ϵ (cid:7) do 4: for player

Original: i
Revised: i

Original: , m do 5: ∆ max a
Revised: , m do 5: ∆ max a

Original: ∈A a π k − i )
Revised: ∈A a π k − i )

Original: − r i ( π k ) 6: a i = a ∈A i r i a i , π k − i ) − r i ( π k ) 7: if max i ∈ [ m ] ∆ i ≤ ϵ then 8: return π k 9:
Revised: − r i ( π k ) 6: a i = a ∈A i r i a i , π k − i ) − r i ( π k ) 7: if max i ∈ [ m ] ∆ i ≤ ϵ then 8: return π k 9:

Original: = i 10: π a j , π i π
Revised: = i 10: π a j , π i π

Original: i
Revised: i

Original: , for all
Revised: , for all

Original: i
Revised: i

Original: =
Revised: =

Original: j
Revised: j

Original: 4.2 Algorithm for Bandit Feedback
Revised: 4.2 Algorithm for Bandit Feedback

Original: When the players can only receive bandit feedback, estimating ˆ r k,f directly for each f ∈ F is no longer feasible.
Revised: When the players can only receive bandit feedback, estimating ˆ r k,f directly for each f ∈ F is no longer feasible.

Original: However, notice that the reward function r i ( a )
Revised: However, notice that the reward function r i ( a )

Original: =
Revised: =

Original: (cid:80) f ∈ a i r f ( n f ( a )) can be seen as an inner product between vectors characterized by action a and reward function r f ( · ) .
Revised: (cid:80) f ∈ a i r f ( n f ( a )) can be seen as an inner product between vectors characterized by action a and reward function r f ( · ) .

Original: Therefore, under bandit feedback, we can treat it as a linear bandit and use ridge regression to build the reward estimator ˜ r ki and corresponding bonus term ˜ b k, r , whose index i is dropped since it is the same for all players.
Revised: Therefore, under bandit feedback, we can treat it as a linear bandit and use ridge regression to build the reward estimator ˜ r ki and corresponding bonus term ˜ b k, r , whose index i is dropped since it is the same for all players.

Original: The new algorithm will use these two terms to replace ˆ r ki and b k, r i in line 4 of Algorithm 1.
Revised: The new algorithm will use these two terms to replace ˆ r ki and b k, r i in line 4 of Algorithm 1.

Original: In particular, define θ ∈ [0 , 1] ˜ d with ˜ d
Revised: In particular, define θ ∈ [0 , 1] ˜ d with ˜ d

Original: = mF to be the vector such that r f ( n ) = θ n + m ( f − 1) .
Revised: = mF to be the vector such that r f ( n ) = θ n + m ( f − 1) .

Original: Meanwhile, for player i ∈ [ m ] , define A i : A (cid:55)→ { 0 , 1 } ˜ d to be the vector-valued function such that [ A i ( a )]
Revised: Meanwhile, for player i ∈ [ m ] , define A i : A (cid:55)→ { 0 , 1 } ˜ d to be the vector-valued function such that [ A i ( a )]

Original: j = 1 (cid:8) j = n + m ( f − 1) , f ∈ a i , n = n f ( a ) (cid:9) .
Revised: j = 1 (cid:8) j = n + m ( f − 1) , f ∈ a i , n = n f ( a ) (cid:9) .

Original: In other words, A i ( a ) is a 0-1 vector with element 1 only at indices corresponding to those in θ that represents r f ( n ) for f ∈ a i and n = n f ( a ) .
Revised: In other words, A i ( a ) is a 0-1 vector with element 1 only at indices corresponding to those in θ that represents r f ( n ) for f ∈ a i and n = n f ( a ) .

Original: Now, with these definitions, the reward function can be written as r i ( a ) =
Revised: Now, with these definitions, the reward function can be written as r i ( a ) =

Original: ⟨ A i ( a ) , θ ⟩ .
Revised: ⟨ A i ( a ) , θ ⟩ .

Original: Then, we build the reward estimator and the bonus term through ridge regression and corresponding confidence bound, which are defined as the following:
Revised: Then, we build the reward estimator and the bonus term through ridge regression and corresponding confidence bound, which are defined as the following:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where (cid:98) θ k = (cid:0) V k (cid:1) − 1 (cid:80) k − 1 k ′ =1 (cid:80) mi =1 A
Revised: where (cid:98) θ k = (cid:0) V k (cid:1) − 1 (cid:80) k − 1 k ′ =1 (cid:80) mi =1 A

Original: i ( a k ′ )
Revised: i ( a k ′ )

Original: r k ′ i , V k
Revised: r k ′ i , V k

Original: = I + (cid:80) k − 1 k ′ =1 (cid:80) mi
Revised: = I + (cid:80) k − 1 k ′ =1 (cid:80) mi

Original: =1 A i ( a k ′ )
Revised: =1 A i ( a k ′ )

Original: A i ( a k ′ )
Revised: A i ( a k ′ )

Original: ⊤ and (cid:113) ˜
Revised: ⊤ and (cid:113) ˜

Original: β k = (cid:112)
Revised: β k = (cid:112)

Original: ˜
Revised: ˜

Original: d + (cid:114) ˜
Revised: d + (cid:114) ˜

Original: d log (cid:16) 1 + mkF ˜ d (cid:17) + ˜ ι .
Revised: d log (cid:16) 1 + mkF ˜ d (cid:17) + ˜ ι .

Original: Note that we cannot bound the sum of this bonus terms by directly applying the elliptical potential lemma.
Revised: Note that we cannot bound the sum of this bonus terms by directly applying the elliptical potential lemma.

Original: We instead prove its variant in Lemma 4.
Revised: We instead prove its variant in Lemma 4.

Original: 4.3 Regret Analysis
Revised: 4.3 Regret Analysis

Original: The Nash regret bounds for the two versions of Algorithm 1 are formally presented in Theorem 1.
Revised: The Nash regret bounds for the two versions of Algorithm 1 are formally presented in Theorem 1.

Original: The proof details are deferred to Appendix C.
Revised: The proof details are deferred to Appendix C.

Original: Theorem 1.
Revised: Theorem 1.

Original: Let ϵ = 1 /K .
Revised: Let ϵ = 1 /K .

Original: For congestion games with semi-bandit feedback, by running Algorithmwith reward estimator and bonus term in (1) , with probability at least 1 − δ , we can achieve that
Revised: For congestion games with semi-bandit feedback, by running Algorithmwith reward estimator and bonus term in (1) , with probability at least 1 − δ , we can achieve that

Original: [Equation]
Revised: [Equation]

Original: Furthermore, if we only have bandit feedback, then by running Algorithm 1 with reward estimator and bonus term in (2) , with probability at least 1 − δ , we can achieve that
Revised: Furthermore, if we only have bandit feedback, then by running Algorithm 1 with reward estimator and bonus term in (2) , with probability at least 1 − δ , we can achieve that

Original: [Equation]
Revised: [Equation]

Original: Remark 2 .
Revised: Remark 2 .

Original: Since each action is a subset of F , the size of each player’s action space can be 2 F .
Revised: Since each action is a subset of F , the size of each player’s action space can be 2 F .

Original: As a result, directly applying Nash-VI in [Liu et al., 2021] leads to a regret bound exponential in F .
Revised: As a result, directly applying Nash-VI in [Liu et al., 2021] leads to a regret bound exponential in F .

Original: Remark 3 .
Revised: Remark 3 .

Original: Note that we assume r f ∈ [0 , 1] , which implies r i ∈ [0 , F ] for each player i ∈ [ m ] .
Revised: Note that we assume r f ∈ [0 , 1] , which implies r i ∈ [0 , F ] for each player i ∈ [ m ] .

Original:  Decentralized Algorithms for Congestion Games
Revised:  Decentralized Algorithms for Congestion Games

Original: In this section, we present a decentralized algorithm for congestion games.
Revised: In this section, we present a decentralized algorithm for congestion games.

Original: Due to limited space, we only introduce the version of bandit feedback as in Section 4.2.
Revised: Due to limited space, we only introduce the version of bandit feedback as in Section 4.2.

Original: The algorithmic details for the semi-bandit feedback setting are deferred into Appendix D.3.
Revised: The algorithmic details for the semi-bandit feedback setting are deferred into Appendix D.3.

Original: We will show that under both settings, even though each player can only observe her own actions and rewards, our decentralized algorithm still enjoys sublinear Nash regret with polynomial dependence on m and F .
Revised: We will show that under both settings, even though each player can only observe her own actions and rewards, our decentralized algorithm still enjoys sublinear Nash regret with polynomial dependence on m and F .

Original: We first define the vector-valued function ϕ
Revised: We first define the vector-valued function ϕ

Original: i : A i (cid:55)→ { 0 , 1 } F i to be the feature map of player i such that [ ϕ i ( a i )]
Revised: i : A i (cid:55)→ { 0 , 1 } F i to be the feature map of player i such that [ ϕ i ( a i )]

Original: f = 1 { f ∈ a i } for a i ∈
Revised: f = 1 { f ∈ a i } for a i ∈

Original: A i and f ∈ (cid:83) a i ∈A i a i .
Revised: A i and f ∈ (cid:83) a i ∈A i a i .

Original: Here, F i is the size of (cid:83) a i ∈A
Revised: Here, F i is the size of (cid:83) a i ∈A

Original: i a i ⊆ F
Revised: i a i ⊆ F

Original: and we can immediately see that F i ≤ F for any i ∈ [ m ] .
Revised: and we can immediately see that F i ≤ F for any i ∈ [ m ] .

Original: The core idea of our algorithm is that the Nash equilibrium can be found by reaching the stationary points of the potential function since all congestion games are potential games.
Revised: The core idea of our algorithm is that the Nash equilibrium can be found by reaching the stationary points of the potential function since all congestion games are potential games.

Original: Here, the UCBlike algorithms used in the centralized setting are not applicable because their policy computation requires value functions for all players (e.g., line 5 of Algorithm 1), which are not available in the decentralized setting.
Revised: Here, the UCBlike algorithms used in the centralized setting are not applicable because their policy computation requires value functions for all players (e.g., line 5 of Algorithm 1), which are not available in the decentralized setting.

Original: Summarized in Algorithm 3, the decentralized algorithm is developed based on the Frank-Wolfe method and has the following three major components.
Revised: Summarized in Algorithm 3, the decentralized algorithm is developed based on the Frank-Wolfe method and has the following three major components.

Original: Gradient Estimator.
Revised: Gradient Estimator.

Original: In line 7, the algorithm builds the estimator (cid:98) ∇ ki Φ defined in (4) by using the τ reward samples collected from line 5.
Revised: In line 7, the algorithm builds the estimator (cid:98) ∇ ki Φ defined in (4) by using the τ reward samples collected from line 5.

Original: Here, (cid:98) ∇ ki Φ estimates the gradient of potential function Φ with respect to the policy π ki .
Revised: Here, (cid:98) ∇ ki Φ estimates the gradient of potential function Φ with respect to the policy π ki .

Original: Recall that for a congestion game, we have Φ( a )
Revised: Recall that for a congestion game, we have Φ( a )

Original: = (cid:80) f ∈F (cid:80) n f ( a ) i =1 r f ( i )
Revised: = (cid:80) f ∈F (cid:80) n f ( a ) i =1 r f ( i )

Original: 1: Input ; π  , initial policy. 
Revised: 1: Input: γ, ν , mixture weights; π 1 i , initial policy. 3: for

Original:  do 4: do 5:
Revised: K do 4: do 5:

Revised: Each takes

Original: ∼ π ki , observes reward r k,ti .
Revised: ∼ π ki , observes reward r k,ti .

Original: 6: for player i = 1 , · · · , m do 7:
Revised: 6: for player i = 1 , · · · , m do 7:

Original: Compute (cid:98) ∇ ki Φ( a i ) by the formula in (4) for all a i ∈
Revised: Compute (cid:98) ∇ ki Φ( a i ) by the formula in (4) for all a i ∈

Original: A i 8:
Revised: A i 8:

Original: Compute (cid:101)
Revised: Compute (cid:101)

Original: π k +

Revised: π k +

Original: i ← argmax π i ∈ ∆( A i ) (cid:68)
Revised: i ← argmax π i ∈ ∆( A i ) (cid:68)

Original: π i , (cid:98) ∇ ki Φ (cid:69) 9:
Revised: π i , (cid:98) ∇ ki Φ (cid:69) 9:

Original: Update π k +

Revised: Update π k +

Original: i ← (
Revised: i ← (

Original: [Equation]
Revised: [Equation]

Original: γ
Revised: γ

Original: [Equation]
Revised: [Equation]

Original: ( ν (cid:101)
Revised: ( ν (cid:101)

Original: [Equation]
Revised: [Equation]

Original: k +1 i
Revised: k +1 i

Original: [Equation]
Revised: [Equation]

Original: i and Φ( π ) =
Revised: i and Φ( π ) =

Original: E a ∼ π [Φ( a )] .
Revised: E a ∼ π [Φ( a )] .

Original: Then we can define ∇ i Φ :=
Revised: Then we can define ∇ i Φ :=

Original: ∇ π i Φ as a vector of dimension |A
Revised: ∇ π i Φ as a vector of dimension |A

Original: i | .
Revised: i | .

Original: For the component indexed by some a i ∈
Revised: For the component indexed by some a i ∈

Original: A i , we can see that Φ( π ) =
Revised: A i , we can see that Φ( π ) =

Original: π i ( a i ) E a − i ∼ π − i [ r i ( a i , a − i )]
Revised: π i ( a i ) E a − i ∼ π − i [ r i ( a i , a − i )]

Original: + const , where const does not depend on π i ( a i ) .
Revised: + const , where const does not depend on π i ( a i ) .

Original: Therefore, we have ∇
Revised: Therefore, we have ∇

Original: [Equation]
Revised: [Equation]

Revised: 

Revised: [Equation]

Revised: (cid:88)

Revised: [Equation]

Original: where
Revised: where

Original: [ θ i ( π )] f = E a − i ∼ π − i (cid:2) r f ( n f ( a − i )
Revised: [ θ i ( π )] f = E a − i ∼ π − i (cid:2) r f ( n f ( a − i )

Original: + 1) (cid:3) .
Revised: + 1) (cid:3) .

Original: Meanwhile, the mean of the t -th reward that player i received at episode k satisfies
Revised: Meanwhile, the mean of the t -th reward that player i received at episode k satisfies

Original: [Equation]
Revised: [Equation]

Original: where [ θ k,ti ( a k,t − i )]
Revised: where [ θ k,ti ( a k,t − i )]

Original: f = r f ( n f ( a k,t − i )
Revised: f = r f ( n f ( a k,t − i )

Original: + 1) and its mean is [ θ i ( π k )] f .
Revised: + 1) and its mean is [ θ i ( π k )] f .

Original: Therefore, we can use linear regression to estimate θ i ( π k ) .
Revised: Therefore, we can use linear regression to estimate θ i ( π k ) .

Original: In particular, we have (cid:98) θ ki ( π k )
Revised: In particular, we have (cid:98) θ ki ( π k )

Original: = 1 τ (cid:80) τt =1 (cid:0)
Revised: = 1 τ (cid:80) τt =1 (cid:0)

Original: Σ ki (cid:1) − 1 ϕ
Revised: Σ ki (cid:1) − 1 ϕ

Original: i ( a k,ti ) r k,ti , with the covariance matrix Σ ki = E a i ∼
Revised: i ( a k,ti ) r k,ti , with the covariance matrix Σ ki = E a i ∼

Original: [Equation]
Revised: [Equation]

Original: ki
Revised: ki

Original: [Equation]
Revised: [Equation]

Original:  i ( a i ) ϕi (
Revised: ϕ i ( a i ) ϕ

Original: [Equation]

Original: i  ) ⊤ (cid:3) .
Revised: i ( a i ) ⊤ (cid:3) .

Original: Then, we have the unbiased gradient estimate
Revised: Then, we have the unbiased gradient estimate

Original: [Equation]
Revised: [Equation]

Original: Remark 4 .
Revised: Remark 4 .

Original: One difference between Algorithm 3 (decentralized) and Algorithm 1 (centralized) is that in the decentralized algorithm, each player is required to play the same policy for τ times before an update can be applied.
Revised: One difference between Algorithm 3 (decentralized) and Algorithm 1 (centralized) is that in the decentralized algorithm, each player is required to play the same policy for τ times before an update can be applied.

Original: An episode is thus defined for convenience as the time period during which the players’ policies are fixed.
Revised: An episode is thus defined for convenience as the time period during which the players’ policies are fixed.

Original: We make this artificial design mainly for controlling the variance of the gradient estimator (cid:98) ∇ ki Φ( a i ) .
Revised: We make this artificial design mainly for controlling the variance of the gradient estimator (cid:98) ∇ ki Φ( a i ) .

Original: However, we conjecture that with more careful design and analysis, it should be possible to improve Algorithm 3 so that only one sample is required per episode [Zhang et al., 2020].
Revised: However, we conjecture that with more careful design and analysis, it should be possible to improve Algorithm 3 so that only one sample is required per episode [Zhang et al., 2020].

Original: G-optimal Design.
Revised: G-optimal Design.

Original: In line 8 and 9, the algorithm performs standard Frank-Wolfe update and mixes the updated policy with an exploration policy ρ i , which is defined as the G-optimal allocation for features { ϕ i ( a i ) } a i ∈A i .
Revised: In line 8 and 9, the algorithm performs standard Frank-Wolfe update and mixes the updated policy with an exploration policy ρ i , which is defined as the G-optimal allocation for features { ϕ i ( a i ) } a i ∈A i .

Original: To be specific, we have
Revised: To be specific, we have

Original: [Equation]
Revised: [Equation]

Original: Here ρ i guarantees that Σ ki is invertible and the variance of (cid:98) ∇ ki Φ( a i ) =
Revised: Here ρ i guarantees that Σ ki is invertible and the variance of (cid:98) ∇ ki Φ( a i ) =

Original: (cid:68) ϕ
Revised: (cid:68) ϕ

Original: i ( a i ) , (cid:98) θ ki ( π k ) (cid:69) depends only on F instead of the size of action space (Lemma 9) because by  Kiefer-Wolfowitz theorem, we have 
Revised: i ( a i ) , (cid:98) θ ki ( π k ) (cid:69) depends only on F instead of the size of action space (Lemma 9) because by the famous Kiefer-Wolfowitz theorem, we have max a i ∈A i ∥ ϕ i

Original: [Equation]
Revised: [Equation]

Revised: a

Original:  [Lattimore and Szepesvári, 2020].
Revised: [Equation] = F [Equation] ≤ F [Lattimore and Szepesvári, 2020].

Original: Frank-Wolfe Update.
Revised: Frank-Wolfe Update.

Original: Finally, we emphasize that it is crucial to use Frank-Wolfe update because it is compatible with L 1 norm and we can show that Φ is mF -smooth with respect to the L 1 norm (Lemma 11).
Revised: Finally, we emphasize that it is crucial to use Frank-Wolfe update because it is compatible with L 1 norm and we can show that Φ is mF -smooth with respect to the L 1 norm (Lemma 11).

Original: In contrast, its smoothness for L 2 norm will depend on the size of the action space.
Revised: In contrast, its smoothness for L 2 norm will depend on the size of the action space.

Original: Before the game starts, each player i can compute her ρ i based on her own action set A i .
Revised: Before the game starts, each player i can compute her ρ i based on her own action set A i .

Original: During the game, all players only have access to their own actions and rewards, which means that Algorithm 3 is fully decentralized.
Revised: During the game, all players only have access to their own actions and rewards, which means that Algorithm 3 is fully decentralized.

Original: The Nash regret bound for this algorithm is formally stated in Theorem 2 and the proof details are given in Appendix D.1 and D.2.
Revised: The Nash regret bound for this algorithm is formally stated in Theorem 2 and the proof details are given in Appendix D.1 and D.2.

Original: Theorem 2.
Revised: Theorem 2.

Original: Let T =
Revised: Let T =

Original: Kτ .
Revised: Kτ .

Original: For congestion game with bandit feedback, by running Algorithm 3 with gradient estimator (cid:98) ∇ ki Φ  and exploration distribution ρ idefined in(4) , (5) ,  with probability at least 1 − δ , we have 
Revised: For congestion game with bandit feedback, by running Algorithmwith gradient estimator (cid:98) ∇ ki Φ in (4) and exploration distribution ρ i in (5) , if K ≥ 2 Fm , then with probability at least 1 − δ , we have Nash [Equation] ( T ) :=

Original: [Equation]
Revised: [Equation]

Original: For congestion game with semi-bandit feedback, by running Algorithm 3 with gradient estimator (cid:101) ∇ ki Φ( a i ) and exploration distribution ˜ 
Revised: For congestion game with semi-bandit feedback, by running Algorithm 3 with gradient estimator (cid:101) ∇ ki Φ( a i ) and exploration distribution ˜ ρ

Original: ρ i  in Appendix D.3,  with probability at least 1 − δ , we have
Revised: i defined in Appendix D.3, if K ≥ 2 √ Fm , then with probability at least 1 − δ , we have

Original: [Equation]
Revised: [Equation]

Original:  Extension to Independent Markov Congestion Games
Revised:  Extension to Independent Markov Congestion Games

Original: In this section, we propose and analyze a Markov extension of the congestion games, called the independent Markov congestion games (IMCGs).
Revised: In this section, we propose and analyze a Markov extension of the congestion games, called the independent Markov congestion games (IMCGs).

Original: 6.1 Problem Formulation
Revised: 6.1 Problem Formulation

Original: General-sum Markov Games.
Revised: General-sum Markov Games.

Original: A finite-horizon time-inhomogeneous tabular general-sum Markov game is defined by M = {S , {A i } mi =1 , H, P, R, s 0 } , where S is the state space, m is the number of players, A i is the action space of player i , A
Revised: A finite-horizon time-inhomogeneous tabular general-sum Markov game is defined by M = {S , {A i } mi =1 , H, P, R, s 0 } , where S is the state space, m is the number of players, A i is the action space of player i , A

Original: =
Revised: =

Original: A 1 × · · · × A m is the whole action space, H is the time horizon, s 0 is the initial state 1 , P =
Revised: A 1 × · · · × A m is the whole action space, H is the time horizon, s 0 is the initial state 1 , P =

Original: ( P 1 , P 2 , · · · , P H ) with P h ∈ [0 , 1] S ×
Revised: ( P 1 , P 2 , · · · , P H ) with P h ∈ [0 , 1] S ×

Original: A × S as the transition kernel at timestep h , R = { R h ( ·| s h , a h ) }
Revised: A × S as the transition kernel at timestep h , R = { R h ( ·| s h , a h ) }

Original: Hh =1 with R h ( ·| s h , a h ) as the reward distribution on [0 , r max ] m with mean r h ( s h , a h ) ∈ [0 , r max ] m at timestep h ∈
Revised: Hh =1 with R h ( ·| s h , a h ) as the reward distribution on [0 , r max ] m with mean r h ( s h , a h ) ∈ [0 , r max ] m at timestep h ∈

Original: [ H ] .
Revised: [ H ] .

Original: At timestep h , all players choose their actions simultaneously and a reward vector is sampled r h ∼ R h ( ·| s h , a h ) , where s h is the current state and a h
Revised: At timestep h , all players choose their actions simultaneously and a reward vector is sampled r h ∼ R h ( ·| s h , a h ) , where s h is the current state and a h

Original: = ( a h, 1 , a h, 2 , · · · , a h,m ) is the joint action.
Revised: = ( a h, 1 , a h, 2 , · · · , a h,m ) is the joint action.

Original: Each player i receives reward r h,i and the state transits to s h +1 ∼ P h ( ·| s h , a h ) .
Revised: Each player i receives reward r h,i and the state transits to s h +1 ∼ P h ( ·| s h , a h ) .

Original: The objective for each player is to maximize her own total reward.
Revised: The objective for each player is to maximize her own total reward.

Original: We assume that the initial state s 1 is fixed.
Revised: We assume that the initial state s 1 is fixed.

Original: A (Markov) policy π is a collection of H functions { π h : S (cid:55)→ ∆( A ) } Hh =1 , each of which maps a state to a distribution over the action space.
Revised: A (Markov) policy π is a collection of H functions { π h : S (cid:55)→ ∆( A ) } Hh =1 , each of which maps a state to a distribution over the action space.

Original: π is a product policy if π h ( · | s ) is a product policy for each ( h, s ) ∈
Revised: π is a product policy if π h ( · | s ) is a product policy for each ( h, s ) ∈

Original: [ H ] × S .
Revised: [ H ] × S .

Original: The value function and Q -value function of player i at timestep h under policy π are defined as
Revised: The value function and Q -value function of player i at timestep h under policy π are defined as

Original: [Equation]
Revised: [Equation]

Original: The best responses and Nash regret can be defined similarly as those for matrix games.
Revised: The best responses and Nash regret can be defined similarly as those for matrix games.

Original: In particular, given a policy π , player i ’s best response policy is π † h,i ( · | s ) = argmax µ ∈ ∆( A i ) V
Revised: In particular, given a policy π , player i ’s best response policy is π † h,i ( · | s ) = argmax µ [Equation] ( A i ) V

Original: µ,π − i h,i ( s ) and the corresponding value function is denoted as V † ,π − i h,i .
Revised: µ,π − i h,i ( s ) and the corresponding value function is denoted as V † ,π − i h,i .

Original: Definition 3.
Revised: Definition 3.

Original: With π k being the policy at k th episode, the Nash regret after K episodes is define as
Revised: With π k being the policy at k th episode, the Nash regret after K episodes is define as

Original: [Equation]
Revised: [Equation]

Original: Independent Markov Congestion Game.
Revised: Independent Markov Congestion Game.

Original: A general-sum Markov game is an independent Markov congestion game (IMCG) if there exists a facility set F such that a i ⊆ F for any
Revised: A general-sum Markov game is an independent Markov congestion game (IMCG) if there exists a facility set F such that a i ⊆ F for any

Original: a i ∈
Revised: a i ∈

Original: A i , a state space S =
Revised: A i , a state space S =

Original: (cid:81)
Revised: (cid:81)

Original: f ∈F S f , a set of facility reward distributions { R fh } h ∈
Revised: f ∈F S f , a set of facility reward distributions { R fh } h ∈

Original: [ H ] ,f ∈F such that if the joint action at s h is a , we have r h,i = (cid:80)
Revised: [ H ] ,f ∈F such that if the joint action at s h is a , we have r h,i = (cid:80)

Original: f ∈ a
Revised: f ∈ a

Original: i r fh , where r fh ∼ R fh ( ·| s h , n f ( a )) with support on [0 , 1] and mean r fh ( s h , n f ( a )) , and a set of transition matrices { P fh } h ∈ [ H ] ,f ∈F such that P h ( s ′ | s, a ) =
Revised: i r fh , where r fh ∼ R fh ( ·| s h , n f ( a )) with support on [0 , 1] and mean r fh ( s h , n f ( a )) , and a set of transition matrices { P fh } h ∈ [ H ] ,f ∈F such that P h ( s ′ | s, a ) =

Original: (cid:81)
Revised: (cid:81)

Original: f ∈F P fh ( s ′ f | s f , n f ( a )) .
Revised: f ∈F P fh ( s ′ f | s f , n f ( a )) .

Original: In other words, at each timestep h and state s ∈ S , the players are in a congestion game.
Revised: In other words, at each timestep h and state s ∈ S , the players are in a congestion game.

Original: Meanwhile, each facility has its own state and independent state transition, which only depends on its current state and number of players using that facility.
Revised: Meanwhile, each facility has its own state and independent state transition, which only depends on its current state and number of players using that facility.

Original: This transition kernel can be viewed as a special case of that in factored MDPs [Szita and L˝orincz, 2009].
Revised: This transition kernel can be viewed as a special case of that in factored MDPs [Szita and L˝orincz, 2009].

Original: The IMCG also admits two types of feedback, semi-bandit feedback and bandit feedback, just like the congestion game.
Revised: The IMCG also admits two types of feedback, semi-bandit feedback and bandit feedback, just like the congestion game.

Original: In this paper, we will consider both types of feedback.
Revised: In this paper, we will consider both types of feedback.

Original: 6.2 Theoretical Guarantee
Revised: 6.2 Theoretical Guarantee

Original: Summarized in Algorithm 5, our centralized algorithm for IMCGs is naturally extended from the
Revised: Summarized in Algorithm 5, our centralized algorithm for IMCGs is naturally extended from the

Original: Nash-UCB (Algorithm 1) by incorporating transition kernel estimators, corresponding bonus terms and Bellman backward update.
Revised: Nash-UCB (Algorithm 1) by incorporating transition kernel estimators, corresponding bonus terms and Bellman backward update.

Original: The key idea is to utilize the independent transition structure to remove the dependence on the exponential size of the state space S =
Revised: The key idea is to utilize the independent transition structure to remove the dependence on the exponential size of the state space S =

Original: (cid:81)
Revised: (cid:81)

Original: f ∈F S f .
Revised: f ∈F S f .

Original: We tackle this issue by adapting technique from factored MDP [Chen et al., 2020].
Revised: We tackle this issue by adapting technique from factored MDP [Chen et al., 2020].

Original: The algorithmic details for both types of feedback are deferred into Appendix E. The Nash regret bounds for the two versions of Algorithm 5 are stated in Theorem 3 and the proof details are deferred to Appendix F.
Revised: The algorithmic details for both types of feedback are deferred into Appendix E. The Nash regret bounds for the two versions of Algorithm 5 are stated in Theorem 3 and the proof details are deferred to Appendix F.

Original: Theorem 3.
Revised: Theorem 3.

Original: For independent Markov congestion game with semi-bandit feedback, by running the centralized Algorithm 5, with probability at least 1 − δ , we can achieve that
Revised: For independent Markov congestion game with semi-bandit feedback, by running the centralized Algorithm 5, with probability at least 1 − δ , we can achieve that

Original: [Equation]
Revised: [Equation]

Original: Furthermore, if we only have bandit feedback, then by running Algorithm 5 with reward estimator and bonus term in (12) and (13) , with probability at least 1 − δ , we can achieve thatNash-Regret ( K ) ≤ (cid:101) O  (cid:88) f ∈F
Revised: Furthermore, if we only have bandit feedback, then by running Algorithm 5 with reward estimator and bonus term in (12) and (13) , with probability at least 1 − δ , we can achieve that

Original: [Equation]

Original: T 

Original: [Equation]

Original: (cid:101)

Original: O 

Original: [Equation]

Original: m

Original: [Equation]
Revised: [Equation]

Original: F (cid:88) f ̸

Original: [Equation]

Original: f ′ (cid:16)

Original: [Equation]

Original: S f ′ (cid:17)  .

Original: The regret bound in [Liu et al., 2021] is (cid:101) O
Revised: The regret bound in [Liu et al., 2021] is (cid:101) O

Original: ( (cid:112) H 3 S 2 (Π mi =1 A i ) T ) , where both A i and S = (cid:81)
Revised: ( (cid:112) H 3 S 2 (Π mi =1 A i ) T ) , where both A i and S = (cid:81)

Original: f ∈F S f can be exponential in F .
Revised: f ∈F S f can be exponential in F .

Original: Our bounds have polynomial dependence on all the parameters.
Revised: Our bounds have polynomial dependence on all the parameters.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we study sample-efficient learning in congestion games by utilizing the special reward structure.
Revised: In this paper, we study sample-efficient learning in congestion games by utilizing the special reward structure.

Original: We propose both centralized and decentralized algorithms for congestion games with two types of feedback, all achieving sample complexities only polynomial in the number of facilities.
Revised: We propose both centralized and decentralized algorithms for congestion games with two types of feedback, all achieving sample complexities only polynomial in the number of facilities.

Original: To the best of our knowledge, each one of them is the first sample-efficient learning algorithm for congestion games in its own setting.
Revised: To the best of our knowledge, each one of them is the first sample-efficient learning algorithm for congestion games in its own setting.

Original: We further define the independent Markov congestion game (IMCG) as a natural extension of the congestion game into the Markov setting together with a sample-efficient centralized algorithm for both types of feedback.
Revised: We further define the independent Markov congestion game (IMCG) as a natural extension of the congestion game into the Markov setting together with a sample-efficient centralized algorithm for both types of feedback.

Original: One promising future direction is to find a sample-efficient decentralized algorithm such that from each player’s own perspective, the algorithm is still no-regret.
Revised: One promising future direction is to find a sample-efficient decentralized algorithm such that from each player’s own perspective, the algorithm is still no-regret.

Original: In other words, diminishing regret is guaranteed for the player by running this algorithm even though other players may use policies from different algorithms.
Revised: In other words, diminishing regret is guaranteed for the player by running this algorithm even though other players may use policies from different algorithms.

Original: Another important future direction is to find sample-efficient centralized/decentralized algorithms that can explicitly find an approximate Nash equilibrium policy.
Revised: Another important future direction is to find sample-efficient centralized/decentralized algorithms that can explicitly find an approximate Nash equilibrium policy.

Original: Acknowledgements
Revised: Acknowledgements

Original:  This work was supported in part by NSF TRIPODS II-DMS 2023166, NSF CCF 2007036, NSF IIS2110170, NSF DMS 2134106, NSF CCF 2212261, NSF IIS 2143493, NSF CCF 2019844.
Revised: We sincerely thank Jing Dong for pointing out a mistake in the initial draft of this paper. This work was supported in part by NSF TRIPODS II-DMS 2023166, NSF CCF 2007036, NSF IIS

Original: E Algorithms for Independent Markov Congestion Games
Revised: E Algorithms for Independent Markov Congestion Games

Original: E.1 Algorithm for Semi-bandit Feedback . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: E.1 Algorithm for Semi-bandit Feedback . . . . . . . . . . . . . . . . . . . . . . . . .

Original: E.2 Algorithm for Bandit Feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: E.2 Algorithm for Bandit Feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: F Analysis for Algorithm
Revised: F Analysis for Algorithm

Original: F.
Revised: F.

Original: Bellman Equations for Genera-sum Markov Games . . . . . . . . . . . . . . . . .
Revised: Bellman Equations for Genera-sum Markov Games . . . . . . . . . . . . . . . . .

Original: F.
Revised: F.

Original: Proof of Theorem 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Proof of Theorem 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: F.
Revised: F.

Original: Lemmas for Semi-bandit Feedback . . . .
Revised: Lemmas for Semi-bandit Feedback . . . .

Original: F.
Revised: F.

Original: Additional Lemmas for Bandit Feedback . . . . . . . . . . . . . . . . . . . . . . .
Revised: Additional Lemmas for Bandit Feedback . . . . . . . . . . . . . . . . . . . . . . .

Original: F.
Revised: F.

Original: Technical Lemmas . .
Revised: Technical Lemmas . .

Original: A Additional Motivating Examples
Revised: A Additional Motivating Examples

Original: In this section, we present two additional motivating examples of our proposed models.
Revised: In this section, we present two additional motivating examples of our proposed models.

Original: Example 2 ( Web Advertisements) .
Revised: Example 2 ( Web Advertisements) .

Original: Consider a set of websites as the facility set and companies who want to advertise their products as the players.
Revised: Consider a set of websites as the facility set and companies who want to advertise their products as the players.

Original: Due to budget constraints, each company may only choose some of these websites to put its product ad.
Revised: Due to budget constraints, each company may only choose some of these websites to put its product ad.

Original: For each website, the probability that a user will click on a certain ad (and then buy the product) depends on how many ads are put on the website.
Revised: For each website, the probability that a user will click on a certain ad (and then buy the product) depends on how many ads are put on the website.

Original: If a website receives too many ads, the probability that a user can see a certain ad will decrease, thus making it congested.
Revised: If a website receives too many ads, the probability that a user can see a certain ad will decrease, thus making it congested.

Original: The reward each company will receive is measured by the amount of products sold during certain period of time, which is bandit feedback.
Revised: The reward each company will receive is measured by the amount of products sold during certain period of time, which is bandit feedback.

Original: Example 3 ( Server Usage) .
Revised: Example 3 ( Server Usage) .

Original: Consider a set of servers in a company as the facility set and server users as the players.
Revised: Consider a set of servers in a company as the facility set and server users as the players.

Original: Each user needs to request several servers to finish her computation task and the cost triggered from each server depends on the number of users requesting that server.
Revised: Each user needs to request several servers to finish her computation task and the cost triggered from each server depends on the number of users requesting that server.

Original: Each user will try to minimize the total cost incurred from the servers she requested.
Revised: Each user will try to minimize the total cost incurred from the servers she requested.

Original: As each user can see the cost from all the servers she requested, this is semi-bandit feedback.
Revised: As each user can see the cost from all the servers she requested, this is semi-bandit feedback.

Original: B Compute ϵ -approximate Nash Equilibrium in Potential Games
Revised: B Compute ϵ -approximate Nash Equilibrium in Potential Games

Original: In this section, we show that the ϵ -N ASH ( · ) operation in Algorithm 1 can be computed efficiently by using Algorithm 2.
Revised: In this section, we show that the ϵ -N ASH ( · ) operation in Algorithm 1 can be computed efficiently by using Algorithm 2.

Original: In particular, we first show that the matrix game with reward functions Q k 1 ( · ) , . . .
Revised: In particular, we first show that the matrix game with reward functions Q k 1 ( · ) , . . .

Original: , Q km ( · ) used in
Revised: , Q km ( · ) used in

Original: Algorithm 1 is a potential game in Lemma 1.
Revised: Algorithm 1 is a potential game in Lemma 1.

Original: Then, we show that Algorithm 2 can efficiently compute an ϵ -approximate Nash equilibrium for potential games and output a product policy as shown in
Revised: Then, we show that Algorithm 2 can efficiently compute an ϵ -approximate Nash equilibrium for potential games and output a product policy as shown in

Original: Lemma 2.
Revised: Lemma 2.

Original: Lemma 1.
Revised: Lemma 1.

Original: In line 5 of Algorithm 1, the matrix game with reward functions Q k 1 ( · ) , . . .
Revised: In line 5 of Algorithm 1, the matrix game with reward functions Q k 1 ( · ) , . . .

Original: , Q km ( · ) forms a potential game for both settings of semi-bandit feedback and bandit feedback.
Revised: , Q km ( · ) forms a potential game for both settings of semi-bandit feedback and bandit feedback.

Original: Proof.
Revised: Proof.

Original: In the setting of semi-bandit feedback, since Q ki ( a ) = (cid:80) f ∈ a i (ˆ r k,f + b k,f, r )( a ) , the reward functions Q k 1 ( · ) , . . .
Revised: In the setting of semi-bandit feedback, since Q ki ( a ) = (cid:80) f ∈ a i (ˆ r k,f + b k,f, r )( a ) , the reward functions Q k 1 ( · ) , . . .

Original: , Q km ( · ) form a congestion game, which we know is a potential game [Monderer and Shapley, 1996].
Revised: , Q km ( · ) form a congestion game, which we know is a potential game [Monderer and Shapley, 1996].

Original: In the setting of bandit feedback, notice that by defining ˜ r k,f ( i )
Revised: In the setting of bandit feedback, notice that by defining ˜ r k,f ( i )

Original: =
Revised: =

Original: (cid:98) θ ki
Revised: (cid:98) θ ki

Original: + m ( f − 1) for ( i, f ) ∈ [ m ] × F , we can have ˜ r ki ( a )
Revised: + m ( f − 1) for ( i, f ) ∈ [ m ] × F , we can have ˜ r ki ( a )

Original: =
Revised: =

Original: (cid:68)
Revised: (cid:68)

Original: A i ( a ) , (cid:98) θ k (cid:69) =
Revised: A i ( a ) , (cid:98) θ k (cid:69) =

Original: (cid:80) f ∈ a i ˜
Revised: (cid:80) f ∈ a i ˜

Original: r k,f ( n f ( a )) .
Revised: r k,f ( n f ( a )) .

Original: Therefore, we claim that the desired potential function is
Revised: Therefore, we claim that the desired potential function is

Original: [Equation]
Revised: [Equation]

Original: where (cid:101)
Revised: where (cid:101)

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: f
Revised: f

Original: [Equation]
Revised: [Equation]

Original: To see this, by referring to the definition of potential function in congestion game [Monderer and
Revised: To see this, by referring to the definition of potential function in congestion game [Monderer and

Original: Shapley, 1996], since ˜ r ki ( a ) =
Revised: Shapley, 1996], since ˜ r ki ( a ) =

Original: (cid:80) f ∈ a i ˜
Revised: (cid:80) f ∈ a i ˜

Original: r k,f ( n f ( a )) , we have that
Revised: r k,f ( n f ( a )) , we have that

Original: [Equation]
Revised: [Equation]

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: − ˜ r i ( a
Revised: − ˜ r i ( a

Original: [Equation]
Revised: [Equation]

Original: i , a
Revised: i , a

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: As a result, we have
Revised: As a result, we have

Original: [Equation]
Revised: [Equation]

Original: which means that Q k 1 ( · ) , . .
Revised: which means that Q k 1 ( · ) , . .

Original: , Q km ( · ) form a potential game.
Revised: , Q km ( · ) form a potential game.

Original: Lemma 2.
Revised: Lemma 2.

Original: Algorithm 2 can output an ϵ -approximate Nash equilibrium.
Revised: Algorithm 2 can output an ϵ -approximate Nash equilibrium.

Original: Proof.
Revised: Proof.

Original: Note that if at round k , we have max i ∈ [ m ] ∆ i ≤ ϵ , then π k is an ϵ -approximate Nash equilibrium.
Revised: Note that if at round k , we have max i ∈ [ m ] ∆ i ≤ ϵ , then π k is an ϵ -approximate Nash equilibrium.

Original: So we only need to prove that max i ∈ [ m ] ∆ i ≤ ϵ is satisfied at some round k ∈ { 1 , . . .
Revised: So we only need to prove that max i ∈ [ m ] ∆ i ≤ ϵ is satisfied at some round k ∈ { 1 , . . .

Original: , (cid:6) mr max ϵ (cid:7) } .
Revised: , (cid:6) mr max ϵ (cid:7) } .

Original: Suppose the potential game ( {A i } mi =1 , { r i } mi =1 ) is associated with potential function Φ ∈ [0 , Φ max ] .
Revised: Suppose the potential game ( {A i } mi =1 , { r i } mi =1 ) is associated with potential function Φ ∈ [0 , Φ max ] .

Original: Set π ∗ = argmax π ∈ (cid:81) i ∈ [ m ] ∆( A i ) Φ( π ) .
Revised: Set π ∗ = argmax π ∈ (cid:81) i ∈ [ m ] ∆( A i ) Φ( π ) .

Original: Then for any π ∈ (cid:81) i ∈ [ m ] ∆( A i ) , we have
Revised: Then for any π ∈ (cid:81) i ∈ [ m ] ∆( A i ) , we have

Original: [Equation]
Revised: [Equation]

Original: As a result, we can set Φ max = mr max .
Revised: As a result, we can set Φ max = mr max .

Original: On the other hand, if j = argmax i ∈ [ m ] ∆ i for round k , we have
Revised: On the other hand, if j = argmax i ∈ [ m ] ∆ i for round k , we have

Original: [Equation]
Revised: [Equation]

Original: So there must exist k ∈ { 1 , . . .
Revised: So there must exist k ∈ { 1 , . . .

Original: , (cid:6) mr max ϵ (cid:7) } such that max i ∈ [ m ] ∆ i ≤ ϵ , otherwise Φ( π k ) increase at least ϵ at each round, which contradicts Φ ∈ [0 , mr max ] .
Revised: , (cid:6) mr max ϵ (cid:7) } such that max i ∈ [ m ] ∆ i ≤ ϵ , otherwise Φ( π k ) increase at least ϵ at each round, which contradicts Φ ∈ [0 , mr max ] .

Original: C Analysis for Algorithm
Revised: C Analysis for Algorithm

Original: Recall that the update rule in Algorithm 1 is Q ki ( a )
Revised: Recall that the update rule in Algorithm 1 is Q ki ( a )

Original: =
Revised: =

Original: ˆ r ki ( a )
Revised: ˆ r ki ( a )

Original: + b k, r i ( a ) , where we have b k, r i ( a )
Revised: + b k, r i ( a ) , where we have b k, r i ( a )

Original: =
Revised: =

Original: (cid:88)
Revised: (cid:88)

Original: f ∈ a i b k,f, r ( a ) , and b k,f, r ( a ) =
Revised: f ∈ a i b k,f, r ( a ) , and b k,f, r ( a ) =

Original: (cid:115) ˜
Revised: (cid:115) ˜

Original: ι
Revised: ι

Original: N k,f ( n f ( a )) ∨ 1 .
Revised: N k,f ( n f ( a )) ∨ 1 .

Original: For proof convenience, we define auxiliary value functions
Revised: For proof convenience, we define auxiliary value functions

Original: [Equation]
Revised: [Equation]

Original: With these definitions, we now begin to prove Theorem 1.
Revised: With these definitions, we now begin to prove Theorem 1.

Original: Proof of Theorem 1.
Revised: Proof of Theorem 1.

Original: Semi-bandit Feedback.
Revised: Semi-bandit Feedback.

Original: By the update rules in Algorithm 1, in the setting of semi-bandit feedback, with probability at least 1 − δ , simultaneously for all ( k, i, a ) ∈
Revised: By the update rules in Algorithm 1, in the setting of semi-bandit feedback, with probability at least 1 − δ , simultaneously for all ( k, i, a ) ∈

Original: [ K ] × [ m ] ×A , we have
Revised: [ K ] × [ m ] ×A , we have

Original: [Equation]
Revised: [Equation]

Original: The second inequality above is obtained by using standard Hoeffding’s inequality and union bound,
Revised: The second inequality above is obtained by using standard Hoeffding’s inequality and union bound,

Original: Therefore, we have Q k i ( a ) ≥ r i ( a ) .
Revised: Therefore, we have Q k i ( a ) ≥ r i ( a ) .

Original: Then, since π k is the ϵ -approximate Nash equilibrium policy of Q k 1 , . . .
Revised: Then, since π k is the ϵ -approximate Nash equilibrium policy of Q k 1 , . . .

Original: , Q km , we have
Revised: , Q km , we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Meanwhile, by definition of Q ki ( a ) and V ki , we can similarly show that Q ki ( a ) ≤ r i ( a ) and
Revised: Meanwhile, by definition of Q ki ( a ) and V ki , we can similarly show that Q ki ( a ) ≤ r i ( a ) and

Original: V ki ≤ V π k i .
Revised: V ki ≤ V π k i .

Original: Therefore, we can have V † ,π k − i
Revised: Therefore, we can have V † ,π k − i

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: Now, we define (cid:101) Q k ( a ) = max i ∈ [ m ] 2 b k, r i ( a ) and (cid:101)
Revised: Now, we define (cid:101) Q k ( a ) = max i ∈ [ m ] 2 b k, r i ( a ) and (cid:101)

Original: V k
Revised: V k

Original: = E a ∼ π k
Revised: = E a ∼ π k

Original: [ (cid:101) Q k ( a )] .
Revised: [ (cid:101) Q k ( a )] .

Original: Then, we can notice that
Revised: Then, we can notice that

Original: [Equation]
Revised: [Equation]

Original: We further define M k = E a ∼ π k (cid:104) (cid:101) Q k ( a ) (cid:105) − (cid:101) Q k ( a k )
Revised: We further define M k = E a ∼ π k (cid:104) (cid:101) Q k ( a ) (cid:105) − (cid:101) Q k ( a k )

Original: = (cid:101) V k − (cid:101) Q k
Revised: = (cid:101) V k − (cid:101) Q k

Original: ( a k ) .
Revised: ( a k ) .

Original: It is not hard to verify that M k is a martingale difference sequence with respect to the history from episode 1 to k − 1 .
Revised: It is not hard to verify that M k is a martingale difference sequence with respect to the history from episode 1 to k − 1 .

Original: Meanwhile, since (cid:12)(cid:12) b k, r ( a ) (cid:12)(cid:12) =
Revised: Meanwhile, since (cid:12)(cid:12) b k, r ( a ) (cid:12)(cid:12) =

Original: (cid:80) f ∈F (cid:113)
Revised: (cid:80) f ∈F (cid:113)

Original: [Equation]
Revised: [Equation]

Original: N k,f
Revised: N k,f

Original: [Equation]
Revised: [Equation]

Original: n f
Revised: n f

Original: [Equation]
Revised: [Equation]

Original: a ))
Revised: a ))

Original: ∨ 1 ≤ F √ ˜ ι .
Revised: ∨ 1 ≤ F √ ˜ ι .

Original: Thus, by Azuma-Hoeffding inequality, we have
Revised: Thus, by Azuma-Hoeffding inequality, we have

Original: [Equation]
Revised: [Equation]

Original: K
Revised: K

Original: [Equation]
Revised: [Equation]

Original: M k = (cid:101)
Revised: M k = (cid:101)

Original: O (cid:16) F √ K (cid:17) .
Revised: O (cid:16) F √ K (cid:17) .

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: (Since the value is always bounded by F .)
Revised: (Since the value is always bounded by F .)

Original: [Equation]
Revised: [Equation]

Original: taking ϵ =
Revised: taking ϵ =

Original: [Equation]
Revised: [Equation]

Original: K
Revised: K

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (By Lemma 6.)
Revised: (By Lemma 6.)

Original: Bandit Feedback.
Revised: Bandit Feedback.

Original: By using Lemma 3, which guarantees optimistic estimation, we can similarly show that
Revised: By using Lemma 3, which guarantees optimistic estimation, we can similarly show that

Original: [Equation]
Revised: [Equation]

Original: ) , F (cid:111)
Revised: ) , F (cid:111)

Original: + Kϵ.
Revised: + Kϵ.

Original: To have an upper bound on M k here, recall that ˜ b k, r ( a ) =
Revised: To have an upper bound on M k here, recall that ˜ b k, r ( a ) =

Original: max i ∈ [ m ] ∥
Revised: max i ∈ [ m ] ∥

Original: A i ( a ) ∥ ( V k ) − 1 (cid:113) ˜
Revised: A i ( a ) ∥ ( V k ) − 1 (cid:113) ˜

Original: β k and (cid:113) ˜ β K
Revised: β k and (cid:113) ˜ β K

Original: =
Revised: =

Original: (cid:101) O (cid:16)(cid:112) ˜
Revised: (cid:101) O (cid:16)(cid:112) ˜

Original: d (cid:17) = (cid:101) O (cid:16) √ mF (cid:17) .
Revised: d (cid:17) = (cid:101) O (cid:16) √ mF (cid:17) .

Original: Meanwhile, we have ∥
Revised: Meanwhile, we have ∥

Original: A i ( a ) ∥ ( V k ) − 1 ≤ ∥
Revised: A i ( a ) ∥ ( V k ) − 1 ≤ ∥

Original: A i ( a ) ∥ I = ∥
Revised: A i ( a ) ∥ I = ∥

Original: A i ( a ) ∥ 2 ≤ √ F .
Revised: A i ( a ) ∥ 2 ≤ √ F .

Original: Thus, we have (cid:12)(cid:12) M k (cid:12)(cid:12) ≤ (cid:101) O (cid:16) √ mF 2 (cid:17) , which by Azuma-Hoeffding inequality implies (cid:80)
Revised: Thus, we have (cid:12)(cid:12) M k (cid:12)(cid:12) ≤ (cid:101) O (cid:16) √ mF 2 (cid:17) , which by Azuma-Hoeffding inequality implies (cid:80)

Original: [Equation]
Revised: [Equation]

Original: Then the sum of the bonus terms can be bounded by using Lemma 4.
Revised: Then the sum of the bonus terms can be bounded by using Lemma 4.

Original: In particular, with ϵ = 1 /K , we have
Revised: In particular, with ϵ = 1 /K , we have

Original: [Equation]
Revised: [Equation]

Original: C.1 Lemmas for Bandit Feedback
Revised: C.1 Lemmas for Bandit Feedback

Original: The following lemma, as a direct corollary of the confidence bound for least square estimators, shows that the reward estimation error can be bounded by the reward bonus term.
Revised: The following lemma, as a direct corollary of the confidence bound for least square estimators, shows that the reward estimation error can be bounded by the reward bonus term.

Original: Lemma 3.
Revised: Lemma 3.

Original: With probability at least 1 − δ , simultaneously for all ( i, k, a ) , it holds that | (˜ r ki − r i )( a ) | ≤ ˜ b k, r ( a ) , where ˜ r ki and ˜
Revised: With probability at least 1 − δ , simultaneously for all ( i, k, a ) , it holds that | (˜ r ki − r i )( a ) | ≤ ˜ b k, r ( a ) , where ˜ r ki and ˜

Original: b k, r are defined in (2) .
Revised: b k, r are defined in (2) .

Original: Proof.
Revised: Proof.

Original: By construction, we have
Revised: By construction, we have

Original: [Equation]
Revised: [Equation]

Original: (By Theorem 20.5 in Lattimore and Szepesvári [2020].)
Revised: (By Theorem 20.5 in Lattimore and Szepesvári [2020].)

Original: Since each element in θ is bounded in [0 , 1] by construction, we have ∥ θ ∥ 2 ≤ (cid:112) ˜ d .
Revised: Since each element in θ is bounded in [0 , 1] by construction, we have ∥ θ ∥ 2 ≤ (cid:112) ˜ d .

Original: Then, by Lemma 4, we have det (cid:0) V k (cid:1) ≤ (cid:16) 1 + mkF ˜ d (cid:17) ˜ d since by construction ∥
Revised: Then, by Lemma 4, we have det (cid:0) V k (cid:1) ≤ (cid:16) 1 + mkF ˜ d (cid:17) ˜ d since by construction ∥

Original: A i ( a ) ∥ 22 ≤ F .
Revised: A i ( a ) ∥ 22 ≤ F .

Original: Finally, to make this bound valid for all player i ∈
Revised: Finally, to make this bound valid for all player i ∈

Original: [ m ] , we only need to take maximization over i ∈ [ m ] .
Revised: [ m ] , we only need to take maximization over i ∈ [ m ] .

Original: Therefore, with probability at least 1 − δ , we have
Revised: Therefore, with probability at least 1 − δ , we have

Original: [Equation]
Revised: [Equation]

Original: where (cid:113) ˜ β
Revised: where (cid:113) ˜ β

Original: [Equation]
Revised: [Equation]

Original: The following is a variant of the famous elliptical potential lemma, which helps bound the sum of reward bonus under bandit feedback.
Revised: The following is a variant of the famous elliptical potential lemma, which helps bound the sum of reward bonus under bandit feedback.

Original: Here, we apply some techniques from the proof of Lemma 19.
Revised: Here, we apply some techniques from the proof of Lemma 19.

Original: in Lattimore and Szepesvári [2020].
Revised: in Lattimore and Szepesvári [2020].

Original: Lemma 4.
Revised: Lemma 4.

Original: Let K, m ≥ 1 be integers.
Revised: Let K, m ≥ 1 be integers.

Original: Suppose V k
Revised: Suppose V k

Original: = I + (cid:80) k − 1 k ′ =1 (cid:80) m i =
Revised: = I + (cid:80) k − 1 k ′ =1 (cid:80) m i =

Original: 1 A k ′ i (cid:16)
Revised: 1 A k ′ i (cid:16)

Original: A k ′
Revised: A k ′

Original: i (cid:17) ⊤ , where
Revised: i (cid:17) ⊤ , where

Original: A k ′ i ∈
Revised: A k ′ i ∈

Original: R
Revised: R

Original: d and (cid:13)(cid:13)(cid:13)
Revised: d and (cid:13)(cid:13)(cid:13)

Original: A k ′ i (cid:13)(cid:13)(cid:13) 2 2 ≤ F .
Revised: A k ′ i (cid:13)(cid:13)(cid:13) 2 2 ≤ F .

Original: Then, it holds that det
Revised: Then, it holds that det

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For the first upper bound about det (cid:0) V k (cid:1) , we have det
Revised: For the first upper bound about det (cid:0) V k (cid:1) , we have det

Original: [Equation]
Revised: [Equation]

Original: are eigenvalues of V k )
Revised: are eigenvalues of V k )

Original: [Equation]
Revised: [Equation]

Original: For the second upper bound.
Revised: For the second upper bound.

Original: First, we notice that min { 1 , x } ≤ 2 log(1 + x ) for any x ≥ 0 .
Revised: First, we notice that min { 1 , x } ≤ 2 log(1 + x ) for any x ≥ 0 .

Original: Thus, we have
Revised: Thus, we have

Original: [Equation]
Revised: [Equation]

Original: Then, for k ≥
Revised: Then, for k ≥

Original: [Equation]
Revised: [Equation]

Original: we can notice that
Revised: we can notice that

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: By Lemma 5.)
Revised: By Lemma 5.)

Original: [Equation]
Revised: [Equation]

Original: (Since by definition, V 1 = I .)
Revised: (Since by definition, V 1 = I .)

Original: As a result
Revised: As a result

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: C.2 Technical Lemmas
Revised: C.2 Technical Lemmas

Original: Lemma 5.
Revised: Lemma 5.

Original: Let y 1 , . . .
Revised: Let y 1 , . . .

Original: y m ∈ R
Revised: y m ∈ R

Original: d be a set of vectors.
Revised: d be a set of vectors.

Original: Then, it holds that
Revised: Then, it holds that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Since I + (cid:80) mi =

Revised: Since I + (cid:80) mi =

Original: y
Revised: y

Original: i
Revised: i

Original: y ⊤ i ⪰ I
Revised: y ⊤ i ⪰ I

Original: + y
Revised: + y

Original: i
Revised: i

Original: y ⊤ i for any i ∈ [ m ] , we have det (cid:0)
Revised: y ⊤ i for any i ∈ [ m ] , we have det (cid:0)

Original: I + (cid:80) mi =

Revised: I + (cid:80) mi =

Original: y i
Revised: y i

Original: y ⊤ i (cid:1) ≥ det (cid:0)
Revised: y ⊤ i (cid:1) ≥ det (cid:0)

Original: I + y
Revised: I + y

Original: i y ⊤ i (cid:1) for any i ∈ [ m ] .
Revised: i y ⊤ i (cid:1) for any i ∈ [ m ] .

Original: That is, we have
Revised: That is, we have

Original: [Equation]
Revised: [Equation]

Original: The last line above holds because the matrix
Revised: The last line above holds because the matrix

Original: I + y i
Revised: I + y i

Original: y ⊤
Revised: y ⊤

Original: i has eigenvalues 1 + ∥ y i ∥ 22 and 1.
Revised: i has eigenvalues 1 + ∥ y i ∥ 22 and 1.

Original: Lemma 6.
Revised: Lemma 6.

Original: For any f ∈ F , it holds that
Revised: For any f ∈ F , it holds that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Here, we have
Revised: Here, we have

Original: K
Revised: K

Original: [Equation]
Revised: [Equation]

Original: The last equality above is based on a pigeon-hold principle argument similar to Lemma 20.
Revised: The last equality above is based on a pigeon-hold principle argument similar to Lemma 20.

Original: D Analysis for AlgorithmD.1 Exploration Distribution and Smoothness
Revised: D Analysis for AlgorithmD.1 Exploration Distribution and Smoothness

Original: We choose the exploration distribution to be the G-optimal design and we have the following properties.
Revised: We choose the exploration distribution to be the G-optimal design and we have the following properties.

Original: Lemma 7.
Revised: Lemma 7.

Original: (Unbiasedness)
Revised: (Unbiasedness)

Original: For any episode k ∈ [ K ] , i ∈ [ m ] and a ∈
Revised: For any episode k ∈ [ K ] , i ∈ [ m ] and a ∈

Original: A i , we have
Revised: A i , we have

Original: [Equation]
Revised: [Equation]

Original: where E k [ · ] is taken over all the randomness before episode k .
Revised: where E k [ · ] is taken over all the randomness before episode k .

Original: Proof.
Revised: Proof.

Original: By the definition of (cid:98) ∇ ki Φ( a ) , we have
Revised: By the definition of (cid:98) ∇ ki Φ( a ) , we have

Original: [Equation]
Revised: [Equation]

Original: only depends on π ki and θ k, 1 i ( π k ) only depends on π k − i )
Revised: only depends on π ki and θ k, 1 i ( π k ) only depends on π k − i )

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Lemma 8.
Revised: Lemma 8.

Original: For any episode k ∈ [ K ] , i ∈ [ m ] and a ∈
Revised: For any episode k ∈ [ K ] , i ∈ [ m ] and a ∈

Original: A i , we have
Revised: A i , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: As
Revised: As

Original: [Equation]
Revised: [Equation]

Original: have
Revised: have

Original: [Equation]
Revised: [Equation]

Original: and ρ i is the G-optimal design with respect to ϕ i ( · ) , for any action
Revised: and ρ i is the G-optimal design with respect to ϕ i ( · ) , for any action

Original: a ∈
Revised: a ∈

Original: A i we have
Revised: A i we have

Original: [Equation]
Revised: [Equation]

Original: Then for any t ∈ [ τ ]
Revised: Then for any t ∈ [ τ ]

Original: [Equation]
Revised: [Equation]

Original: since | r k
Revised: since | r k

Original: [Equation]
Revised: [Equation]

Original: ti | ≤ F
Revised: ti | ≤ F

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: As a result
Revised: As a result

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Lemma 9.
Revised: Lemma 9.

Original: For any episode k ∈ [ K ] , i ∈ [ m ] and a ∈
Revised: For any episode k ∈ [ K ] , i ∈ [ m ] and a ∈

Original: A i , we have
Revised: A i , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We first show that for any t ∈ [ τ ] , we have
Revised: We first show that for any t ∈ [ τ ] , we have

Original: [Equation]
Revised: [Equation]

Original: Lemma 10.
Revised: Lemma 10.

Original: With probability 1 − δ , for all k ∈ [ K ] , i ∈ [ m ] and a ∈ A i , we have (cid:12)(cid:12) (cid:12) (cid:98) ∇ k
Revised: With probability 1 − δ , for all k ∈ [ K ] , i ∈ [ m ] and a ∈ A i , we have (cid:12)(cid:12) (cid:12) (cid:98) ∇ k

Original: i Φ( a ) − ∇ k i Φ( a ) (cid:12)(cid:12)
Revised: i Φ( a ) − ∇ k i Φ( a ) (cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: ≤ c
Revised: ≤ c

Original: [Equation]
Revised: [Equation]

Original: F
Revised: F

Original: [Equation]
Revised: [Equation]

Original: ) γτ
Revised: ) γτ

Original: Proof.
Revised: Proof.

Original: Recall that
Revised: Recall that

Original: [Equation]
Revised: [Equation]

Original: and ( a k,ti , r k,ti ) are drawn independently at each t ∈ [ τ ] .
Revised: and ( a k,ti , r k,ti ) are drawn independently at each t ∈ [ τ ] .

Original: Lemma 7 shows that (cid:98) ∇ ki Φ( a i ) is an unbiased estimate of ∇ ki Φ( a i )
Revised: Lemma 7 shows that (cid:98) ∇ ki Φ( a i ) is an unbiased estimate of ∇ ki Φ( a i )

Original: In addition, Lemma 8 shows that ϕ ⊤ i ( a i )[ C ki ] − 1 r k,
Revised: In addition, Lemma 8 shows that ϕ ⊤ i ( a i )[ C ki ] − 1 r k,

Original: ti ϕ i
Revised: ti ϕ i

Original: ( a k,ti ) is bounded by
Revised: ( a k,ti ) is bounded by

Original: F 2 /γ and Lemma 9 shows that its second moment is bounded by F 3 /γ .
Revised: F 2 /γ and Lemma 9 shows that its second moment is bounded by F 3 /γ .

Original: Then by Bernstein’s inequality, for a fixed k ∈ [ K ] , i ∈ [ m ] and a ∈
Revised: Then by Bernstein’s inequality, for a fixed k ∈ [ K ] , i ∈ [ m ] and a ∈

Original: A i , with probability 1 − δ , we have
Revised: A i , with probability 1 − δ , we have

Original: [Equation]
Revised: [Equation]

Original: The argument holds by applying the union bound and the fact that |A i | ≤ 2 F .
Revised: The argument holds by applying the union bound and the fact that |A i | ≤ 2 F .

Original: Lemma 11.
Revised: Lemma 11.

Original: Φ( · ) is mF -Lipschitz and mF -smooth with respect to the L1 norm
Revised: Φ( · ) is mF -Lipschitz and mF -smooth with respect to the L1 norm

Original: ∥ · ∥ 1 .
Revised: ∥ · ∥ 1 .

Original: Proof.
Revised: Proof.

Original: Recall that Φ( π ) =
Revised: Recall that Φ( π ) =

Original: E a ∼ π Φ( a ) and Φ( a ) ∈ [0 , mF ] .
Revised: E a ∼ π Φ( a ) and Φ( a ) ∈ [0 , mF ] .

Original: [Equation]
Revised: [Equation]

Original: Similarly we have ∇ π Φ( a i ) =
Revised: Similarly we have ∇ π Φ( a i ) =

Original: E a − i ∼ π − i Φ( a i , a − i ) .
Revised: E a − i ∼ π − i Φ( a i , a − i ) .

Original: As a result, we have
Revised: As a result, we have

Original: [Equation]
Revised: [Equation]

Original: Definition 4.
Revised: Definition 4.

Original: (Frank Wolfe Gap)
Revised: (Frank Wolfe Gap)

Original: The Frank Wolfe gap of a joint strategy π for Φ( · ) is defined as
Revised: The Frank Wolfe gap of a joint strategy π for Φ( · ) is defined as

Original: [Equation]
Revised: [Equation]

Original: Lemma 12.
Revised: Lemma 12.

Original: Suppose the Frank Wolfe gap of π is ϵ .
Revised: Suppose the Frank Wolfe gap of π is ϵ .

Original: Then π is an ϵ -Nash policy.
Revised: Then π is an ϵ -Nash policy.

Original: Proof.
Revised: Proof.

Original: For a fixed player i , suppose player i change her strategy to π ′ i .
Revised: For a fixed player i , suppose player i change her strategy to π ′ i .

Original: [Equation]
Revised: [Equation]

Original: D.2 Analysis for Frank Wolfe in Bandit Feedback
Revised: D.2 Analysis for Frank Wolfe in Bandit Feedback

Original: Theorem 4.
Revised: Theorem 4.

Original: Let T =
Revised: Let T =

Original: Kτ .
Revised: Kτ .

Original: For the congestion game with bandit feedback, by running Algorithmwith gradient estimator (cid:98) ∇ ki Φ  and exploration distribution ρ idefined in(4) and (5) , setting parameters ν=
Revised: For the congestion game with bandit feedback, by running Algorithm 3 with gradient estimator (cid:98) ∇ ki Φ in (4) and exploration distribution ρ i in (5) , setting parameters ν

Original:  Fm √ K , γ = FmK and τ = K 2 ,  with probability 1 − δ , we have
Revised: = Fm √ K , γ = FmK and τ = K 2 , if K ≥ 2 Fm , then with probability 1 − δ , we have

Original: Nash-Regret ( T
Revised: Nash

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Set ∇ k
Revised: Set ∇ k

Original: Φ = ∇
Revised: Φ = ∇

Original: Φ(Π k ) ∈
Revised: Φ(Π k ) ∈

Original: R A and ∇ ki Φ = ∇ k Φ( π i )
Revised: R A and ∇ ki Φ = ∇ k Φ( π i )

Original: ∈
Revised: ∈

Original: R A i .
Revised: R A i .

Original: As we have Φ( · ) is mF -smooth w.r.t.
Revised: As we have Φ( · ) is mF -smooth w.r.t.

Original: ∥ · ∥ 1 , we have
Revised: ∥ · ∥ 1 , we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Define the true target policy at episode k
Revised: Define the true target policy at episode k

Original: [Equation]
Revised: [Equation]

Original: and the Frank Wolfe gap of joint strategy π
Revised: and the Frank Wolfe gap of joint strategy π

Original: [Equation]
Revised: [Equation]

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: γτ
Revised: γτ

Original: Apply it to the previous bound
Revised: Apply it to the previous bound

Original: and we have
Revised: and we have

Original: [Equation]
Revised: [Equation]

Original: Summing over k ∈ [ K ] and we get
Revised: Summing over k ∈ [ K ] and we get

Original: K (cid:88)
Revised: K (cid:88)

Original: [Equation]

Original: =

Original: [Equation]
Revised: [Equation]

Original: Since Φ( · ) is bounded in [0, mF].

Original: Set ν
Revised: Set ν

Original: =
Revised: =

Original: Fm √ K , γ = FmK , τ = K 2, and we have{+notice that when K ≥ 
+}
Revised: Fm √ K , γ = FmK , τ = K 2 and {+notice that when K ≥ 
+}

Revised: Fm , we have 1 − γ ≥ 12 .

Revised: Since Φ( · ) is bounded in [0 , mF ] , we can have

Original: [Equation]
Revised: [Equation]

Original: Then by Lemma 12, for T =
Revised: Then by Lemma 12, for T =

Original: Kτ , we have
Revised: Kτ , we have

Revised: Nash-Regret ( T

Original: [Equation]
Revised: [Equation]

Original: D.3 Algorithm and Analysis for Semi-bandit Feedback
Revised: D.3 Algorithm and Analysis for Semi-bandit Feedback

Original: In the setting of semi-bandit feedback, we will need a different gradient estimator (cid:101) ∇ ki Φ( a i ) and a different exploration distribution ˜ ρ i to utilize the extra reward information from each chosen facility.
Revised: In the setting of semi-bandit feedback, we will need a different gradient estimator (cid:101) ∇ ki Φ( a i ) and a different exploration distribution ˜ ρ i to utilize the extra reward information from each chosen facility.

Original: Based on the analysis in Section 5, using (3), we have ∇ ki Φ( a i ) =
Revised: Based on the analysis in Section 5, using (3), we have ∇ ki Φ( a i ) =

Original: (cid:80) f ∈ a i
Revised: (cid:80) f ∈ a i

Original: [ θ i ( π k )] f , where
Revised: [ θ i ( π k )] f , where

Original: [ θ i ( π k )] f = E a − i ∼ π k − i (cid:2) r f ( n f ( a − i ) + 1) (cid:3) .
Revised: [ θ i ( π k )] f = E a − i ∼ π k − i (cid:2) r f ( n f ( a − i ) + 1) (cid:3) .

Original: Meanwhile, in semi-bandit feedback, the mean of tth reward player i received for facility f at episode k is r f ( n f ( a k,ti , a k,t − i )) .
Revised: Meanwhile, in semi-bandit feedback, the mean of tth reward player i received for facility f at episode k is r f ( n f ( a k,ti , a k,t − i )) .

Original: Therefore, we can use inverse propensity score (IPS) estimator to estimate [ θ i ( π k )] f .
Revised: Therefore, we can use inverse propensity score (IPS) estimator to estimate [ θ i ( π k )] f .

Original: In particular, we have
Revised: In particular, we have

Original: [Equation]
Revised: [Equation]

Original: where [ (cid:101) θ k,
Revised: where [ (cid:101) θ k,

Original: [Equation]
Revised: [Equation]

Original: Then, we can naturally have
Revised: Then, we can naturally have

Original: [Equation]
Revised: [Equation]

Original: Furthermore, by Lemma 14, we can see that by using ˜ ρ i computed by Algorithm 4, for all players, we have P a i ∼ π ki ( f ∈ a i ) ≥ γ 2 F for all f ∈ (cid:83) a i ∈A i a i .
Revised: Furthermore, by Lemma 14, we can see that by using ˜ ρ i computed by Algorithm 4, for all players, we have P a i ∼ π ki ( f ∈ a i ) ≥ γ 2 F for all f ∈ (cid:83) a i ∈A i a i .

Original: Properties of the IPS estimator are summarized in Lemma 15.
Revised: Properties of the IPS estimator are summarized in Lemma 15.

Original: By using these properties, we can have the following lemma.
Revised: By using these properties, we can have the following lemma.

Original: Lemma 13.
Revised: Lemma 13.

Original: With probability 1 − δ , for all k ∈ [ K ] , i ∈ [ m ] and a i ∈
Revised: With probability 1 − δ , for all k ∈ [ K ] , i ∈ [ m ] and a i ∈

Original: A i , we have
Revised: A i , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By Lemma 15 and Bernstein’s inequality, simultaneously for all ( i, k, f ) ∈ [ m ] × [ K ] × F , with probability at least 1 − δ , we have
Revised: By Lemma 15 and Bernstein’s inequality, simultaneously for all ( i, k, f ) ∈ [ m ] × [ K ] × F , with probability at least 1 − δ , we have

Original: [Equation]
Revised: [Equation]

Original: Since (cid:101) ∇ ki Φ( a i ) =
Revised: Since (cid:101) ∇ ki Φ( a i ) =

Original: (cid:80) f ∈ a i
Revised: (cid:80) f ∈ a i

Original: [ (cid:101) θ ki ( π k )]
Revised: [ (cid:101) θ ki ( π k )]

Original: f , by triangle inequality, we have
Revised: f , by triangle inequality, we have

Original: [Equation]
Revised: [Equation]

Original: With this more refined gradient estimator, we can now have the following theorem.
Revised: With this more refined gradient estimator, we can now have the following theorem.

Original: Theorem 5.
Revised: Theorem 5.

Original: Let T =
Revised: Let T =

Original: Kτ .
Revised: Kτ .

Original: For the congestion game with semi-bandit feedback, by running Algorithm 3 with gradient estimator (cid:101) ∇ ki Φ in (6) and exploration distribution ˜ϕ
Revised: For the congestion game with semi-bandit feedback, by running Algorithm 3 with gradient estimator (cid:101) ∇ ki Φ in (6) and exploration distribution ˜

Original:  i in Algorithm 4, setting parameters ν = √ F m √ K , γ = √ FmK and τ = K 2 ,  with probability 1 − δ , we have
Revised: ρ i in Algorithm 4, setting parameters ν = √ F m √ K , γ = √ FmK and τ = K 2 , if K ≥ 2 √ Fm , then with probability 1 − δ , we have

Original: Nash-Regret ( T
Revised: Nash-Regret ( T

Original: [Equation]
Revised: [Equation]

Original: = 
Revised: = τ

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By following the proof of Theorem 4 and applying the concentration inequality in Lemma
Revised: By following the proof of Theorem 4 and applying the concentration inequality in Lemma

Original: [Equation]
Revised: [Equation]

Original: we can have
Revised: we can have

Original: [Equation]
Revised: [Equation]

Original: Summing over k ∈ [ K ] and we get
Revised: Summing over k ∈ [ K ] and we get

Original: K (cid:88)

Original: [Equation]
Revised: [Equation]

Original: Set ν = √ F m √ K , γ = √ FmK , τ = K 2 and  we can have
Revised: Set ν = √ F m √ K , γ = √ FmK , τ = K 2 and notice that when K ≥ 2 √ Fm , we have 1 − γ ≥ 12 . Thus, we can have

Original: [Equation]
Revised: [Equation]

Original: Then by Lemma 12, for T =
Revised: Then by Lemma 12, for T =

Original: Kτ , we have
Revised: Kτ , we have

Original: [Equation]

Original: Regret (

Original: [Equation]
Revised: [Equation]

Original: D.4 Lemmas for Semi-bandit Feedback
Revised: D.4 Lemmas for Semi-bandit Feedback

Original:  1: Input: A i , player i -th action set 2: Initialize (cid:101)
Revised: Algorithm 1: Input: A i , player i -th action set 2: Initialize (cid:101)

Original: A i ← ∅ 3: for a i in A i do 4: if ∃ f ∈ a i such that f / ∈ (cid:83) a ′ i ∈ (cid:101)
Revised: A i ← ∅ 3: for a i in A i do 4: if ∃ f ∈ a i such that f / ∈ (cid:83) a ′ i ∈ (cid:101)

Original: A i a ′ i
Revised: A i a ′ i

Original: then 5: (cid:101)
Revised: then 5: (cid:101)

Original: A i ← (cid:101)
Revised: A i ← (cid:101)

Original: A i ∪ { a i } 6: if F i =
Revised: A i ∪ { a i } 6: if F i =

Original: (cid:83) a ′ i ∈ (cid:101)
Revised: (cid:83) a ′ i ∈ (cid:101)

Original: A i a ′ i
Revised: A i a ′ i

Original: then 7: break 8: Assign ˜ ρ i ( a i ) ← 12 F for each a i ∈ (cid:101)
Revised: then 7: break 8: Assign ˜ ρ i ( a i ) ← 12 F for each a i ∈ (cid:101)

Original: A i 9: Assign remaining probability mass arbitrarily to actions in A \ (cid:101)
Revised: A i 9: Assign remaining probability mass arbitrarily to actions in A \ (cid:101)

Original: A i 10: return ˜ ρ i
Revised: A i 10: return ˜ ρ i

Original: Lemma 14.
Revised: Lemma 14.

Original: Let F i
Revised: Let F i

Original: =
Revised: =

Original: (cid:83) a i ∈A i a i .
Revised: (cid:83) a i ∈A i a i .

Original: For any player i , if ˜ ρ i is the output of Algorithm 4 and π ki contains a mixture of ˜
Revised: For any player i , if ˜ ρ i is the output of Algorithm 4 and π ki contains a mixture of ˜

Original: ρ i with weight γ , then we have P a i ∼ π ki ( f ∈ a i ) ≥ γ 2 F for any f ∈ F i .
Revised: ρ i with weight γ , then we have P a i ∼ π ki ( f ∈ a i ) ≥ γ 2 F for any f ∈ F i .

Original: Proof.
Revised: Proof.

Original: By Algorithm 4, whenever a new action is added into (cid:101)
Revised: By Algorithm 4, whenever a new action is added into (cid:101)

Original: A i , it contains facility not appeared in current (cid:101)
Revised: A i , it contains facility not appeared in current (cid:101)

Original: A i .
Revised: A i .

Original: Then, since there are at most |F i | ≤ F distinct facilities in the action set A i , the final (cid:101)
Revised: Then, since there are at most |F i | ≤ F distinct facilities in the action set A i , the final (cid:101)

Original: A i must satisfy | (cid:101)
Revised: A i must satisfy | (cid:101)

Original: A i | ≤ F .
Revised: A i | ≤ F .

Original: Therefore, ˜ ρ i is a valid distribution over A i .
Revised: Therefore, ˜ ρ i is a valid distribution over A i .

Original: Since π ki contains a mixture of ˜ ρ i with weight γ , for any a i ∈
Revised: Since π ki contains a mixture of ˜ ρ i with weight γ , for any a i ∈

Original: A i , we have π ki ( a i ) ≥ γ ˜
Revised: A i , we have π ki ( a i ) ≥ γ ˜

Original: ρ
Revised: ρ

Original: i ( a i ) .
Revised: i ( a i ) .

Original: Thus, we have
Revised: Thus, we have

Original: [Equation]
Revised: [Equation]

Original: The last inequality above holds since by construction, (cid:101)
Revised: The last inequality above holds since by construction, (cid:101)

Original: A i contains all facilities contained in A i .
Revised: A i contains all facilities contained in A i .

Original: Lemma 15.
Revised: Lemma 15.

Original: If π ki contains a mixture of ˜ ρ i given in Algorithm 4 with weight γ .
Revised: If π ki contains a mixture of ˜ ρ i given in Algorithm 4 with weight γ .

Original: Then, the IPS estimator [ (cid:101) θ ki ( π k )]
Revised: Then, the IPS estimator [ (cid:101) θ ki ( π k )]

Original: f satisfies
Revised: f satisfies

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For the first property, since E k (cid:2) r k,t,f | a k,t (cid:3) =
Revised: For the first property, since E k (cid:2) r k,t,f | a k,t (cid:3) =

Original: r f ( n f ( a k,ti , a k,t − i )) and a k,t ∼ π k , We have
Revised: r f ( n f ( a k,ti , a k,t − i )) and a k,t ∼ π k , We have

Original: [Equation]
Revised: [Equation]

Original: =[
Revised: =[

Original: θ i ( π k )] f .
Revised: θ i ( π k )] f .

Original: The equality (i) above holds because E a i ∼ π ki (cid:2)
Revised: The equality (i) above holds because E a i ∼ π ki (cid:2)

Original: r f ( n f ( a i , a − i )) | a − i , f ∈ a i (cid:3) =
Revised: r f ( n f ( a i , a − i )) | a − i , f ∈ a i (cid:3) =

Original: r f ( n f ( a − i ) + 1) and f ∈ a
Revised: r f ( n f ( a − i ) + 1) and f ∈ a

Original: i does not depend on a − i .
Revised: i does not depend on a − i .

Original: For the second property, since P a i ∼ π ki ( f ∈ a i ) ≥ γ 2 F by Lemma 14 and r k,t,f ∈ [0 , 1] , we can immediately have | [ (cid:101) θ k,ti ( π k )]
Revised: For the second property, since P a i ∼ π ki ( f ∈ a i ) ≥ γ 2 F by Lemma 14 and r k,t,f ∈ [0 , 1] , we can immediately have | [ (cid:101) θ k,ti ( π k )]

Original: f | ≤ 2 Fγ .
Revised: f | ≤ 2 Fγ .

Original: For the third property, we have
Revised: For the third property, we have

Original: [Equation]
Revised: [Equation]

Original: E Algorithms for Independent Markov Congestion Games
Revised: E Algorithms for Independent Markov Congestion Games

Original: In this section, present missing details of our centralized algorithm for independent Markov congestion games, which is summarized in Algorithm 5.
Revised: In this section, present missing details of our centralized algorithm for independent Markov congestion games, which is summarized in Algorithm 5.

Original: The proof of its theoretical guarantee is given in
Revised: The proof of its theoretical guarantee is given in

Original: Appendix F.
Revised: Appendix F.

Original: E.1 Algorithm for Semi-bandit Feedback
Revised: E.1 Algorithm for Semi-bandit Feedback

Original: Under the semi-bandit feedback, the players can receive reward information from all facilities they choose.
Revised: Under the semi-bandit feedback, the players can receive reward information from all facilities they choose.

Original: Therefore, we can similarly define
Revised: Therefore, we can similarly define

Original: [Equation]
Revised: [Equation]

Original: 1: Input: ϵ , accuracy parameter for Nash equilibrium computation 2: Initialize: V k H +1 ,i ( s ) = 0 for all ( i, k, s ) ∈ [ m ] ×
Revised: 1: Input: ϵ , accuracy parameter for Nash equilibrium computation 2: Initialize: V k H +1 ,i ( s ) = 0 for all ( i, k, s ) ∈ [ m ] ×

Original: [ K ] × S 3: for episode k = 1 , . .
Revised: [ K ] × S 3: for episode k = 1 , . .

Original: , K do 4: for step h
Revised: , K do 4: for step h

Original: = H, H − 1 , . .
Revised: = H, H − 1 , . .

Original: , 1 do 5: for player
Revised: , 1 do 5: for player

Original: i = 1 , . .
Revised: i = 1 , . .

Original: , m do 6:
Revised: , m do 6:

Original: Q k h,i ( s, a ) ← min (cid:110) (
Revised: Q k h,i ( s, a ) ← min (cid:110) (

Original: ˆ r kh,i + (cid:98) P kh V k h
Revised: ˆ r kh,i + (cid:98) P kh V k h

Original: +1 ,i + b kh )
Revised: +1 ,i + b kh )

Original: ( s, a ) , HF (cid:111) for all ( s, a ) ∈ S ×
Revised: ( s, a ) , HF (cid:111) for all ( s, a ) ∈ S ×

Original: A 7: for s ∈ S do 8: π kh ( · | s ) ← ϵ -N ASH ( Q k h, 1 ( s, · ) , · · · , Q k h,m ( s, · )) 9: for player i = 1 , . . .
Revised: A 7: for s ∈ S do 8: π kh ( · | s ) ← ϵ -N ASH ( Q k h, 1 ( s, · ) , · · · , Q k h,m ( s, · )) 9: for player i = 1 , . . .

Original: , m do 10:
Revised: , m do 10:

Original: V k h,i ( s ) ← E a ∼ π kh
Revised: V k h,i ( s ) ← E a ∼ π kh

Original: [ Q k h,i ( s, a )] 11: for step h
Revised: [ Q k h,i ( s, a )] 11: for step h

Original: = 1 , . .
Revised: = 1 , . .

Original: , H do 12:
Revised: , H do 12:

Original: Take action a kh ∼ π kh ( · | s kh ) , observe reward r k,fh and next state s kh + 13:
Revised: Take action a kh ∼ π kh ( · | s kh ) , observe reward r k,fh and next state s kh + 13:

Original: Update reward estimator ˆ r kh,i , transition estimator (cid:98) P kh and bonus term b kh ˆ r k,fh ( s f , n ) = (cid:80)
Revised: Update reward estimator ˆ r kh,i , transition estimator (cid:98) P kh and bonus term b kh ˆ r k,fh ( s f , n ) = (cid:80)

Original: k k ′ =1 r k ′ ,
Revised: k k ′ =1 r k ′ ,

Original: f h 1 (cid:110) ( s k ′ ,f h , n f ( a k ′ h ))
Revised: f h 1 (cid:110) ( s k ′ ,f h , n f ( a k ′ h ))

Original: =
Revised: =

Original: ( s f , n ) (cid:111)
Revised: ( s f , n ) (cid:111)

Original: N k,fh ( s f , n ) ∨, (cid:98) P k,fh ( s ′ f | s f , n ) =
Revised: N k,fh ( s f , n ) ∨, (cid:98) P k,fh ( s ′ f | s f , n ) =

Original: (cid:80) kk ′
Revised: (cid:80) kk ′

Original: =1 1 (cid:110) ( s k ′ ,f h +1 , s k ′ ,f h , n f
Revised: =1 1 (cid:110) ( s k ′ ,f h +1 , s k ′ ,f h , n f

Original: [Equation]
Revised: [Equation]

Original: Then, the estimators for the reward function and transition kernel can be defined as
Revised: Then, the estimators for the reward function and transition kernel can be defined as

Original: [Equation]
Revised: [Equation]

Original: Then, with ι = 2 log(4( m + 1)( (cid:80) f ∈F S f ) T/δ ) , we define the bonus term to be b kh ( s, a ) =
Revised: Then, with ι = 2 log(4( m + 1)( (cid:80) f ∈F S f ) T/δ ) , we define the bonus term to be b kh ( s, a ) =

Original: b k, pv h ( s, a ) + b k, r h ( s, a ) , which is a sum of transition bonus and reward bonus.
Revised: b k, pv h ( s, a ) + b k, r h ( s, a ) , which is a sum of transition bonus and reward bonus.

Original: In particular, we have b k,
Revised: In particular, we have b k,

Original: pv h
Revised: pv h

Original: [Equation]
Revised: [Equation]

Original: For convenience, we define ( (cid:98) P kh V )
Revised: For convenience, we define ( (cid:98) P kh V )

Original: ( s, a ) =
Revised: ( s, a ) =

Original: E s ′ ∼ (cid:98) P kh
Revised: E s ′ ∼ (cid:98) P kh

Original: ·| s, a ) [ V ( s ′ )] with value function V : S (cid:55)→ R .
Revised: ·| s, a ) [ V ( s ′ )] with value function V : S (cid:55)→ R .

Original: Remark 5 .
Revised: Remark 5 .

Original: Unlike Algorithm 1 for congestion game, here, Q k h, 1 ( s, · ) , . . .
Revised: Unlike Algorithm 1 for congestion game, here, Q k h, 1 ( s, · ) , . . .

Original: , Q k h,m ( s, · ) in line 6 of
Revised: , Q k h,m ( s, · ) in line 6 of

Original: Algorithm 5 in general does not form a potential game.
Revised: Algorithm 5 in general does not form a potential game.

Original: Therefore, we cannot use Algorithm 2 and ϵ -N ASH is not always computationally efficient.
Revised: Therefore, we cannot use Algorithm 2 and ϵ -N ASH is not always computationally efficient.

Original: E.2 Algorithm for Bandit Feedback
Revised: E.2 Algorithm for Bandit Feedback

Original: In bandit feedback scenario, since players’ observation about state transitions remains unaffected, we only need to modify the reward estimator ˆ r kh,i defined in (7) and reward bonus term b k, r h ( s, a ) defined in (9).
Revised: In bandit feedback scenario, since players’ observation about state transitions remains unaffected, we only need to modify the reward estimator ˆ r kh,i defined in (7) and reward bonus term b k, r h ( s, a ) defined in (9).

Original: Similar to the congestion game with bandit feedback introduced in Section 4.2, for IMCGs, we can also write its reward function as r h,i ( s, a ) =
Revised: Similar to the congestion game with bandit feedback introduced in Section 4.2, for IMCGs, we can also write its reward function as r h,i ( s, a ) =

Original: ⟨ A i ( s, a ) , θ h ⟩ , where θ h is unknown and A i ( s, a ) is a 0-1 vector.
Revised: ⟨ A i ( s, a ) , θ h ⟩ , where θ h is unknown and A i ( s, a ) is a 0-1 vector.

Original: In particular, define θ h ∈ [0 , 1] d with d = m (cid:80) f
Revised: In particular, define θ h ∈ [0 , 1] d with d = m (cid:80) f

Original: ∈F S f to be the vector such that θ h,i =
Revised: ∈F S f to be the vector such that θ h,i =

Original: r fh ( s f , n ) for some f ∈ F and ( s f , n ) ∈ S f × [ m ] .
Revised: r fh ( s f , n ) for some f ∈ F and ( s f , n ) ∈ S f × [ m ] .

Original: Then, we can similarly build estimator ˆ r kh,i through ridge regression as the following.
Revised: Then, we can similarly build estimator ˆ r kh,i through ridge regression as the following.

Original: design matrix: V kh
Revised: design matrix: V kh

Original: =
Revised: =

Original: I + k − 1 (cid:88) k ′ =
Revised: I + k − 1 (cid:88) k ′ =

Original: m (cid:88) i =
Revised: m (cid:88) i =

Original: A i ( s k ′ h , a k ′ h )
Revised: A i ( s k ′ h , a k ′ h )

Original: A i ( s k ′ h , a k ′ h ) ⊤ , (10) θ h estimator: (cid:98) θ
Revised: A i ( s k ′ h , a k ′ h ) ⊤ , (10) θ h estimator: (cid:98) θ

Original: kh = (cid:0) V kh (cid:1) − 1 k − 1 (cid:88)
Revised: kh = (cid:0) V kh (cid:1) − 1 k − 1 (cid:88)

Original: k ′ =
Revised: k ′ =

Original: m (cid:88) i =
Revised: m (cid:88) i =

Original: A i ( s k ′ h , a k ′ h ) r k ′ h,i , (11) reward estimator: ˜ r kh,i (
Revised: A i ( s k ′ h , a k ′ h ) r k ′ h,i , (11) reward estimator: ˜ r kh,i (

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: = (cid:68)
Revised: = (cid:68)

Original: [Equation]
Revised: [Equation]

Original: ( s,
Revised: ( s,

Original: [Equation]
Revised: [Equation]

Original: ) , (cid:98) θ kh (cid:69) , (12) reward bonus: ˜ b k, r h ( s, a ) = max i ∈ [ m ] ∥
Revised: ) , (cid:98) θ kh (cid:69) , (12) reward bonus: ˜ b k, r h ( s, a ) = max i ∈ [ m ] ∥

Original: A i ( s, a ) ∥ ( V kh ) − 1 (cid:112) β k , (13) where √ β k = √ d
Revised: A i ( s, a ) ∥ ( V kh ) − 1 (cid:112) β k , (13) where √ β k = √ d

Original: + (cid:113) d log (cid:0) 1 + mkFd (cid:1) + ι .
Revised: + (cid:113) d log (cid:0) 1 + mkFd (cid:1) + ι .

Original: F Analysis for AlgorithmF.1 Bellman Equations for Genera-sum Markov Games
Revised: F Analysis for AlgorithmF.1 Bellman Equations for Genera-sum Markov Games

Original: Before analyzing Algorithm 5, we first give a brief review of the Bellman equations for general-sum
Revised: Before analyzing Algorithm 5, we first give a brief review of the Bellman equations for general-sum

Original: Markov games.
Revised: Markov games.

Original: These equations are well-known among the literature Bai and Jin [2020], Liu et al.
Revised: These equations are well-known among the literature Bai and Jin [2020], Liu et al.

Original: [2021], Jin et al.
Revised: [2021], Jin et al.

Original: [2021b].
Revised: [2021b].

Original: Fixed policies.
Revised: Fixed policies.

Original: Given a fixed policy π , for any ( h, i, s, a ) ∈
Revised: Given a fixed policy π , for any ( h, i, s, a ) ∈

Original: [ H ] × [ m ] × S × A , it holds that
Revised: [ H ] × [ m ] × S × A , it holds that

Original: [Equation]
Revised: [Equation]

Original: (14) where V πH +1 ,i ( s ) = 0 for any ( i, s ) ∈ [ m ] × S .
Revised: (14) where V πH +1 ,i ( s ) = 0 for any ( i, s ) ∈ [ m ] × S .

Original: Best responses.
Revised: Best responses.

Original: Given a fixed policy π , define the best response value functions for player i as
Revised: Given a fixed policy π , define the best response value functions for player i as

Original: Q † ,π − i h,i ( s, a ) = max π i ∈ ∆
Revised: Q † ,π − i h,i ( s, a ) = max π i ∈ ∆

Original: ( A i ) Q π i ,π − i h,i ( s, a ) and V † ,π − i h,i ( s ) = max π i ∈ ∆( A i )
Revised: ( A i ) Q π i ,π − i h,i ( s, a ) and V † ,π − i h,i ( s ) = max π i ∈ ∆( A i )

Original: V π i ,π − i h,i ( s ) .
Revised: V π i ,π − i h,i ( s ) .

Original: Then, for any ( h, i, s, a ) ∈
Revised: Then, for any ( h, i, s, a ) ∈

Original: [ H ] × [ m ] × S × A , it holds that
Revised: [ H ] × [ m ] × S × A , it holds that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: i h,i
Revised: i h,i

Original: [Equation]
Revised: [Equation]

Original: where V † ,π − i H
Revised: where V † ,π − i H

Original: +1 ,i ( s )
Revised: +1 ,i ( s )

Original: = 

Revised: =

Original: for any ( i, s ) ∈ [ m ] × S .
Revised: for any ( i, s ) ∈ [ m ] × S .

Original: F.2 Proof of Theorem
Revised: F.2 Proof of Theorem

Original: Recall that the update rule in Algorithm 5 is
Revised: Recall that the update rule in Algorithm 5 is

Original: [Equation]
Revised: [Equation]

Original: Similar to the proof of Theorem 1, we define auxiliary value functions
Revised: Similar to the proof of Theorem 1, we define auxiliary value functions

Original: Q
Revised: Q

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We now begin to prove the first part of Theorem 3.
Revised: We now begin to prove the first part of Theorem 3.

Original: Proof of Theorem 3.
Revised: Proof of Theorem 3.

Original: Step 1.
Revised: Step 1.

Original: We first consider the setting of semi-bandit feedback.
Revised: We first consider the setting of semi-bandit feedback.

Original: Assume the result in Lemma 17 holds since it is a high-probability event.
Revised: Assume the result in Lemma 17 holds since it is a high-probability event.

Original: Then, for any ( k, s ) ∈
Revised: Then, for any ( k, s ) ∈

Original: [ K ] × S , it holds that
Revised: [ K ] × S , it holds that

Original: [Equation]
Revised: [Equation]

Original: By the update rules in Algorithm 5, we can notice the following recursive relations
Revised: By the update rules in Algorithm 5, we can notice the following recursive relations

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Thus, we define (cid:101) V
Revised: Thus, we define (cid:101) V

Original: kH +

Revised: kH +

Original: ( s )
Revised: ( s )

Original: = 

Revised: =

Original: for any s ∈ S and (cid:101) Q kh ,
Revised: for any s ∈ S and (cid:101) Q kh ,

Original: [Equation]
Revised: [Equation]

Original: V kh recursively as
Revised: V kh recursively as

Original: [Equation]
Revised: [Equation]

Original: Obviously, we have max i ∈ [ m ] ( V k h,i − V kh,i )( s ) ≤ (cid:101)
Revised: Obviously, we have max i ∈ [ m ] ( V k h,i − V kh,i )( s ) ≤ (cid:101)

Original: V kH +1 .
Revised: V kH +1 .

Original: Then, by inductively assuming the same relation holds for h + 1 , we can have
Revised: Then, by inductively assuming the same relation holds for h + 1 , we can have

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: by induction
Revised: by induction

Original: [Equation]
Revised: [Equation]

Original: for any h ∈ [ H ]
Revised: for any h ∈ [ H ]

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: As a result
Revised: As a result

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Step 2, Semi-bandit Feedback.
Revised: Step 2, Semi-bandit Feedback.

Original: We define the martingale difference sequences
Revised: We define the martingale difference sequences

Original: [Equation]
Revised: [Equation]

Original: It is not hard to check that M kh ( (cid:101) Q ) and M kh ( (cid:101) V ) are both indeed martingale difference sequences with respect to the history till episode k and time step h .
Revised: It is not hard to check that M kh ( (cid:101) Q ) and M kh ( (cid:101) V ) are both indeed martingale difference sequences with respect to the history till episode k and time step h .

Original: With these definitions
Revised: With these definitions

Original: [Equation]
Revised: [Equation]

Original: we can now decompose the regret bound as
Revised: we can now decompose the regret bound as

Original: [Equation]
Revised: [Equation]

Original: The above inequality (i) holds by applying Lemma 17 and the fact (cid:101) V kh ( s ) ≤ HF , which comes from the definition in (17).
Revised: The above inequality (i) holds by applying Lemma 17 and the fact (cid:101) V kh ( s ) ≤ HF , which comes from the definition in (17).

Original: Then, by unrolling this relation from h = 1 to h = H and noticing (cid:101) V kH +

Revised: Then, by unrolling this relation from h = 1 to h = H and noticing (cid:101) V kH +

Original: = 0 , we can have
Revised: = 0 , we can have

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Step 3, Bandit Feedback.
Revised: Step 3, Bandit Feedback.

Original: In the setting of bandit feedback, we only modify the reward estimator
Revised: In the setting of bandit feedback, we only modify the reward estimator

Original: ˜ r kh,i and its corresponding bonus term ˜ b k, r h .
Revised: ˜ r kh,i and its corresponding bonus term ˜ b k, r h .

Original: Thus, by going through the proof of Lemma 17, we can notice that to have the same result for bandit feedback, it suffice to use Lemma 18 to show that the reward estimation error is bounded by the reward bonus term.
Revised: Thus, by going through the proof of Lemma 17, we can notice that to have the same result for bandit feedback, it suffice to use Lemma 18 to show that the reward estimation error is bounded by the reward bonus term.

Original: Then, by the inequality (18), we can notice that to achieve the final Nash-regret bound, we only need to bound the summation (cid:80)
Revised: Then, by the inequality (18), we can notice that to achieve the final Nash-regret bound, we only need to bound the summation (cid:80)

Original: Kk =1 (cid:80)
Revised: Kk =1 (cid:80)

Original: Hh =1 ˜ b k, r h ( s kh , a kh ) , which is
Revised: Hh =1 ˜ b k, r h ( s kh , a kh ) , which is

Original: [Equation]
Revised: [Equation]

Original: (By definition of ˜ b k, r h in (13).)
Revised: (By definition of ˜ b k, r h in (13).)

Original: [Equation]
Revised: [Equation]

Original: (By definition of β
Revised: (By definition of β

Original: [Equation]
Revised: [Equation]

Original: and Lemma 19.)
Revised: and Lemma 19.)

Original: [Equation]
Revised: [Equation]

Original: (Since d = m (cid:80)
Revised: (Since d = m (cid:80)

Original: [Equation]
Revised: [Equation]

Original: S f .)
Revised: S f .)

Original: Therefore, by (18), with ϵ = 1 /T , under bandit feedback, we have
Revised: Therefore, by (18), with ϵ = 1 /T , under bandit feedback, we have

Original: Nash-Regret ( K ) ≤
Revised: Nash-Regret ( K ) ≤

Original: [Equation]
Revised: [Equation]

Original: F.3 Lemmas for Semi-bandit Feedback
Revised: F.3 Lemmas for Semi-bandit Feedback

Original: The following two lemmas shows that our value function estimations are indeed optimistic.
Revised: The following two lemmas shows that our value function estimations are indeed optimistic.

Original: Lemma 16.
Revised: Lemma 16.

Original: With probability at least 1 − δ , simultaneously for arbitrary value function V ∈ [0 , HF ] S and any tuple ( k, h, s, a ) , it holds that | ( (cid:98) P kh − P h ) V ( s, a ) | ≤ b k, pv h
Revised: With probability at least 1 − δ , simultaneously for arbitrary value function V ∈ [0 , HF ] S and any tuple ( k, h, s, a ) , it holds that | ( (cid:98) P kh − P h ) V ( s, a ) | ≤ b k, pv h

Original: ( s, a ) , where b k,
Revised: ( s, a ) , where b k,

Original: pv h ( s, a ) is defined in (8) .
Revised: pv h ( s, a ) is defined in (8) .

Original: Proof.
Revised: Proof.

Original: We define P fh to be the operator such that for some value function V f : S f
Revised: We define P fh to be the operator such that for some value function V f : S f

Original: (cid:55)→ R , we have ( P fh V
Revised: (cid:55)→ R , we have ( P fh V

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: s,
Revised: s,

Original: [Equation]
Revised: [Equation]

Original: ) =
Revised: ) =

Original: E s ′
Revised: E s ′

Original: [Equation]
Revised: [Equation]

Original: ∼ P fh ( ·| s f ,n f ( a )) (cid:2) V f
Revised: ∼ P fh ( ·| s f ,n f ( a )) (cid:2) V f

Original: ( s ′ f ) (cid:3) .
Revised: ( s ′ f ) (cid:3) .

Original: We also define (cid:98) P k,fh similarly.
Revised: We also define (cid:98) P k,fh similarly.

Original: Then, by definition of our transition kernel, for operators P h and (cid:98) P kh , it holds that
Revised: Then, by definition of our transition kernel, for operators P h and (cid:98) P kh , it holds that

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Therefore, by Lemma E.1 in Chen et al.
Revised: Therefore, by Lemma E.1 in Chen et al.

Original: [2020], since ∥ V ∥ ∞ ≤ HF , we have
Revised: [2020], since ∥ V ∥ ∞ ≤ HF , we have

Original: [Equation]
Revised: [Equation]

Original: where errp k,fh ( s, a ) =
Revised: where errp k,fh ( s, a ) =

Original: ∥ (cid:98) P k,fh ( · | s f , n f ( a )) − P fh ( · | s f , n f ( a ))
Revised: ∥ (cid:98) P k,fh ( · | s f , n f ( a )) − P fh ( · | s f , n f ( a ))

Original: ∥ 1 .
Revised: ∥ 1 .

Original: Now, notice that (cid:16)(cid:81) f ′ ̸
Revised: Now, notice that (cid:16)(cid:81) f ′ ̸

Original: = f P f ′ h (cid:17) V ( s, a ) can be seen as some value function from S f to [0 , HF ] .
Revised: = f P f ′ h (cid:17) V ( s, a ) can be seen as some value function from S f to [0 , HF ] .

Original: Therefore, by Lemma 12 in Bai and Jin [2020], with probability at least 1 − δ 2 , simultaneously for any V and ( k, h, s, a ) , it holds that
Revised: Therefore, by Lemma 12 in Bai and Jin [2020], with probability at least 1 − δ 2 , simultaneously for any V and ( k, h, s, a ) , it holds that

Original: [Equation]
Revised: [Equation]

Original: where ι = 2 log(4( m + 1)( (cid:80) f ∈F S f ) T/δ )
Revised: where ι = 2 log(4( m + 1)( (cid:80) f ∈F S f ) T/δ )

Original: Meanwhile, by standard Hoeffding’s inequality and union bound, with probability at least 1 − δ 2 , simultaneously for any ( k, h, s, a ) , it holds that
Revised: Meanwhile, by standard Hoeffding’s inequality and union bound, with probability at least 1 − δ 2 , simultaneously for any ( k, h, s, a ) , it holds that

Original: [Equation]
Revised: [Equation]

Original: Finally, by plugging above two concentration inequalities back into (19), we can have
Revised: Finally, by plugging above two concentration inequalities back into (19), we can have

Original: [Equation]
Revised: [Equation]

Original: Lemma 17.
Revised: Lemma 17.

Original: With probability at least 1 − δ , for any ( k, h, i, s, a ) ∈
Revised: With probability at least 1 − δ , for any ( k, h, i, s, a ) ∈

Original: [ K ] ×
Revised: [ K ] ×

Original: [ H ] × [ m ] × S × A , it holds that
Revised: [ H ] × [ m ] × S × A , it holds that

Original: [Equation]
Revised: [Equation]

Original: where Q k h,k and V kh,i are defined in (16) .
Revised: where Q k h,k and V kh,i are defined in (16) .

Original: Proof.
Revised: Proof.

Original: The proof is adapted from Liu et al.
Revised: The proof is adapted from Liu et al.

Original: [2021] and goes by induction from h = H
Revised: [2021] and goes by induction from h = H

Original: + 1 to h = 1 .
Revised: + 1 to h = 1 .

Original: We can see that inequalities (21) obviously hold when h = H + 1 since by definition we have
Revised: We can see that inequalities (21) obviously hold when h = H + 1 since by definition we have

Original: V k H +1 ,i ( s ) =
Revised: V k H +1 ,i ( s ) =

Original: V kH +1 ,i ( s )
Revised: V kH +1 ,i ( s )

Original: = 

Revised: =

Original: for any ( k, i, s ) .
Revised: for any ( k, i, s ) .

Original: Now, suppose inequalities (21) hold for h +1 .
Revised: Now, suppose inequalities (21) hold for h +1 .

Original: Then, if we have Q k h,i ( s, a ) =
Revised: Then, if we have Q k h,i ( s, a ) =

Original: HF , it holds trivially that Q k h,i ( s, a ) ≥ Q † ,π k − i h,i ( s, a ) .
Revised: HF , it holds trivially that Q k h,i ( s, a ) ≥ Q † ,π k − i h,i ( s, a ) .

Original: Otherwise, by Bellman equations (15) and update rule in Algorithm 5, we have
Revised: Otherwise, by Bellman equations (15) and update rule in Algorithm 5, we have

Original: [Equation]
Revised: [Equation]

Original: Now, recall that b kh ( s, a ) =
Revised: Now, recall that b kh ( s, a ) =

Original: b k, pv h ( s, a ) + b k, r h ( s, a ) .
Revised: b k, pv h ( s, a ) + b k, r h ( s, a ) .

Original: By reward definition in congestion game, we have
Revised: By reward definition in congestion game, we have

Original: [Equation]
Revised: [Equation]

Original: Thus, by using standard Hoefding’s inequality and union bound, we can immediately have | (A) | ≤ b k, r h ( s, a ) .
Revised: Thus, by using standard Hoefding’s inequality and union bound, we can immediately have | (A) | ≤ b k, r h ( s, a ) .

Original: Then, since V † ,π k − i h,i ∈ [0 , HF ] S , by Lemma 16, we have | (C) | ≤ b k, pv h
Revised: Then, since V † ,π k − i h,i ∈ [0 , HF ] S , by Lemma 16, we have | (C) | ≤ b k, pv h

Original: ( s, a ) .
Revised: ( s, a ) .

Original: That is, we have (A) + (C) + b kh ( s, a ) ≥ 0 .
Revised: That is, we have (A) + (C) + b kh ( s, a ) ≥ 0 .

Original: Then, by inductive hypothesis, we know that V k h
Revised: Then, by inductive hypothesis, we know that V k h

Original: +1 ,i ≥ V † ,π k − i
Revised: +1 ,i ≥ V † ,π k − i

Original: h
Revised: h

Original: +1 ,i − ( H − h ) ϵ , which implies (B) ≥ 0 .
Revised: +1 ,i − ( H − h ) ϵ , which implies (B) ≥ 0 .

Original: Therefore, we have Q k h,i ( s, a ) − Q † ,π k − i h,i ( s, a ) ≥ − ( H − h ) ϵ .
Revised: Therefore, we have Q k h,i ( s, a ) − Q † ,π k − i h,i ( s, a ) ≥ − ( H − h ) ϵ .

Original: For V k h,i and V † ,π k − i h,i , we notice that in Algorithm 5, π k is computed as the ϵ -approximate Nash equilibrium of ( Q k h, 1 , . . .
Revised: For V k h,i and V † ,π k − i h,i , we notice that in Algorithm 5, π k is computed as the ϵ -approximate Nash equilibrium of ( Q k h, 1 , . . .

Original: , Q k h,m ) .
Revised: , Q k h,m ) .

Original: Therefore, it holds that
Revised: Therefore, it holds that

Original: [Equation]
Revised: [Equation]

Original: By Bellman equations (15), we also have
Revised: By Bellman equations (15), we also have

Original: [Equation]
Revised: [Equation]

Original: ν,π kh, − i )( ·| s ) (cid:104) Q † ,π k − i h,i ( s, a ′ )
Revised: ν,π kh, − i )( ·| s ) (cid:104) Q † ,π k − i h,i ( s, a ′ )

Original: (cid:105) .
Revised: (cid:105) .

Original: Since Q k h,i ( s, a ) − Q † ,π k − i h,i ( s, a ) ≥ − ( H − h )
Revised: Since Q k h,i ( s, a ) − Q † ,π k − i h,i ( s, a ) ≥ − ( H − h )

Original: ϵ , we immediately have V k h,i ( s ) − V † ,π k − i
Revised: ϵ , we immediately have V k h,i ( s ) − V † ,π k − i

Original: h,i ( s ) ≥ − ( H − h
Revised: h,i ( s ) ≥ − ( H − h

Original: + 1) ϵ .
Revised: + 1) ϵ .

Original: Thus, by induction, we have that Q k h,i ( s, a ) ≥ Q † ,π k − i h,i ( s, a ) − ( H − h ) ϵ and
Revised: Thus, by induction, we have that Q k h,i ( s, a ) ≥ Q † ,π k − i h,i ( s, a ) − ( H − h ) ϵ and

Original: V k h,i ( s ) ≥ V † ,π k − i h,i ( s ) − ( H − h + 1) ϵ for all h ∈
Revised: V k h,i ( s ) ≥ V † ,π k − i h,i ( s ) − ( H − h + 1) ϵ for all h ∈

Original: [ H ] .
Revised: [ H ] .

Original: The inequalities for V kh,i and Q k h,i can be proved similarly.
Revised: The inequalities for V kh,i and Q k h,i can be proved similarly.

Original: F.4 Additional Lemmas for Bandit Feedback
Revised: F.4 Additional Lemmas for Bandit Feedback

Original: The following lemma shows that the reward estimation error can be bounded by the reward bonus term.
Revised: The following lemma shows that the reward estimation error can be bounded by the reward bonus term.

Original: Lemma 18.
Revised: Lemma 18.

Original: With probability at least 1 − δ , simultaneously for all ( i, k, h, s, a ) , it holds that | (˜ r kh,i − r h,i )( s, a ) | ≤ ˜ b k, r h ( s, a ) , where
Revised: With probability at least 1 − δ , simultaneously for all ( i, k, h, s, a ) , it holds that | (˜ r kh,i − r h,i )( s, a ) | ≤ ˜ b k, r h ( s, a ) , where

Original: ˜ r kh,i and ˜ b k, r h are defined in (12) and (13) .
Revised: ˜ r kh,i and ˜ b k, r h are defined in (12) and (13) .

Original: Proof.
Revised: Proof.

Original: The proof is extremely similar to Lemma 3.
Revised: The proof is extremely similar to Lemma 3.

Original: By construction, we have
Revised: By construction, we have

Original: [Equation]
Revised: [Equation]

Original: (By Theorem 20.5 in Lattimore and Szepesvári [2020].)
Revised: (By Theorem 20.5 in Lattimore and Szepesvári [2020].)

Original: Since each element in θ h is bounded in [0 , 1] by construction, we have ∥ θ h ∥ 2 ≤ √ d .
Revised: Since each element in θ h is bounded in [0 , 1] by construction, we have ∥ θ h ∥ 2 ≤ √ d .

Original: Then, by Lemma 4, we have det (cid:0) V kh (cid:1) ≤ (cid:0) 1 + mkFd (cid:1) d since by construction ∥
Revised: Then, by Lemma 4, we have det (cid:0) V kh (cid:1) ≤ (cid:0) 1 + mkFd (cid:1) d since by construction ∥

Original: A i ( s, a ) ∥ 22 ≤ F .
Revised: A i ( s, a ) ∥ 22 ≤ F .

Original: Finally, to make this bound valid for all player i ∈
Revised: Finally, to make this bound valid for all player i ∈

Original: [ m ] , we only need to take maximization over i ∈ [ m ] .
Revised: [ m ] , we only need to take maximization over i ∈ [ m ] .

Original: Therefore, with probability at least 1 − δ , we have
Revised: Therefore, with probability at least 1 − δ , we have

Original: [Equation]
Revised: [Equation]

Original: where √ β k = √ d
Revised: where √ β k = √ d

Original: + (cid:113) d log (cid:0) 1 + mkFd (cid:1) + ι .
Revised: + (cid:113) d log (cid:0) 1 + mkFd (cid:1) + ι .

Original: The follow lemma bound the sum of reward bonus under bandit feedback.
Revised: The follow lemma bound the sum of reward bonus under bandit feedback.

Original: Lemma 19.
Revised: Lemma 19.

Original: For any h ∈ [ H ] , it holds that
Revised: For any h ∈ [ H ] , it holds that

Original: K
Revised: K

Original: [Equation]
Revised: [Equation]

Original: where d = m (cid:80)
Revised: where d = m (cid:80)

Original: f ∈F S f .
Revised: f ∈F S f .

Original: Proof.
Revised: Proof.

Original: First, since V kh = I + (cid:80) k − 1 k ′ =1 (cid:80) mi
Revised: First, since V kh = I + (cid:80) k − 1 k ′ =1 (cid:80) mi

Original: =1 A i ( s k ′ h , a k ′ h )
Revised: =1 A i ( s k ′ h , a k ′ h )

Original: A i ( s k ′ h , a k ′ h ) ⊤ , we have V kh ⪰ I and thus (cid:0) V kh (cid:1) − 1 ⪯ I .
Revised: A i ( s k ′ h , a k ′ h ) ⊤ , we have V kh ⪰ I and thus (cid:0) V kh (cid:1) − 1 ⪯ I .

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: For simplicity, let A kh,i =
Revised: For simplicity, let A kh,i =

Original: A i ( s kh , a kh ) .
Revised: A i ( s kh , a kh ) .

Original: Then, as a result, we have
Revised: Then, as a result, we have

Original: K
Revised: K

Original: [Equation]
Revised: [Equation]

Original: k =
Revised: k =

Original: [Equation]
Revised: [Equation]

Original: F.5 Technical Lemmas
Revised: F.5 Technical Lemmas

Original: Lemma 20.
Revised: Lemma 20.

Original: For any f ∈ F , it holds that
Revised: For any f ∈ F , it holds that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Here, we have
Revised: Here, we have

Original: K (cid:88)
Revised: K (cid:88)

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: (By standard technique)
Revised: (By standard technique)

Original: [Equation]
Revised: [Equation]

Original: The last line above holds because (cid:80)
Revised: The last line above holds because (cid:80)

Original: Hh =1 (cid:80) s
Revised: Hh =1 (cid:80) s

Original: f ∈S f (cid:80) mn =0 N K,fh ( s f , n ) =
Revised: f ∈S f (cid:80) mn =0 N K,fh ( s f , n ) =

Original: T .
Revised: T .

Original: This is based on a pigeonhole principle argument.
Revised: This is based on a pigeonhole principle argument.

Original: In particular, whenever the players take one more action, for any f ∈ F , the count for some tuple ( h, s f , n ) will increase exactly by 1.
Revised: In particular, whenever the players take one more action, for any f ∈ F , the count for some tuple ( h, s f , n ) will increase exactly by 1.

Original: Lemma 21 (Chen et al.
Revised: Lemma 21 (Chen et al.

Original: For any f, f ′ ∈ F and f ̸ = f ′ , it holds that
Revised: For any f, f ′ ∈ F and f ̸ = f ′ , it holds that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We define the joint empirical counter
Revised: We define the joint empirical counter

Original: N
Revised: N

Original: [Equation]
Revised: [Equation]

Original: Obviously, we have N f,f ′ h ( s f , s f ′ , n, n ′ )
Revised: Obviously, we have N f,f ′ h ( s f , s f ′ , n, n ′ )

Original: ≤ min
Revised: ≤ min

Original: [Equation]
Revised: [Equation]

Original: s
Revised: s

Original: [Equation]
Revised: [Equation]

Original: s
Revised: s

Original: [Equation]
Revised: [Equation]

Original: n ′ ) (cid:111) , which implies
Revised: n ′ ) (cid:111) , which implies

Original: N k
Revised: N k

Original: [Equation]
Revised: [Equation]

Original: s
Revised: s

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]