Original: Learning in Congestion Games with Bandit Feedback
Revised: Learning in Congestion Games with Bandit Feedback

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Learning Nash equilibria is a central problem in multi-agent systems.
Revised: Learning Nash equilibria is a central problem in multi-agent systems.

Original: In this paper,we investigate congestion games, a class of games with benign theoretical structureand broad real-world applications.
Revised: In this paper,we investigate congestion games, a class of games with benign theoretical structureand broad real-world applications.

Original: We first propose a centralized algorithm basedon the optimism in the face of uncertainty principle for congestion games with(semi-)bandit feedback, and obtain finite-sample guarantees.
Revised: We first propose a centralized algorithm basedon the optimism in the face of uncertainty principle for congestion games with(semi-)bandit feedback, and obtain finite-sample guarantees.

Original: Then we proposea decentralized algorithm via a novel combination of the Frank-Wolfe methodand G-optimal design.
Revised: Then we proposea decentralized algorithm via a novel combination of the Frank-Wolfe methodand G-optimal design.

Original: By exploiting the structure of the congestion game, weshow the sample complexity of both algorithms depends only polynomially on thenumber of players and the number of facilities, but not the size of the action set,which can be exponentially large in terms of the number of facilities.
Revised: By exploiting the structure of the congestion game, weshow the sample complexity of both algorithms depends only polynomially on thenumber of players and the number of facilities, but not the size of the action set,which can be exponentially large in terms of the number of facilities.

Original: We furtherdefine a new problem class, Markov congestion games, which allows us to modelthe non-stationarity in congestion games.
Revised: We furtherdefine a new problem class, Markov congestion games, which allows us to modelthe non-stationarity in congestion games.

Original: We propose a centralized algorithm for
Revised: We propose a centralized algorithm for

Original: Markov congestion games, whose sample complexity again has only polynomialdependence on all relevant problem parameters, but not the size of the action set.
Revised: Markov congestion games, whose sample complexity again has only polynomialdependence on all relevant problem parameters, but not the size of the action set.

Original:  Introduction
Revised:  Introduction

Original: Nash equilibrium (NE) is a widely adopted concept in game theory community, used to describe thebehavior of multi-agent systems with selfish players [Roughgarden, 2010].
Revised: Nash equilibrium (NE) is a widely adopted concept in game theory community, used to describe thebehavior of multi-agent systems with selfish players [Roughgarden, 2010].

Original: At the Nash equilibrium,no player has the incentive to change its own strategy unilaterally, which implies it is a steady state ofthe system.
Revised: At the Nash equilibrium,no player has the incentive to change its own strategy unilaterally, which implies it is a steady state ofthe system.

Original: For a general-sum game, computing the Nash equilibrium is PPAD-hard
Revised: For a general-sum game, computing the Nash equilibrium is PPAD-hard

Original: [Daskalakis,2013] and the query complexity is exponential in the number of players [Rubinstein, 2016].
Revised: [Daskalakis,2013] and the query complexity is exponential in the number of players [Rubinstein, 2016].

Original: To helpaddress these issues, a natural approach is to consider games with special structures.
Revised: To helpaddress these issues, a natural approach is to consider games with special structures.

Original: In this paper, wefocus on congestion games.
Revised: In this paper, wefocus on congestion games.

Original: Congestion games are general-sum games with facilities (resources) shared among players
Revised: Congestion games are general-sum games with facilities (resources) shared among players

Original: [Rosenthal,1973].
Revised: [Rosenthal,1973].

Original: During the game, each player will decide what combination of facilities to utilize, and popularfacilities will become congested, which results in a possibly higher cost on each user.
Revised: During the game, each player will decide what combination of facilities to utilize, and popularfacilities will become congested, which results in a possibly higher cost on each user.

Original: One exampleof congestion game is the routing game
Revised: One exampleof congestion game is the routing game

Original: [Fotakis et al., 2002], where each player needs to travel froma given starting point to a destination point through some shared routes.
Revised: [Fotakis et al., 2002], where each player needs to travel froma given starting point to a destination point through some shared routes.

Original: These routes are representedas a traffic graph and the facilities are the edges.
Revised: These routes are representedas a traffic graph and the facilities are the edges.

Original: Each player will decide her path to go, and the moreplayers use the same edge, the longer the edge travel time will be.
Revised: Each player will decide her path to go, and the moreplayers use the same edge, the longer the edge travel time will be.

Original: Congestion games also have wideapplications in electrical grids [Ibars et al., 2010], internet routing [Al-Kashoash
Revised: Congestion games also have wideapplications in electrical grids [Ibars et al., 2010], internet routing [Al-Kashoash

Original: et al., 2017] andrate allocation [Johari and Tsitsiklis, 2004].
Revised: et al., 2017] andrate allocation [Johari and Tsitsiklis, 2004].

Original: In many real-world scenarios, players can only have(semi-)bandit feedback, i.e., players know only the payoff of the facilities they choose.
Revised: In many real-world scenarios, players can only have(semi-)bandit feedback, i.e., players know only the payoff of the facilities they choose.

Original: This kind oflearning under uncertainty has been widely studied in bandits and in reinforcement learning for thesingle-agent setting, while theoretical understanding for the multi-agent case is still largely missing.
Revised: This kind oflearning under uncertainty has been widely studied in bandits and in reinforcement learning for thesingle-agent setting, while theoretical understanding for the multi-agent case is still largely missing.

Original: There are two types of algorithms in multi-agent systems, namely centralized algorithms and decentralized algorithms.
Revised: There are two types of algorithms in multi-agent systems, namely centralized algorithms and decentralized algorithms.

Original: For centralized algorithms, there exists a central authority that can controland receive feedback from all players in the game.
Revised: For centralized algorithms, there exists a central authority that can controland receive feedback from all players in the game.

Original: As we have global coordination, centralizedalgorithms usually have favorable performance.
Revised: As we have global coordination, centralizedalgorithms usually have favorable performance.

Original: On the other hand, such a central authority may not
Revised: On the other hand, such a central authority may not

Original: [Table]
Revised: [Table]

Original: -Wolfe with Exploration
Revised: -Wolfe with Exploration

Original: I
Revised: I

Original: [Table]
Revised: [Table]

Original: Frank-Wolfe with Exploration II
Revised: Frank-Wolfe with Exploration II

Original: [Table]
Revised: [Table]

Original: always be available in practice
Revised: always be available in practice

Original: [Table]
Revised: [Table]

Original: and thus people turn to decentralized algorithms,
Revised: and thus people turn to decentralized algorithms,

Original: i
Revised: i

Original: [Table]
Revised: [Table]

Original: e., each playermakes decisions individually and can only observe her own feedback.
Revised: e., each playermakes decisions individually and can only observe her own feedback.

Original: However, decentralized algorithms are vulnerable to nonstationarity because each player is making decisions in a nonstationaryenvironment as others’ strategies are changing [Zhang et al., 2021a].
Revised: However, decentralized algorithms are vulnerable to nonstationarity because each player is making decisions in a nonstationaryenvironment as others’ strategies are changing [Zhang et al., 2021a].

Original: In this paper, we will studyboth centralized and decentralized algorithms in congestion games with bandit feedback, and we willprovide motivating scenarios for both algorithms in Section 1.2.
Revised: In this paper, we will studyboth centralized and decentralized algorithms in congestion games with bandit feedback, and we willprovide motivating scenarios for both algorithms in Section 1.2.

Original: The main challenge in designing algorithms for m -player congestion games with bandit feedbackis the curse of exponential action set, i.e., the number of actions can be exponential in the numberof facilities F because every subset of facilities can be an action.
Revised: The main challenge in designing algorithms for m -player congestion games with bandit feedbackis the curse of exponential action set, i.e., the number of actions can be exponential in the numberof facilities F because every subset of facilities can be an action.

Original: As a result, an efficient algorithmshould have sample complexity polynomial in m and F and has no dependence on the size of theaction space.
Revised: As a result, an efficient algorithmshould have sample complexity polynomial in m and F and has no dependence on the size of theaction space.

Original: One closely related type of general-sum game is the potential game, in which eachindividual’s payoff changes, resulting from strategy modification, can be quantified by a commonpotential function.
Revised: One closely related type of general-sum game is the potential game, in which eachindividual’s payoff changes, resulting from strategy modification, can be quantified by a commonpotential function.

Original: It is well-known that all congestion games are potential games and each potentialgame has an equivalent congestion game formulation [Monderer and Shapley, 1996].
Revised: It is well-known that all congestion games are potential games and each potentialgame has an equivalent congestion game formulation [Monderer and Shapley, 1996].

Original: However,existing algorithms designed for potential games all have sample complexity scaling at least linearlyin the number of actions [Leonardos et al., 2021, Ding et al., 2022], which is inefficient for congestiongames.
Revised: However,existing algorithms designed for potential games all have sample complexity scaling at least linearlyin the number of actions [Leonardos et al., 2021, Ding et al., 2022], which is inefficient for congestiongames.

Original: This motivates the following question:
Revised: This motivates the following question:

Original: Can we design provably sample-efficient centralized and decentralized algorithms that can converge to NE in congestion games with bandit feedback?
Revised: Can we design provably efficient centralized and decentralized algorithms that can converge to NE incongestion games with bandit feedback?

Original: We provide the first affirmative answer to this question.
Revised: We provide the first affirmative answer to this question.

Original: To clarify, we use a notion of non-asymptoticconvergence that commonly appears in the literature of online learning and reinforcement learning[Orabona, 2019, Ding et al., 2022, Liu et al., 2021], which is different from the concept of asymptoticconvergence that is more widely used in the literature of algorithmic game theory [Heliou et al.

Original: ,2017, Krichene et al., 2015].

Original: In particular, our notion of convergence focuses on finite-time analysisand accumulative reward throughout the learning process.

Original: We call an algorithm convergent if its

Original: Nash-regret (formally defined in Section 3) is sub-linear.

Original: These two notions of convergence are notdirectly comparable since they are defined over different regimes (non-asymptotic v.s. asymptotic).

Original: We highlight our contributions below and compare our results with previous algorithms in Table 1.
Revised: We highlight our contributions below andcompare our results with previous algorithms in Table 1.

Original: Inparticular, our algorithms are shaded and we prove sublinear Nash regrets.

Original: In Table 1, the sample online-

Original: -batch in Section 3.1 of [Jin et al., 2018].

Original: 1.1 Main Novelties and Contributions
Revised: 1.1 Main Novelties and Contributions

Original: Centralized algorithm for congestion game.
Revised: Centralized algorithm for congestion game.

Original: We adapt the principle of optimism in the face ofuncertainty in stochastic bandits to ensure sufficient exploration in congestion games.
Revised: We adapt the principle of optimism in the face ofuncertainty in stochastic bandits to ensure sufficient exploration in congestion games.

Original: We begin withcongestion games with semi-bandit feedback, in which each player can observe the reward of everyfacility in the action.
Revised: We begin withcongestion games with semi-bandit feedback, in which each player can observe the reward of everyfacility in the action.

Original: Instead of estimating the action reward as in stochastic multi-armed bandits, we estimate the facility rewards directly, which removes the dependence on the size of action space.
Revised: Instead of estimating the action reward as in stochastic multi-armed bandits,we estimate the facility rewards directly, which removes the dependence on the size of action space.

Original: Furthermore, we consider congestion games with bandit feedback, in which each player can onlyobserve the overall reward.
Revised: Furthermore, we consider congestion games with bandit feedback, in which each player can onlyobserve the overall reward.

Original: In this setting, we borrow ideas from linear bandits to estimate the rewardfunction and analyze the algorithm.
Revised: In this setting, we borrow ideas from linear bandits to estimate the reward function and analyze the algorithm.

Original: The algorithm is provably sample efficient in both cases.
Revised: The algorithm is provably sample efficient in both cases.

Original: Decentralized algorithm for congestion game
Revised: Decentralized algorithm for congestion game.

Original: Our decentralized algorithm is a Frank-Wolfe method with exploration, in which each player only observes her own actions and rewards.
Revised: Our decentralized algorithm is a Frank-Wolfemethod with exploration, in which each player only observes her own actions and rewards.

Original: To efficiently explore in the congestion game, we utilize the G-optimal design allocation for bandit feedback and a specific distribution for semi-bandit feedback.
Revised: To efficiently explore in the congestion game, we utilize the G-optimal design allocation for banditfeedback and a specific distribution for semi-bandit feedback.

Original: As a result, the sample complexitydoes not depend on the number of actions.
Revised: As a result, the sample complexitydoes not depend on the number of actions.

Original: In addition, the L 1 smoothnessparameter of the potentialfunction does not depend on the number of actions, which is exploited by the Frank-Wolfe method.
Revised: In addition, the L 1 smoothness of the potential functiondoes not depend on the number of actions, which is exploited by the Frank-Wolfe method.

Original: With the help of these two specific algorithmic designs for congestion games, we give the firstdecentralized algorithm for both semi-bandit feedback and bandit feedback that has no dependenceon the size of the action space in congestion games.
Revised: With the help of these two specific algorithmic designs for congestion games, we give the first decentralizedalgorithm for both semi-bandit feedback and bandit feedback that has no dependence on the size ofthe action space in congestion games.3.

Original: Centralized algorithm for independent Markov congestion game.
Revised: Centralized algorithm for independent Markov congestion game.

Original: We extend the formulationof congestion game into a Markov setting and propose the independent Markov congestion game IMCG ), in which each facility has its own internal state and state transition happens independentlyamong all the facilities.
Revised: We extend the formulationof congestion game into a Markov setting and propose the independent Markov congestion game(IMCG), in which each facility has its own internal state and state transition happens independentlyamong all the facilities.

Original: In Section 1.2, we give some examples that fit in this model.
Revised: In Section 1.2, we give some examples that fit in this model.

Original: By utilizingtechniques from factored MDPs, we extend our centralized algorithms for congestion games toefficiently solve IMCGs, with both semi bandit and bandit feedback.
Revised: By utilizingtechniques from factored MDPs, we extend our centralized algorithms for congestion games toefficiently solve IMCGs, with both semi-bandit and bandit feedback.

Original: 1.2 Motivating Examples
Revised: 1.2 Motivating Examples

Original: We provide an exmple here to motivate our proposed models.
Revised: We provide several examples here to motivate our proposed models.

Original: See Section 3 for the formal definitionof (semi-)bandit feedback and (Markov) congestion games and Appendix A for additional examples.
Revised: See Section 3 for the formaldefinition of (semi-)bandit feedback and (Markov) congestion games. Example 1 ( Web Advertisements) .

Revised: Consider a set of websites as the facility set and companies whowant to advertise their products as the players.

Revised: Due to the budget constraints, each company mayonly choose some of these websites to put its product ad.

Original:  Example 1 ( Routing Games) .
Revised: For each website, the probability that auser will click certain ad (and then buy the product) depends on how many ads are put on the website. If a website receives too many ads, the probability that a user can see certain ad will decrease, thusmaking it congested. The reward each company will receive is measured by the amount of productssold during certain period of time, which is bandit feedback. Example 2 ( Server Usage) .

Revised: Consider a set of servers in a company as the facility set and server usersas the players.

Revised: Each user needs to request several servers to finish her computation task and the costtriggered from each server depends on the number of users requesting that server.

Revised: Each user will tryto minimize the total cost incurred from the servers she requested.

Revised: As each user can see the cost fromall the servers she requested, this is semi-bandit feedback.

Revised: Example 3 ( Routing Games) .

Original: For a routing game, there exists multiple players in a traffic graphtravelling from starting points to destination points, and the facilities are the edges.
Revised: For a routing game, there exists multiple players in a traffic graphtravelling from starting points to destination points, and the facilities are the edges.

Original: The cost of eachedge is the waiting time, which depends on the number of players using that edge.
Revised: The cost of eachedge is the waiting time, which depends on the number of players using that edge.

Original: • Centralized algorithm for routing games: Imagine each player is using Google Maps 
Revised: • Centralized algorithm for routing games: Imagine each player is using Google Maps to navigate.

Original: to

Original: navigate Then Google Maps can serve as a center that knows the starting points and the destination points, aswell as the real-time feedback of the waiting time on each edge of all the players.
Revised: Then Google Maps can serve as a center that knows the starting points and the destination points, aswell as the real-time feedback of the waiting time on each edge of all the players.

Original: Google Maps itselfalso has the incentive to assign paths according to the Nash equilibrium strategy as then each playerwill find out that deviating from the navigation has no benefit and thus stick to the app.
Revised: Google Maps itselfalso has the incentive to assign paths according to the Nash equilibrium strategy as then each playerwill find out that deviating from the navigation has no benefit and thus stick to the app.

Original: • Decentralized algorithm for routing games: Consider the case that players are still using GoogleMaps but due to privacy concerns or limited bandwidth, they only use the offline version, which hasaccess only to the information of each single user.
Revised: • Decentralized algorithm for routing games: Consider the case that players are still using GoogleMaps but due to privacy concerns or limited bandwidth, they only use the offline version, which onlyhave access to the information of each single user.

Original: Then Google Maps needs to use decentralizedalgorithms so that it can still assign Nash equilibrium strategy to each user after repeated plays.
Revised: Then Google Maps need to use decentralizedalgorithms so that it can still assign Nash equilibrium strategy to each user after repeated plays.

Original: • Markov routing games: For Markov routing games, the time cost on each edge will change amongdifferent timesteps, which is a more accurate model of the real-world.
Revised: • Markov routing games: For Markov routing games, the time cost on each edge will change amongdifferent timesteps, which is a more accurate model of the real-world.

Original: For instance, some roads areprone to car accidents, which will result in an increasing cost on the next timestep, and the chance ofaccidents also depends on the number of players using that edge currently.
Revised: For instance, some roads are prone to car accidents, which will result in an increasing cost on the next timestep, and the chance ofaccidents also depends on the number of players using that edge currently.

Original: This is modelled by theMarkovian facility state transition in independent Markov congestion games.
Revised: This is modelled by theMarkovian facility state transition in independent Markov congestion games.

Original:  Related Work
Revised:  Related Work

Original: Potential Games.
Revised: Potential Games.

Original: Potential games are general-sum games that admit a common potential functionto quantify the changes in individual’s payoff [Monderer and Shapley, 1996].
Revised: Potential games are general-sum games that admit a common potential functionto quantify the changes in individual’s payoff [Monderer and Shapley, 1996].

Original: Algorithmic gametheory community has studied how different dynamics converge to the Nash equilibium, e.g., bestresponse dynamic [Durand, 2018, Swenson et al., 2018] and no-regret dynamic [Heliou et al., 2017,
Revised: Algorithmic gametheory community has studied how different dynamics converge to the Nash equilibium, e.g., bestresponse dynamic [Durand, 2018, Swenson et al., 2018] and no-regret dynamic [Heliou et al., 2017,

Original: Cheung and Piliouras, 2020], while usually they provide only asymptotic convergence, with eitherfull information setting or bandit feedback setting.
Revised: Cheung and Piliouras, 2020], while usually they provide only asymptotic convergence, with eitherfull information setting or bandit feedback setting.

Original: Recently, reinforcement learning communitystudied Markov potential games with bandit feedback, which can be applied to standard potentialgames.
Revised: Recently, reinforcement learning communitystudied Markov potential games with bandit feedback, which can be applied to standard potentialgames.

Original: See the Markov Games part below for more details.
Revised: See the Markov Games part below for more details.

Original: Congestion Games.
Revised: Congestion Games.

Original: Congestion games are developed in the nominal work [Rosenthal, 1973]
Revised: Congestion games are developed in the nominal work [Rosenthal, 1973]

Original: andlater Monderer and Shapley [1996] builds the close connection between congestion games andpotential games.
Revised: andlater Monderer and Shapley [1996] builds the close connection between congestion games andpotential games.

Original: Congestion games are further divided into atomic congestion games and non-atomic congestion games depending on if each player is separable.
Revised: Congestion games are further divided into atomic congestion games and non-atomiccongestion games depending on if each player is separable.

Original: Plenty of works consider non-atomiccongestion games with non-decreasing cost function, which implies a convex potential function[Roughgarden and Tardos, 2004].
Revised: Plenty of works consider non-atomiccongestion games with non-decreasing cost function, which implies a convex potential function[Roughgarden and Tardos, 2004].

Original: We consider the more difficult atomic congestion game as thepotential function can be non-convex.
Revised: We consider the more difficult atomic congestion game as thepotential function can be non-convex.

Original: For online non-atomic case, [Krichene et al., 2015] considerspartial information setting while they provide convergence in the sense of Cesaro means.
Revised: For online non-atomic case, [Krichene et al., 2015] considerspartial information setting while they provide convergence in the sense of Cesaro means.

Original: [Kleinberget al., 2009, Krichene et al., 2014] shows that some no-regret online learning algorithm asymptoticallyconverges to Nash equilibrium.
Revised: [Kleinberget al., 2009, Krichene et al., 2014] shows that some no-regret online learning algorithm asymptoticallyconverges to Nash equilibrium.

Original: [Chen and Lu, 2015, 2016] are two closely related works that considerbandit feedback in atomic congestion games and provide non-asymptotic convergence.
Revised: [Chen and Lu, 2015, 2016] are two closely related works that considerbandit feedback in atomic congestion games and provide non-asymptotic convergence.

Original: However,they still assume a convex potential function and the sample complexity has exponential dependenceon the number of facilities, which is far from ideal.
Revised: However,they still assume a convex potential function and the sample complexity has exponential dependenceon the number of facilities, which is far from ideal.

Original: Markov Games.
Revised: Markov Games.

Original: Markov games are widely studied since the seminal work [Shapley, 1953].
Revised: Markov games are widely studied since the seminal work [Shapley, 1953].

Original: Recently, it received much attention due to the advances in reinforcement learning theory.
Revised: Recently, it received much attention due to the advances in reinforcement learning theory.

Original: Liu et al.[2021] provides a centralized algorithm for learning the Nash equilibrium in general-sum Markovgames, and [Jin et al., 2021a, Song et al., 2021] provide decentralized algorithms for learning the(coarse)
Revised: Liu et al.[2021] provides a centralized algorithm for learning the Nash equilibrium in general-sum Markovgames, and [Jin et al., 2021a, Song et al., 2021] provide decentralized algorithms for learning the(coarse)

Original: correlated equilibrium.
Revised: correlated equilibrium.

Original: One closely related line of research is about Markov potential games[Leonardos et al., 2021, Zhang et al., 2021b,
Revised: One closely related line of research is about Markov potential games[Leonardos et al., 2021, Zhang et al., 2021b,

Original: Fox et al., 2021, Cen et al., 2022, Ding et al., 2022].
Revised: Fox et al., 2021, Cen et al., 2022, Ding et al., 2022].

Original: However, applying their algorithms to congestion games leads to explicit dependence on the numberof actions, which would be exponentially worse than our algorithms.
Revised: However, applying their algorithms to congestion games leads to explicit dependence on the numberof actions, which would be exponentially worse than our algorithms.

Original: See Table 1 for comparisons.
Revised: See Table 1 for comparisons.

Original: Our independent Markov congestion game is motivated by the state-based potential games studied in
Revised: Our independent Markov congestion game is motivated by the state-based potential games studied in

Original: Marden [2012] and Macua et al.
Revised: Marden [2012] and Macua et al.

Original: [2018], and its transition kernel is closely related to the factored
Revised: [2018], and its transition kernel is closely related to the factored

Original: MDPs, whose single agent algorithms are studied in [Osband and Van Roy, 2014, Chen et al., 2020,
Revised: MDPs, whose single agent algorithms are studied in [Osband and Van Roy, 2014, Chen et al., 2020,

Original: Xu and Tewari, 2020, Tian et al., 2020, Rosenberg and Mansour, 2021].
Revised: Xu and Tewari, 2020, Tian et al., 2020, Rosenberg and Mansour, 2021].

Original:  Preliminaries
Revised:  Preliminaries

Revised: 3.1 Matrix Games

Original: General-sum Matrix Games.
Revised: General-sum Matrix Games.

Original: We consider the model of general-sum matrix games, defined by thetuple G =
Revised: We consider the model of general-sum matrix games, defined by thetuple G =

Original: ( {A i } mi =1 , R ) , where m is the number of players, A i is the action space of player i and
Revised: ( {A i } mi =1 , R ) , where m is the number of players, A i is the action space of player i and

Original: R ( ·| a ) is the reward distribution on [0 , r max ] m with mean r ( a ) .
Revised: R ( ·| a ) is the reward distribution on [0 , r max ] m with mean r ( a ) .

Original: Let A =
Revised: Let A =

Original: A 1 × · · · × A m be thewhole action space and denote its element as a = ( a 1 , . . .
Revised: A 1 × · · · × A m be thewhole action space and denote its element as a = ( a 1 , . . .

Original: , a m ) ∈ A .
Revised: , a m ) ∈ A .

Original: After all these players takeactions a ∈ A , a reward vector is sampled r ∼ R ( ·| a ) and player i will receive reward r i ∈ [0 , r max ]with mean r i ( a ) .
Revised: After all these players takeactions a ∈ A , a reward vector is sampled r ∼ R ( ·| a ) and player i will receive reward r i ∈ [0 , r max ]with mean r i ( a ) .

Original: Each player’s objective is to maximize her own reward.
Revised: Each player’s objective is to maximize her own reward.

Original: A general policy π is defined as a vector in ∆( A ) , the probability simplex over the action space A .
Revised: A general policy π is defined as a vector in ∆( A ) , the probability simplex over the action space A .

Original: A product policy π = ( π 1 , . . .
Revised: A product policy π = ( π 1 , . . .

Original: , π m ) is defined as a tuple in ∆( A 1 ) × · · · × ∆( A m ) , in which a =( a 1 , . .
Revised: , π m ) is defined as a tuple in ∆( A 1 ) × · · · × ∆( A m ) , in which a =( a 1 , . .

Original: . , a m ) ∼ π represents a i i .
Revised: . , a m ) ∼ π represents a i i .

Original: i .
Revised: i .

Original: d . ∼ π i .
Revised: d . ∼ π i .

Original: The value of policy π for player i is V πi =
Revised: The value of policy π for player i is V πi =

Original: E a ∼ π
Revised: E a ∼ π

Original: [ r i ( a )] .
Revised: [ r i ( a )] .

Original: Nash Equilibrium and Nash Regret.
Revised: Nash Equilibrium and Nash Regret.

Original: Given a general policy π , let π −
Revised: Given a general policy π , let π −

Original: i be the marginal jointpolicy of players 1 , . .
Revised: i be the marginal jointpolicy of players 1 , . .

Original: , i − 1 , i + 1 , . . .
Revised: , i − 1 , i + 1 , . . .

Original: , m .
Revised: , m .

Original: Then, the best response of player i under policy π isπ † i = argmax µ ∈ ∆( A i ) V
Revised: Then, the best response of player i under policy π isπ † i = argmax µ ∈ ∆( A i ) V

Original: µ,π − i
Revised: µ,π − i

Original: i and the corresponding value is V † ,π − i
Revised: i and the corresponding value is V † ,π − i

Original: i :=
Revised: i :=

Original: V π † i ,π − i i .
Revised: V π † i ,π − i i .

Original: Our goal is to findthe approximate Nash equilibrium of the matrix game, which is defined below.
Revised: Our goal is to findthe approximate Nash equilibrium of the matrix game, which is defined below.

Original: Definition 1.
Revised: Definition 1.

Original: A product policy π is an ϵ -approximate Nash equilibrium if max i ( V † ,π − i
Revised: A product policy π is an ϵ -approximate Nash equilibrium if max i ( V † ,π − i

Original: i − V πi ) ≤ ϵ .
Revised: i − V πi ) ≤ ϵ .

Original: An ϵ -approximate Nash equilibrium can be obtained by achieving a sublinear Nash regret, which isdefined below.
Revised: An ϵ -approximate Nash equilibrium can be obtained by achieving a sublinear Nash regret, which isdefined below.

Original: See Section 3 in Ding et al.
Revised: See Section 3 in Ding et al.

Original: [2022] for a more detailed discussion.
Revised: [2022] for a more detailed discussion.

Original: Definition 2.
Revised: Definition 2.

Original: With π k being the policy at k -th episode, the Nash regret after K episodes is define as
Revised: With π k being the policy at k -th episode, the Nash regret after K episodes is define as

Original: [Equation]
Revised: [Equation]

Original: Remark 1 .

Original: Here, if we replace max i ∈ [ m ] by (cid:80) mi =1 in the definition of Nash regret, the single-step

Original: Nash regret at episode k will become the Nikaido-Isoda (NI) function evaluated at π k , which is apopular objective for equilibrium computation [Nikaidô and Isoda, 1955, Raghunathan et al., 2019].

Original: Replacing max i ∈ [ m ] by (cid:80) m i

Original: =1 will make our regret bounds multiplied by a factor of m , while ourconclusion will not be affected.

Original: Potential Games.
Revised: Potential Games.

Original: A potential game is a general-sum game such that there exists a potential function Φ : ∆( A ) →
Revised: A potential game is a general-sum game such that there exists a potential function Φ : ∆( A ) →

Original: [0 , Φ max ] such that for any player i ∈
Revised: [0 , Φ max ] such that for any player i ∈

Original: [ m ] and policies π i , π ′ i , π − i , it satisfies
Revised: [ m ] and policies π i , π ′ i , π − i , it satisfies

Original: [Equation]
Revised: [Equation]

Original: We can immediately see that a policy that maximizes the potential function is a Nash equilibrium.
Revised: We can immediately see that a policy that maximizes the potential function is a Nash equilibrium.

Original: Congestion Games.
Revised: Congestion Games.

Original: A congestion game is defined by G =
Revised: A congestion game is defined by G =

Original: ( F , {A i } mi =1 , (cid:8) R f (cid:9) f ∈F ) , where
Revised: ( F , {A i } mi =1 , (cid:8) R f (cid:9) f ∈F ) , where

Original: F =
Revised: F =

Original: [ F ] is called the facility set and R f ( ·| n ) ∈ [0 , 1] is the reward distribution for facility
Revised: [ F ] is called the facility set and R f ( ·| n ) ∈ [0 , 1] is the reward distribution for facility

Original: f withmean r f ( n ) , where n ∈ [ m ] .
Revised: f withmean r f ( n ) , where n ∈ [ m ] .

Original: Each action a i ∈
Revised: Each action a i ∈

Original: A i is a subset of F (i.e., a i ⊆ F ).
Revised: A i is a subset of F (i.e., a i ⊆ F ).

Original: Suppose the jointaction chosen by all the players is a ∈ A , then a random reward is sampled r f ∼ R f ( ·| n f ( a )) foreach facility f , where n f ( a ) = (cid:80) mi =1 1 { f ∈ a i } is the number of players using facility f .
Revised: Suppose thejoint action chosen by all the players is a ∈ A , then a random reward is sampled r f ∼ R f ( ·| n f ( a ))for each facility f , where n f ( a ) = (cid:80) mi =1 1 { f ∈ a i } is the number of players using facility f .

Original: Thereward collected by player i is r i
Revised: Thereward collected by player i is r i

Original: = (cid:80) f ∈ a i r f with mean r i ( a )
Revised: = (cid:80) f ∈ a i r f with mean r i ( a )

Original: = (cid:80) f ∈ a i r f ( n f ( a )) ∈ [0 , F ] .
Revised: = (cid:80) f ∈ a i r f ( n f ( a )) ∈ [0 , F ] .

Original: Connection to Potential Games [Monderer and Shapley, 1996].
Revised: Connection to Potential Games [Monderer and Shapley, 1996].

Original: As a special class of potentialgame, all congestion games have the potential function: Φ( a ) = (cid:80) f ∈F (cid:80) n f ( a ) i =1 r f ( i ) .
Revised: As a special class of potentialgame, all congestion games have the potential function: Φ( a ) = (cid:80) f ∈F (cid:80) n f ( a ) i =1 r f ( i ) .

Original: To seethis, we can easily verify that Φ( a i , a − i ) − Φ( a ′
Revised: To seethis, we can easily verify that Φ( a i , a − i ) − Φ( a ′

Original: i , a − i )
Revised: i , a − i )

Original: =
Revised: =

Original: r i ( a i , a − i ) − r i ( a ′
Revised: r i ( a i , a − i ) − r i ( a ′

Original: i , a − i ) holds.
Revised: i , a − i ) holds.

Original: Then, bydefining Φ( π ) =
Revised: Then, bydefining Φ( π ) =

Original: E a ∼ π [Φ( a )] , we can have Φ( π i , π − i ) − Φ( π ′ i , π − i ) =
Revised: E a ∼ π [Φ( a )] , we can have Φ( π i , π − i ) − Φ( π ′ i , π − i ) =

Original: V π
Revised: V π

Original: i ,π − i i −
Revised: i ,π − i i −

Original: V π ′
Revised: V π ′

Original: i ,π − i i .
Revised: i ,π − i i .

Original: Types of feedback.
Revised: Types of feedback.

Original: There are in general two types of reward feedback for the congestion games,semi-bandit feedback and bandit feedback, both of which are reasonable under different scenarios.
Revised: There are in general two types of reward feedback for the congestion games,semi-bandit feedback and bandit feedback, both of which are reasonable under different scenarios.

Original: In semi-bandit feedback, after taking the action, player i will receive reward information r f foreach f ∈ a i ; in bandit feedback, after taking the action, player i will only receive the rewardr i
Revised: In semi-bandit feedback, after taking the action, player i will receive reward information r f foreach f ∈ a i ; in bandit feedback, after taking the action, player i will only receive the rewardr i

Original: = (cid:80) f ∈ a i r f with no knowledge about each r f .
Revised: = (cid:80) f ∈ a i r f with no knowledge about each r f .

Original: In this paper, we will address both of them, withmore focus on the bandit feedback, which can be directly generalized to semi-bandit feedback.
Revised: In this paper, we will address both of them, withmore focus on the bandit feedback, which can be directly generalized to semi-bandit feedback.

Original: Centralized Algorithms for Congestion Games
Revised: 3.2 Markov Games

Revised: General-sum Markov Games. A finite-horizon time-inhomogeneous tabular general-sum Markovgame is defined by M = {S , {A i } mi =1 , H, P, R } , where S is the state space, m is the numberof players, A i is the action space of player i , A =

Revised: A 1 × · · · × A m is the whole action space,

Revised: H is the time horizon, P =

Revised: ( P 1 , P 2 , · · · , P H ) with P h ∈ [0 , 1] S ×A× S as the transition kernel attimestep h , R =

Revised: { R h ( ·| s h , a h ) }

Revised: Hh =1 with R h ( ·| s h , a h ) as the reward distribution on [0 , r max ] m withmean r h

Revised: ( s h , a h ) ∈ [0 , r max ] m at timestep h ∈

Revised: [ H ] .

Revised: At timestep h , all players choose their actionssimultaneously and a reward vector is sampled r h ∼ R h ( ·| s h , a h ) , where s h is the current stateand a h

Revised: = ( a h, 1 , a h, 2 , · · · , a h,m ) is the joint action.

Revised: Each player i receives reward r h,i and the statetransits to s h +1 ∼ P h ( ·| s h , a h ) .

Revised: The objective for each player is to maximize her own total reward.

Revised: We assume that the initial state s 1 is fixed, but the results can immediately be extended to the casewhere s 1 follows some fixed distribution.

Revised: A (Markov) policy π is a collection of H functions { π h : S (cid:55)→ ∆( A ) } , each of which maps a stateto a distribution over the action space.

Revised: π is a product policy if π h ( · | s ) is a product policy for each( h, s ) ∈

Revised: [ H ] × S .

Revised: The value function and Q -value function of player i at timestep h under policy πare defined as

Revised: [Equation]

Revised: The best responses and Nash regret can be defined similarly as those for matrix games.

Revised: In particular,given a policy π , player i ’s best response policy is π † h,i ( · | s ) = argmax µ ∈ ∆( A i ) V

Revised: µ,π − i h,i ( s ) and thecorresponding value function is denoted as V † ,π − i h,i .

Revised: Definition 3.

Revised: With π k being the policy at k th episode, the Nash regret after K episodes is define as

Revised: [Equation]

Revised: In this paper, we will define and study the independent Markov congestion game, which is a naturalextension of the congestion game.

Revised: Independent Markov Congestion Game.

Revised: A general-sum Markov game is an independent Markovcongestion game (IMCG) if there exists a facility set F such that a i ⊆ F for any

Revised: a i ∈

Revised: A i , a state space S =

Revised: (cid:81)

Revised: f ∈F S f , a set of facility reward distributions { R fh } h ∈

Revised: [ H ] ,f ∈F such that if the jointaction at s h is a , we have r h,i = (cid:80)

Revised: f ∈ a

Revised: i r fh , where r fh ∼ R fh ( ·| s h , n f ( a )) with support on [0 , 1]and mean r fh ( s h , n f ( a )) , and a set of transition matrices { P fh } h ∈ [ H ] ,f ∈F such that P h ( s ′ | s, a ) =(cid:81)

Revised: f ∈F P fh ( s ′ f | s f , n f ( a )) .

Revised: In other words, at each timestep h and state s ∈ S , the players are in acongestion game.

Revised: Meanwhile, each facility has its own state and independent state transition, whichonly depends on its current state and number of players using that facility.

Revised: This transition kernel canbe viewed as a special case of that in factored MDPs [Szita and L˝orincz, 2009].

Revised: The IMCG also admits two types of feedback, semi-bandit feedback and bandit feedback, just likethe congestion game.

Revised: In this paper, we will consider both types of feedback.

Revised:  Centralized Algorithms for Congestion Games

Original: In this section, we introduce two centralized algorithms for congestion games – one for the semibandit feedback and one for the bandit feedback.
Revised: In this section, we introduce two centralized algorithms for congestion games – one for the semibandit feedback and one for the bandit feedback.

Original: We will see that both of them can achieve sublinear
Revised: We will see that both of them can achieve sublinear

Original: Nash regret with polynomial dependence on both m and F .
Revised: Nash regret with polynomial dependence on both m and F .

Original: 4.1 Algorithm for Semi-bandit Feedback
Revised: 4.1 Algorithm for Semi-bandit Feedback

Original: Summarized in Algorithm 1, Nash upper confidence bound (Nash-UCB) for congestion games isdeveloped based on optimism in the face of uncertainty.
Revised: Summarized in Algorithm 1, Nash upper confidence bound (Nash-UCB) for congestion games isdeveloped based on optimism in the face of uncertainty.

Original: In particular, the algorithm estimates thereward matrices optimistically in line 4, computes its Nash equilibrium policy in line 5 and thenfollows this policy.
Revised: In particular, the algorithm estimates thereward matrices optimistically in line 5, computes its Nash equilibrium policy in line 6 and thenfollows this policy.

Original: For convenience, we define the empirical counter N k,f ( n ) =
Revised: For convenience, we define the empirical counter N k,f ( n ) =

Original: (cid:80) kk ′
Revised: (cid:80) kk ′

Original: =1 1 (cid:110) n f ( a k ′ )
Revised: =1 1 (cid:110) n f ( a k ′ )

Original: = n (cid:111) and ˜ ι =2 log(4( m + 1) K/δ ) .
Revised: = n (cid:111) and ˜ ι =2 log(4( m + 1) K/δ ) .

Original: Then, the reward estimator for f and the bonus term are defined as
Revised: Then, the reward estimator for f and the bonus term are defined as

Original: [Equation]
Revised: [Equation]

Revised: r k,f

Revised: [Equation]

Revised: n

Original:  where r k,f ∈ [0 , 1] is the random reward realization of r f ( n f ( a k )) .
Revised: [Equation] = [Equation] where r k,f ∈ [0 , 1] is the random reward realization of r f ( n f ( a k )) .

Original: Naturally, the reward estimatorfor player
Revised: Naturally, the reward estimatorfor player

Original: i is ˆ r ki ( a ) = (cid:80)
Revised: i is ˆ r ki ( a ) = (cid:80)

Original: f ∈ a i ˆ r k,f ( n f ( a )) .
Revised: f ∈ a i ˆ r k,f ( n f ( a )) .

Original: 1: Input: ϵ , accuracy parameter

Original: Algorithm 1 is motivated by the Nash-VI algorithm in [Liu et al., 2021] plus a deliberate utilizationof the special reward structure in the congestion games.
Revised: Algorithm 1 is motivated by the Nash-VI algorithm in [Liu et al., 2021] plus a deliberate utilizationof the special reward structure in the congestion games.

Original: Moreover, notice that a matrix game withreward functions Q k 1 ( · ) , . . .
Revised: Moreover, notice that a matrix game withreward functions Q k 1 ( · ) , . . .

Original: , Q km ( · ) forms a potential game (see Lemma 1).
Revised: , Q km ( · ) forms a potential game (see Lemma 1).

Original: As a result, in line 5, we can efficiently compute the ϵ -approximate Nash equilibrium π k for that matrix game by utilizing
Revised: As a result, in line 6,we can efficiently compute the ϵ -approximate Nash equilibrium π k for that matrix game by utilizing

Original: Algorithm 2, (see Lemma 2).
Revised: Algorithm 3, which is given in Appendix A.

Original: It is a simple greedy algorithm such that in each round, it modifies oneplayer’s policy whose modification can increase the potential function most.

Original: In addition, Algorithm 2always outputs a deterministic product policy.Algorithm ∈A i
Revised: In addition, Algorithm 3 guarantees that π k is always adeterministic product policy.

Original: −

Original: ∈ m i k 9: i 10:

Original: 4.2 Algorithm for Bandit Feedback
Revised: 4.2 Algorithm for Bandit Feedback

Original: When the players can only receive bandit feedback, estimating ˆ r k,f directly for each f ∈ F is nolonger feasible.
Revised: When the players can only receive bandit feedback, estimating ˆ r k,f directly for each f ∈ F is nolonger feasible.

Original: However, notice that the reward function r i ( a )
Revised: However, notice that the reward function r i ( a )

Original: =
Revised: =

Original: (cid:80) f ∈ a i r f ( n f ( a )) can be seen asan inner product between vectors characterized by action a and reward function r f ( · ) .
Revised: (cid:80) f ∈ a i r f ( n f ( a )) can be seen as an inner product between vectors characterized by action a and reward function r f ( · ) .

Original: Therefore,under bandit feedback, we can treat it as a linear bandit and use ridge regression to build the rewardestimator ˜ r ki and corresponding bonus term ˜ b k, r , whose index i is dropped since it is the same for allplayers.
Revised: Therefore,under bandit feedback, we can treat it as a linear bandit and use ridge regression to build the rewardestimator ˜ r ki and corresponding bonus term ˜ b k, r .

Original: The new algorithm will use these two terms to replace ˆ r ki and b k, r i in line 4 of Algorithm 1. {+
+}
Revised: The new algorithm will use these two terms toreplace ˆ r ki and b k, r i in line 5 of Algorithm 1. {+
+}

Original: In particular, define θ ∈ [0 , 1] ˜ d with ˜ d
Revised: In particular, define θ ∈ [0 , 1] ˜ d with ˜ d

Original: = mF to be the vector such that r f ( n ) = θ n + m ( f − 1) .
Revised: = mF to be the vector such that r f ( n ) = θ n + m ( f − 1) .

Original: Meanwhile, for player i ∈ [ m ] , define A i : A (cid:55)→ { 0 , 1 } ˜ d to be the vector-valued function such that
Revised: Meanwhile, for player i ∈ [ m ] , define A i : A (cid:55)→ { 0 , 1 } ˜ d to be the vector-valued function such that

Original: [Equation]
Revised: [Equation]

Original: In other words, A i ( a ) is a 0-1 vector with element 1 only at indices corresponding to those in θ thatrepresents r f ( n ) for f ∈ a i and n = n f ( a ) .
Revised: In other words, A i ( a ) is a 0-1 vector with element 1 only at indices corresponding to those in θ thatrepresents r f ( n ) for f ∈ a i and n = n f ( a ) .

Original: Now, with these definitions, the reward function can bewritten as r i ( a ) =
Revised: Now, with these definitions, the reward function can bewritten as r i ( a ) =

Original: ⟨ A i ( a ) , θ ⟩ .
Revised: ⟨ A i ( a ) , θ ⟩ .

Original: Then, we build the reward estimator and the bonus term through ridgeregression and corresponding confidence bound, which are defined as the following:
Revised: Then, we build the reward estimator and the bonus term through ridgeregression and corresponding confidence bound, which are defined as the following:

Original: [Equation]
Revised: [Equation]

Original: where (cid:98) θ k = (cid:0) V k (cid:1) − 1 (cid:80) k − 1 k ′ =1 (cid:80) m i
Revised: where (cid:98) θ k = (cid:0) V k (cid:1) − 1 (cid:80) k − 1 k ′ =1 (cid:80) mi =1 A

Original: =1 A i ( a k ′ )
Revised: i ( a k ′ )

Original: r k ′ i , V k
Revised: r k ′ i , V k

Original: = I + (cid:80) k − 1 k ′ =1 (cid:80) m i =
Revised: = I + (cid:80) k − 1 k ′ =1 (cid:80) mi

Original: 1 A i ( a k ′ )
Revised: =1 A i ( a k ′ )

Original: A i ( a k ′ )
Revised: A i ( a k ′ )

Original: ⊤ and(cid:113)
Revised: ⊤ and(cid:113)

Original: ˜ β
Revised: ˜ β

Original: k
Revised: k

Original: = (cid:112)
Revised: [Equation]

Original: ˜ d + (cid:114) ˜ d log (cid:16) 1 + mkF ˜ d (cid:17) + ˜ ι .
Revised: + [Equation]

Original:  Note that we cannot bound the sum of this bonus terms bydirectly applying the elliptical potential lemma.
Revised: 4 Note that we cannot bound the sum of this bonus terms bydirectly applying the elliptical potential lemma.

Original: We instead prove its variant in Lemma 4.4.
Revised: We instead prove its variant in Lemma 4.

Original: 3 Regret Analysis
Revised: 4.3 Regret Analysis

Original: The Nash regret bounds for the two versions of Algorithm 1 are formally presented in Theorem 1.
Revised: The Nash regret bounds for the two versions of Algorithm 1 are formally presented in Theorem 1.

Original: The proof details are deferred to Appendix D.
Revised: The proof details are deferred to Appendix C.

Original: Theorem 1.
Revised: Theorem 1.

Original: Let ϵ = 1 /K .
Revised: Let ϵ = 1 /K .

Original: For congestion games with semi-bandit feedback, by running Algorithmwith reward estimator and bonus term in (1) , with probability at least 1 − δ , we can achieve that
Revised: For congestion games with semi-bandit feedback, by running Algorithmwith reward estimator and bonus term in (1) , with probability at least 1 − δ , we can achieve that

Original: [Equation]
Revised: [Equation]

Original: Furthermore, if we only have bandit feedback, then by running Algorithm 1 with reward estimator and bonus term in (2) , with probability at least 1 − δ , we can achieve that
Revised: Furthermore, if we only have bandit feedback, then by running Algorithm 1 with reward estimator and bonus term in (2) , with probability at least 1 − δ , we can achieve that

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: + (

Original: [Table]

Original: −

Original: [Table]

Original: [Table]

Original: ki )

Original: + γρ

Original: i

Original: Remark 2 .
Revised: Remark 1 .

Original: Since each action is a subset of F , the size of each player’s action space can be 2 F .
Revised: Since each action is a subset of F , the size of each player’s action space can be 2 F .

Original: As aresult, directly applying Nash-VI in [Liu et al., 2021] leads to a regret bound exponential in F .
Revised: As aresult, directly applying Nash-VI in [Liu et al., 2021] leads to a regret bound exponential in F .

Original: Remark 3 .
Revised: Remark 2 .

Original: Note that we assume r f ∈ [0 , 1] , which implies r i ∈ [0 , F ] for each player i ∈ [ m ] .
Revised: Note that we assume r f ∈ [0 , 1] , which implies r i ∈ [0 , F ] for each player i ∈ [ m ] .

Original:  Decentralized Algorithms for Congestion Games
Revised:  Decentralized Algorithms for Congestion Games

Original: In this section, we present a decentralized algorithm for congestion games.
Revised: In this section, we present a decentralized algorithm for congestion games.

Original: Due to limited space, weonly introduce the version of bandit feedback as in Section 4.2.
Revised: Due to limited space, weonly introduce the version of bandit feedback as in Section 4.2.

Original: The algorithmic details for the settingof semi-bandit feedback are deferred into Appendix E.3.
Revised: The algorithmic details for the settingof semi-bandit feedback are deferred into Appendix D.3.

Original: We will show that under both settings, eventhough each player can only observe her own actions and rewards, our decentralized algorithm stillenjoys sublinear Nash regret with polynomial dependence on m and F .
Revised: We will show that under both settings, eventhough each player can only observe her own actions and rewards, our decentralized algorithm stillenjoys sublinear Nash regret with polynomial dependence on m and F .

Original: We first define the vector-valued function ϕ
Revised: We first define the vector-valued function ϕ

Original: i : A i (cid:55)→ { 0 , 1 } F i to be the feature map of player i suchthat
Revised: i : A i (cid:55)→ { 0 , 1 } F i to be the feature map of player i suchthat

Original: [ ϕ i ( a i )]
Revised: [ ϕ i ( a i )]

Original: f = 1 { f ∈ a i } for a i ∈
Revised: f = 1 { f ∈ a i } for a i ∈

Original: A i and f ∈ (cid:83) a i ∈A i a i .
Revised: A i and f ∈ (cid:83) a i ∈A i a i .

Original: Here, F i is the size of (cid:83) a i ∈A
Revised: Here, F i is the size of (cid:83) a i ∈A

Original: i a i ⊆
Revised: i a i ⊆

Original: Fand we can immediately see that F i ≤ F for any i ∈ [ m ] .
Revised: Fand we can immediately see that F i ≤ F for any i ∈ [ m ] .

Original: The core idea of our algorithm is that the Nash equilibrium can be found by reaching the stationarypoints of the potential function since all congestion games are potential games.
Revised: The core idea of our algorithm is that the Nash equilibrium can be found by reaching the stationarypoints of the potential function since all congestion games are potential games.

Original: Summarized in
Revised: Summarized in

Original: Algorithm 3, our decentralized algorithm is developed based on the Frank-Wolfe method and has thefollowing three major components. 
Revised: Algorithm 2, our decentralized algorithm is developed based on the Frank-Wolfe method and has thefollowing three major components. Algorithm m do 7:

Revised: Compute (cid:98) ∇ ki Φ( a i by the formula all

Revised: ∈

Original: Gradient Estimator.{+A Compute (cid:101) π k +
 i ← argmax π i ∈ ∆( A i ) (cid:68)+}
Revised: {+A Compute (cid:101) π k +
 i ← argmax π i ∈ ∆( A i ) (cid:68)+}

Revised: π i , (cid:98) ∇ ki Φ (cid:69) 9:

Revised: Update π k +1 i ← (1 − γ )( ν (cid:101)

Revised: π

Revised: k +1 i

Revised: + (1 − ν )

Revised: π ki )

Original:  In line 7, the algorithm builds the estimator (cid:98) ∇ ki Φ defined in (4) by using the τreward samples collected from line 5.
Revised: + γρ i Gradient Estimator. In line 7, the algorithm builds the estimator (cid:98) ∇ ki Φ defined in (4) by using the τreward samples collected from line 5.

Original: Here, (cid:98) ∇ ki Φ estimates the gradient of potential function Φ withrespect to the policy π ki .
Revised: Here, (cid:98) ∇ ki Φ estimates the gradient of potential function Φ withrespect to the policy π ki .

Original: Recall that for a congestion game, we have Φ( a )
Revised: Recall that for a congestion game, we have Φ( a )

Original: = (cid:80) f ∈F (cid:80) n f ( a ) i =1 r f ( i )and Φ( π ) =
Revised: = (cid:80) f ∈F (cid:80) n f ( a ) i =1 r f ( i )and Φ( π ) =

Original: E a ∼ π [Φ( a )] .
Revised: E a ∼ π [Φ( a )] .

Original: Then we can define ∇ i Φ :=
Revised: Then we can define ∇ i Φ :=

Original: ∇ π i Φ as a vector of dimension |A
Revised: ∇ π i Φ as a vector of dimension |A

Original: i | .
Revised: i | .

Original: For thecomponent indexed by some a i ∈
Revised: For thecomponent indexed by some a i ∈

Original: A i , we can see that Φ( π ) =
Revised: A i , we can see that Φ( π ) =

Original: π i ( a i ) E a − i ∼ π − i [ r i ( a i , a − i )]
Revised: π i ( a i ) E a − i ∼ π − i [ r i ( a i , a − i )]

Original: + const ,where const does not depend on π i ( a i ) .
Revised: + const ,where const does not depend on π i ( a i ) .

Original: Therefore, we have
Revised: Therefore, we have

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  where
Revised: 3) where

Original: [ θ i ( π )] f = E a − i ∼ π − i (cid:2) r f ( n f ( a − i )
Revised: [ θ i ( π )] f = E a − i ∼ π − i (cid:2) r f ( n f ( a − i )

Original: + 1) (cid:3) .
Revised: + 1) (cid:3) .

Original: Meanwhile, the mean of the t -th reward that player i received at episode k satisfies
Revised: Meanwhile, the mean of the t -th reward that player i received at episode k satisfies

Original: [Equation]
Revised: [Equation]

Original: where [ θ k,ti ( a k,t − i )]
Revised: where [ θ k,ti ( a k,t − i )]

Original: f = r f ( n f ( a k,t − i )
Revised: f = r f ( n f ( a k,t − i )

Original: + 1) and its mean is [ θ i ( π k )] f .
Revised: + 1) and its mean is [ θ i ( π k )] f .

Original: Therefore, we can use linearregression to estimate θ i ( π k ) .
Revised: Therefore, we can use linearregression to estimate θ i ( π k ) .

Original: In particular, we have (cid:98) θ k i ( π k )=
Revised: In particular, we have (cid:98) θ ki ( π k )

Original:  1 τ (cid:80) τ t =1 (cid:0)Σ
Revised: = 1 τ (cid:80) τt =1 (cid:0)

Original: k i (cid:1) − 1 ϕ
Revised: Σ ki (cid:1) − 1 ϕ

Original: i ( a k,t i ) r k,t i , withthe covariance matrix Σ ki
Revised: i ( a k,ti ) r k,ti , withthe covariance matrix Σ ki

Original: =
Revised: =

Original: E a i ∼ π ki (cid:2) 
Revised: E a i ∼ π ki (cid:2) ϕ i

Revised: [Equation]

Revised: a i

Revised: [Equation]

Original: ϕ
Revised: ϕ

Original: i ( a i ) ϕ
Revised: i [Equation]

Original: i ( a i ) ⊤ (cid:3) .
Revised: a i ) ⊤ (cid:3) .

Original: Then, we have the unbiased gradient estimate
Revised: Then, we have the unbiased gradient estimate

Original: [Equation]
Revised: [Equation]

Original: G-optimal Design.
Revised: G-optimal Design.

Original: In line 8 and 9, the algorithm performs standard Frank-Wolfe update and mixesthe updated policy with an exploration policy ρ i , which is defined as the G-optimal allocation forfeatures { ϕ i ( a i ) } a i ∈A i .
Revised: In line 8 and 9, the algorithm performs standard Frank-Wolfe update and mixes the updated policy with an exploration policy ρ i , which is defined as the G-optimal allocation forfeatures { ϕ i ( a i ) } a i ∈A i .

Original: To be specific, we have
Revised: To be specific, we have

Original: [Equation]
Revised: [Equation]

Original: Here ρ i guarantees that Σ ki is invertible and the variance of (cid:98) ∇ ki Φ( a i ) =
Revised: Here ρ i guarantees that Σ ki is invertible and the variance of (cid:98) ∇ ki ( a i ) =

Original: (cid:68) ϕ
Revised: (cid:68) ϕ

Original: i ( a i ) , (cid:98) θ ki ( π k ) (cid:69) dependsonly on F instead of the size of action space (Lemma 9) because by Kiefer-Wolfowitz theorem, wehave max a i ∈A i ∥ϕ i ( a i ) ∥ 2 E a ′ i ∼ ρi
Revised: i ( a i ) , (cid:98) θ ki ( π k ) (cid:69) dependsonly on F instead of the size of action space (Lemma 9). This is because by Kiefer-Wolfowitztheorem, we have max a i ∈A i ∥

Original: [ ϕ i ( a ′ i ) ϕ
Revised: ϕ i ( a [Equation] a

Original: i ( a ′ i ) ⊤ ] − 1 = F i ≤ F [Lattimore and Szepesvári, 2020].
Revised: [Equation] [Pukelsheim, 2006].

Original: Frank-Wolfe Update.
Revised: Frank-Wolfe Update.

Original: Finally, we emphasize that it is crucial to use Frank-Wolfe update becauseit is compatible with L 1 norm and we can show that Φ is mF -smooth with respect to the L 1 norm(Lemma 11).
Revised: Finally, we emphasize that it is crucial to use Frank-Wolfe update becauseit is compatible with L 1 norm and we can show that Φ is mF -smooth with respect to the L 1 norm(Lemma 11).

Original: In contrast, its smoothness for L 2 norm will depend on the size of the action space.
Revised: In contrast, its smoothness for L 2 norm will depend on the size of the action space.

Original: Before the game starts, each player i can compute her ρ i based on her own action set A i .
Revised: Before the game starts, each player i can compute her ρ i based on her own action set A i .

Original: During thegame, all players only have access to their own actions and rewards, which means that Algorithm 3 isfully decentralized.
Revised: During thegame, all players only have access to their own actions and rewards, which means that Algorithm 2 isfully decentralized.

Original: The Nash regret bound for this algorithm is formally stated in Theorem 2 and theproof details are given in Appendix E.1 and E.2.
Revised: The Nash regret bound for this algorithm is formally stated in Theorem 2 and theproof details are given in Appendix D.1 and D.2.

Original: Theorem 2.
Revised: Theorem 2.

Original: Let T =
Revised: Let T =

Original: Kτ .
Revised: Kτ .

Original: For congestion game with bandit feedback, by running Algorithm 3 with gradient estimator (cid:98) ∇ ki Φ and exploration distribution ρ i defined in (4) , (5) , with probability at least 1 − δ , we have
Revised: For congestion game with bandit feedback, by running Algorithm 2 with gradient estimator (cid:98) ∇ ki Φ and exploration distribution ρ i defined in (4) , (5) , with probability at least 1 − δ , we have

Original: [Equation]
Revised: [Equation]

Original: For congestion game with semi-bandit feedback, by running Algorithm 3 with gradient estimator (cid:101) ∇ ki Φ( a i ) and exploration distribution ˜ 
Revised: For congestion game with semi-bandit feedback, by running Algorithm 2 with gradient estimator (cid:101) ∇ ki Φ( a i ) and exploration distribution ˜ ρ

Original: ρ i  in Appendix E.3, with probability at least 1 − δ , we have
Revised: i described in Appendix D.3, with probability at least 1 − δ , we have

Original: Nash-Regret ( T ) ≤ (cid:101)

Original: [Equation]
Revised: [Equation]

Original:  Extension to Independent Markov Congestion Games
Revised:  Algorithms for Independent Markov Congestion Games

Original: In this section, we propose and analyze a Markov extension of the congestion games, called theindependent Markov congestion games (IMCGs).

Original: 6.1 Problem Formulation

Original: General-sum Markov Games.

Original: A finite-horizon time-inhomogeneous tabular general-sum Markovgame is defined by M = {S , {A i } mi =1 , H, P, R } , where S is the state space, m is the numberof players, A i is the action space of player i , A

Original: =

Original: A 1 × · · · × A m is the whole action space,

Original: H is the time horizon, P = ( P 1 , P 2 , · · · , P H ) with P h ∈ [0 , 1] S × A × S as the transition kernel attimestep h , R = { R h ( ·| s h , a h ) } H h
Revised: In this section, we present the theoretical guarantee of our centralized algorithms for independent

Original: =1 with R h ( ·| s h , a h ) as the reward distribution on [0 , r max ] m withmean r h ( s h , a h ) ∈ [0 , r max ] m at timestep h ∈

Original: [ H ] .

Original: At timestep h , all players choose their actionssimultaneously and a reward vector is sampled r h ∼ R h ( ·| s h , a h ) , where s h is the current stateand a h

Original: = ( a h, 1 , a h, 2 , · · · , a h,m ) is the joint action.

Original: Each player i receives reward r h,i and the statetransits to s h +1 ∼ P h ( ·| s h , a h ) .

Original: The objective for each player is to maximize her own total reward.

Original: We assume that the initial state s 1 is fixed.

Original: A (Markov) policy π is a collection of H functions { π h : S (cid:55)→ ∆( A ) } Hh =1 , each of which maps astate to a distribution over the action space.

Original: π is a product policy if π h ( · | s ) is a product policy foreach ( h, s ) ∈

Original: [ H ] × S .

Original: The value function and Q -value function of player i at timestep h underpolicy π are defined as

Original: [Equation]

Original: The best responses and Nash regret can be defined similarly as those for matrix games.

Original: In particular,given a policy π , player i ’s best response policy is π † h,i ( · | s ) = argmax µ ∈ ∆( A i ) V

Original: µ,π − i h,i ( s ) and thecorresponding value function is denoted as V † ,π − i h,i .

Original: Definition 3.

Original: With π k being the policy at k th episode, the Nash regret after K episodes is define as

Original: Nash-Regret ( K ) =

Original: K (cid:88) k

Original: = max i ∈ [ m ] (cid:18)

Original: V † ,π k − i 1 ,i − V

Original: π

Original: k 1 ,i (cid:19) ( s 1 ) .

Original: Independent Markov Congestion Game.
Revised: Markov congestion games.

Original: A general-sum Markov game is an independent Markovcongestion game (IMCG) if there exists a facility set F such that a i ⊆ F for any

Original: a i ∈

Original: A i , a statespace S =

Original: (cid:81)

Original: f ∈F S f , a set of facility reward distributions { R fh } h ∈

Original: [ H ] ,f ∈F such that if the jointaction at s h is a , we have r h,i = (cid:80)

Original: f ∈ a

Original: i r fh , where r fh ∼ R fh ( ·| s h , n f ( a )) with support on [0 , 1]and mean r fh ( s h , n f ( a )) , and a set of transition matrices { P fh } h ∈ [ H ] ,f ∈F such that P h ( s ′ | s, a ) =(cid:81)

Original: f ∈F P fh ( s ′ f | s f , n f ( a )) .

Original: In other words, at each timestep h and state s ∈ S , the players are in acongestion game.

Original: Meanwhile, each facility has its own state and independent state transition, whichonly depends on its current state and number of players using that facility.

Original: This transition kernel canbe viewed as a special case of that in factored MDPs [Szita and L˝orincz, 2009].

Original: The IMCG alsoadmits two types of feedback, semi-bandit feedback and bandit feedback, just like the congestiongame.

Original: In this paper, we will consider both types of feedback.

Original: 6.2 Theoretical Guarantee

Original: Summarized in Algorithm 4, our centralized algorithm for IMCGs is naturally extended from the Nash-UCB (Algorithm 1) by incorporating transition kernel estimators, corresponding bonus termsand Bellman backward update.
Revised: Summarized in Algorithm 4, it is naturally extended from Nash-UCB(Algorithm 1) by incorporating transition kernel estimators, corresponding bonus terms and Bellmanbackward update.

Original: The key idea is to utilize the independent transition structure toremove the dependence on the exponential size of the state space S =
Revised: The key idea is to utilize the independent transition structure to remove thedependence on the exponential size of the state space S =

Original: (cid:81)
Revised: (cid:81)

Original: f ∈F S f .
Revised: f ∈F S f .

Original: We tackle thisissue by adapting technique from factored MDP [Chen et al., 2020].
Revised: We tackle this issue byadapting technique from factored MDP [Chen et al., 2020].

Original: The algorithmic details for bothtypes of feedback are deferred into Appendix C.
Revised: Due to space constraint, the algorithmicdetails for both types of feedback are deferred into Appendix B.

Original: The Nash regret bounds for the two versions of Algorithm 4 are stated in Theorem 3 and the proof details are deferred to Appendix F.
Revised: The Nash regret bounds for the twoversions of Algorithm 4 are stated in Theorem 3 and the proof details are deferred to Appendix E.

Original: Theorem 3.
Revised: Theorem 3.

Original: For independent Markov congestion game with semi-bandit feedback, by running the centralized Algorithm 4, with probability at least 1 − δ , we can achieve that
Revised: For independent Markov congestion game with semi-bandit feedback, by running the centralized Algorithm 4, with probability at least 1 − δ , we can achieve that

Original: [Equation]
Revised: [Equation]

Original: Furthermore, if we only have bandit feedback, then by running Algorithm 4 with reward estimator and bonus term in (11) and (12) , with probability at least 1 − δ , we can achieve that
Revised: Furthermore, if we only have bandit feedback, then by running Algorithm 4 with reward estimator and bonus term in (11) and (12) , with probability at least 1 − δ , we can achieve that

Original: [Equation]
Revised: [Equation]

Revised: Remark 3 .

Original: The regret bound in [Liu et al., 2021] is (cid:101) O
Revised: The regret bound in [Liu et al., 2021] is (cid:101) O

Original: ( (cid:112) H 3 S 2 (Π mi =1 A i ) T ) , where both A i and S = (cid:81)
Revised: ( (cid:112) H 3 S 2 (Π mi =1 A i ) T ) , where both A i and S = (cid:81)

Original: f ∈F S fcan be exponential in F .
Revised: f ∈F S f can be exponential in F .

Original: Our bounds have polynomial dependence on all the parameters.
Revised: Our bounds have polynomial dependence on all theparameters.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we study sample-efficient learning in congestion games by utilizing the special rewardstructure.
Revised: In this paper, we study sample-efficient learning in congestion games by utilizing the special rewardstructure.

Original: We propose both centralized and decentralized algorithms for congestion games with twotypes of feedback,  all achieving sample complexities only polynomial in the number of facilities.
Revised: We propose both centralized and decentralized algorithms for congestion games withthe two types of feedback, namely semi-bandit feedback and bandit feedback, all achieving samplecomplexities only polynomial in the number of facilities, given that the size of the action space can beexponential in it.

Original: To the best of our knowledge, each one of them is the first sample-efficient learning algorithm forcongestion games in its own setting.
Revised: To the best of our knowledge, each one of them is the first sample-efficient learningalgorithm for congestion games in its own setting.

Original: We further define the independent Markov congestion game(IMCG) as a natural extension of the congestion game into the Markov setting together with asample-efficient centralized algorithm for both types of feedback.
Revised: We further define the independent Markov congestion game (IMCG) as a natural extension of thecongestion game into the Markov setting. We also propose centralized algorithms for IMCGs withboth types of feedback, which achieve sample complexities only polynomial in the number offacilities.

Original: One promising future direction is to find a sample-efficient decentralized algorithm such that fromeach player’s own perspective, the algorithm is still no-regret.
Revised: One promising future direction is to find a sample-efficient decentralized algorithm such that fromeach player’s own perspective, the algorithm is still no-regret.

Original: In other words, diminishing regret isguaranteed for the player by running this algorithm even though other players may use policies fromdifferent algorithms.
Revised: In other words, diminishing regret isguaranteed for the player by running this algorithm even though other players may use policies fromdifferent algorithms.

Original: Another important future direction is to find a sample-efficient decentralizedalgorithm for IMCGs.
Revised: Another important future direction is to find a sample-efficient decentralizedalgorithm for IMCGs.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [No] (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: [No](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]2.
Revised: [Yes]2.

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes](b) Did you include complete proofs of all theoretical results?
Revised: [Yes](b) Did you include complete proofs of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [N/A]
Revised: [N/A](b)

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
Revised: Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you report error bars (e
Revised: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experiments multiple times)?
Revised: g., with respect to the random seed after running experiments multiple times)?

Original: [N/A]
Revised: [N/A](d)

Original: (d) Did you include the total amount of compute and the type of resources used (e g., type of GPUs, internal cluster, or cloud provider)?
Revised: Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [N/A]
Revised: [N/A]

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/ curating contains personally identifiableinformation or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A]5.
Revised: [N/A]

Original: If youused crowdsourcing or conducted research with human subjects...
Revised: If you crowdsourcing or conducted research with human subjects.

Original: (a) 
Revised: (a) Did

Original: Did you include the full text of instructions given to participants and screenshots , if
Revised: you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A
Revised: [N/A]