Original: Stochastic Online Learning with Feedback Graphs: Finite-Time and Asymptotic Optimality
Revised: Stochastic Online Learning with Feedback Graphs: Finite-Time and Asymptotic Optimality

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: We revisit the problem of stochastic online learning with feedback graphs, with thegoal of devising algorithms that are optimal, up to constants, both asymptoticallyand in ﬁnite time.
Revised: We revisit the problem of stochastic online learning with feedback graphs, with thegoal of devising algorithms that are optimal, up to constants, both asymptoticallyand in ﬁnite time.

Original: We show that, surprisingly, the notion of optimal ﬁnite-timeregret is not a uniquely deﬁned property in this context and that, in general, it isdecoupled from the asymptotic rate.
Revised: We show that, surprisingly, the notion of optimal ﬁnite-timeregret is not a uniquely deﬁned property in this context and that, in general, it isdecoupled from the asymptotic rate.

Original: We discuss alternative choices and proposea notion of ﬁnite-time optimality that we argue is meaningful .
Revised: We discuss alternative choices and proposea notion of ﬁnite-time optimality that we argue is meaningful .

Original: For that notion,we give an algorithm that admits quasi-optimal regret both in ﬁnite-time andasymptotically.
Revised: For that notion,we give an algorithm that admits quasi-optimal regret both in ﬁnite-time andasymptotically.

Original:  Introduction
Revised:  Introduction

Original: Online learning is a sequential decision making game in which, at each round, the learner selects onearm (or expert) out of a ﬁnite set of K arms.
Revised: Online learning is a sequential decision making game in which, at each round, the learner selects onearm (or expert) out of a ﬁnite set of K arms.

Original: In the stochastic setting, each arm admits some rewarddistribution and the learner receives a reward drawn from the distribution corresponding to the armselected.
Revised: In the stochastic setting, each arm admits some rewarddistribution and the learner receives a reward drawn from the distribution corresponding to the armselected.

Original: In the bandit setting , the learner observes only that reward [Lai et al., 1985, Auer et al.
Revised: In the bandit setting , the learner observes only that reward [Lai et al., 1985, Auer et al.

Original: ,2002a,b], while in the full information setting , the rewards of all K arms are observed [Littlestoneand Warmuth, 1994, Freund and Schapire, 1997].
Revised: ,2002a,b], while in the full information setting , the rewards of all K arms are observed [Littlestoneand Warmuth, 1994, Freund and Schapire, 1997].

Original: Both settings are special instances of a more general model of online learning with side informationintroduced by Mannor and Shamir [2011], where the information supplied to the learner is speciﬁedby a feedback graph .
Revised: Both settings are special instances of a more general model of online learning with side informationintroduced by Mannor and Shamir [2011], where the information supplied to the learner is speciﬁedby a feedback graph .

Original: In an undirected feedback graph, each vertex represents an arm and an edgebetween between arm v and w indicates that the reward of w is observed when v is selected andvice-versa.
Revised: In an undirected feedback graph, each vertex represents an arm and an edgebetween between arm v and w indicates that the reward of w is observed when v is selected andvice-versa.

Original: The bandit setting corresponds to a graph reduced to self-loops at each vertex, the fullinformation to a fully connected graph.
Revised: The bandit setting corresponds to a graph reduced to self-loops at each vertex, the fullinformation to a fully connected graph.

Original: The problem of online learning with stochastic rewards andfeedback graphs has been studied by several publications in the last decade or so.
Revised: The problem of online learning with stochastic rewards andfeedback graphs has been studied by several publications in the last decade or so.

Original: The performance ofan algorithm in this problem is expressed in terms of its pseudo-regret, that is the different betweenthe expected reward achieved by always pulling the best arm and the expected cumulative rewardobtained by the algorithm.
Revised: The performance ofan algorithm in this problem is expressed in terms of its pseudo-regret, that is the different betweenthe expected reward achieved by always pulling the best arm and the expected cumulative rewardobtained by the algorithm.

Original: The UCB algorithm of Auer et al.
Revised: The UCB algorithm of Auer et al.

Original: [2002a] designed for the bandit setting forms a baseline for thisscenario.
Revised: [2002a] designed for the bandit setting forms a baseline for thisscenario.

Original: For general feedback graphs, Caron et al.
Revised: For general feedback graphs, Caron et al.

Original: [2012] designed a UCB -type algorithm, UCB - N ,as well as a closely related variant.
Revised: [2012] designed a UCB -type algorithm, UCB - N ,as well as a closely related variant.

Original: The pseudo-regret guarantee of UCB - N is expressed in terms ofthe most favorable clique covering of the graph, that is its partitioning into cliques.
Revised: The pseudo-regret guarantee of UCB - N is expressed in terms ofthe most favorable clique covering of the graph, that is its partitioning into cliques.

Original: This guarantee isalways at least as favorable as the bandit one [Auer et al., 2002a], which coincides with the speciﬁcchoice of the trivial clique covering.
Revised: This guarantee isalways at least as favorable as the bandit one [Auer et al., 2002a], which coincides with the speciﬁcchoice of the trivial clique covering.

Original: However, the bound depends on the ratio of the maximum andminimum mean reward gaps within each clique, which, in general, can be quite large.
Revised: However, the bound depends on the ratio of the maximum andminimum mean reward gaps within each clique, which, in general, can be quite large.

Original: Cohen et al.
Revised: Cohen et al.

Original: [2016] presented an action-elimination-type algorithm
Revised: [2016] presented an action-elimination-type algorithm

Original: [Even-Dar et al., 2006], whoseguarantee depends on the least favorable maximal independent set.
Revised: [Even-Dar et al., 2006], whoseguarantee depends on the least favorable maximal independent set.

Original: While there are instances inwhich this guarantee is worse compared to the bound presented in Caron et al.
Revised: While there are instances inwhich this guarantee is worse compared to the bound presented in Caron et al.

Original: [2012], in general it could be much more favorable compared to the clique partition guarantee of Caron et al.
Revised: [2012], in general it could be much more favorable compared to the clique partition guarantee of Caron et al.

Original: Thealgorithm of Cohen et al.
Revised: Thealgorithm of Cohen et al.

Original: [2016] does not require access to the full feedback graph, but only to theout-neighborhood of the arm selected at each round and the results also hold for time-varying graphs.
Revised: [2016] does not require access to the full feedback graph, but only to theout-neighborhood of the arm selected at each round and the results also hold for time-varying graphs.

Original: Later, Lykouris et al.
Revised: Later, Lykouris et al.

Original: [2020] presented an improved analysis of the UCB - N algorithm based on a newlayering technique, which showed that UCB - N beneﬁts, in fact, from a more favorable guaranteebased on the independence number of the graph, at the price of some logarithmic factors.
Revised: [2020] presented an improved analysis of the UCB - N algorithm based on a newlayering technique, which showed that UCB - N beneﬁts, in fact, from a more favorable guaranteebased on the independence number of the graph, at the price of some logarithmic factors.

Original: Theiranalysis also implied a similar guarantee for a variant of arm-elimination and Thompson sampling,as well as some improvement of the bound of Cohen et al.
Revised: Theiranalysis also implied a similar guarantee for a variant of arm-elimination and Thompson sampling,as well as some improvement of the bound of Cohen et al.

Original: [2016] in the case of a ﬁxed feedbackgraph.
Revised: [2016] in the case of a ﬁxed feedbackgraph.

Original: Buccapatnam et al.
Revised: Buccapatnam et al.

Original: [2014] gave an action-elimination-type algorithm
Revised: [2014] gave an action-elimination-type algorithm

Original: [Even-Dar et al., 2006],
Revised: [Even-Dar et al., 2006],

Original: UCB - LP , that leverages the solution of a linear-programming (LP) problem.
Revised: UCB - LP , that leverages the solution of a linear-programming (LP) problem.

Original: The guarantee presenteddepends only on the domination number of the graph, which can be substantially smaller than theindependence number.
Revised: The guarantee presenteddepends only on the domination number of the graph, which can be substantially smaller than theindependence number.

Original: A follow-up publication [Buccapatnam et al., 2017a] presents an analysis foran extension of the scenario of online learning with stochastic feedback graphs.
Revised: A follow-up publication [Buccapatnam et al., 2017a] presents an analysis foran extension of the scenario of online learning with stochastic feedback graphs.

Original: We will show that the algorithms just discussed do not achieve asymptotically optimal pseudo-regretguarantees and that it is also unclear how tight their ﬁnite-time instance-dependent bounds are.
Revised: We will show that the algorithms just discussed do not achieve asymptotically optimal pseudo-regretguarantees and that it is also unclear how tight their ﬁnite-time instance-dependent bounds are.

Original: Wuet al.
Revised: Wuet al.

Original: [2015] and Li et al.
Revised: [2015] and Li et al.

Original: [2020] proposed asymptotically optimal algorithms with matching lowerbounds.
Revised: [2020] proposed asymptotically optimal algorithms with matching lowerbounds.

Original: However, the corresponding ﬁnite-time regret guarantees are far from optimal and includeterms that can dominate the pseudo-regret for any reasonable time horizon.
Revised: However, the corresponding ﬁnite-time regret guarantees are far from optimal and includeterms that can dominate the pseudo-regret for any reasonable time horizon.

Original: We brieﬂy discuss other work related to online learning with feedback graphs.
Revised: We brieﬂy discuss other work related to online learning with feedback graphs.

Original: When rewards areadversarial, there has been a vast amount of work studying different settings for the feedback graphsuch as the graph evolving throughout the game or the graph not being observable before the start ofeach round [Alon et al., 2013, 2015, 2017].
Revised: When rewards areadversarial, there has been a vast amount of work studying different settings for the feedback graphsuch as the graph evolving throughout the game or the graph not being observable before the start ofeach round [Alon et al., 2013, 2015, 2017].

Original: The setting in which only noisy feedback is provided bythe graph is addressed in Kocák et al.
Revised: The setting in which only noisy feedback is provided bythe graph is addressed in Kocák et al.

Original: First order regret bounds, that is bounds which dependon the reward of the best arm, are derived in Lykouris et al.
Revised: First order regret bounds, that is bounds which dependon the reward of the best arm, are derived in Lykouris et al.

Original: [2018], Lee et al.
Revised: [2018], Lee et al.

Original: The settingof sleeping experts is studied in Cortes et al.
Revised: The settingof sleeping experts is studied in Cortes et al.

Original: Cortes et al.
Revised: Cortes et al.

Original: [2020] study stochastic rewardswhen the feedback graph evolves throughout the game, however, they do not assume that the rewardsand the graph are statistically independent.
Revised: [2020] study stochastic rewardswhen the feedback graph evolves throughout the game, however, they do not assume that the rewardsand the graph are statistically independent.

Original: Another instance in which the feedback and rewards arecorrelated is that of online learning with abstention [Cortes et al., 2018].
Revised: Another instance in which the feedback and rewards arecorrelated is that of online learning with abstention [Cortes et al., 2018].

Original: In this setting the player canchoose to abstain from making a prediction.
Revised: In this setting the player canchoose to abstain from making a prediction.

Original: The more general problem of Reinforcement Learningwith graph feedback has been studied by Dann et al.
Revised: The more general problem of Reinforcement Learningwith graph feedback has been studied by Dann et al.

Original: For additional work on online learningwith feedback graphs we recommend the survey of Valko [2016].
Revised: For additional work on online learningwith feedback graphs we recommend the survey of Valko [2016].

Original: We revisit the problem of stochastic online learning with feedback graphs, with the goal of devisingalgorithms that are optimal, up to constants, both asymptotically and in ﬁnite time.
Revised: We revisit the problem of stochastic online learning with feedback graphs, with the goal of devisingalgorithms that are optimal, up to constants, both asymptotically and in ﬁnite time.

Original: We show that,surprisingly, the notion of optimal ﬁnite-time regret is not a uniquely deﬁned property in this contextand that, in general, it is decoupled from the asymptotic rate.
Revised: We show that,surprisingly, the notion of optimal ﬁnite-time regret is not a uniquely deﬁned property in this contextand that, in general, it is decoupled from the asymptotic rate.

Original: Let T denote the time horizon and
Revised: Let T denote the time horizon and

Original: Reg A ( T ) the pseudo-regret of algorithm
Revised: Reg A ( T ) the pseudo-regret of algorithm

Original: A after T rounds.
Revised: A after T rounds.

Original: When A is clear from the context, we dropthe subscript.
Revised: When A is clear from the context, we dropthe subscript.

Original: It is known that c ∗ , the value of the LP considered by Buccapatnam et al.
Revised: It is known that c ∗ , the value of the LP considered by Buccapatnam et al.

Original: [2014], Wuet al.
Revised: [2014], Wuet al.

Original: [2015], Li et al.
Revised: [2015], Li et al.

Original: [2020], is asymptotically a lower bound for Reg A ( T )/ log ( T ) .
Revised: [2020], is asymptotically a lower bound for Reg A ( T )/ log ( T ) .

Original: We prove that noalgorithm A can achieve a ﬁnite-time pseudo-regret guarantee of the form Reg A ( T ) ≤ O ( c ∗ log ( T )) .
Revised: We prove that noalgorithm A can achieve a ﬁnite-time pseudo-regret guarantee of the form Reg A ( T ) ≤ O ( c ∗ log ( T )) .

Original: Moreover, we show that there exists a feedback graph G for which any algorithm suffers a regretof at least Ω ( K 1 8 ( c ∗  + 1 ∆ min )) , where ∆ min is the minimum reward gap.
Revised: Moreover, we show that there exists a feedback graph G for which any algorithm suffers a regret of atleast Ω ( K 1 8 ( c ∗ log ( T ) + [Equation] min )) , where ∆ min is the minimum reward gap.

Original: We discuss alternativechoices and propose a notion of ﬁnite-time optimality that we argue is meaningful , based on a regretquantity d ∗ that we show any algorithm must incur in the worst case.
Revised: We discuss alternativechoices and propose a notion of ﬁnite-time optimality that we argue is meaningful , based on a regretquantity d ∗ that we show any algorithm must incur in the worst case.

Original: For that notion, we give analgorithm whose pseudo-regret is quasi-optimal, both in ﬁnite-time and asymptotically and can beupper bounded by O ( c ∗ log ( T ) + d ∗ ) .
Revised: For that notion, we give analgorithm whose pseudo-regret is quasi-optimal, both in ﬁnite-time and asymptotically and can beupper bounded by O ( c ∗ log ( T ) + d ∗ ) .

Original:  Learning scenario
Revised:  Learning scenario

Original: We consider the problem of online learning with stochastic rewards and a ﬁxed undirected feedbackgraph.
Revised: We consider the problem of online learning with stochastic rewards and a ﬁxed undirected feedbackgraph.

Original: As in the familiar multi-armed bandit problem, the learner can choose one of K ≥ 1 arms.
Revised: As in the familiar multi-armed bandit problem, the learner can choose one of K ≥ 1 arms.

Original: Each arm i ∈
Revised: Each arm i ∈

Original: [ K ] admits a reward distribution, with mean µ i .
Revised: [ K ] admits a reward distribution, with mean µ i .

Original: For all our lower bounds, we assumethat the distribution of the reward of each arm is Gaussian with variance 1 / √ 2 .
Revised: For all our lower bounds, we assumethat the distribution of the reward of each arm is Gaussian with variance 1 / √ 2 .

Original: For our upper bounds,we only assume that the distribution of each arm is sub-Gaussian with variance proxy bounded by1 .
Revised: For our upper bounds,we only assume that the distribution of each arm is sub-Gaussian with variance proxy bounded by1 .

Original: We assume that the means are always bounded in [ 0 , 1 ] .
Revised: We assume that the means are always bounded in [ 0 , 1 ] .

Original: For arm i , we denote by ∆ i = µ ∗ − µ iits mean gap to the best µ ∗ =
Revised: For arm i , we denote by ∆ i = µ ∗ − µ iits mean gap to the best µ ∗ =

Original: max i ∈
Revised: max i ∈

Original: [ K ] µ i .
Revised: [ K ] µ i .

Original: We will also denote by ∆ min the smallest and by ∆ maxthe largest of these gaps.
Revised: We will also denote by ∆ min the smallest and by ∆ maxthe largest of these gaps.

Original: At each round t ∈ [ T ] , the learner selects an arm i t and receives a rewardr t,i t drawn from the reward distribution of arm i t .
Revised: At each round t ∈ [ T ] , the learner selects an arm i t and receives a rewardr t,i t drawn from the reward distribution of arm i t .

Original: In addition to observing that reward, the learner observes the reward of some other arms, as speciﬁed by an undirected graph G =
Revised: In addition to observing that reward, the learner observes the reward of some other arms, as speciﬁed by an undirected graph G =

Original: ( V,E ) , where thevertex set V coincides with [ K ] : an edge e ∈ E between vertices i and j indicates that the learnerobserves the reward of arm j when selecting arm i and vice-versa.
Revised: ( V,E ) , where thevertex set V coincides with [ K ] : an edge e ∈ E between vertices i and j indicates that the learnerobserves the reward of arm j when selecting arm i and vice-versa.

Original: We will denote by N i the set ofneighbors of arm i in G , N i = { j ∈ V ∶( i,j ) ∈ E } , and will assume self-loops at every vertex, that is,we have i ∈ N i for all i ∈ V .
Revised: We will denote by N i the set ofneighbors of arm i in G , N i = { j ∈ V ∶( i,j ) ∈ E } , and will assume self-loops at every vertex, that is,we have i ∈ N i for all i ∈ V .

Original: The objective of the learner A is to minimize its pseudo-regret , that isthe expected cumulative gap between the reward of an optimal arm i ∗ and its reward:
Revised: The objective of the learner A is to minimize its pseudo-regret , that isthe expected cumulative gap between the reward of an optimal arm i ∗ and its reward:

Original: [Equation]
Revised: [Equation]

Original: where the expectation is taken over the random draw of a reward from an arm’s distribution and thepossibly randomized selection strategy of the learner.
Revised: where the expectation is taken over the random draw of a reward from an arm’s distribution and thepossibly randomized selection strategy of the learner.

Original: In the following, we may sometimes abusivelyuse the shorter term regret instead of pseudo-regret.
Revised: In the following, we may sometimes abusivelyuse the shorter term regret instead of pseudo-regret.

Original: We will denote by I ∗ the set of optimal arms,that is, arms with mean reward µ ∗ , and, for any t ∈
Revised: We will denote by I ∗ the set of optimal arms,that is, arms with mean reward µ ∗ , and, for any t ∈

Original: [ T ] will denote by r t the vector of all rewards r t,iat time t .
Revised: [ T ] will denote by r t the vector of all rewards r t,iat time t .

Original: When discussing asymptotic or ﬁnite-time optimality, we assume the setting of Gaussianrewards.
Revised: When discussing asymptotic or ﬁnite-time optimality, we assume the setting of Gaussianrewards.

Original: We will assume an informed setting where the graph G is ﬁxed and accessible to the learner before thestart of the game.
Revised: We will assume an informed setting where the graph G is ﬁxed and accessible to the learner before thestart of the game.

Original: Our analysis makes use of the following standard graph theory notions [Goddardand Henning, 2013].
Revised: Our analysis makes use of the following standard graph theory notions [Goddardand Henning, 2013].

Original: A subset of the vertices is independent if no two vertices in it are adjacent.
Revised: A subset of the vertices is independent if no two vertices in it are adjacent.

Original: Theindependence number of G , α ( G ) , is the size of the maximum independent set in G .
Revised: Theindependence number of G , α ( G ) , is the size of the maximum independent set in G .

Original: A dominatingset of G is a subset S ⊆ V such that every vertex not in S is adjacent to S .
Revised: A dominatingset of G is a subset S ⊆ V such that every vertex not in S is adjacent to S .

Original: The domination numberof G , γ ( G ) , is the minimum size of a dominating set.
Revised: The domination numberof G , γ ( G ) , is the minimum size of a dominating set.

Original: It is known that for any graph G , we haveγ ( G ) ≤ α ( G ) .
Revised: It is known that for any graph G , we haveγ ( G ) ≤ α ( G ) .

Original: The difference between the domination and independence numbers can be substantialin many cases.
Revised: The difference between the domination and independence numbers can be substantialin many cases.

Original: For example, for a star graph with n vertices, we have γ ( G ) =
Revised: For example, for a star graph with n vertices, we have γ ( G ) =

Original: 1 and α ( G )
Revised: 1 and α ( G )

Original: =
Revised: =

Original: n − 1 .
Revised: n − 1 .

Original: Inthe following, in the absence of any ambiguity, we simply drop the graph arguments and write α orγ .
Revised: Inthe following, in the absence of any ambiguity, we simply drop the graph arguments and write α orγ .

Original: We will denote by D( G ′ )
Revised: We will denote by D( G ′ )

Original: the minimum dominating set of a sub-graph G ′ ⊆
Revised: the minimum dominating set of a sub-graph G ′ ⊆

Original: G and by I( G ′ )
Revised: G and by I( G ′ )

Original: themaximum independent set.
Revised: themaximum independent set.

Original: When the minimum dominating set is not unique, D( G ′ ) can be selectedin an arbitrary but ﬁxed way.
Revised: When the minimum dominating set is not unique, D( G ′ ) can be selectedin an arbitrary but ﬁxed way.

Original:  Sub-optimality of previous algorithms
Revised:  Sub-optimality of previous algorithms

Original: In this section, we discuss in more detail the previous work the most closely related to ours [Buccapatnam et al., 2014, Wu et al., 2015, Buccapatnam et al., 2017b, Li et al., 2020] and demonstrate theirsub-optimality.
Revised: In this section, we discuss in more detail the previous work the most closely related to ours [Buccapatnam et al., 2014, Wu et al., 2015, Buccapatnam et al., 2017b, Li et al., 2020] and demonstrate theirsub-optimality.

Original: These algorithms all seek to achieve instance-dependent optimal regret bounds bysolving and playing according to the following linear program (LP), which is known to characterizethe instance-dependent asymptotic regret for this problem when the rewards follow a Gaussiandistribution:
Revised: These algorithms all seek to achieve instance-dependent optimal regret bounds bysolving and playing according to the following linear program (LP), which is known to characterizethe instance-dependent asymptotic regret for this problem when the rewards follow a Gaussiandistribution:

Original: [Equation]
Revised: [Equation]

Original: We note that these prior work algorithms can work in more general settings, but we will restrict ourdiscussion to their use in the informed setting with a ﬁxed feedback graph that we consider in thisstudy.
Revised: We note that these prior work algorithms can work in more general settings, but we will restrict ourdiscussion to their use in the informed setting with a ﬁxed feedback graph that we consider in thisstudy.

Original: The UCB - LP algorithm of Buccapatnam et al.
Revised: The UCB - LP algorithm of Buccapatnam et al.

Original: [2014, 2017b] is based on the following modiﬁcationof LP1: min x ∈ R K + ⟨ x, 1 ⟩ subject to ∑ j ∈ N i x j ≥ 1 , for all i ∈
Revised: [2014, 2017b] is based on the following modiﬁcationof LP1: min x ∈ R K + ⟨ x, 1 ⟩ subject to ∑ j ∈ N i x j ≥ 1 , for all i ∈

Original: [ K ] , in which the gap informationis eliminated, working with gaps such that ∆ min =
Revised: [ K ] , in which the gap informationis eliminated, working with gaps such that ∆ min =

Original: Θ ( ∆ max ) .
Revised: Θ ( ∆ max ) .

Original: This modiﬁed problem is the LPrelaxation of the minimum dominating set integer program of graph G .
Revised: This modiﬁed problem is the LPrelaxation of the minimum dominating set integer program of graph G .

Original: The algorithm ﬁrst solves this minimum dominating set relaxation and then proceeds as an actionelimination-algorithm in O ( log ( T )) phases.
Revised: The algorithm ﬁrst solves this minimum dominating set relaxation and then proceeds as an actionelimination-algorithm in O ( log ( T )) phases.

Original: During the ﬁrst O ( log ( K )) rounds, their algorithmplays by exploring based on the solution of their LP.
Revised: During the ﬁrst O ( log ( K )) rounds, their algorithmplays by exploring based on the solution of their LP.

Original: Once the exploration rounds have concluded,it simply behaves as a bandit action-elimination algorithm.
Revised: Once the exploration rounds have concluded,it simply behaves as a bandit action-elimination algorithm.

Original: We argue below that this algorithm issub-optimal, in at least two ways.
Revised: We argue below that this algorithm issub-optimal, in at least two ways.

Original: Star graph with equal gaps.
Revised: Star graph with equal gaps.

Original: Consider the case where the feedback graph is a star graph (Figure 1(a)): there is one root or revealing vertex r adjacent to all other vertices.
Revised: Consider the case where the feedback graph is a star graph (Figure 1(a)): there is one root or revealing vertex r adjacent to all other vertices.

Original: In our construction, theoptimal arm is chosen uniformly at random among the leaves of the graph.
Revised: In our construction, theoptimal arm is chosen uniformly at random among the leaves of the graph.

Original: The rewards are chosen sothat all sub-optimal arms admit the same expected reward with gap to the best ∆ ≤ O ( 1 / K 1 + (cid:15) ) , (cid:15) > 0 .
Revised: The rewards are chosen sothat all sub-optimal arms admit the same expected reward with gap to the best ∆ ≤ O ( 1 / K 1 + (cid:15) ) , (cid:15) > 0 .

Original: In this case, an optimal strategy consists of playing the revealing arm for Θ ( 1 / ∆ 2 ) rounds to identify
Revised: In this case, an optimal strategy consists of playing the revealing arm for Θ ( 1 / ∆ 2 ) rounds to identify

Original: [Figure]
Revised: [Figure]

Original: the optimal arm, and thus incurs regret at most O ( 1∆ ) .
Revised: the optimal arm, and thus incurs regret at most O ( 1∆ ) .

Original: On the other hand the UCB - LP strategy incursregret at least Ω ( K log ( T ) ∆ ) .
Revised: On the other hand the UCB - LP strategy incursregret at least Ω ( K log ( T ) ∆ ) .

Original: Even if we ignore the dependence on the time horizon, the dependenceon K is clearly sub-optimal.
Revised: Even if we ignore the dependence on the time horizon, the dependenceon K is clearly sub-optimal.

Original: Sub-optimality of using the minimum dominating set relaxation.
Revised: Sub-optimality of using the minimum dominating set relaxation.

Original: In the second problem instance,given in Figure 1(b), we consider a star-like graph in which we have a revealing vertex r 1 , adjacentto all other vertices.
Revised: In the second problem instance,given in Figure 1(b), we consider a star-like graph in which we have a revealing vertex r 1 , adjacentto all other vertices.

Original: We also have an "almost" revealing vertex r 2 which is adjacent to all verticesbut a single leaf vertex (leaves are the vertices with degree 1 and 2 in this case).
Revised: We also have an "almost" revealing vertex r 2 which is adjacent to all verticesbut a single leaf vertex (leaves are the vertices with degree 1 and 2 in this case).

Original: The optimal armis again chosen uniformly among the leaves.
Revised: The optimal armis again chosen uniformly among the leaves.

Original: Rewards are set so that the gap at r 1 is ∆ max and theremaining gaps are ∆ min .
Revised: Rewards are set so that the gap at r 1 is ∆ max and theremaining gaps are ∆ min .

Original: The solution to the LP of Buccapatnam et al.
Revised: The solution to the LP of Buccapatnam et al.

Original: [2014, 2017b] puts all theweights on r 1 .
Revised: [2014, 2017b] puts all theweights on r 1 .

Original: However, the optimal policy for this problem consists of playing r 2 and the leaf vertexnot adjacent to r 2 until all arms but the optimal arm are eliminated.
Revised: However, the optimal policy for this problem consists of playing r 2 and the leaf vertexnot adjacent to r 2 until all arms but the optimal arm are eliminated.

Original: The instance optimal regret inthis case is O ( log ( T ) ∆ min ) , while UCB - LP incurs regret Ω ( K log ( T ) ∆ min ) .
Revised: The instance optimal regret inthis case is O ( log ( T ) ∆ min ) , while UCB - LP incurs regret Ω ( K log ( T ) ∆ min ) .

Original: Next, we discuss [Wu et al., 2015] and [Li et al., 2020].
Revised: Next, we discuss [Wu et al., 2015] and [Li et al., 2020].

Original: Their instance-dependent algorithms arebased on iteratively solving empirical approximations to LP1.
Revised: Their instance-dependent algorithms arebased on iteratively solving empirical approximations to LP1.

Original: For simplicity, we only discuss theinstance-dependent regret bound in [Li et al., 2020].
Revised: For simplicity, we only discuss theinstance-dependent regret bound in [Li et al., 2020].

Original: A similar bound can be found in [Wu et al.
Revised: A similar bound can be found in [Wu et al.

Original: Let x ( ∆ ) denote the solution of LP1 and deﬁne the following perturbed solution
Revised: Let x ( ∆ ) denote the solution of LP1 and deﬁne the following perturbed solution

Original: [Equation]
Revised: [Equation]

Original: The solution x ( ∆ ,(cid:15) ) is the solution of LP1 with (cid:15) -perturbed gaps.
Revised: The solution x ( ∆ ,(cid:15) ) is the solution of LP1 with (cid:15) -perturbed gaps.

Original: [Li et al., 2020][Theorem 4] states that the expected regret of their algorithm is bounded as follows:
Revised: [Li et al., 2020][Theorem 4] states that the expected regret of their algorithm is bounded as follows:

Original: Reg ( T ) ≤
Revised: Reg ( T ) ≤

Original: O (∑ i ∈[ K ] log ( T ) x i ( ∆ ,(cid:15) )
Revised: O (∑ i ∈[ K ] log ( T ) x i ( ∆ ,(cid:15) )

Original: ∆ i + ∑ Tt = 1 exp (− β ( t ) (cid:15) 2 K ) + K ) , for any (cid:15) > 0 and β ( t ) =
Revised: ∆ i + ∑ Tt = 1 exp (− β ( t ) (cid:15) 2 K ) + K ) , for any (cid:15) > 0 and β ( t ) =

Original: o ( t ) .
Revised: o ( t ) .

Original: For thestandard bandit problem with Gaussian rewards, we can compute the perturbed solution: x i ( ∆ ,(cid:15) )
Revised: For thestandard bandit problem with Gaussian rewards, we can compute the perturbed solution: x i ( ∆ ,(cid:15) )

Original: =
Revised: =

Original: max ( 1 ( ∆ i + (cid:15) ) 2 , 1 ( ∆ i − (cid:15) ) 2 ) .
Revised: max ( 1 ( ∆ i + (cid:15) ) 2 , 1 ( ∆ i − (cid:15) ) 2 ) .

Original: Thus, for a meaningful regret bound, we would need (cid:15) =
Revised: Thus, for a meaningful regret bound, we would need (cid:15) =

Original: Θ ( ∆ min ) .
Revised: Θ ( ∆ min ) .

Original: If (cid:15) ismuch smaller, then the term ∑ Tt = 1 exp (− β ( t ) (cid:15) 2 K ) becomes too large and otherwise we risk makingx i ( ∆ ,(cid:15) ) too large.
Revised: If (cid:15) ismuch smaller, then the term ∑ Tt = 1 exp (− β ( t ) (cid:15) 2 K ) becomes too large and otherwise we risk makingx i ( ∆ ,(cid:15) ) too large.

Original: To analyze the second term more carefully, we allow β ( t ) =
Revised: To analyze the second term more carefully, we allow β ( t ) =

Original: t .
Revised: t .

Original: The ﬁrst K ∆ 2min termsof ∑ T t = 1 exp ( − β ( t ) (cid:15) 2 K ) are now at least 1 e and thus this sum is at least ∑ T t = 1 exp ( − β ( t ) (cid:15) 2 K ) ≥ Ke ∆ 2min .
Revised: The ﬁrst K ∆ 2minterms of ∑ T t = 1 exp ( − β ( t ) (cid:15) 2 K ) are now at least 1 e and thus this sum is at least ∑ T t = 1 exp ( − β ( t ) (cid:15) 2 K ) ≥ K ∆ 2min .

Original: Thus, the bandit regret bound evaluates to at least Ω (∑ i ∈[ K ] log ( T ) ∆ i + K ∆ 2min ) .
Revised: Thus, the bandit regret bound evaluates to at least Ω (∑ i ∈[ K ] log ( T ) ∆ i + K ∆ 2min ) .

Original: While this bound isasymptotically optimal, since the second term does not have dependence on T , it admits a very poordependence on the smallest gap.
Revised: While this bound isasymptotically optimal, since the second term does not have dependence on T , it admits a very poordependence on the smallest gap.

Original: We can repeat the argument above with a star-graph construction inwhich the revealing vertex has gap ∆ min .
Revised: We can repeat the argument above with a star-graph construction inwhich the revealing vertex has gap ∆ min .

Original: In this case, the optimal strategy given by the solution to
Revised: In this case, the optimal strategy given by the solution to

Original: LP1 consists of playing the revealing vertex for 1∆ 2min times and incurs regret at most O ( log ( T ) ∆ min ) .
Revised: LP1 consists of playing the revealing vertex for 1∆ 2min times and incurs regret at most O ( log ( T ) ∆ min ) .

Original: Theregret bound of the algorithm of Li et al.
Revised: Theregret bound of the algorithm of Li et al.

Original: [2020], however, amounts to Ω ( log ( T ) ∆ min + K ∆ 2min ) .
Revised: [2020], however, amounts to Ω ( log ( T ) ∆ min + K ∆ 2min ) .

Original:  Instance-dependent ﬁnite-time bounds
Revised:  Instance-dependent ﬁnite-time bounds

Original: In this section, we provide an in-depth discussion of what ﬁnite-time optimality actually means.
Revised: In this section, we provide an in-depth discussion of what ﬁnite-time optimality actually means.

Original: Finite-time bounds are statements of the form Reg ( T ) ≤ f ( T ) , which hold for any T > 0 .
Revised: Finite-time bounds are statements of the form Reg ( T ) ≤ f ( T ) , which hold for any T > 0 .

Original: Speciﬁcally,we are considering functions of the type f ( T ) =
Revised: Speciﬁcally,we are considering functions of the type f ( T ) =

Original: c ∗ log ( T ) + d, 1 which we know to exist from prior work.
Revised: c ∗ log ( T ) + d, 1 which we know to exist from prior work.

Original: The question of what the optimal expression of d might be seems easy to answer at ﬁrst.
Revised: The question of what the optimal expression of d might be seems easy to answer at ﬁrst.

Original: Indeed, for bandits with Gaussian rewards, one can achieve d = O (∑ i ∆ i ) , which in general is muchsmaller than c ∗ =
Revised: Indeed, for bandits with Gaussian rewards, one can achieve d = O (∑ i ∆ i ) , which in general is muchsmaller than c ∗ =

Original: Ω (∑ i 1∆ i )
Revised: Ω (∑ i 1∆ i )

Original: [Lattimore and Szepesvári, 2020], and hence will be dominated by thetime-dependent part of the regret for almost all reasonable lengths of the time horizon T .
Revised: [Lattimore and Szepesvári, 2020], and hence will be dominated by thetime-dependent part of the regret for almost all reasonable lengths of the time horizon T .

Original: In fullinformation, we obtain a meaningful optimal value d ∗ for a given gap vector by considering theworst-case regret of any algorithm under any permutation of the arms.
Revised: In fullinformation, we obtain a meaningful optimal value d ∗ for a given gap vector by considering theworst-case regret of any algorithm under any permutation of the arms.

Original: This leads to d ∗ =
Revised: This leads to d ∗ =

Original: O ( ln ( K ) ∆ min )
Revised: O ( ln ( K ) ∆ min )

Original: [Mourtada and Gaïffas, 2019].
Revised: [Mourtada and Gaïffas, 2019].

Original: Note that in the full information setting we have c ∗ = 0 .
Revised: Note that in the full information setting we have c ∗ = 0 .

Original: One might hope for a similar structure for feedback graphs, where the optimal d depends only onthe “full-information structure”, that is the gaps of arms neighboring an optimal arm.
Revised: One might hope for a similar structure for feedback graphs, where the optimal d depends only onthe “full-information structure”, that is the gaps of arms neighboring an optimal arm.

Original: All other armscontribute to c ∗ and we might assume that their complexity is already captured in the c ∗ log ( T ) termas it is the case for bandits.
Revised: All other armscontribute to c ∗ and we might assume that their complexity is already captured in the c ∗ log ( T ) termas it is the case for bandits.

Original: However, the situation is more complicated, as we show next.
Revised: However, the situation is more complicated, as we show next.

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: For anyK ≥ 32 and ∆ min = O( 1 √ K) , there exists a graph G with K vertices andan instance with  unique best arm and minimal gap ∆ min , such that for any algorithm and any
Revised: For any ∆ min and K , there exists a graph G such that for any algorithm, there existsan instance with a unique optimal arm, such that the regret of the algorithm satisﬁes

Original:  T ≥ Ω ( K 3 / 4 ∆ min ), the regret is at least Reg ( T ) c ∗ + 1∆min = Ω ( K 18 ) .
Revised: [Table] =˜Ω ( K 18 ) , for any T ≥ Ω ( K 3 / 4 ∆ min ) .

Original: Theorem 4.1 shows that there exists a problem instance, in which d ≫ c ∗ dominates the ﬁnitetime regret for any T ≤ O ( exp ( K 1 / 8 )) .
Revised: Theorem 4.1 shows that there exists a problem instance, in which d ≫ c ∗ dominates the ﬁnitetime regret for any T ≤ O ( exp ( K 1 / 8 )) .

Original: We note that this is not simply due to the full-informationstructure of the feedback graph G , as c ∗ is positive.
Revised: We note that this is not simply due to the full-informationstructure of the feedback graph G , as c ∗ is positive.

Original: Furthermore, our results suggest that there isno simple characterization of d in terms of c ∗ , e.g., d = Θ ( c ∗ / ∆ min ) .
Revised: Furthermore, our results suggest that there isno simple characterization of d in terms of c ∗ , e.g., d = Θ ( c ∗ / ∆ min ) .

Original: As shown in Section 7, thereexists a non-trivial family of graphs, for which for any rewards instance, we have d = O ( c ∗ ) .
Revised: As shown in Section 7, thereexists a non-trivial family of graphs, for which for any rewards instance, we have d = O ( c ∗ ) .

Original: Having established that d could be the dominating term in the regret for any reasonable time horizon,we now discuss the hardness of deﬁning an optimal d .
Revised: Having established that d could be the dominating term in the regret for any reasonable time horizon,we now discuss the hardness of deﬁning an optimal d .

Original: Let us ﬁrst consider a simple two-arm fullinformation problem and inspect algorithms of the style: “Play arm 1, unless the cumulative rewardof arm 2 exceeds that of arm 1 by a threshold of τ .”
Revised: Let us ﬁrst consider a simple two-arm fullinformation problem and inspect algorithms of the style: “Play arm 1, unless the cumulative rewardof arm 2 exceeds that of arm 1 by a threshold of τ .”

Original: This kind of algorithm has small regret (smalld ) if arm 1 is optimal, and large otherwise.
Revised: This kind of algorithm has small regret (smalld ) if arm 1 is optimal, and large otherwise.

Original: Tuning τ yields different trade-offs between the twoscenarios.
Revised: Tuning τ yields different trade-offs between the twoscenarios.

Original: The same issue appears in learning with graph feedback on a larger scale.
Revised: The same issue appears in learning with graph feedback on a larger scale.

Original: Given twoinstances deﬁned by gap vectors ∆ and ∆ ′ respectively, an agent can trade off the constant regretpart d ( ∆ ) and d ( ∆ ′ ) in the two instances.
Revised: Given twoinstances deﬁned by gap vectors ∆ and ∆ ′ respectively, an agent can trade off the constant regretpart d ( ∆ ) and d ( ∆ ′ ) in the two instances.

Original: Take for example two algorithms A and B and assumethe respective values of c ∗ and d for the two instances and algorithms are given by Table 1.
Revised: Take for example two algorithms A and B and assumethe respective values of c ∗ and d for the two instances and algorithms are given by Table 1.

Original: As weshow in the Appendix A, there exist indeed a feedback graph and instances that are consistent withthe table.
Revised: As weshow in the Appendix A, there exist indeed a feedback graph and instances that are consistent withthe table.

Original: Which algorithm is more “optimal”, A or B ?
Revised: Which algorithm is more “optimal”, A or B ?

Original: B ensures that max δ ∈{ ∆ , ∆ ′ } d ( δ ) c ∗
Revised: B ensures that max δ ∈{ ∆ , ∆ ′ } d ( δ ) c ∗

Original: =
Revised: =

Original: O( 1 ) and c ∗ d for Alg.
Revised: O( 1 ) and c ∗ d for Alg.

Original: A d for Alg.
Revised: A d for Alg.

Original: B
Revised: B

Original: Instance
Revised: Instance

Original: [Table]
Revised: [Table]

Original: 202 we can write the regret function as f ( T ) =
Revised: 202 we can write the regret function as f ( T ) =

Original: O( c ∗ log ( T )) without the need of a constant term d at all.
Revised: O( c ∗ log ( T )) without the need of a constant term d at all.

Original: This algorithm minimizes the competitive ratio of d and c ∗ .
Revised: This algorithm minimizes the competitive ratio of d and c ∗ .

Original: A minimizes the worst-case absoluteregret A = argmin a ∈{A , B} max δ ∈{ ∆ , ∆ ′ } d ( δ,a ) .
Revised: A minimizes the worst-case absoluteregret A = argmin a ∈{A , B} max δ ∈{ ∆ , ∆ ′ } d ( δ,a ) .

Original: Thus, we argue that the notion of optimality is subject to a choice and that there is no uniquecorrect answer.
Revised: Thus, we argue that the notion of optimality is subject to a choice and that there is no uniquecorrect answer.

Original: In this paper, we opt for the worst-case absolute regret, minimized by A in theabove example, for the following reasons: 1.
Revised: In this paper, we opt for the second choice, a minimax notion of optimality, for thefollowing reasons: 1.

Original: Theorem 4.1 shows that a constant competitive ratio isgenerally unachievable;2.
Revised: Theorem 4.1 shows that a constant competitive ratio is generally unachievable;

Original: Optimizing regret in general is a different objective than that of competitiveratio.
Revised: Optimizing regret in general is a different objective than that of competitive ratio.

Original: Optimizing for a mixture implies a counter-intuitive preferences such as: “In a hard environmentwhere I cannot avoid suffering a loss of 1000, it does not matter much if I suffer an additional 1000on top, as long as I do better on easier environments.”
Revised: Optimizing for amixture implies a counter-intuitive preferences such as: “In a hard environment where I cannot avoidsuffering a loss of 1000, it does not matter much if I suffer an additional 1000 on top, as long as I dobetter on easier environments.”

Original: Moreover, note that, even if one wereinterested in optimizing the competitive ratio between c ∗ and d , it is unclear if one could achieve thatcomputationally efﬁciently.
Revised: Moreover, note that, even if one were interested in optimizing thecompetitive ratio between c ∗ and d , it is unclear if one could achieve that computationally efﬁciently.

Original: We present our ﬁnal deﬁnition for an optimal notion of d ∗ in Section 6.
Revised: We present our ﬁnal deﬁnition for an optimal notion of d ∗ in Section 6.

Original: The high level idea is to takeall confusing instances, where the means are perturbed by less than ∆ s and consider the worst-caseregret any algorithm suffers over these instances until identifying all gaps up to ∆ s precision.
Revised: The high level idea is to takeall confusing instances, where the means are perturbed by less than ∆ s and consider the worst-caseregret any algorithm suffers over these instances until identifying all gaps up to ∆ s precision.

Original:  Algorithm and regret upper bounds
Revised:  Algorithm and regret upper bounds

Original: Our algorithm works by approximating the gaps ( ∆ i ) i ∈
Revised: Our algorithm works by approximating the gaps ( ∆ i ) i ∈

Original: [ K ] and then solving a version of LP1.
Revised: [ K ] and then solving a version of LP1.

Original: First,note that all arms i with gaps ∆ i ≤ 1 T can be ignored as the total contribution to the regret is atmost O ( 1 ) .
Revised: First,note that all arms i with gaps ∆ i ≤ 1 T can be ignored as the total contribution to the regret is atmost O ( 1 ) .

Original: We now segment the interval [ 1 T , 1 ] , containing each relevant gap, into sub-intervals[ 2 − s , 2 − s + 1 ] , where s ∈ [⌈ log 2 ( T )⌉] .
Revised: We now segment the interval [ 1 T , 1 ] , containing each relevant gap, into sub-intervals[ 2 − s , 2 − s + 1 ] , where s ∈ [⌈ log 2 ( T )⌉] .

Original: The algorithm now proceeds in phases corresponding to eachof the ⌈ log 2 ( T )⌉ intervals.
Revised: The algorithm now proceeds in phases corresponding to eachof the ⌈ log 2 ( T )⌉ intervals.

Original: During phase s , all arms with gaps ∆ i ∈ [ 2 − s , 2 − s + 1 ] will be observedsufﬁciently many times to be identiﬁed as sub-optimal.
Revised: During phase s , all arms with gaps ∆ i ∈ [ 2 − s , 2 − s + 1 ] will be observedsufﬁciently many times to be identiﬁed as sub-optimal.

Original: For phase s , let ∆ s = 2 − s denote the smallest possible gap that can be part of the interval[ 2 − s , 2 − s + 1 ] and deﬁne the clipped gap vector as ∆ s ∈ R K , ∆ si = ∆ s ∨ ∆ i .
Revised: For phase s , let ∆ s = 2 − s denote the smallest possible gap that can be part of the interval[ 2 − s , 2 − s + 1 ] and deﬁne the clipped gap vector as ∆ s ∈ R K , ∆ si = ∆ s ∨ ∆ i .

Original: Further, deﬁne theset Γ s = { i ∈
Revised: Further, deﬁne theset Γ s = { i ∈

Original: [ K ]∶ ∆ i ≤ 2∆ s } .
Revised: [ K ]∶ ∆ i ≤ 2∆ s } .

Original: Γ s consists of all optimal arms I ∗ and all sub-optimal arms with gapssmall enough, making them impossible to distinguish from optimal arms.
Revised: Γ s consists of all optimal arms I ∗ and all sub-optimal arms with gapssmall enough, making them impossible to distinguish from optimal arms.

Original: Deﬁne the following LP
Revised: Deﬁne the following LP

Original: [Equation]
Revised: [Equation]

Original: For any arm i such that ∆ i ∈
Revised: For any arm i such that ∆ i ∈

Original: [ 2 − s , 2 − s + 1 ] observing i for 1∆ 2 s times is sufﬁcient to identify i as asub-optimal arm.
Revised: [ 2 − s , 2 − s + 1 ] observing i for 1∆ 2 s times is sufﬁcient to identify i as asub-optimal arm.

Original: Further, information theory dictates that i needs to be observed at least 1∆ 2 s − 1 timesto be distinguished as sub-optimal.
Revised: Further, information theory dictates that i needs to be observed at least 1∆ 2 s − 1 timesto be distinguished as sub-optimal.

Original: Thus, the constraints of LP2 are necessary and sufﬁcient foridentifying the sub-optimal arms i with ∆ i ∈ [ 2 − s , 2 − s + 1 ] .
Revised: Thus, the constraints of LP2 are necessary and sufﬁcient foridentifying the sub-optimal arms i with ∆ i ∈ [ 2 − s , 2 − s + 1 ] .

Original: Furthermore, since there is no sufﬁcientinformation to distinguish between any two arms i and j with gaps ∆ i ≤ ∆ j < ∆ s , we choose to treatall of them as equal in the objective of the LP.
Revised: Furthermore, since there is no sufﬁcientinformation to distinguish between any two arms i and j with gaps ∆ i ≤ ∆ j < ∆ s , we choose to treatall of them as equal in the objective of the LP.

Original: Indeed, Lemma 6.1 shows that for any graph G andany algorithm, there exists an assignment of the gaps ∆ i <
Revised: Indeed, Lemma 6.1 shows that for any graph G andany algorithm, there exists an assignment of the gaps ∆ i <

Original: ∆ s so that the algorithm will suffer regretproportional to the value of LP2.
Revised: ∆ s so that the algorithm will suffer regretproportional to the value of LP2.

Original: In practice, it is impossible to devise an algorithm that solves and plays according to LP2 because evenduring phase s , there is still no complete knowledge of the gaps ∆ i > ∆ s , but, rather only empiricalestimators, and so there is no access to ∆ s .
Revised: In practice, it is impossible to devise an algorithm that solves and plays according to LP2 because evenduring phase s , there is still no complete knowledge of the gaps ∆ i > ∆ s , but, rather only empiricalestimators, and so there is no access to ∆ s .

Original: We also replace the constraints by a conﬁdence intervalterm of the order log ( 1 / δ s ) ∆ 2 s .
Revised: We also replace the constraints by a conﬁdence intervalterm of the order log ( 1 / δ s ) ∆ 2 s .

Original: This enables us to bound the probability of failure for the algorithm by δ sduring phase s .
Revised: This enables us to bound the probability of failure for the algorithm by δ sduring phase s .

Original: We note that standard choices of δ s such as δ s =
Revised: We note that standard choices of δ s such as δ s =

Original: Θ ( 1 T ) from UCB-type strategieswill result in a regret bound that has a sub-optimal time-horizon dependence.
Revised: Θ ( 1 T ) from UCB-type strategieswill result in a regret bound that has a sub-optimal time-horizon dependence.

Original: This suggests that amore careful choice of δ s must be determined.
Revised: This suggests that amore careful choice of δ s must be determined.

Original: 5.1 Algorithm
Revised: 5.1 Algorithm

Original: To describe our algorithm, we will adopt the following deﬁnitions and notation.
Revised: To describe our algorithm, we will adopt the following deﬁnitions and notation.

Original: Let τ s denote thelast time-step of phase s .
Revised: Let τ s denote thelast time-step of phase s .

Original: We will denote by n i ( s ) the total number of times the reward of arm i isobserved up to and including s , n i ( s ) =
Revised: We will denote by n i ( s ) the total number of times the reward of arm i isobserved up to and including s , n i ( s ) =

Original: ∑ τ s t = 1 I ( i t ∈ N i ) , and by r i ( s ) the average reward observed,ˆ r i ( s ) =
Revised: ∑ τ s t = 1 I ( i t ∈ N i ) , and by r i ( s ) the average reward observed,ˆ r i ( s ) =

Original: [∑ τ s t = 1 r t,i I ( i t ∈ N i )]/ n i ( s ) .
Revised: [∑ τ s t = 1 r t,i I ( i t ∈ N i )]/ n i ( s ) .

Original: We also denote by ˆ∆
Revised: We also denote by ˆ∆

Original: i ( s ) a lower bound on ∆ s with a shrinkingconﬁdence interval b i ( s ) and by ˆΓ s the empirical version of the set Γ s :
Revised: i ( s ) a lower bound on ∆ s with a shrinkingconﬁdence interval b i ( s ) and by ˆΓ s the empirical version of the set Γ s :

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: s ) , where b i ( s ) =
Revised: s ) , where b i ( s ) =

Original: [Equation]
Revised: [Equation]

Original: Our algorithm solves an empirical version of (LP2) at each phase, which is the following LP:
Revised: Our algorithm solves an empirical version of (LP2) at each phase, which is the following LP:

Original: [Equation]
Revised: [Equation]

Original: Pseudocode can be found in Algorithm 1.
Revised: Pseudocode can be found in Algorithm 1.

Original: In the ﬁrst ⌈ log ( K )⌉ rounds, the algorithm just playsaccording to the minimum dominating set of G .
Revised: In the ﬁrst ⌈ log ( K )⌉ rounds, the algorithm just playsaccording to the minimum dominating set of G .

Original: This is because there is not enough informationregarding any of the gaps.
Revised: This is because there is not enough informationregarding any of the gaps.

Original: Denote the approximate solution of LP3 as x ∗ LP 3 at phase s .
Revised: Denote the approximate solution of LP3 as x ∗ LP 3 at phase s .

Original: Thenat every round of phase s we play each arm exactly ⌈( x ∗ LP 3 )
Revised: Thenat every round of phase s we play each arm exactly ⌈( x ∗ LP 3 )

Original: i ⌉ many times.
Revised: i ⌉ many times.

Original: Phase s then endsafter ∑ j ∈[ K ] ⌈( x ∗ LP 3 )
Revised: Phase s then endsafter ∑ j ∈[ K ] ⌈( x ∗ LP 3 )

Original: i ⌉ rounds.
Revised: i ⌉ rounds.

Original: We note that it is sufﬁcient to approximately solve LP3 so that theconstraints are satisﬁed up to some multiplicative factor and the value of the solution is bounded by amultiplicative factor in the value of the LP.
Revised: We note that it is sufﬁcient to approximately solve LP3 so that theconstraints are satisﬁed up to some multiplicative factor and the value of the solution is bounded by amultiplicative factor in the value of the LP.

Original: Input : Graph G =
Revised: Input : Graph G =

Original: ( V,E ) ,
Revised: ( V,E ) ,

Original: conﬁdence parameter δ , time horizon T 1 Initialize t = 0 , s = 0 , ˆ r i ( 0 ) = 0 , ∀ i ∈
Revised: conﬁdence parameter δ , time horizon T 1 Initialize t = 0 , s = 0 , ˆ r i ( 0 ) = 0 , ∀ i ∈

Original: [ K ] 2 Compute (approximate) minimum dominating set ˆ D( G ) 3 while s ≤ ⌈ log ( K )⌉ do
Revised: [ K ] 2 Compute (approximate) minimum dominating set ˆ D( G ) 3 while s ≤ ⌈ log ( K )⌉ do

Original: Play each arm i ∈
Revised: Play each arm i ∈

Original: ˆ D( G ) for α ′ log ( K ∆ s + 1 ) ∆ 2 s rounds
Revised: ˆ D( G ) for α ′ log ( K ∆ s + 1 ) ∆ 2 s rounds

Original: Update t and s .
Revised: Update t and s .

Original: 6 while t ≤ T do
Revised: 6 while t ≤ T do

Original: Compute a (approximate) solution x ∗ LP 3 to LP3.
Revised: Compute a (approximate) solution x ∗ LP 3 to LP3.

Original: Play each action i for ⌈( x ∗ LP 3 ) i ⌉ rounds and update t .
Revised: Play each action i for ⌈( x ∗ LP 3 ) i ⌉ rounds and update t .

Original: Update the phase s + = 1 .
Revised: Update the phase s + = 1 .

Original: 5.2 Regret bound
Revised: 5.2 Regret bound

Original: The ﬁrst step in the regret analysis of Algorithm 1 is to relate the value of LP3 to the value of LP4based on the true gaps given below.
Revised: The ﬁrst step in the regret analysis of Algorithm 1 is to relate the value of LP3 to the value of LP4based on the true gaps given below.

Original: [Equation]
Revised: [Equation]

Original: We do so by showing that ˆΓ s + 1 ⊆
Revised: We do so by showing that ˆΓ s + 1 ⊆

Original: Γ s and that ˆ∆ ( s )
Revised: Γ s and that ˆ∆ ( s )

Original: =
Revised: =

Original: Θ ( ∆ s ) .
Revised: Θ ( ∆ s ) .

Original: This allows us to upper upper boundthe value of LP3 by the value of LP4 in the following way.
Revised: This allows us to upper upper boundthe value of LP3 by the value of LP4 in the following way.

Original: Lemma 5.1.
Revised: Lemma 5.1.

Original: Let D LP 3 ( s ) be the value of LP3 at phase s and let D LP 4 ( s ) be the value of LP4 atphase s .
Revised: Let D LP 3 ( s ) be the value of LP3 at phase s and let D LP 4 ( s ) be the value of LP4 atphase s .

Original: For any s ≥ log ( K ) ∨ 10 holds that D LP 3 ( s + 1 ) ≤ 4 D LP 4 ( s ) , with probability at least1 − 3 ( ∆ s / 2 + 1 K ) α − 2 .
Revised: For any s ≥ log ( K ) ∨ 10 holds that D LP 3 ( s + 1 ) ≤ 4 D LP 4 ( s ) , with probability at least1 − 3 ( ∆ s / 2 + 1 K ) α − 2 .

Original: Further, for any s ≥ log (∣ I ∗ ∣/( 4∆ min )) ∨ 10 it holds that the regret incurred forplaying according to LP3 is at most 16 α ′ c ∗ ( G,µ ) with the same probability.
Revised: Further, for any s ≥ log (∣ I ∗ ∣/( 4∆ min )) ∨ 10 it holds that the regret incurred forplaying according to LP3 is at most 16 α ′ c ∗ ( G,µ ) with the same probability.

Original: Lemma 5.1 shows that playing Algorithm 1 is already asymptotically optimal, as the incurred regretduring any phase s ≥ log (∣ I ∗ ∣/ ∆ min ) starts being bounded by O ( c ∗ ) .
Revised: Lemma 5.1 shows that playing Algorithm 1 is already asymptotically optimal, as the incurred regretduring any phase s ≥ log (∣ I ∗ ∣/ ∆ min ) starts being bounded by O ( c ∗ ) .

Original: There are two challengingparts in proving Lemma 5.1.
Revised: There are two challengingparts in proving Lemma 5.1.

Original: First is how to handle the concentration of ˆ∆ i ( s ) for actions i /∈ ˆΓ swhich have been eliminated prior to phase s .
Revised: First is how to handle the concentration of ˆ∆ i ( s ) for actions i /∈ ˆΓ swhich have been eliminated prior to phase s .

Original: This challenge arises because α ′ needs to be setas a time-independent parameter as the time-horizon part of the regret incurred by the algorithmwill depend on α ′ .
Revised: This challenge arises because α ′ needs to be setas a time-independent parameter as the time-horizon part of the regret incurred by the algorithmwill depend on α ′ .

Original: We notice that for any phase s ≥ 2 the event that the empirical reward, ˆ r i ( t ) ,concentrates uniformly around its mean µ i in the interval t ∈
Revised: We notice that for any phase s ≥ 2 the event that the empirical reward, ˆ r i ( t ) ,concentrates uniformly around its mean µ i in the interval t ∈

Original: [ s / 2 ,s ] can be controlled with highprobability.
Revised: [ s / 2 ,s ] can be controlled with highprobability.

Original: This in turn guarantees that the empirical gap estimator ˆ∆ ( t ) is small enough and henceaction i is observed sufﬁciently many times in phases [ s / 2 ,s ] .
Revised: This in turn guarantees that the empirical gap estimator ˆ∆ ( t ) is small enough and henceaction i is observed sufﬁciently many times in phases [ s / 2 ,s ] .

Original: The second challenge is to analyze the regret of the solution of LP3 directly, for any s ≥ log ( ∣ I ∗ ∣ K )so that we can bound this regret by c ∗ .
Revised: The second challenge is to analyze the regret of the solution of LP3 directly, for any s ≥ log ( ∣ I ∗ ∣ K )so that we can bound this regret by c ∗ .

Original: The key observation is that there exists a ˆ x ∗ which isfeasible (with high probability) for LP1 with the property that ⟨ ˆ x ∗ , ˆ∆ ( s )⟩ ≤ O ( c ∗ ) and further∑ i /∈
Revised: The key observation is that there exists a ˆ x ∗ which isfeasible (with high probability) for LP1 with the property that ⟨ ˆ x ∗ , ˆ∆ ( s )⟩ ≤ O ( c ∗ ) and further∑ i /∈

Original: I ∗ x ∗ LP 3 ,i ˆ∆
Revised: I ∗ x ∗ LP 3 ,i ˆ∆

Original: i ( s ) ≤ 2 ∑ i /∈
Revised: i ( s ) ≤ 2 ∑ i /∈

Original: I ∗ ˆ x ∗ i ˆ∆
Revised: I ∗ ˆ x ∗ i ˆ∆

Original: i ( s ) .
Revised: i ( s ) .

Original: This is sufﬁcient to conclude that D LP 3 ( s ) ≤ O ( c ∗ )
Revised: This is sufﬁcient to conclude that D LP 3 ( s ) ≤ O ( c ∗ )

Original: Lemma 5.1 can now be combined with the observation that the constraints of LP2 are a subset of theconstraints of LP4, up to a logarithmic factor in 1∆ min , to argue the following upper regret bound.
Revised: Lemma 5.1 can now be combined with the observation that the constraints of LP2 are a subset of theconstraints of LP4, up to a logarithmic factor in 1∆ min , to argue the following upper regret bound.

Original: Theorem 5.2.
Revised: Theorem 5.2.

Original: Let d ∗ ( G,µ )
Revised: Let d ∗ ( G,µ )

Original: = max s ≤ log (∣ I ∗ ∣/ ∆ min ) D LP 2 ( s ) .
Revised: = max s ≤ log (∣ I ∗ ∣/ ∆ min ) D LP 2 ( s ) .

Original: There exists an algorithm with expectedregret Reg ( T ) bounded as Reg ( T ) ≤
Revised: There exists an algorithm with expectedregret Reg ( T ) bounded as Reg ( T ) ≤

Original: O ( log 2 ( 1 ∆ min ) d ∗ + log ( T ) c ∗ + γ ( G ) K log ( K )) .
Revised: O ( log 2 ( 1 ∆ min ) d ∗ + log ( T ) c ∗ + γ ( G ) K log ( K )) .

Original: We note that Algorithm 1 can incur additional regret of order O ( K ) per phase due to the rounding,⌈ x ∗ LP 3 ⌉ , of the solution to LP3.
Revised: We note that Algorithm 1 can incur additional regret of order O ( K ) per phase due to the rounding,⌈ x ∗ LP 3 ⌉ , of the solution to LP3.

Original: Thus its regret will only be asymptotically optimal in the settingwhen ∆ min
Revised: Thus its regret will only be asymptotically optimal in the settingwhen ∆ min

Original: ≤ O ( 1 / K ) .
Revised: ≤ O ( 1 / K ) .

Original: To ﬁx this minor issue, we present an algorithm with more careful roundingin Appendix B.1, which enjoys the regret bound of Theorem 5.2.
Revised: To ﬁx this minor issue, we present an algorithm with more careful roundingin Appendix B.1, which enjoys the regret bound of Theorem 5.2.

Original:  Regret lower bounds
Revised:  Regret lower bounds

Original: Lower bound with d ∗ .
Revised: Lower bound with d ∗ .

Original: We are able to show the following result for any algorithm.
Revised: We are able to show the following result for any algorithm.

Original: Lemma 6.1.
Revised: Lemma 6.1.

Original: Fix any instance µ s.t.
Revised: Fix any instance µ s.t.

Original: µ i ≤ 1 − 2∆ s ,i ∈ I ∗ .
Revised: µ i ≤ 1 − 2∆ s ,i ∈ I ∗ .

Original: Let Λ s ( µ ) be the set of problem instanceswith means µ ′ ∈ µ + [ 0 , 2∆ s ] k .
Revised: Let Λ s ( µ ) be the set of problem instanceswith means µ ′ ∈ µ + [ 0 , 2∆ s ] k .

Original: Then for any algorithm, there exists an instance in Λ s ( µ ) such thatthe regret is lower bounded by LP2.
Revised: Then for any algorithm, there exists an instance in Λ s ( µ ) such thatthe regret is lower bounded by LP2.

Original: Motivated by Lemma 6.1, the quantity d ∗ ( G,µ ) is a meaningful deﬁnition of ﬁnite-time optimality.
Revised: Motivated by Lemma 6.1, the quantity d ∗ ( G,µ ) is a meaningful deﬁnition of ﬁnite-time optimality.

Original: We note that d ∗ is indeed independent of the time-horizon and only depends on the topology of G andthe instance µ .
Revised: We note that d ∗ is indeed independent of the time-horizon and only depends on the topology of G andthe instance µ .

Original: The result in Lemma 6.1 is a companion to the upper bound in Theorem 5.2.
Revised: The result in Lemma 6.1 is a companion to the upper bound in Theorem 5.2.

Original: It showsthat for any instance µ and number of observations which are not sufﬁcient to distinguish the armswith smallest positive gaps as sub-optimal, any algorithm will necessarily incur large regret of orderd ∗ .
Revised: It showsthat for any instance µ and number of observations which are not sufﬁcient to distinguish the armswith smallest positive gaps as sub-optimal, any algorithm will necessarily incur large regret of orderd ∗ .

Original: This happens because the algorithm will not be able to distinguish µ from some environment µ ′which is identical to µ except for the reward of a single arm which is only slightly perturbed.
Revised: This happens because the algorithm will not be able to distinguish µ from some environment µ ′which is identical to µ except for the reward of a single arm which is only slightly perturbed.

Original: The deﬁnition of d ∗ as a maximum over different values of s might seem surprising, as one couldexpect that the value of LP2 strictly increases when s grows, after all this is precisely what happens inthe bandit setting.
Revised: The deﬁnition of d ∗ as a maximum over different values of s might seem surprising, as one couldexpect that the value of LP2 strictly increases when s grows, after all this is precisely what happens inthe bandit setting.

Original: This is not the case for general graphs, where the value can also decrease betweenphases s and s + 1 .
Revised: This is not the case for general graphs, where the value can also decrease betweenphases s and s + 1 .

Original: Intuitively this happens when the approximate minimum weighted dominating setchosen by the LP’s solution increases between phases.
Revised: Intuitively this happens when the approximate minimum weighted dominating setchosen by the LP’s solution increases between phases.

Original: The result in Lemma 6.1 has a min-max ﬂavor in the sense that all possible instances which are closeto µ are considered.
Revised: The result in Lemma 6.1 has a min-max ﬂavor in the sense that all possible instances which are closeto µ are considered.

Original: It is reasonable to ask if d ∗ can be further bounded by a favorable instancedependent quantity.
Revised: It is reasonable to ask if d ∗ can be further bounded by a favorable instancedependent quantity.

Original: The answer to this question is complicated and certainly depends on the topologyof the feedback graph as we show next.
Revised: The answer to this question is complicated and certainly depends on the topologyof the feedback graph as we show next.

Original: Sketch of proof of Theorem 4.1.
Revised: Sketch of proof of Theorem 4.1.

Original: We now show that any ﬁnitetime term d has to exceed c ∗ by at least a multiplicative polynomialfactor in the number of actions K .
Revised: We now show that any ﬁnitetime term d has to exceed c ∗ by at least a multiplicative polynomialfactor in the number of actions K .

Original: To do so we exhibit a speciﬁcfeedback graph G , found in Figure 2, on which any algorithm willhave to incur regret at least Ω ( K 18 c ∗ ) for some µ s.t.
Revised: To do so we exhibit a speciﬁcfeedback graph G , found in Figure 2, on which any algorithm willhave to incur regret at least Ω ( K 18 c ∗ ) for some µ s.t.

Original: c ∗ ≥ 1∆ min .
Revised: c ∗ ≥ 1∆ min .

Original: Formally the graph is deﬁned to have a vertex set V = N 1 ⋃N 2 ⋃N 3of 2 K + 1 arms, with each of N
Revised: Formally the graph is deﬁned to have a vertex set V = N 1 ⋃N 2 ⋃N 3of 2 K + 1 arms, with each of N

Original: i ’s disjoint and N
Revised: i ’s disjoint and N

Original: 1 = { 2 i ∶ 1 ≤ i ≤ K } ,
Revised: 1 = { 2 i ∶ 1 ≤ i ≤ K } ,

Original: N 2 = { 2 i + 1 ∶ 0 ≤ i ≤ K } , N 3 = { 0 } .
Revised: N 2 = { 2 i + 1 ∶ 0 ≤ i ≤ K } , N 3 = { 0 } .

Original: The set of edges is deﬁnedas follows.
Revised: The set of edges is deﬁnedas follows.

Original: Every vertex in N 1 is adjacent to the vertex in N 3 andthe 2 i vertex is adjacent to both 2 i + 1 and 2 i − 1 in N 2 modulo2 K + 1 .
Revised: Every vertex in N 1 is adjacent to the vertex in N 3 andthe 2 i vertex is adjacent to both 2 i + 1 and 2 i − 1 in N 2 modulo2 K + 1 .

Original: Finally vertex 2 i + 1 in N 2 is further adjacent to to the next⌈ K 1 / 8 ⌉ vertices in N 2 modulo 2 K + 1 .
Revised: Finally vertex 2 i + 1 in N 2 is further adjacent to to the next⌈ K 1 / 8 ⌉ vertices in N 2 modulo 2 K + 1 .

Original: The base instance, E , is deﬁned by a scalar ν ∈ [ 0 , 1 ] andgap parameter ∆ so that the expected reward of every action in N 1 is equal to ν − ∆ , the expectedreward of every action in N 2 is equal to ν − K 1 / 4 ∆ and the expected reward of the action in N 3is ν − √ K ∆ .
Revised: The base instance, E , is deﬁned by a scalar ν ∈ [ 0 , 1 ] andgap parameter ∆ so that the expected reward of every action in N 1 is equal to ν − ∆ , the expectedreward of every action in N 2 is equal to ν − K 1 / 4 ∆ and the expected reward of the action in N 3is ν − √ K ∆ .

Original: We assume that all rewards follow a Gaussian with variance 1 √ 2 .
Revised: We assume that all rewards follow a Gaussian with variance 1 √ 2 .

Original: We denote by∆ 1 =
Revised: We denote by∆ 1 =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The lower bound now ﬁxes an algorithm A and considers two cases.
Revised: The lower bound now ﬁxes an algorithm A and considers two cases.

Original: First, A could commit to oftenplaying arms in N 1 .
Revised: First, A could commit to oftenplaying arms in N 1 .

Original: In this case we show that there could be a large gap to the arms in N 1 whichwould not be detectable by A as arms in N 2 are not observed often enough.
Revised: In this case we show that there could be a large gap to the arms in N 1 whichwould not be detectable by A as arms in N 2 are not observed often enough.

Original: This is indeed the caseas A needs to play Ω ( K ) actions in N 1 to cover N 2 .
Revised: This is indeed the caseas A needs to play Ω ( K ) actions in N 1 to cover N 2 .

Original: This ﬁrst case corresponds to assuming that thenumber of arms played from N 2 is at most O ( K 78 / ∆ 22 ) in the ﬁrst O ( K / ∆ 22 ) rounds.
Revised: This ﬁrst case corresponds to assuming that thenumber of arms played from N 2 is at most O ( K 78 / ∆ 22 ) in the ﬁrst O ( K / ∆ 22 ) rounds.

Original: The secondcase considers the scenario in which actions in N 2 are played for more than Ω ( K 78 / ∆ 22 ) times in theﬁrst O
Revised: The secondcase considers the scenario in which actions in N 2 are played for more than Ω ( K 78 / ∆ 22 ) times in theﬁrst O

Original: ( K / ∆ 22 ) rounds.
Revised: ( K / ∆ 22 ) rounds.

Original: In this case, A would suffer large regret if the gap at actions in N 1 is smallenough, so that the optimal strategy is to cover N 2 by playing arms in N 1 .
Revised: In this case, A would suffer large regret if the gap at actions in N 1 is smallenough, so that the optimal strategy is to cover N 2 by playing arms in N 1 .

Original: More formally, we begin by showing that there always exists an arm n ∗ ∈ N 2 which is observedfor only O ( 1 / ∆ 22 ) times.
Revised: More formally, we begin by showing that there always exists an arm n ∗ ∈ N 2 which is observedfor only O ( 1 / ∆ 22 ) times.

Original: Next, we change the expected reward of n ∗ depending on which of theabove two cases occur.
Revised: Next, we change the expected reward of n ∗ depending on which of theabove two cases occur.

Original: In the ﬁrst case we change the environment by setting the reward of n ∗to have expectation ν + ∆ 2 .
Revised: In the ﬁrst case we change the environment by setting the reward of n ∗to have expectation ν + ∆ 2 .

Original: We can now argue that the regret of A will be at least Ω ( K 34 / ∆ ) asn ∗ will not be played often enough in the new environment.
Revised: We can now argue that the regret of A will be at least Ω ( K 34 / ∆ ) asn ∗ will not be played often enough in the new environment.

Original: The value of c ∗ , however, is at most
Revised: The value of c ∗ , however, is at most

Original: O ( K 78 / ∆ 2 ) =
Revised: O ( K 78 / ∆ 2 ) =

Original: O ( K 58 / ∆ ) , as playing each action in a minimum dominating set over N 2 for 1∆ 22 rounds is feasible for LP1.
Revised: O ( K 58 / ∆ ) , as playing each action in a minimum dominating set over N 2 for 1∆ 22 rounds is feasible for LP1.

Original: For the second case, we set the expected reward of n ∗ to equal ν .
Revised: For the second case, we set the expected reward of n ∗ to equal ν .

Original: Theoptimal strategy now has regret at most O (√ K / ∆ ) by playing the action in N 3 for 1∆ 2 and everyaction in N 1 for 1∆ 22 rounds.
Revised: Theoptimal strategy now has regret at most O (√ K / ∆ ) by playing the action in N 3 for 1∆ 2 and everyaction in N 1 for 1∆ 22 rounds.

Original: On the other hand A will incur at least Ω ( K 58 / ∆ ) , as again n ∗ is notplayed often enough.
Revised: On the other hand A will incur at least Ω ( K 58 / ∆ ) , as again n ∗ is notplayed often enough.

Original: This argument implies the result presented in Theorem 4.1.
Revised: This argument implies the result presented in Theorem 4.1.

Original:  Characterizing the value of d ∗
Revised:  Characterizing the value of d ∗

Original: Theorem 4.1 suggests that we take into account the topology of G explicitly when trying to bound d ∗ ,independently of the instance µ .
Revised: Theorem 4.1 suggests that we take into account the topology of G explicitly when trying to bound d ∗ ,independently of the instance µ .

Original: In this section, we ﬁrst show a bound on d ∗ that depends only onindependent sets of G .
Revised: In this section, we ﬁrst show a bound on d ∗ that depends only onindependent sets of G .

Original: Then, we show a set of graphs G for which d ∗ ≤ O ( c ∗ ) on any instance µ .
Revised: Then, we show a set of graphs G for which d ∗ ≤ O ( c ∗ ) on any instance µ .

Original: Let us recall the regret bounds presented in [Lykouris et al., 2020].
Revised: Let us recall the regret bounds presented in [Lykouris et al., 2020].

Original: Denote by I ( G ) the set of allindependent sets for the graph G .
Revised: Denote by I ( G ) the set of allindependent sets for the graph G .

Original: Then the regret bounds presented in [Lykouris et al., 2020] are ofthe order Reg ( T ) ≤
Revised: Then the regret bounds presented in [Lykouris et al., 2020] are ofthe order Reg ( T ) ≤

Original: O ( max I ∈
Revised: O ( max I ∈

Original: I
Revised: I

Original: ( G )
Revised: ( G )

Original: ∑ i ∈ I log 2 ( T ) ∆ i ) .
Revised: ∑ i ∈ I log 2 ( T ) ∆ i ) .

Original: It is possible to show, as we do in Appendix
Revised: It is possible to show, as we do in Appendix

Original: D.1,that d ∗ ( G,µ ) ≤ max I ∈ I ( G )
Revised: D.1,that d ∗ ( G,µ ) ≤ max I ∈ I ( G )

Original: ∑ i ∈
Revised: ∑ i ∈

Original: I 1∆ i
Revised: I 1∆ i

Original: Thus, our algorithm enjoys regret bounds which are better thanwhat is known for the algorithms studied in [Cohen et al., 2016, Lykouris et al., 2020].
Revised: Thus, our algorithm enjoys regret bounds which are better thanwhat is known for the algorithms studied in [Cohen et al., 2016, Lykouris et al., 2020].

Original: The abovebound, however, could be very loose as was discussed in the beginning of the paper, especially whenconsidering star-graphs, as the bound would just reduce to the bandit case.
Revised: The abovebound, however, could be very loose as was discussed in the beginning of the paper, especially whenconsidering star-graphs, as the bound would just reduce to the bandit case.

Original: It turns out, however, thatd ∗ ≤ O ( c ∗ ) in this case.
Revised: It turns out, however, thatd ∗ ≤ O ( c ∗ ) in this case.

Original: In fact we can state a sufﬁcient condition on G so that d ∗ ≤ c ∗ + ∣ I ∗ ∣ ∆ min for amore general family of graphs.
Revised: In fact we can state a sufﬁcient condition on G so that d ∗ ≤ c ∗ + ∣ I ∗ ∣ ∆ min for amore general family of graphs.

Original: We begin by deﬁning the following operation on G .
Revised: We begin by deﬁning the following operation on G .

Original: Deﬁnition 7.1.
Revised: Deﬁnition 7.1.

Original: Let ∼ be the equivalence class deﬁned by u ∼ v iff N u
Revised: Let ∼ be the equivalence class deﬁned by u ∼ v iff N u

Original: =
Revised: =

Original: N v
Revised: N v

Original: and let C be the mappingwhich sends G to the quotient G / ∼ through the operation of collapsing any sub-graph of G into itsequivalence class.
Revised: and let C be the mappingwhich sends G to the quotient G / ∼ through the operation of collapsing any sub-graph of G into itsequivalence class.

Original: We note that C is well-deﬁned as the relation ∼ is an equivalent relation.
Revised: We note that C is well-deﬁned as the relation ∼ is an equivalent relation.

Original: The equivalence classesdeﬁned by ∼ are cliques with the following property.
Revised: The equivalence classesdeﬁned by ∼ are cliques with the following property.

Original: For any v in an equivalence class [ v ] it holdsthat u ∈ [ v ] , ∀ u ∈ N v , that is the vertices in the equivalence class clique only have neighbors inthe clique to which they belong.
Revised: For any v in an equivalence class [ v ] it holdsthat u ∈ [ v ] , ∀ u ∈ N v , that is the vertices in the equivalence class clique only have neighbors inthe clique to which they belong.

Original: For any instance µ of the problem, this allows us to collapse eachequivalence class [ v ] to a vertex v with the maximum expected reward in [ v ] .
Revised: For any instance µ of the problem, this allows us to collapse eachequivalence class [ v ] to a vertex v with the maximum expected reward in [ v ] .

Original: The next lemma statesa sufﬁcient condition on G under which d ∗ is bounded.
Revised: The next lemma statesa sufﬁcient condition on G under which d ∗ is bounded.

Original: Lemma 7.2.
Revised: Lemma 7.2.

Original: If the graph G is such that C ( G ) has no path of length greater than two between anytwo vertices, then, for any instance µ , the following inequality holds: c ∗ + ∣ I ∗ ∣ ∆ min ≥ d ∗ .
Revised: If the graph G is such that C ( G ) has no path of length greater than two between anytwo vertices, then, for any instance µ , the following inequality holds: c ∗ + ∣ I ∗ ∣ ∆ min ≥ d ∗ .

Original:  Conclusion
Revised:  Conclusion

Original: We presented a detailed study of the problem of stochastic online learning with feedback graphs in aﬁnite time setting.
Revised: We presented a detailed study of the problem of stochastic online learning with feedback graphs in aﬁnite time setting.

Original: We pointed out the surprising issue of deﬁning optimal ﬁnite-time regret for thisproblem.
Revised: We pointed out the surprising issue of deﬁning optimal ﬁnite-time regret for thisproblem.

Original: We gave an instance on which no algorithm can hope to match, in ﬁnite time, the quantityc ∗ , which characterizes asymptotic optimality.
Revised: We gave an instance on which no algorithm can hope to match, in ﬁnite time, the quantityc ∗ , which characterizes asymptotic optimality.

Original: Next, we derived an asymptotically optimal algorithmthat is also min-max optimal in a ﬁnite-time sense and admits more favorable regret guarantees thanthose given in prior work.
Revised: Next, we derived an asymptotically optimal algorithmthat is also min-max optimal in a ﬁnite-time sense and admits more favorable regret guarantees thanthose given in prior work.

Original: Finally, we described a family of feedback graphs for which matching theasymptotically optimal rate is possible in ﬁnite time.
Revised: Finally, we described a family of feedback graphs for which matching theasymptotically optimal rate is possible in ﬁnite time.

Original: There are several interesting questions that follow from this work.
Revised: There are several interesting questions that follow from this work.

Original: First, while the condition on 375 C ( G ) in Lemma 7.2 is sufﬁcient, it is not necessary.
Revised: First, while the condition on 374 C ( G ) in Lemma 7.2 is sufﬁcient, it is not necessary.

Original: For example, a star-like graph in which twoleaf vertices are also neighbors will have the property that d ∗ ≤ O ( c ∗ ) for any instance µ .
Revised: For example, a star-like graph in which twoleaf vertices are also neighbors will have the property that d ∗ ≤ O ( c ∗ ) for any instance µ .

Original: We askwhat would be a necessary and sufﬁcient condition on G for which d ∗ =
Revised: We askwhat would be a necessary and sufﬁcient condition on G for which d ∗ =

Original: Θ ( c ∗ ) on any instanceµ ?
Revised: Θ ( c ∗ ) on any instanceµ ?

Original: Another interesting question is how to address the setting of evolving feedback graphs.
Revised: Another interesting question is how to address the setting of evolving feedback graphs.

Original: It isunclear what conditions on the graph sequence would allow us to recover bounds that improve on theexisting independence number results.
Revised: It isunclear what conditions on the graph sequence would allow us to recover bounds that improve on theexisting independence number results.

Original: Further, can we use our approach to show improved results forthe setting of dependent rewards and feedback graphs studied in [Cortes et al., 2020]?
Revised: Further, can we use our approach to show improved results forthe setting of dependent rewards and feedback graphs studied in [Cortes et al., 2020]?

Original: Finally, ourmethodology crucially relies on the informed setting assumption.
Revised: Finally, ourmethodology crucially relies on the informed setting assumption.

Original: We ask if it is possible to achievesimilar bounds to Theorem 5.2 in the uninformed setting. 
Revised: We ask if it is possible to achievesimilar bounds to Theorem 5.2 in the uninformed setting. Checklist

Revised: The checklist follows the references.

Revised: Please read the checklist guidelines carefully for information onhow to answer these questions.

Revised: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .

Revised: For all authors.

Revised: (a)

Revised: Do

Revised: the main claims

Revised: made in the abstract and introduction accurately reﬂect the paper’s

Revised: contributions

Revised: and

Revised: scope

Revised: [Yes]

Revised: (b) Did

Revised: you describe the limitations of your work

Revised: Yes

Revised: ] Both upper and lower bounds arediscussed in the respective sections.

Revised: We further have a broad discussion about the typeof bounds we choose to pursue in Section 4 and Appenidx A.

Revised: (c) Did you discuss any potential negative societal impacts of your work?

Revised: [N/A]

Revised: This work

Revised: is theoretical and it is hard to judge societal impact.

Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Revised: them?

Revised: [Yes]

Revised: If you are including theoretical results...

Revised: (a) Did you state the full set of assumptions of all theoretical results?

Revised: [Yes] Please refer to

Revised: the results in the stated theorems.

Revised: (b) Did you include complete proofs of all theoretical results?

Revised: [Yes] Please refer to the

Revised: appendix for complete proofs of the main results.

Revised: If you ran experiments...

Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Revised: mental results (either in the supplemental material or as a URL)?

Revised: [N/A]

Revised: There is no

Revised: code available for this work.

Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Revised: how they

Revised: were chosen)?

Revised: [N/A]

Revised: (c) Did you report error bars (e

Revised: g., with respect to the random seed after running experi-

Revised: ments multiple times)?

Revised: [N/A]

Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Revised: [N/A]

Revised: If you are using existing assets (e

Revised: g., code, data, models) or curating/releasing new assets...

Revised: (a) If your work uses existing assets, did you cite the creators?

Revised: [N/A]

Revised: (b) Did you mention the license of the assets?

Revised: [N/A]

Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Revised: [N/A]

Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Revised: using/curating?

Revised: [N/A]

Revised: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Revised: information or offensive content?

Revised: [N/A]

Revised: If you used crowdsourcing or conducted research with human subjects...

Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Revised: applicable?

Revised: [N/A]

Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Revised: Board (IRB) approvals, if applicable?

Revised: [N/A]

Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Revised: spent on participant compensation?

Revised: [N/A]