Original: Stochastic Online Learning with Feedback Graphs: Finite-Time and Asymptotic Optimality
Revised: Stochastic Online Learning with Feedback Graphs: Finite-Time and Asymptotic Optimality

Original: Abstract
Revised: Abstract

Original: We revisit the problem of stochastic online learning with feedback graphs, with the goal of devising algorithms that are optimal, up to constants, both asymptotically and in ﬁnite time.
Revised: We revisit the problem of stochastic online learning with feedback graphs, with the goal of devising algorithms that are optimal, up to constants, both asymptotically and in ﬁnite time.

Original: We show that, surprisingly, the notion of optimal ﬁnite-time regret is not a uniquely deﬁned property in this context and that, in general, it is decoupled from the asymptotic rate.
Revised: We show that, surprisingly, the notion of optimal ﬁnite-time regret is not a uniquely deﬁned property in this context and that, in general, it is decoupled from the asymptotic rate.

Original: We discuss alternative choices and propose a notion of ﬁnite-time optimality that we argue is meaningful .
Revised: We discuss alternative choices and propose a notion of ﬁnite-time optimality that we argue is meaningful .

Original: For that notion, we give an algorithm that admits quasi-optimal regret both in ﬁnite-time and asymptotically
Revised: For that notion, we give an algorithm that admits quasi-optimal regret both in ﬁnite-time and asymptotically

Original:  Introduction
Revised:  Introduction

Original: Online learning is a sequential decision making game in which, at each round, the learner selects one arm (or expert) out of a ﬁnite set of K arms.
Revised: Online learning is a sequential decision making game in which, at each round, the learner selects one arm (or expert) out of a ﬁnite set of K arms.

Original: In the stochastic setting, each arm admits some reward distribution and the learner receives a reward drawn from the distribution corresponding to the arm selected.
Revised: In the stochastic setting, each arm admits some reward distribution and the learner receives a reward drawn from the distribution corresponding to the arm selected.

Original: In the bandit setting , the learner observes only that reward [Lai et al., 1985, Auer et al., 2002a,b], while in the full information setting , the rewards of all K arms are observed [Littlestone and Warmuth, 1994, Freund and Schapire, 1997].
Revised: In the bandit setting , the learner observes only that reward [Lai et al., 1985, Auer et al., 2002a,b], while in the full information setting , the rewards of all K arms are observed [Littlestone and Warmuth, 1994, Freund and Schapire, 1997].

Original: Both settings are special instances of a more general model of online learning with side information introduced by Mannor and Shamir [2011], where the information supplied to the learner is speciﬁed by a feedback graph .
Revised: Both settings are special instances of a more general model of online learning with side information introduced by Mannor and Shamir [2011], where the information supplied to the learner is speciﬁed by a feedback graph .

Original: In an undirected feedback graph, each vertex represents an arm and an edge between between arm v and w indicates that the reward of w is observed when v is selected and vice-versa.
Revised: In an undirected feedback graph, each vertex represents an arm and an edge between between arm v and w indicates that the reward of w is observed when v is selected and vice-versa.

Original: The bandit setting corresponds to a graph reduced to self-loops at each vertex, the full information to a fully connected graph.
Revised: The bandit setting corresponds to a graph reduced to self-loops at each vertex, the full information to a fully connected graph.

Original: The problem of online learning with stochastic rewards and feedback graphs has been studied by several publications in the last decade or so.
Revised: The problem of online learning with stochastic rewards and feedback graphs has been studied by several publications in the last decade or so.

Original: The performance of an algorithm in this problem is expressed in terms of its pseudo-regret, that is the difference between the expected reward achieved by always pulling the best arm and the expected cumulative reward obtained by the algorithm.
Revised: The performance of an algorithm in this problem is expressed in terms of its pseudo-regret, that is the difference between the expected reward achieved by always pulling the best arm and the expected cumulative reward obtained by the algorithm.

Original: The UCB algorithm of Auer et al.
Revised: The UCB algorithm of Auer et al.

Original: [2002a] designed for the bandit setting forms a baseline for this scenario.
Revised: [2002a] designed for the bandit setting forms a baseline for this scenario.

Original: For general feedback graphs, Caron et al.
Revised: For general feedback graphs, Caron et al.

Original: [2012] designed a UCB -type algorithm, UCB - N , as well as a closely related variant.
Revised: [2012] designed a UCB -type algorithm, UCB - N , as well as a closely related variant.

Original: The pseudo-regret guarantee of UCB - N is expressed in terms of the most favorable clique covering of the graph, that is its partitioning into cliques.
Revised: The pseudo-regret guarantee of UCB - N is expressed in terms of the most favorable clique covering of the graph, that is its partitioning into cliques.

Original: This guarantee is always at least as favorable as the bandit one [Auer et al., 2002a], which coincides with the speciﬁc choice of the trivial clique covering.
Revised: This guarantee is always at least as favorable as the bandit one [Auer et al., 2002a], which coincides with the speciﬁc choice of the trivial clique covering.

Original: However, the bound depends on the ratio of the maximum and minimum mean reward gaps within each clique, which, in general, can be quite large.
Revised: However, the bound depends on the ratio of the maximum and minimum mean reward gaps within each clique, which, in general, can be quite large.

Original: Cohen et al.
Revised: Cohen et al.

Original: [2016] presented an action-elimination-type algorithm
Revised: [2016] presented an action-elimination-type algorithm

Original: [Even-Dar et al., 2006], whose guarantee depends on the least favorable maximal independent set.
Revised: [Even-Dar et al., 2006], whose guarantee depends on the least favorable maximal independent set.

Original: While there are instances in which this guarantee is worse compared to the bound presented in Caron et al.
Revised: While there are instances in which this guarantee is worse compared to the bound presented in Caron et al.

Original: [2012], in general it could be much more favorable compared to the clique partition guarantee of Caron et al.
Revised: [2012], in general it could be much more favorable compared to the clique partition guarantee of Caron et al.

Original: The algorithm of Cohen et al.
Revised: The algorithm of Cohen et al.

Original: [2016] does not require access to the full feedback graph, but only to the out-neighborhood of the arm selected at each round and the results also hold for time-varying graphs.
Revised: [2016] does not require access to the full feedback graph, but only to the out-neighborhood of the arm selected at each round and the results also hold for time-varying graphs.

Original: Later, Lykouris et al.
Revised: Later, Lykouris et al.

Original: [2020] presented an improved analysis of the UCB - N algorithm based on a new layering technique, which showed that UCB - N beneﬁts, in fact, from a more favorable guarantee based on the independence number of the graph, at the price of some logarithmic factors.
Revised: [2020] presented an improved analysis of the UCB - N algorithm based on a new layering technique, which showed that UCB - N beneﬁts, in fact, from a more favorable guarantee based on the independence number of the graph, at the price of some logarithmic factors.

Original: Their analysis also implied a similar guarantee for a variant of arm-elimination and Thompson sampling, as well as some improvement of the bound of Cohen et al.
Revised: Their analysis also implied a similar guarantee for a variant of arm-elimination and Thompson sampling, as well as some improvement of the bound of Cohen et al.

Original: [2016] in the case of a ﬁxed feedback graph.
Revised: [2016] in the case of a ﬁxed feedback graph.

Original: Buccapatnam et al.
Revised: Buccapatnam et al.

Original: [2014] gave an action-elimination-type algorithm
Revised: [2014] gave an action-elimination-type algorithm

Original: [Even-Dar et al., 2006],
Revised: [Even-Dar et al., 2006],

Original: UCB - LP , that leverages the solution of a linear-programming (LP) problem.
Revised: UCB - LP , that leverages the solution of a linear-programming (LP) problem.

Original: The guarantee presented depends only on the domination number of the graph, which can be substantially smaller than the independence number.
Revised: The guarantee presented depends only on the domination number of the graph, which can be substantially smaller than the independence number.

Original: A follow-up publication [Buccapatnam et al., 2017a] presents an analysis for an extension of the scenario of online learning with stochastic feedback graphs.
Revised: A follow-up publication [Buccapatnam et al., 2017a] presents an analysis for an extension of the scenario of online learning with stochastic feedback graphs.

Original: We will show that the algorithms just discussed do not achieve asymptotically optimal pseudo-regret guarantees and that it is also unclear how tight their ﬁnite-time instance-dependent bounds are.
Revised: We will show that the algorithms just discussed do not achieve asymptotically optimal pseudo-regret guarantees and that it is also unclear how tight their ﬁnite-time instance-dependent bounds are.

Original: Wu et al.
Revised: Wu et al.

Original: [2015] and Li et al.
Revised: [2015] and Li et al.

Original: [2020] proposed asymptotically optimal algorithms with matching lower bounds.
Revised: [2020] proposed asymptotically optimal algorithms with matching lower bounds.

Original: However, the corresponding ﬁnite-time regret guarantees are far from optimal and include terms that can dominate the pseudo-regret for any reasonable time horizon.
Revised: However, the corresponding ﬁnite-time regret guarantees are far from optimal and include terms that can dominate the pseudo-regret for any reasonable time horizon.

Original: We brieﬂy discuss other work related to online learning with feedback graphs.
Revised: We brieﬂy discuss other work related to online learning with feedback graphs.

Original: When rewards are adversarial, there has been a vast amount of work studying different settings for the feedback graph such as the graph evolving throughout the game or the graph not being observable before the start of each round [Alon et al., 2013, 2015, 2017].
Revised: When rewards are adversarial, there has been a vast amount of work studying different settings for the feedback graph such as the graph evolving throughout the game or the graph not being observable before the start of each round [Alon et al., 2013, 2015, 2017].

Original: The setting in which only noisy feedback is provided by the graph is addressed in Kocák et al.
Revised: The setting in which only noisy feedback is provided by the graph is addressed in Kocák et al.

Original: First order regret bounds, that is bounds which depend on the reward of the best arm, are derived in Lykouris et al.
Revised: First order regret bounds, that is bounds which depend on the reward of the best arm, are derived in Lykouris et al.

Original: [2018], Lee et al.
Revised: [2018], Lee et al.

Original: The setting of sleeping experts is studied in Cortes et al.
Revised: The setting of sleeping experts is studied in Cortes et al.

Original: Cortes et al.
Revised: Cortes et al.

Original: [2020] study stochastic rewards when the feedback graph evolves throughout the game, however, they do not assume that the rewards and the graph are statistically independent.
Revised: [2020] study stochastic rewards when the feedback graph evolves throughout the game, however, they do not assume that the rewards and the graph are statistically independent.

Original: Another instance in which the feedback and rewards are correlated is that of online learning with abstention [Cortes et al., 2018].
Revised: Another instance in which the feedback and rewards are correlated is that of online learning with abstention [Cortes et al., 2018].

Original: In this setting the player can choose to abstain from making a prediction.
Revised: In this setting the player can choose to abstain from making a prediction.

Original: The more general problem of Reinforcement Learning with graph feedback has been studied by Dann et al.
Revised: The more general problem of Reinforcement Learning with graph feedback has been studied by Dann et al.

Original: For additional work on online learning with feedback graphs we recommend the survey of Valko [2016].
Revised: For additional work on online learning with feedback graphs we recommend the survey of Valko [2016].

Original: We revisit the problem of stochastic online learning with feedback graphs, with the goal of devising algorithms that are optimal, up to constants, both asymptotically and in ﬁnite time.
Revised: We revisit the problem of stochastic online learning with feedback graphs, with the goal of devising algorithms that are optimal, up to constants, both asymptotically and in ﬁnite time.

Original: We show that, surprisingly, the notion of optimal ﬁnite-time regret is not a uniquely deﬁned property in this context and that, in general, it is decoupled from the asymptotic rate.
Revised: We show that, surprisingly, the notion of optimal ﬁnite-time regret is not a uniquely deﬁned property in this context and that, in general, it is decoupled from the asymptotic rate.

Original: Let T denote the time horizon and
Revised: Let T denote the time horizon and

Original: Reg A ( T ) the pseudo-regret of algorithm
Revised: Reg A ( T ) the pseudo-regret of algorithm

Original: A after T rounds.
Revised: A after T rounds.

Original: When A is clear from the context, we drop the subscript.
Revised: When A is clear from the context, we drop the subscript.

Original: It is known that c ∗ , the value of the LP considered by Buccapatnam et al.
Revised: It is known that c ∗ , the value of the LP considered by Buccapatnam et al.

Original: [2014], Wu et al.
Revised: [2014], Wu et al.

Original: [2015], Li et al.
Revised: [2015], Li et al.

Original: [2020], is asymptotically a lower bound for Reg A ( T )/ log ( T ) .
Revised: [2020], is asymptotically a lower bound for Reg A ( T )/ log ( T ) .

Original: We prove that no algorithm A can achieve a ﬁnite-time pseudo-regret guarantee of the form Reg A ( T ) ≤ O ( c ∗ log ( T )) .
Revised: We prove that no algorithm A can achieve a ﬁnite-time pseudo-regret guarantee of the form Reg A ( T ) ≤ O ( c ∗ log ( T )) .

Original: Moreover, we show that there exists a feedback graph G for which any algorithm suffers a regret of at least Ω ( K 1 8 ( c ∗ + 1 ∆ min )) , where ∆ min is the minimum reward gap.
Revised: Moreover, we show that there exists a feedback graph G for which any algorithm suffers a regret of at least Ω ( K 1 8 ( c ∗ + 1 ∆ min )) , where ∆ min is the minimum reward gap.

Original: We discuss alternative choices and propose a notion of ﬁnite-time optimality that we argue is meaningful , based on a regret quantity d ∗ that we show any algorithm must incur in the worst case.
Revised: We discuss alternative choices and propose a notion of ﬁnite-time optimality that we argue is meaningful , based on a regret quantity d ∗ that we show any algorithm must incur in the worst case.

Original: For that notion, we give an algorithm whose pseudo-regret is quasi-optimal, both in ﬁnite-time and asymptotically and can be upper bounded by O ( c ∗ log ( T ) + d ∗ ) .
Revised: For that notion, we give an algorithm whose pseudo-regret is quasi-optimal, both in ﬁnite-time and asymptotically and can be upper bounded by O ( c ∗ log ( T ) + d ∗ ) .

Original:  Learning scenario
Revised:  Learning scenario

Original: We consider the problem of online learning with stochastic rewards and a ﬁxed undirected feedback graph.
Revised: We consider the problem of online learning with stochastic rewards and a ﬁxed undirected feedback graph.

Original: As in the familiar multi-armed bandit problem, the learner can choose one of K ≥ 1 arms.
Revised: As in the familiar multi-armed bandit problem, the learner can choose one of K ≥ 1 arms.

Original: Each arm i ∈
Revised: Each arm i ∈

Original: [ K ] admits a reward distribution, with mean µ i .
Revised: [ K ] admits a reward distribution, with mean µ i .

Original: For all our lower bounds, we assume that the distribution of the reward of each arm is Gaussian with variance 1 / √ 2 .
Revised: For all our lower bounds, we assume that the distribution of the reward of each arm is Gaussian with variance 1 / √ 2 .

Original: For our upper bounds, we only assume that the distribution of each arm is sub-Gaussian with variance proxy bounded by 1 .
Revised: For our upper bounds, we only assume that the distribution of each arm is sub-Gaussian with variance proxy bounded by 1 .

Original: We assume that the means are always bounded in [ 0 , 1 ] .
Revised: We assume that the means are always bounded in [ 0 , 1 ] .

Original: For arm i , we denote by ∆ i = µ ∗ − µ i its mean gap to the best µ ∗ =
Revised: For arm i , we denote by ∆ i = µ ∗ − µ i its mean gap to the best µ ∗ =

Original: max i ∈
Revised: max i ∈

Original: [ K ] µ i .
Revised: [ K ] µ i .

Original: We will also denote by ∆ min the smallest and by ∆ max the largest of these gaps.
Revised: We will also denote by ∆ min the smallest and by ∆ max the largest of these gaps.

Original: At each round t ∈ [ T ] , the learner selects an arm i t and receives a reward r t,i t drawn from the reward distribution of arm i t .
Revised: At each round t ∈ [ T ] , the learner selects an arm i t and receives a reward r t,i t drawn from the reward distribution of arm i t .

Original: In addition to observing that reward, the learner observes the reward of some other arms, as speciﬁed by an undirected graph G =
Revised: In addition to observing that reward, the learner observes the reward of some other arms, as speciﬁed by an undirected graph G =

Original: ( V,E ) , where the vertex set V coincides with [ K ] : an edge e ∈ E between vertices i and j indicates that the learner observes the reward of arm j when selecting arm i and vice-versa.
Revised: ( V,E ) , where the vertex set V coincides with [ K ] : an edge e ∈ E between vertices i and j indicates that the learner observes the reward of arm j when selecting arm i and vice-versa.

Original: We will denote by N i the set of neighbors of arm i in G , N i = { j ∈ V ∶( i,j ) ∈ E } , and will assume self-loops at every vertex, that is, we have i ∈ N i for all i ∈ V .
Revised: We will denote by N i the set of neighbors of arm i in G , N i = { j ∈ V ∶( i,j ) ∈ E } , and will assume self-loops at every vertex, that is, we have i ∈ N i for all i ∈ V .

Original: The objective of the learner A is to minimize its pseudo-regret , that is the expected cumulative gap between the reward of an optimal arm i ∗ and its reward:
Revised: The objective of the learner A is to minimize its pseudo-regret , that is the expected cumulative gap between the reward of an optimal arm i ∗ and its reward:

Original: [Equation]
Revised: [Equation]

Original: where the expectation is taken over the random draw of a reward from an arm’s distribution and the possibly randomized selection strategy of the learner.
Revised: where the expectation is taken over the random draw of a reward from an arm’s distribution and the possibly randomized selection strategy of the learner.

Original: In the following, we may sometimes abusively use the shorter term regret instead of pseudo-regret.
Revised: In the following, we may sometimes abusively use the shorter term regret instead of pseudo-regret.

Original: We will denote by I ∗ the set of optimal arms, that is, arms with mean reward µ ∗ , and, for any t ∈
Revised: We will denote by I ∗ the set of optimal arms, that is, arms with mean reward µ ∗ , and, for any t ∈

Original: [ T ] will denote by r t the vector of all rewards r t,i at time t .
Revised: [ T ] will denote by r t the vector of all rewards r t,i at time t .

Original: When discussing asymptotic or ﬁnite-time optimality, we assume the setting of Gaussian rewards.
Revised: When discussing asymptotic or ﬁnite-time optimality, we assume the setting of Gaussian rewards.

Original: We will assume an informed setting where the graph G is ﬁxed and accessible to the learner before the start of the game.
Revised: We will assume an informed setting where the graph G is ﬁxed and accessible to the learner before the start of the game.

Original: Our analysis makes use of the following standard graph theory notions [Goddard and Henning, 2013].
Revised: Our analysis makes use of the following standard graph theory notions [Goddard and Henning, 2013].

Original: A subset of the vertices is independent if no two vertices in it are adjacent.
Revised: A subset of the vertices is independent if no two vertices in it are adjacent.

Original: The independence number of G , α ( G ) , is the size of the maximum independent set in G .
Revised: The independence number of G , α ( G ) , is the size of the maximum independent set in G .

Original: A dominating set of G is a subset S ⊆ V such that every vertex not in S is adjacent to S .
Revised: A dominating set of G is a subset S ⊆ V such that every vertex not in S is adjacent to S .

Original: The domination number of G , γ ( G ) , is the minimum size of a dominating set.
Revised: The domination number of G , γ ( G ) , is the minimum size of a dominating set.

Original: It is known that for any graph G , we have γ ( G ) ≤ α ( G ) .
Revised: It is known that for any graph G , we have γ ( G ) ≤ α ( G ) .

Original: The difference between the domination and independence numbers can be substantial in many cases.
Revised: The difference between the domination and independence numbers can be substantial in many cases.

Original: For example, for a star graph with n vertices, we have γ ( G ) =
Revised: For example, for a star graph with n vertices, we have γ ( G ) =

Original: 1 and α ( G )
Revised: 1 and α ( G )

Original: =
Revised: =

Original: n − 1 .
Revised: n − 1 .

Original: In the following, in the absence of any ambiguity, we simply drop the graph arguments and write α or γ .
Revised: In the following, in the absence of any ambiguity, we simply drop the graph arguments and write α or γ .

Original: We will denote by D( G ′ )
Revised: We will denote by D( G ′ )

Original: the minimum dominating set of a sub-graph G ′ ⊆
Revised: the minimum dominating set of a sub-graph G ′ ⊆

Original: G and by I( G ′ )
Revised: G and by I( G ′ )

Original: the maximum independent set.
Revised: the maximum independent set.

Original: When the minimum dominating set is not unique, D( G ′ ) can be selected in an arbitrary but ﬁxed way.
Revised: When the minimum dominating set is not unique, D( G ′ ) can be selected in an arbitrary but ﬁxed way.

Original:  Sub-optimality of previous algorithms
Revised:  Sub-optimality of previous algorithms

Original: In this section, we discuss in more detail the previous work most closely related to ours [Buccapatnam et al., 2014, Wu et al., 2015, Buccapatnam et al., 2017b, Li et al., 2020] and demonstrate their sub-optimality.
Revised: In this section, we discuss in more detail the previous work most closely related to ours [Buccapatnam et al., 2014, Wu et al., 2015, Buccapatnam et al., 2017b, Li et al., 2020] and demonstrate their sub-optimality.

Original: A summary of our comparison can be found in Table 1.
Revised: A summary of our comparison can be found in Table 1.

Original: These algorithms all seek to achieve instance-dependent optimal regret bounds by solving and playing according to the following linear program (LP), which is known to characterize the instance-dependent asymptotic regret for this problem when the rewards follow a Gaussian distribution:
Revised: These algorithms all seek to achieve instance-dependent optimal regret bounds by solving and playing according to the following linear program (LP), which is known to characterize the instance-dependent asymptotic regret for this problem when the rewards follow a Gaussian distribution:

Original: [Equation]
Revised: [Equation]

Original: We note that these prior works’ algorithms can work in more general settings, but we will restrict our discussion to their use in the informed setting with a ﬁxed feedback graph that we consider in this study.
Revised: We note that these prior works’ algorithms can work in more general settings, but we will restrict our discussion to their use in the informed setting with a ﬁxed feedback graph that we consider in this study.

Original: The UCB - LP algorithm of Buccapatnam et al.
Revised: The UCB - LP algorithm of Buccapatnam et al.

Original: [2014, 2017b] is based on the following modiﬁcation of LP1: min x ∈ R K + ⟨ x, 1 ⟩ subject to ∑ j ∈ N i x j ≥ 1 , for all i ∈
Revised: [2014, 2017b] is based on the following modiﬁcation of LP1: min x ∈ R K + ⟨ x, 1 ⟩ subject to ∑ j ∈ N i x j ≥ 1 , for all i ∈

Original: [ K ] , in which the gap information is eliminated, working with gaps such that ∆ min =
Revised: [ K ] , in which the gap information is eliminated, working with gaps such that ∆ min =

Original: Θ ( ∆ max ) .
Revised: Θ ( ∆ max ) .

Original: This modiﬁed problem is the LP relaxation of the minimum dominating set integer program of graph G .
Revised: This modiﬁed problem is the LP relaxation of the minimum dominating set integer program of graph G .

Original: The algorithm ﬁrst solves this minimum dominating set relaxation and then proceeds as an actionelimination algorithm in O ( log ( T )) phases.
Revised: The algorithm ﬁrst solves this minimum dominating set relaxation and then proceeds as an actionelimination algorithm in O ( log ( T )) phases.

Original: During the ﬁrst O ( log ( K )) rounds, their algorithm plays by exploring based on the solution of their LP.
Revised: During the ﬁrst O ( log ( K )) rounds, their algorithm plays by exploring based on the solution of their LP.

Original: Once the exploration rounds have concluded, it simply behaves as a bandit action-elimination algorithm.
Revised: Once the exploration rounds have concluded, it simply behaves as a bandit action-elimination algorithm.

Original: We argue below that this algorithm is sub-optimal, in at least two ways.
Revised: We argue below that this algorithm is sub-optimal, in at least two ways.

Original: Star graph with equal gaps.
Revised: Star graph with equal gaps.

Original: Consider the case where the feedback graph is a star graph (Figure 1(a)): there is one root or revealing vertex r adjacent to all other vertices.
Revised: Consider the case where the feedback graph is a star graph (Figure 1(a)): there is one root or revealing vertex r adjacent to all other vertices.

Original: In our construction, the optimal arm is chosen uniformly at random among the leaves of the graph.
Revised: In our construction, the optimal arm is chosen uniformly at random among the leaves of the graph.

Original: The rewards are chosen so that all sub-optimal arms admit the same expected reward with gap to the best ∆ ≤ O ( 1 / K 1 + (cid:15) ) , (cid:15) > 0 .
Revised: The rewards are chosen so that all sub-optimal arms admit the same expected reward with gap to the best ∆ ≤ O ( 1 / K 1 + (cid:15) ) , (cid:15) > 0 .

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: In this case, an optimal strategy consists of playing the revealing arm for Θ ( 1 / ∆ 2 ) rounds to identify the optimal arm, and thus incurs regret at most O ( 1∆ ) .
Revised: In this case, an optimal strategy consists of playing the revealing arm for Θ ( 1 / ∆ 2 ) rounds to identify the optimal arm, and thus incurs regret at most O ( 1∆ ) .

Original: On the other hand the UCB - LP strategy incurs regret at least Ω ( K log ( T ) ∆ ) .
Revised: On the other hand the UCB - LP strategy incurs regret at least Ω ( K log ( T ) ∆ ) .

Original: Even if we ignore the dependence on the time horizon, the dependence on K is clearly sub-optimal.
Revised: Even if we ignore the dependence on the time horizon, the dependence on K is clearly sub-optimal.

Original: Sub-optimality of using the minimum dominating set relaxation.
Revised: Sub-optimality of using the minimum dominating set relaxation.

Original: In the second problem instance, given in Figure 1(b), we consider a star-like graph in which we have a revealing vertex r 1 , adjacent to all other vertices.
Revised: In the second problem instance, given in Figure 1(b), we consider a star-like graph in which we have a revealing vertex r 1 , adjacent to all other vertices.

Original: We also have an "almost" revealing vertex r 2 which is adjacent to all vertices but a single leaf vertex (leaves are the vertices with degree 1 and 2 in this case).
Revised: We also have an "almost" revealing vertex r 2 which is adjacent to all vertices but a single leaf vertex (leaves are the vertices with degree 1 and 2 in this case).

Original: The optimal arm is again chosen uniformly among the leaves.
Revised: The optimal arm is again chosen uniformly among the leaves.

Original: Rewards are set so that the gap at r 1 is ∆ max and the remaining gaps are ∆ min .
Revised: Rewards are set so that the gap at r 1 is ∆ max and the remaining gaps are ∆ min .

Original: The solution to the LP of Buccapatnam et al.
Revised: The solution to the LP of Buccapatnam et al.

Original: [2014, 2017b] puts all the weights on r 1 .
Revised: [2014, 2017b] puts all the weights on r 1 .

Original: However, the optimal policy for this problem consists of playing r 2 and the leaf vertex not adjacent to r 2 until all arms but the optimal arm are eliminated.
Revised: However, the optimal policy for this problem consists of playing r 2 and the leaf vertex not adjacent to r 2 until all arms but the optimal arm are eliminated.

Original: The instance optimal regret in this case is O ( log ( T ) ∆ min ) , while UCB - LP incurs regret Ω ( K log ( T ) ∆ min ) .
Revised: The instance optimal regret in this case is O ( log ( T ) ∆ min ) , while UCB - LP incurs regret Ω ( K log ( T ) ∆ min ) .

Original: Next, we discuss [Wu et al., 2015] and [Li et al., 2020].
Revised: Next, we discuss [Wu et al., 2015] and [Li et al., 2020].

Original: Their instance-dependent algorithms are based on iteratively solving empirical approximations to LP1.
Revised: Their instance-dependent algorithms are based on iteratively solving empirical approximations to LP1.

Original: For simplicity, we only discuss the instance-dependent regret bound in [Li et al., 2020].
Revised: For simplicity, we only discuss the instance-dependent regret bound in [Li et al., 2020].

Original: A similar bound can be found in [Wu et al., 2015].
Revised: A similar bound can be found in [Wu et al., 2015].

Original: Let x ( ∆ ) denote the solution of LP1 and deﬁne the following perturbed solution
Revised: Let x ( ∆ ) denote the solution of LP1 and deﬁne the following perturbed solution

Original: [Equation]
Revised: [Equation]

Original: The solution x ( ∆ ,(cid:15) ) is the solution of LP1 with (cid:15) -perturbed gaps.
Revised: The solution x ( ∆ ,(cid:15) ) is the solution of LP1 with (cid:15) -perturbed gaps.

Original: [Li et al., 2020][Theorem 4] states that the expected regret of their algorithm is bounded as follows:
Revised: [Li et al., 2020][Theorem 4] states that the expected regret of their algorithm is bounded as follows:

Original: Reg ( T ) ≤
Revised: Reg ( T ) ≤

Original: O (∑ i ∈[ K ] log ( T ) x i ( ∆ ,(cid:15) )
Revised: O (∑ i ∈[ K ] log ( T ) x i ( ∆ ,(cid:15) )

Original: ∆ i + ∑ Tt = 1 exp (− β ( t ) (cid:15) 2 K ) + K ) , for any (cid:15) > 0 and β ( t ) =
Revised: ∆ i + ∑ Tt = 1 exp (− β ( t ) (cid:15) 2 K ) + K ) , for any (cid:15) > 0 and β ( t ) =

Original: o ( t ) .
Revised: o ( t ) .

Original: For the standard bandit problem with Gaussian rewards, we can compute the perturbed solution: x i ( ∆ ,(cid:15) )
Revised: For the standard bandit problem with Gaussian rewards, we can compute the perturbed solution: x i ( ∆ ,(cid:15) )

Original: =
Revised: =

Original: max ( 1 ( ∆ i + (cid:15) ) 2 , 1 ( ∆ i − (cid:15) ) 2 ) .
Revised: max ( 1 ( ∆ i + (cid:15) ) 2 , 1 ( ∆ i − (cid:15) ) 2 ) .

Original: Thus, for a meaningful regret bound, we would need (cid:15) =
Revised: Thus, for a meaningful regret bound, we would need (cid:15) =

Original: Θ ( ∆ min ) .
Revised: Θ ( ∆ min ) .

Original: If (cid:15) is much smaller, then the term ∑ Tt = 1 exp (− β ( t ) (cid:15) 2 K ) becomes too large and otherwise we risk making x i ( ∆ ,(cid:15) ) too large.
Revised: If (cid:15) is much smaller, then the term ∑ Tt = 1 exp (− β ( t ) (cid:15) 2 K ) becomes too large and otherwise we risk making x i ( ∆ ,(cid:15) ) too large.

Original: To analyze the second term more carefully, we allow β ( t ) =
Revised: To analyze the second term more carefully, we allow β ( t ) =

Original: t .
Revised: t .

Original: The ﬁrst K ∆ 2min terms of ∑ Tt = 1 exp (− β ( t ) (cid:15) 2 K ) are now at least 1 e and thus this sum is at least ∑ Tt = 1 exp (− β ( t ) (cid:15) 2 K ) ≥
Revised: The ﬁrst K ∆ 2min terms of ∑ Tt = 1 exp (− β ( t ) (cid:15) 2 K ) are now at least 1 e and thus this sum is at least ∑ Tt = 1 exp (− β ( t ) (cid:15) 2 K ) ≥

Original: K e ∆ 2min .
Revised: K e ∆ 2min .

Original: Thus, the bandit regret bound evaluates to at least Ω (∑ i ∈[ K ] log ( T ) ∆ i + K ∆ 2min ) .
Revised: Thus, the bandit regret bound evaluates to at least Ω (∑ i ∈[ K ] log ( T ) ∆ i + K ∆ 2min ) .

Original: While this bound is asymptotically optimal, since the second term does not have dependence on T , it admits a very poor dependence on the smallest gap.
Revised: While this bound is asymptotically optimal, since the second term does not have dependence on T , it admits a very poor dependence on the smallest gap.

Original: We can repeat the argument above with a star-graph construction in which the revealing vertex has gap ∆ min .
Revised: We can repeat the argument above with a star-graph construction in which the revealing vertex has gap ∆ min .

Original: In this case, the optimal strategy given by the solution to
Revised: In this case, the optimal strategy given by the solution to

Original: LP1 consists of playing the revealing vertex for 1∆ 2min times and incurs regret at most O ( log ( T ) ∆ min ) .
Revised: LP1 consists of playing the revealing vertex for 1∆ 2min times and incurs regret at most O ( log ( T ) ∆ min ) .

Original: The regret bound of the algorithm of Li et al.
Revised: The regret bound of the algorithm of Li et al.

Original: [2020], however, amounts to Ω ( log ( T ) ∆ min + K ∆ 2min ) .
Revised: [2020], however, amounts to Ω ( log ( T ) ∆ min + K ∆ 2min ) .

Original:  Instance-dependent ﬁnite-time bounds
Revised:  Instance-dependent ﬁnite-time bounds

Original: In this section, we provide an in-depth discussion of what ﬁnite-time optimality actually means.
Revised: In this section, we provide an in-depth discussion of what ﬁnite-time optimality actually means.

Original: Finite-time bounds are statements of the form Reg ( T ) ≤ f ( T ) , which hold for any T > 0 .
Revised: Finite-time bounds are statements of the form Reg ( T ) ≤ f ( T ) , which hold for any T > 0 .

Original: Speciﬁcally, we are considering functions of the type f ( T ) =
Revised: Speciﬁcally, we are considering functions of the type f ( T ) =

Original: c ∗ log ( T ) + d, 1 which we know to exist from prior work.
Revised: c ∗ log ( T ) + d, 1 which we know to exist from prior work.

Original: The question of what the optimal expression of d might be seems easy to answer at ﬁrst.
Revised: The question of what the optimal expression of d might be seems easy to answer at ﬁrst.

Original: Indeed, for bandits with Gaussian rewards, one can achieve d = O (∑ i ∆ i ) , which in general is much smaller than c ∗ =
Revised: Indeed, for bandits with Gaussian rewards, one can achieve d = O (∑ i ∆ i ) , which in general is much smaller than c ∗ =

Original: Ω (∑ i 1∆ i )
Revised: Ω (∑ i 1∆ i )

Original: [Lattimore and Szepesvári, 2020], and hence will be dominated by the time-dependent part of the regret for almost all reasonable lengths of the time horizon T .
Revised: [Lattimore and Szepesvári, 2020], and hence will be dominated by the time-dependent part of the regret for almost all reasonable lengths of the time horizon T .

Original: In full information, we obtain a meaningful optimal value d ∗ for a given gap vector by considering the worst-case regret of any algorithm under any permutation of the arms.
Revised: In full information, we obtain a meaningful optimal value d ∗ for a given gap vector by considering the worst-case regret of any algorithm under any permutation of the arms.

Original: This leads to d ∗ =
Revised: This leads to d ∗ =

Original: O ( ln ( K ) ∆ min )
Revised: O ( ln ( K ) ∆ min )

Original: [Mourtada and Gaïffas, 2019].
Revised: [Mourtada and Gaïffas, 2019].

Original: Note that in the full information setting we have c ∗ = 0 .
Revised: Note that in the full information setting we have c ∗ = 0 .

Original: One might hope for a similar structure for feedback graphs, where the optimal d depends only on the “full-information structure”, that is the gaps of arms neighboring an optimal arm.
Revised: One might hope for a similar structure for feedback graphs, where the optimal d depends only on the “full-information structure”, that is the gaps of arms neighboring an optimal arm.

Original: All other arms contribute to c ∗ and we might assume that their complexity is already captured in the c ∗ log ( T ) term as it is the case for bandits.
Revised: All other arms contribute to c ∗ and we might assume that their complexity is already captured in the c ∗ log ( T ) term as it is the case for bandits.

Original: However, the situation is more complicated, as we show next.
Revised: However, the situation is more complicated, as we show next.

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: For any K ≥ 32 and ∆ min =
Revised: For any K ≥ 32 and ∆ min =

Original: O( 1 √ K ) , there exists a graph G with K vertices and an instance with unique best arm and minimal gap ∆ min, such that for any algorithm and any T ≥ Ω ( K 3 / 4 ∆ min ) , the regret is at least Reg ( T ) =
Revised: O( 1 √ K ) , there exists a graph G with K vertices, such that for any algorithm there exists an instance with unique best arm and minimal gap ∆ min for which for any T ≥ Ω ( K 3 / 4 ∆ min ) , the regret is at least Reg ( T ) =

Original: Ω ( K 18 ( c ∗ + 1∆ min )) .
Revised: Ω ( K 18 ( c ∗ + 1∆ min )) .

Original: Theorem 4.1 shows that there exists a problem instance, in which d ≫ c ∗ dominates the ﬁnite time regret for any T ≤ O ( exp ( K 1 / 8 )) .
Revised: Theorem 4.1 shows that there exists a problem instance, in which d ≫ c ∗ dominates the ﬁnite time regret for any T ≤ O ( exp ( K 1 / 8 )) .

Original: We note that this is not simply due to the full-information structure of the feedback graph G , as c ∗ is positive.
Revised: We note that this is not simply due to the full-information structure of the feedback graph G , as c ∗ is positive.

Original: Furthermore, combined with examples shown in Section 7, Theorem 4.1 suggests that there is no simple characterization of d independent of the graph in terms of c ∗ , e.g., d = Θ ( c ∗ / ∆ min ) .
Revised: Furthermore, combined with examples shown in Section 7, Theorem 4.1 suggests that there is no simple characterization of d independent of the graph in terms of c ∗ , e.g., d = Θ ( c ∗ / ∆ min ) .

Original: While d = Θ ( K 1 / 8 c ∗ ) holds for the example instance in
Revised: While d = Θ ( K 1 / 8 c ∗ ) holds for the example instance in

Original: Theorem 4.1, there exists a non-trivial family of graphs, for which for any rewards instance, we have d = O ( c ∗ ) , so the shape and relevance of d is highly graph-dependent.
Revised: Theorem 4.1, there exists a non-trivial family of graphs, for which for any rewards instance, we have d = O ( c ∗ ) , so the shape and relevance of d is highly graph-dependent.

Original: Having established that d could be the dominating term in the regret for any reasonable time horizon, we now discuss the hardness of deﬁning an optimal d .
Revised: Having established that d could be the dominating term in the regret for any reasonable time horizon, we now discuss the hardness of deﬁning an optimal d .

Original: Let us ﬁrst consider a simple two-arm fullinformation problem and inspect algorithms of the style: “Play arm 1, unless the cumulative reward of arm 2 exceeds that of arm 1 by a threshold of τ .”
Revised: Let us ﬁrst consider a simple two-arm fullinformation problem and inspect algorithms of the style: “Play arm 1, unless the cumulative reward of arm 2 exceeds that of arm 1 by a threshold of τ .”

Original: This kind of algorithm has small regret (small d ) if arm 1 is optimal, and large otherwise.
Revised: This kind of algorithm has small regret (small d ) if arm 1 is optimal, and large otherwise.

Original: Tuning τ yields different trade-offs between the two scenarios.
Revised: Tuning τ yields different trade-offs between the two scenarios.

Original: The same issue appears in learning with graph feedback on a larger scale.
Revised: The same issue appears in learning with graph feedback on a larger scale.

Original: Given two instances deﬁned by gap vectors ∆ and ∆ ′ respectively, an agent can trade off the constant regret part d ( ∆ ) and d ( ∆ ′ ) in the two instances.
Revised: Given two instances deﬁned by gap vectors ∆ and ∆ ′ respectively, an agent can trade off the constant regret part d ( ∆ ) and d ( ∆ ′ ) in the two instances.

Original: Take for example two algorithms A and B and assume the respective values of c ∗ and d for the two instances and algorithms are given by Table 2.
Revised: Take for example two algorithms A and B and assume the respective values of c ∗ and d for the two instances and algorithms are given by Table 2.

Original: As we show in Appendix A, there exist a feedback graph and instances that are consistent with the table.
Revised: As we show in Appendix A, there exist a feedback graph and instances that are consistent with the table.

Original: Which algorithm is more “optimal”, A or B ?
Revised: Which algorithm is more “optimal”, A or B ?

Original: B ensures that max δ ∈{ ∆ , ∆ ′ } d ( δ ) c ∗
Revised: B ensures that max δ ∈{ ∆ , ∆ ′ } d ( δ ) c ∗

Original: =
Revised: =

Original: O( 1 ) and we can c ∗ d for Alg.
Revised: O( 1 ) and we can c ∗ d for Alg.

Original: A d for Alg.
Revised: A d for Alg.

Original: B
Revised: B

Original: Instance ∆
Revised: Instance ∆

Original: C /
Revised: C /

Original: C /4 C /
Revised: C /4 C /

Original: Instance ∆ ′
Revised: Instance ∆ ′

Original: Cε
Revised: Cε

Original: C /

Revised: C /

Original: Cε write the regret function as f ( T ) =
Revised: Cε write the regret function as f ( T ) =

Original: O( c ∗ log ( T )) without the need of a constant term d at all.
Revised: O( c ∗ log ( T )) without the need of a constant term d at all.

Original: This algorithm minimizes the competitive ratio of d and c ∗ .
Revised: This algorithm minimizes the competitive ratio of d and c ∗ .

Original: A minimizes the worst-case absolute regretA
Revised: A minimizes the worst-case absolute regret

Original: [Equation]
Revised: A = argmin a ∈{A , B} max δ ∈{ ∆ , ∆ ′ } d ( δ,a ) .

Original: Thus, we argue that the notion of optimality is subject to a choice and that there is no unique correct answer.
Revised: Thus, we argue that the notion of optimality is subject to a choice and that there is no unique correct answer.

Original: In this paper, we opt for the worst-case absolute regret, minimized by A in the above example, for the following reasons: 1.
Revised: In this paper, we opt for the worst-case absolute regret, minimized by A in the above example, for the following reasons: 1.

Original: Theorem 4.1 shows that a constant competitive ratio is generally unachievable; 2.
Revised: Theorem 4.1 shows that a constant competitive ratio is generally unachievable; 2.

Original: Optimizing regret in general is a different objective than that of competitive ratio.
Revised: Optimizing regret in general is a different objective than that of competitive ratio.

Original: Optimizing for a mixture implies a counter-intuitive preference such as: “In a hard environment where I cannot avoid suffering a loss of 1000, it does not matter much if I suffer an additional 1000 on top, as long as I do better on easier environments.”
Revised: Optimizing for a mixture implies a counter-intuitive preference such as: “In a hard environment where I cannot avoid suffering a loss of 1000, it does not matter much if I suffer an additional 1000 on top, as long as I do better on easier environments.”

Original: Moreover, note that, even if one were interested in optimizing the competitive ratio between c ∗ and d , it is unclear if one could achieve that objective computationally efﬁciently.
Revised: Moreover, note that, even if one were interested in optimizing the competitive ratio between c ∗ and d , it is unclear if one could achieve that objective computationally efﬁciently.

Original: We present our ﬁnal deﬁnition for an optimal notion of d ∗ in Section 6.
Revised: We present our ﬁnal deﬁnition for an optimal notion of d ∗ in Section 6.

Original: The high level idea is to take all confusing instances, where the means are perturbed by less than ∆ s and consider the worst-case regret any algorithm suffers over these instances until identifying all gaps up to ∆ s precision.
Revised: The high level idea is to take all confusing instances, where the means are perturbed by less than ∆ s and consider the worst-case regret any algorithm suffers over these instances until identifying all gaps up to ∆ s precision.

Original:  Algorithm and regret upper bounds
Revised:  Algorithm and regret upper bounds

Original: Our algorithm works by approximating the gaps ( ∆ i ) i ∈
Revised: Our algorithm works by approximating the gaps ( ∆ i ) i ∈

Original: [ K ] and then solving a version of LP1.
Revised: [ K ] and then solving a version of LP1.

Original: First, note that all arms i with gaps ∆ i ≤ 1 T can be ignored as the total contribution to the regret is at most O ( 1 ) .
Revised: First, note that all arms i with gaps ∆ i ≤ 1 T can be ignored as the total contribution to the regret is at most O ( 1 ) .

Original: We now segment the interval [ 1 T , 1 ] , containing each relevant gap, into sub-intervals [ 2 − s , 2 − s + 1 ] , where s ∈ [⌈ log 2 ( T )⌉] .
Revised: We now segment the interval [ 1 T , 1 ] , containing each relevant gap, into sub-intervals [ 2 − s , 2 − s + 1 ] , where s ∈ [⌈ log 2 ( T )⌉] .

Original: The algorithm now proceeds in phases corresponding to each of the ⌈ log 2 ( T )⌉ intervals.
Revised: The algorithm now proceeds in phases corresponding to each of the ⌈ log 2 ( T )⌉ intervals.

Original: During phase s , all arms with gaps ∆ i ∈ [ 2 − s , 2 − s + 1 ] will be observed sufﬁciently many times to be identiﬁed as sub-optimal.
Revised: During phase s , all arms with gaps ∆ i ∈ [ 2 − s , 2 − s + 1 ] will be observed sufﬁciently many times to be identiﬁed as sub-optimal.

Original: For phase s , let ∆ s = 2 − s denote the smallest possible gap that can be part of the interval [ 2 − s , 2 − s + 1 ] and deﬁne the clipped gap vector as ∆ s ∈ R K , ∆ si = ∆ s ∨ ∆ i .
Revised: For phase s , let ∆ s = 2 − s denote the smallest possible gap that can be part of the interval [ 2 − s , 2 − s + 1 ] and deﬁne the clipped gap vector as ∆ s ∈ R K , ∆ si = ∆ s ∨ ∆ i .

Original: Further, deﬁne the set Γ s = {
Revised: Further, deﬁne the set Γ s = {

Original: i ∈
Revised: i ∈

Original: [ K ]∶ ∆ i ≤ 2∆ s } .
Revised: [ K ]∶ ∆ i ≤ 2∆ s } .

Original: Γ s consists of all optimal arms I ∗ and all sub-optimal arms with gaps small enough, making them impossible to distinguish from optimal arms.
Revised: Γ s consists of all optimal arms I ∗ and all sub-optimal arms with gaps small enough, making them impossible to distinguish from optimal arms.

Original: Deﬁne the following LP
Revised: Deﬁne the following LP

Original: [Equation]
Revised: [Equation]

Original: For any arm i such that ∆ i ∈
Revised: For any arm i such that ∆ i ∈

Original: [ 2 − s , 2 − s + 1 ] observing i for 1∆ 2 s times is sufﬁcient to identify i as a sub-optimal arm.
Revised: [ 2 − s , 2 − s + 1 ] observing i for 1∆ 2 s times is sufﬁcient to identify i as a sub-optimal arm.

Original: Further, information theory dictates that i needs to be observed at least 1∆ 2 s − 1 times to be distinguished as sub-optimal.
Revised: Further, information theory dictates that i needs to be observed at least 1∆ 2 s − 1 times to be distinguished as sub-optimal.

Original: Thus, the constraints of LP2 are necessary and sufﬁcient for identifying the sub-optimal arms i with ∆ i ∈ [ 2 − s , 2 − s + 1 ] .
Revised: Thus, the constraints of LP2 are necessary and sufﬁcient for identifying the sub-optimal arms i with ∆ i ∈ [ 2 − s , 2 − s + 1 ] .

Original: Furthermore, since there is no sufﬁcient information to distinguish between any two arms i and j with gaps ∆ i ≤ ∆ j < ∆ s , we choose to treat all of them as equal in the objective of the LP.
Revised: Furthermore, since there is no sufﬁcient information to distinguish between any two arms i and j with gaps ∆ i ≤ ∆ j < ∆ s , we choose to treat all of them as equal in the objective of the LP.

Original: Indeed, Lemma 6.1 shows that for any graph G and any algorithm, there exists an assignment of the gaps ∆ i <
Revised: Indeed, Lemma 6.1 shows that for any graph G and any algorithm, there exists an assignment of the gaps ∆ i <

Original: ∆ s so that the algorithm will suffer regret proportional to the value of LP2.
Revised: ∆ s so that the algorithm will suffer regret proportional to the value of LP2.

Original: In practice, it is impossible to devise an algorithm that solves and plays according to LP2 because even during phase s , there is still no complete knowledge of the gaps ∆ i > ∆ s , but, rather only empirical estimators, and so there is no access to ∆ s .
Revised: In practice, it is impossible to devise an algorithm that solves and plays according to LP2 because even during phase s , there is still no complete knowledge of the gaps ∆ i > ∆ s , but, rather only empirical estimators, and so there is no access to ∆ s .

Original: We also replace the constraints by a conﬁdence interval term of the order log ( 1 / δ s ) ∆ 2 s .
Revised: We also replace the constraints by a conﬁdence interval term of the order log ( 1 / δ s ) ∆ 2 s .

Original: This enables us to bound the probability of failure for the algorithm by δ s during phase s .
Revised: This enables us to bound the probability of failure for the algorithm by δ s during phase s .

Original: We note that standard choices of δ s such as δ s =
Revised: We note that standard choices of δ s such as δ s =

Original: Θ ( 1 T ) from UCB-type strategies will result in a regret bound that has a sub-optimal time-horizon dependence.
Revised: Θ ( 1 T ) from UCB-type strategies will result in a regret bound that has a sub-optimal time-horizon dependence.

Original: This suggests that a more careful choice of δ s must be determined.
Revised: This suggests that a more careful choice of δ s must be determined.

Original: 5.1 Algorithm
Revised: 5.1 Algorithm

Original: To describe our algorithm, we will adopt the following deﬁnitions and notation.
Revised: To describe our algorithm, we will adopt the following deﬁnitions and notation.

Original: Let τ s denote the last time-step of phase s .
Revised: Let τ s denote the last time-step of phase s .

Original: We will denote by n i ( s ) the total number of times the reward of arm i is observed up to and including s , n i ( s ) =
Revised: We will denote by n i ( s ) the total number of times the reward of arm i is observed up to and including s , n i ( s ) =

Original: ∑ τ s t = 1 I ( i t ∈ N i ) , and by r i ( s ) the average reward observed, ˆ r i ( s ) =
Revised: ∑ τ s t = 1 I ( i t ∈ N i ) , and by r i ( s ) the average reward observed, ˆ r i ( s ) =

Original: [∑ τ s t = 1 r t,i I ( i t ∈ N i )]/ n i ( s ) .
Revised: [∑ τ s t = 1 r t,i I ( i t ∈ N i )]/ n i ( s ) .

Original: We also denote by ˆ∆
Revised: We also denote by ˆ∆

Original: i ( s ) a lower bound on ∆ s with a shrinking conﬁdence interval b i ( s ) and by ˆΓ s the empirical version of the set Γ s :
Revised: i ( s ) a lower bound on ∆ s with a shrinking conﬁdence interval b i ( s ) and by ˆΓ s the empirical version of the set Γ s :

Original: [Equation]
Revised: [Equation]

Original: where b
Revised: where b

Original: [Equation]
Revised: [Equation]

Original: ( s ) =
Revised: ( s ) =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Input : Graph G =
Revised: Input : Graph G =

Original: ( V,E ) ,
Revised: ( V,E ) ,

Original: conﬁdence parameter δ , time horizon T 1 Initialize t = 0 , s = 0 , ˆ r i ( 0 ) = 0 , ∀ i ∈
Revised: conﬁdence parameter δ , time horizon T 1 Initialize t = 0 , s = 0 , ˆ r i ( 0 ) = 0 , ∀ i ∈

Original: [ K ] 2 Compute (approximate) minimum dominating set ˆ D( G ) 3 while s ≤ ⌈ log ( K )⌉ do
Revised: [ K ] 2 Compute (approximate) minimum dominating set ˆ D( G ) 3 while s ≤ ⌈ log ( K )⌉ do

Original: Play each arm i ∈
Revised: Play each arm i ∈

Original: ˆ D( G ) for α ′ log ( K ∆ s + 1 ) ∆ 2 s rounds
Revised: ˆ D( G ) for α ′ log ( K ∆ s + 1 ) ∆ 2 s rounds

Original: Update t and s .
Revised: Update t and s .

Original: 6 while t ≤ T do
Revised: 6 while t ≤ T do

Original: Compute a (approximate) solution x ∗ LP 3 to LP3.
Revised: Compute a (approximate) solution x ∗ LP 3 to LP3.

Original: Play each action i for ⌈ x ∗ LP 3 ) i ⌉ rounds and update t .
Revised: Play each action i for ⌈ x ∗ LP 3 ) i ⌉ rounds and update t .

Original: Update the phase s + = 1 .
Revised: Update the phase s + = 1 .

Original: Our algorithm solves an empirical version of (LP2) at each phase, which is the following LP:
Revised: Our algorithm solves an empirical version of (LP2) at each phase, which is the following LP:

Original: [Equation]
Revised: [Equation]

Original: Pseudocode can be found in Algorithm 1.
Revised: Pseudocode can be found in Algorithm 1.

Original: In the ﬁrst ⌈ log ( K )⌉ rounds, the algorithm just plays according to the minimum dominating set of G .
Revised: In the ﬁrst ⌈ log ( K )⌉ rounds, the algorithm just plays according to the minimum dominating set of G .

Original: This is because there is not enough information regarding any of the gaps.
Revised: This is because there is not enough information regarding any of the gaps.

Original: Denote the approximate solution of LP3 as x ∗ LP 3 at phase s .
Revised: Denote the approximate solution of LP3 as x ∗ LP 3 at phase s .

Original: Then at every round of phase s we play each arm exactly ⌈( x ∗ LP 3 )
Revised: Then at every round of phase s we play each arm exactly ⌈( x ∗ LP 3 )

Original: i ⌉ many times.
Revised: i ⌉ many times.

Original: Phase s then ends after ∑ j ∈[ K ] ⌈( x ∗ LP 3 ) i ⌉ rounds.
Revised: Phase s then ends after ∑ j ∈[ K ] ⌈( x ∗ LP 3 ) i ⌉ rounds.

Original: We note that it is sufﬁcient to approximately solve LP3 so that the constraints are satisﬁed up to some multiplicative factor and the value of the solution is bounded by a multiplicative factor in the value of the LP.
Revised: We note that it is sufﬁcient to approximately solve LP3 so that the constraints are satisﬁed up to some multiplicative factor and the value of the solution is bounded by a multiplicative factor in the value of the LP.

Original: 5.2 Regret bound
Revised: 5.2 Regret bound

Original: The ﬁrst step in the regret analysis of Algorithm 1 is to relate the value of LP3 to the value of LP based on the true gaps given below.
Revised: The ﬁrst step in the regret analysis of Algorithm 1 is to relate the value of LP3 to the value of LP based on the true gaps given below.

Original: [Equation]
Revised: [Equation]

Original: We do so by showing that ˆ Γ s + 1 ⊆
Revised: We do so by showing that ˆ Γ s + 1 ⊆

Original: Γ s and that ˆ ∆ ( s )
Revised: Γ s and that ˆ ∆ ( s )

Original: =
Revised: =

Original: Θ ( ∆ s ) .
Revised: Θ ( ∆ s ) .

Original: This allows us to upper upper bound the value of LP3 by the value of LP4 in the following way.
Revised: This allows us to upper upper bound the value of LP3 by the value of LP4 in the following way.

Original: Lemma 5.1.
Revised: Lemma 5.1.

Original: Let D LP 3 ( s ) be the value of LP3 at phase s and let D LP 4 ( s ) be the value of LP4 at phase s .
Revised: Let D LP 3 ( s ) be the value of LP3 at phase s and let D LP 4 ( s ) be the value of LP4 at phase s .

Original: For any s ≥ log ( K ) ∨ 10 holds that D LP 3 ( s + 1 ) ≤ 4 D LP 4 ( s ) , with probability at least 1 − 3 ( ∆ s / 2 + 1 K ) α − 2 .
Revised: For any s ≥ log ( K ) ∨ 10 holds that D LP 3 ( s + 1 ) ≤ 4 D LP 4 ( s ) , with probability at least 1 − 3 ( ∆ s / 2 + 1 K ) α − 2 .

Original: Further, for any s ≥ log (∣ I ∗ ∣/( 4∆ min )) ∨ 10 it holds that the regret incurred for playing according to LP3 is at most 16 α ′ c ∗ ( G,µ ) with the same probability.
Revised: Further, for any s ≥ log (∣ I ∗ ∣/( 4∆ min )) ∨ 10 it holds that the regret incurred for playing according to LP3 is at most 16 α ′ c ∗ ( G,µ ) with the same probability.

Original: Lemma 5.1 shows that playing Algorithm 1 is already asymptotically optimal, as the incurred regret during any phase s ≥ log (∣ I ∗ ∣/ ∆ min ) starts being bounded by O ( c ∗ ) .
Revised: Lemma 5.1 shows that playing Algorithm 1 is already asymptotically optimal, as the incurred regret during any phase s ≥ log (∣ I ∗ ∣/ ∆ min ) starts being bounded by O ( c ∗ ) .

Original: There are two challenging parts in proving Lemma 5.1.
Revised: There are two challenging parts in proving Lemma 5.1.

Original: First is how to handle the concentration of ˆ∆ i ( s ) for actions i /∈ ˆΓ s which have been eliminated prior to phase s .
Revised: First is how to handle the concentration of ˆ∆ i ( s ) for actions i /∈ ˆΓ s which have been eliminated prior to phase s .

Original: This challenge arises because α ′ needs to be set as a time-independent parameter as the time-horizon part of the regret incurred by the algorithm will depend on α ′ .
Revised: This challenge arises because α ′ needs to be set as a time-independent parameter as the time-horizon part of the regret incurred by the algorithm will depend on α ′ .

Original: We notice that for any phase s ≥ 2 the event that the empirical reward, ˆ r i ( t ) , concentrates uniformly around its mean µ i in the interval t ∈
Revised: We notice that for any phase s ≥ 2 the event that the empirical reward, ˆ r i ( t ) , concentrates uniformly around its mean µ i in the interval t ∈

Original: [ s / 2 ,s ] can be controlled with high probability.
Revised: [ s / 2 ,s ] can be controlled with high probability.

Original: This in turn guarantees that the empirical gap estimator ˆ∆ ( t ) is small enough and hence action i is observed sufﬁciently many times in phases [ s / 2 ,s ] .
Revised: This in turn guarantees that the empirical gap estimator ˆ∆ ( t ) is small enough and hence action i is observed sufﬁciently many times in phases [ s / 2 ,s ] .

Original: The second challenge is to analyze the regret of the solution of LP3 directly, for any s ≥ log ( ∣ I ∗ ∣ K ) so that we can bound this regret by c ∗ .
Revised: The second challenge is to analyze the regret of the solution of LP3 directly, for any s ≥ log ( ∣ I ∗ ∣ K ) so that we can bound this regret by c ∗ .

Original: The key observation is that there exists a ˆ x ∗ which is feasible (with high probability) for LP1 with the property that ⟨ ˆ x ∗ , ˆ∆ ( s )⟩ ≤ O ( c ∗ ) and further ∑ i /∈
Revised: The key observation is that there exists a ˆ x ∗ which is feasible (with high probability) for LP1 with the property that ⟨ ˆ x ∗ , ˆ∆ ( s )⟩ ≤ O ( c ∗ ) and further ∑ i /∈

Original: I ∗ x ∗ LP 3 ,i ˆ∆
Revised: I ∗ x ∗ LP 3 ,i ˆ∆

Original: i ( s ) ≤ 2 ∑ i /∈
Revised: i ( s ) ≤ 2 ∑ i /∈

Original: I ∗ ˆ x ∗ i ˆ∆
Revised: I ∗ ˆ x ∗ i ˆ∆

Original: i ( s ) .
Revised: i ( s ) .

Original: This is sufﬁcient to conclude that D LP 3 ( s ) ≤ O ( c ∗ )
Revised: This is sufﬁcient to conclude that D LP 3 ( s ) ≤ O ( c ∗ )

Original: Lemma 5.1 can now be combined with the observation that the constraints of LP2 are a subset of the constraints of LP4, up to a logarithmic factor in 1∆ min , to argue the following upper regret bound.
Revised: Lemma 5.1 can now be combined with the observation that the constraints of LP2 are a subset of the constraints of LP4, up to a logarithmic factor in 1∆ min , to argue the following upper regret bound.

Original: Theorem 5.2.
Revised: Theorem 5.2.

Original: Let d ∗ ( G,µ )
Revised: Let d ∗ ( G,µ )

Original: = max s ≤ log (
Revised: = max s ≤ log (

Original: [Equation]
Revised: [Equation]

Original: I
Revised: I

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: D LP 2 ( s ) .
Revised: D LP 2 ( s ) .

Original: There exists an algorithm with expected regret Reg ( T ) bounded as
Revised: There exists an algorithm with expected regret Reg ( T ) bounded as

Original: [Equation]
Revised: [Equation]

Original: We note that Algorithm 1 can incur additional regret of order O ( K ) per phase due to the rounding, ⌈ x ∗ LP 3 ⌉ , of the solution to LP3.
Revised: We note that Algorithm 1 can incur additional regret of order O ( K ) per phase due to the rounding, ⌈ x ∗ LP 3 ⌉ , of the solution to LP3.

Original: Thus its regret will only be asymptotically optimal in the setting when ∆ min ≤ O ( 1 / K ) .
Revised: Thus its regret will only be asymptotically optimal in the setting when ∆ min ≤ O ( 1 / K ) .

Original: To ﬁx this minor issue, we present an algorithm with more careful rounding in Appendix B.1, which enjoys the regret bound of Theorem 5.2.
Revised: To ﬁx this minor issue, we present an algorithm with more careful rounding in Appendix B.1, which enjoys the regret bound of Theorem 5.2.

Original:  Regret lower bounds
Revised:  Regret lower bounds

Original: Lower bound with d ∗ .
Revised: Lower bound with d ∗ .

Original: We are able to show the following result for any algorithm.
Revised: We are able to show the following result for any algorithm.

Original: Lemma 6.1.
Revised: Lemma 6.1.

Original: Fix any instance µ s.t.
Revised: Fix any instance µ s.t.

Original: µ i ≤ 1 − 2∆ s ,i ∈ I ∗ .
Revised: µ i ≤ 1 − 2∆ s ,i ∈ I ∗ .

Original: Let Λ s ( µ ) be the set of problem instances with means µ ′ ∈ µ + [ 0 , 2∆ s ] k .
Revised: Let Λ s ( µ ) be the set of problem instances with means µ ′ ∈ µ + [ 0 , 2∆ s ] k .

Original: Then for any algorithm, there exists an instance in Λ s ( µ ) such that the regret is lower bounded by LP2.
Revised: Then for any algorithm, there exists an instance in Λ s ( µ ) such that the regret is lower bounded by LP2.

Original: Motivated by Lemma 6.1, the quantity d ∗ ( G,µ ) is a meaningful deﬁnition of ﬁnite-time optimality.
Revised: Motivated by Lemma 6.1, the quantity d ∗ ( G,µ ) is a meaningful deﬁnition of ﬁnite-time optimality.

Original: We note that d ∗ is indeed independent of the time-horizon and only depends on the topology of G and the instance µ .
Revised: We note that d ∗ is indeed independent of the time-horizon and only depends on the topology of G and the instance µ .

Original: The result in Lemma 6.1 is a companion to the upper bound in Theorem 5.2.
Revised: The result in Lemma 6.1 is a companion to the upper bound in Theorem 5.2.

Original: It shows that for any instance µ and number of observations which are not sufﬁcient to distinguish the arms with smallest positive gaps as sub-optimal, any algorithm will necessarily incur large regret of order d ∗ .
Revised: It shows that for any instance µ and number of observations which are not sufﬁcient to distinguish the arms with smallest positive gaps as sub-optimal, any algorithm will necessarily incur large regret of order d ∗ .

Original: This happens because the algorithm will not be able to distinguish µ from some environment µ ′ which is identical to µ except for the reward of a single arm which is only slightly perturbed.
Revised: This happens because the algorithm will not be able to distinguish µ from some environment µ ′ which is identical to µ except for the reward of a single arm which is only slightly perturbed.

Original: The deﬁnition of d ∗ as a maximum over different values of s might seem surprising, as one could expect that the value of LP2 strictly increases when s grows, after all this is precisely what happens in the bandit setting.
Revised: The deﬁnition of d ∗ as a maximum over different values of s might seem surprising, as one could expect that the value of LP2 strictly increases when s grows, after all this is precisely what happens in the bandit setting.

Original: This is not the case for general graphs, where the value can also decrease between phases s and s + 1 .
Revised: This is not the case for general graphs, where the value can also decrease between phases s and s + 1 .

Original: Intuitively this happens when the approximate minimum weighted dominating set chosen by the LP’s solution increases between phases.
Revised: Intuitively this happens when the approximate minimum weighted dominating set chosen by the LP’s solution increases between phases.

Original: The result in Lemma 6.1 has a min-max ﬂavor in the sense that all possible instances which are close to µ are considered.
Revised: The result in Lemma 6.1 has a min-max ﬂavor in the sense that all possible instances which are close to µ are considered.

Original: It is reasonable to ask if d ∗ can be further bounded by a favorable instancedependent quantity.
Revised: It is reasonable to ask if d ∗ can be further bounded by a favorable instancedependent quantity.

Original: The answer to this question is complicated and certainly depends on the topology of the feedback graph as we show next.
Revised: The answer to this question is complicated and certainly depends on the topology of the feedback graph as we show next.

Original: Sketch of proof of Theorem 4.1.
Revised: Sketch of proof of Theorem 4.1.

Original: We now show that any ﬁnite time term d has to exceed c ∗ by at least a multiplicative polynomial factor in the number of actions K .
Revised: We now show that any ﬁnite time term d has to exceed c ∗ by at least a multiplicative polynomial factor in the number of actions K .

Original: To do so we exhibit a speciﬁc feedback graph G , found in Figure 2, on which any algorithm will have to incur regret at least Ω ( K 18 c ∗ ) for some µ s.t. c ∗ ≥ 1∆ min .
Revised: To do so we exhibit a speciﬁc feedback graph G , found in Figure 2, on which any algorithm will have to incur regret at least Ω ( K 18 c ∗ ) for some µ s.t. c ∗ ≥ 1∆ min .

Original: Formally the graph is deﬁned to have a vertex set V = N 1 ⋃N 2 ⋃Nof 2 K + 1 arms, with each of N
Revised: Formally the graph is deﬁned to have a vertex set V = N 1 ⋃N 2 ⋃Nof 2 K + 1 arms, with each of N

Original: i ’s disjoint and N
Revised: i ’s disjoint and N

Original: 1 = { 2 i ∶ 1 ≤ i ≤ K } ,
Revised: 1 = { 2 i ∶ 1 ≤ i ≤ K } ,

Original: N 2 = { 2 i + 1 ∶ 0 ≤ i ≤ K } , N 3 = { 0 } .
Revised: N 2 = { 2 i + 1 ∶ 0 ≤ i ≤ K } , N 3 = { 0 } .

Original: The set of edges is deﬁned as follows.
Revised: The set of edges is deﬁned as follows.

Original: Every vertex in N 1 is adjacent to the vertex in N 3 and the 2 i vertex is adjacent to both 2 i + 

Revised: Every vertex in N 1 is adjacent to the vertex in N 3 and the 2 i vertex is adjacent to both 2 i +

Original: and 2 i − 1 in N 2 modulo 2 K + 1 .
Revised: and 2 i − 1 in N 2 modulo 2 K + 1 .

Original: Finally vertex 2 i + 1 in N 2 is further adjacent to to the next ⌈ K 1 / 8 ⌉ vertices in N 2 modulo 2 K + 1 .
Revised: Finally vertex 2 i + 1 in N 2 is further adjacent to to the next ⌈ K 1 / 8 ⌉ vertices in N 2 modulo 2 K + 1 .

Original: The base instance, E , is deﬁned by a scalar ν ∈ [ 0 , 1 ] and gap parameter ∆ so that the expected reward of every action in N 1 is equal to ν − ∆ , the expected reward of every action in N 2 is equal to ν − K 1 / 4 ∆ and the expected reward of the action in N is ν − √ K ∆ .
Revised: The base instance, E , is deﬁned by a scalar ν ∈ [ 0 , 1 ] and gap parameter ∆ so that the expected reward of every action in N 1 is equal to ν − ∆ , the expected reward of every action in N 2 is equal to ν − K 1 / 4 ∆ and the expected reward of the action in N is ν − √ K ∆ .

Original: We assume that all rewards follow a Gaussian with variance 1 √ 2 .
Revised: We assume that all rewards follow a Gaussian with variance 1 √ 2 .

Original: We denote by ∆ 1 = ∆ , ∆ 2 = K 14 ∆ , ∆ 3 = √ K ∆ .
Revised: We denote by ∆ 1 = ∆ , ∆ 2 = K 14 ∆ , ∆ 3 = √ K ∆ .

Original: The lower bound now ﬁxes an algorithm A and considers two cases.
Revised: The lower bound now ﬁxes an algorithm A and considers two cases.

Original: First, A could commit to often playing arms in N 1 .
Revised: First, A could commit to often playing arms in N 1 .

Original: In this case we show that there could be a large gap to the arms in N 1 which would not be detectable by A as arms in N 2 are not observed often enough.
Revised: In this case we show that there could be a large gap to the arms in N 1 which would not be detectable by A as arms in N 2 are not observed often enough.

Original: This is indeed the case as A needs to play Ω ( K ) actions in N 1 to cover N 2 .
Revised: This is indeed the case as A needs to play Ω ( K ) actions in N 1 to cover N 2 .

Original: This ﬁrst case corresponds to assuming that the number of arms played from N 2 is at most O ( K 78 / ∆ 22 ) in the ﬁrst O ( K / ∆ 22 ) rounds.
Revised: This ﬁrst case corresponds to assuming that the number of arms played from N 2 is at most O ( K 78 / ∆ 22 ) in the ﬁrst O ( K / ∆ 22 ) rounds.

Original: The second case considers the scenario in which actions in N 2 are played for more than Ω ( K 78 / ∆ 22 ) times in the ﬁrst O ( K / ∆ 22 ) rounds.
Revised: The second case considers the scenario in which actions in N 2 are played for more than Ω ( K 78 / ∆ 22 ) times in the ﬁrst O ( K / ∆ 22 ) rounds.

Original: In this case, A would suffer large regret if the gap at actions in N 1 is small enough, so that the optimal strategy is to cover N 2 by playing arms in N 1 .
Revised: In this case, A would suffer large regret if the gap at actions in N 1 is small enough, so that the optimal strategy is to cover N 2 by playing arms in N 1 .

Original: More formally, we begin by showing that there always exists an arm n ∗ ∈ N 2 which is observed for only O ( 1 / ∆ 22 ) times.
Revised: More formally, we begin by showing that there always exists an arm n ∗ ∈ N 2 which is observed for only O ( 1 / ∆ 22 ) times.

Original: Next, we change the expected reward of n ∗ depending on which of the above two cases occur.
Revised: Next, we change the expected reward of n ∗ depending on which of the above two cases occur.

Original: In the ﬁrst case we change the environment by setting the reward of n ∗ to have expectation ν + ∆ 2 .
Revised: In the ﬁrst case we change the environment by setting the reward of n ∗ to have expectation ν + ∆ 2 .

Original: We can now argue that the regret of A will be at least Ω ( K 34 / ∆ ) as n ∗ will not be played often enough in the new environment.
Revised: We can now argue that the regret of A will be at least Ω ( K 34 / ∆ ) as n ∗ will not be played often enough in the new environment.

Original: The value of c ∗ , however, is at most
Revised: The value of c ∗ , however, is at most

Original: O ( K 78 / ∆ 2 ) =
Revised: O ( K 78 / ∆ 2 ) =

Original: O ( K 58 / ∆ ) , as playing each action in a minimum dominating set over N 2 for 1∆rounds is feasible for LP1.
Revised: O ( K 58 / ∆ ) , as playing each action in a minimum dominating set over N 2 for 1∆rounds is feasible for LP1.

Original: For the second case, we set the expected reward of n ∗ to equal ν .
Revised: For the second case, we set the expected reward of n ∗ to equal ν .

Original: The optimal strategy now has regret at most O (√ K / ∆ ) by playing the action in N 3 for 1∆ 2 and every action in N 1 for 1∆ 22 rounds.
Revised: The optimal strategy now has regret at most O (√ K / ∆ ) by playing the action in N 3 for 1∆ 2 and every action in N 1 for 1∆ 22 rounds.

Original: On the other hand A will incur at least Ω ( K 58 / ∆ ) , as again n ∗ is not played often enough.
Revised: On the other hand A will incur at least Ω ( K 58 / ∆ ) , as again n ∗ is not played often enough.

Original: This argument implies the result presented in Theorem 4.1.
Revised: This argument implies the result presented in Theorem 4.1.

Original:  Characterizing the value of d ∗
Revised:  Characterizing the value of d ∗

Original: Theorem 4.1 suggests that we take into account the topology of G explicitly when trying to bound d ∗ , independently of the instance µ .
Revised: Theorem 4.1 suggests that we take into account the topology of G explicitly when trying to bound d ∗ , independently of the instance µ .

Original: In this section, we ﬁrst show a bound on d ∗ that depends only on independent sets of G .
Revised: In this section, we ﬁrst show a bound on d ∗ that depends only on independent sets of G .

Original: Then, we show a set of graphs G for which d ∗ ≤ O ( c ∗ ) on any instance µ .
Revised: Then, we show a set of graphs G for which d ∗ ≤ O ( c ∗ ) on any instance µ .

Original: Let us recall the regret bounds presented in [Lykouris et al., 2020].
Revised: Let us recall the regret bounds presented in [Lykouris et al., 2020].

Original: Denote by I ( G ) the set of all independent sets for the graph G .
Revised: Denote by I ( G ) the set of all independent sets for the graph G .

Original: Then the regret bounds presented in [Lykouris et al., 2020] are of the order
Revised: Then the regret bounds presented in [Lykouris et al., 2020] are of the order

Original: Reg ( T ) ≤ O ( max I ∈
Revised: Reg ( T ) ≤ O ( max I ∈

Original: I ( G )
Revised: I ( G )

Original: ∑ i ∈ I log 2 ( T ) ∆ i ) .
Revised: ∑ i ∈ I log 2 ( T ) ∆ i ) .

Original: It is possible to show, as we do in Appendix D.1, that d ∗ ( G,µ ) ≤ max
Revised: It is possible to show, as we do in Appendix D.1, that d ∗ ( G,µ ) ≤ max

Original: I ∈
Revised: I ∈

Original: I ( G )
Revised: I ( G )

Original: ∑ i ∈
Revised: ∑ i ∈

Original: I 1∆ i
Revised: I 1∆ i

Original: Thus, our algorithm enjoys regret bounds which are better than what is known for the algorithms studied in [Cohen et al., 2016, Lykouris et al., 2020].
Revised: Thus, our algorithm enjoys regret bounds which are better than what is known for the algorithms studied in [Cohen et al., 2016, Lykouris et al., 2020].

Original: The above bound, however, could be very loose as was discussed in the beginning of the paper, especially when considering star-graphs, as the bound would just reduce to the bandit case.
Revised: The above bound, however, could be very loose as was discussed in the beginning of the paper, especially when considering star-graphs, as the bound would just reduce to the bandit case.

Original: It turns out, however, that d ∗ ≤ O ( c ∗ ) in this case.
Revised: It turns out, however, that d ∗ ≤ O ( c ∗ ) in this case.

Original: In fact we can state a sufﬁcient condition on G so that d ∗ ≤ c ∗ + ∣ I ∗ ∣ ∆ min for a more general family of graphs.
Revised: In fact we can state a sufﬁcient condition on G so that d ∗ ≤ c ∗ + ∣ I ∗ ∣ ∆ min for a more general family of graphs.

Original: We begin by deﬁning the following operation on G .
Revised: We begin by deﬁning the following operation on G .

Original: Deﬁnition 7.1.
Revised: Deﬁnition 7.1.

Original: Let ∼ be the equivalence class deﬁned by u ∼ v iff N u
Revised: Let ∼ be the equivalence class deﬁned by u ∼ v iff N u

Original: =
Revised: =

Original: N v
Revised: N v

Original: and let C be the mapping which sends G to the quotient G / ∼ through the operation of collapsing any sub-graph of G into its equivalence class.
Revised: and let C be the mapping which sends G to the quotient G / ∼ through the operation of collapsing any sub-graph of G into its equivalence class.

Original: We note that C is well-deﬁned as the relation ∼ is an equivalence relation.
Revised: We note that C is well-deﬁned as the relation ∼ is an equivalence relation.

Original: The equivalence classes deﬁned by ∼ are cliques with the following property.
Revised: The equivalence classes deﬁned by ∼ are cliques with the following property.

Original: For any v in an equivalence class [ v ] it holds that u ∈ [ v ] , ∀ u ∈ N v , that is the vertices in the equivalence class clique only have neighbors in the clique to which they belong.
Revised: For any v in an equivalence class [ v ] it holds that u ∈ [ v ] , ∀ u ∈ N v , that is the vertices in the equivalence class clique only have neighbors in the clique to which they belong.

Original: An example can be found in Figure 3, where the graph contains four cliques adjacent to the vertex r .
Revised: An example can be found in Figure 3, where the graph contains four cliques adjacent to the vertex r .

Original: In general, the vertex r can also be a clique as well and the graph can have multiple disjoint components of the type presented in the ﬁgure.
Revised: In general, the vertex r can also be a clique as well and the graph can have multiple disjoint components of the type presented in the ﬁgure.

Original: For any instance µ of the problem, this allows us to collapse each equivalence class [ v ] to a vertex v with the maximum expected reward in [ v ] .
Revised: For any instance µ of the problem, this allows us to collapse each equivalence class [ v ] to a vertex v with the maximum expected reward in [ v ] .

Original: The next lemma states a sufﬁcient condition on G under which d ∗ is bounded.
Revised: The next lemma states a sufﬁcient condition on G under which d ∗ is bounded.

Original: Lemma 7.2.
Revised: Lemma 7.2.

Original: If the graph G is such that C ( G ) has no path of length greater than two between any two vertices, then, for any instance µ , the following inequality holds: c ∗ + ∣ I ∗ ∣ ∆ min ≥ d ∗ .
Revised: If the graph G is such that C ( G ) has no path of length greater than two between any two vertices, then, for any instance µ , the following inequality holds: c ∗ + ∣ I ∗ ∣ ∆ min ≥ d ∗ .

Original:  Conclusion
Revised:  Conclusion

Original: We presented a detailed study of the problem of stochastic online learning with feedback graphs in a ﬁnite time setting.
Revised: We presented a detailed study of the problem of stochastic online learning with feedback graphs in a ﬁnite time setting.

Original: We pointed out the surprising issue of deﬁning optimal ﬁnite-time regret for this problem.
Revised: We pointed out the surprising issue of deﬁning optimal ﬁnite-time regret for this problem.

Original: We gave an instance on which no algorithm can hope to match, in ﬁnite time, the quantity c ∗ , which characterizes asymptotic optimality.
Revised: We gave an instance on which no algorithm can hope to match, in ﬁnite time, the quantity c ∗ , which characterizes asymptotic optimality.

Original: Next, we derived an asymptotically optimal algorithm that is also min-max optimal in a ﬁnite-time sense and admits more favorable regret guarantees than those given in prior work.
Revised: Next, we derived an asymptotically optimal algorithm that is also min-max optimal in a ﬁnite-time sense and admits more favorable regret guarantees than those given in prior work.

Original: Finally, we described a family of feedback graphs for which matching the asymptotically optimal rate is possible in ﬁnite time.
Revised: Finally, we described a family of feedback graphs for which matching the asymptotically optimal rate is possible in ﬁnite time.

Original: There are several interesting questions that follow from this work.
Revised: There are several interesting questions that follow from this work.

Original: First, while the condition on C ( G ) in Lemma 7.2 is sufﬁcient, it is not necessary.
Revised: First, while the condition on C ( G ) in Lemma 7.2 is sufﬁcient, it is not necessary.

Original: For example, a star-like graph in which two leaf vertices are also neighbors will have the property that d ∗ ≤ O ( c ∗ ) for any instance µ .
Revised: For example, a star-like graph in which two leaf vertices are also neighbors will have the property that d ∗ ≤ O ( c ∗ ) for any instance µ .

Original: We ask what would be a necessary and sufﬁcient condition on G for which d ∗ =
Revised: We ask what would be a necessary and sufﬁcient condition on G for which d ∗ =

Original: Θ ( c ∗ ) on any instance µ ?
Revised: Θ ( c ∗ ) on any instance µ ?

Original: Another interesting question is how to address the setting of evolving feedback graphs.
Revised: Another interesting question is how to address the setting of evolving feedback graphs.

Original: It is unclear what conditions on the graph sequence would allow us to recover bounds that improve on the existing independence number results.
Revised: It is unclear what conditions on the graph sequence would allow us to recover bounds that improve on the existing independence number results.

Original: Further, can we use our approach to show improved results for the setting of dependent rewards and feedback graphs studied in [Cortes et al., 2020]?
Revised: Further, can we use our approach to show improved results for the setting of dependent rewards and feedback graphs studied in [Cortes et al., 2020]?

Original: Finally, our methodology crucially relies on the informed setting assumption.
Revised: Finally, our methodology crucially relies on the informed setting assumption.

Original: We ask if it is possible to achieve similar bounds to Theorem 5.2 in the uninformed setting.A Reﬁned example for hardness of determining optimal d
Revised: We ask if it is possible to achieve similar bounds to Theorem 5.2 in the uninformed setting.

Original: To understand better why it is difﬁcult to deﬁne a notion of optimality for the constant term d in the ﬁnite-time bound, consider the following toy problem.

Original: The graph is given by Figure 4.

Original: There are n disjoint copies of an open cube graph with 8 vertices each.

Original: [-We let V 
-]
Revised: [-We let V 
-]

Original: =

Original: { ν i, 1 ,ν i, 2 ,ν i, 3 ,ν i, 4 } i ∈[ n ] and V 2 = { ν i, 5 ,ν i, 6 ,ν i, 7 ,ν i, 8 } i ∈[ n ] .

Original: We assume that we have oracle knowledge of the mean rewards of all arms µ ( ν )

Original: [-= 
-]
Revised: [-= 
-]

Original: for any ν ∈ V 2 and µ ( ν ) = 12 − ∆ for all ν ∈ V 1 , with one exception.

Original: There is one arm in V 1 , chosen uniformly at random, that is optimal with a mean µ ∗ ∈ { 12 + 2∆ , 12 + ε ∆ } .

Original: We note that we do not know the index of the optimal arm and so the problem reduces to identifying the optimal arm and the respective environment (i.e. value of µ ∗ ).

Original: The best we can do is to collect equally many samples for each arm in V 1 until we have sufﬁcient statistics to ﬁgure out either the environment or the optimal arm.

Original: Under Env.

Original: A we need to collect 1 /( 3∆ ) 2 samples and under Env.

Original: B we need to collect 1 /(( 1 + (cid:15) ) ∆ ) 2 samples.

Original: There are two canonical base strategies corresponding to algorithm A and B in Section 4: either play all arms in V 2 for N ( env ) times (Algorithm A ), depending on the environment, or play all arms in V 1 for N ( env )/ 4 many times (Algorithm B ).

Original: The following table shows the regret each strategy suffers for collecting sufﬁcient samples to distinguish the environments.

Original: [Equation]

Original: 4 n /( 9∆ ) 4 nε /(( 1 + ε ) 2 ∆ )

Original: Under Env.

Original: A we have c ∗ Env.

Original: A = n ( 3∆ ) and under Env.

Original: B we have c ∗ Env.

Original: B = nε ( 1 + (cid:15) ) 2 ∆ .

Original: Which strategy is the “optimal” one?

Original: One possible answer is to say that A is optimal, since it minimizes the worst-case regret.

Original: One might be tempted to say that B is better, since we can absorb the constant term in the leading O( c ∗ log ( T )) without the need of adding a constant d at all!

Original: That is, B minimizes the competitive ration.

Original: The implicit assumption made for the second choice of optimality is: “In a bad environment, where it is inevitable to suffer a loss of 100000, suffering an additional 100000 is just as bad as suffering an additional loss of 10 in an environment where one cannot avoid a loss of 10.”

Original: We argue that this notion of optimality is not aligned with the principle of regret as a benchmark.

Original: In regret, unlike the competitive ratio, we care about the absolute value of suboptimality.

Original: Hence, we claim that considering strategy 1 optimal in our toy experiment independent of the value of c ∗ in environment A and B is a meaningful choice.

Original: The same argument implies that hiding arbitrarily large constants in the O -notation will obscure critical information about the practicalities of an algorithm, which our work unfortunately does as well.

Original: The regret upper bounds presented in this work hide only universal constants which are independent of the problem parameters, including the topology of the feedback graph.

Original: B Regret upper bound proofs

Original: For the rest of the appendix we are going to assume that each gap ∆ i is such that ∆ i = 2 − f ( i ) for some function f

Original: ∶[ K ] → ⌈ log ( T )⌉ .

Original: This is without loss of generality as every ∆ i is in [ 2 − s , 2 − s + 1 ] for some s .

Original: Thus, we can clip every ∆ i to 2 − s i for some s i and change the constraints and objective of LP1 by at most a factor of 2 .

Original: Thus the value of c ∗ would change by at most a factor of 2 .

Original: B.1 Algorithm modiﬁcation

Original: Since Algorithm 1 plays ⌈ x ∗ LP 3 ,i ⌉ we need to take care of the difference ⌈ x ∗ LP 3 ,i ⌉ − x ∗ LP 3 ,i .

Original: At worst, playing according to the rounded solution of LP3 can result in a Ω ( K ) additive factor on top of

Original: D LP 3 ( s ) .

Original: This can accumulate regret up to an Ω ( K log ( T )) factor in the ﬁnal bound.

Original: Our goal is to give asymptotically optimal bounds together with the ﬁnite time bounds and such a term might be sub-optimal in the case when ∆ min ≥ ω ( 1 K ) .

Original: To avoid the additional K -factor we modify Algorithm 1 in the following way.

Original: Note that for any

Original: Algorithm 2: Modiﬁcation of Algorithminput : Graph G =

Original: ( V,E ) ,

Original: conﬁdence parameter δ , time horizon T 1 Initialize t = 0 , s = 0 , ˆ r i ( 0 ) = 0 , ∀ i ∈

Original: [ K ] , B = [ 0 ] K 2 Compute (approximate) minimum dominating set ˆ D( G ) 3 while s ≤ ⌈ log ( K )⌉ do

Original: Play each arm i ∈ log ( K ∆ s + 1 ) ∆ 2 s rounds do do x ∗ LP 3 ,i ≥ 1 , the following inequality holds: ⌈ x ∗ LP 3 ,i ⌉ ≤ 2 x ∗ LP 3 ,i , and thus playing such arms will only increase the incurred regret by a multiplicative factor of at most 2 .

Original: Thus, we only need to consider x ∗ LP 3 ,i < 1 .

Original: We introduce a buffer B ∈ R K which will inform us when to play an arm i for which x ∗ LP 3 ,i < 1 .

Original: The ﬁrst time the solution of the LP informs us to play i for less than a single round, we play i for a single round and update the buffer as B i + = x ∗ LP 3 ,i .

Original: We observe that we have now overplayed i and have a buffer of 1 − x ∗ LP 3 ,i extra plays of i .

Original: Thus, at the next phase at which x ∗ LP 3 ,i < 1 , we can check if x ∗ LP 3 ,i can be covered by the remaining buffer.

Original: If so, then there is no need to play arm i again as we still have sufﬁcient number of observations provided by playing i .

Original: If the buffer is exceeded, we again play i for one round and take into account the additional overplay.

Original: Thus, at the end of phase s , the total number of arm i has been played does not exceed

Original: [Equation]

Original: where x ∗ LP 3 ,i ( t ) is the solution to LP3 at phase t .

Original: The above implies the following lemma.

Original: Lemma B.1.

Original: Let x ∗ LP 3 ,i ( t ) be the solution to LP3 at phase t .

Original: The at the end of phase s of Algorithmthe total number of plays of arm i is at most

Original: [Equation]

Original: The above lemma implies that we can, at the price of a constant multiplicative factor of 3 , consider the solution of LP3 instead of the rounded solution played by Algorithm 1.

Original: Hence, for the rest of the appendix, we do so.

Original: B.2 Proof of Theorem 5.

Original: We begin with a somewhat standard concentration result.

Original: Lemma B.2.

Original: For any s ∈ [ 10 , log ( T )] ,K ≥ 2 ,α ′ ≤ 3072 , the following inequality holds

Original: [Equation]

Original: Proof.

Original: We use Theorem 1 from Zhao et al.

Original: [2016] which states that for a sum of zero-mean, 1 /sub-Gaussian random variables ( X i ) ti = 1 the following inequality holds

Original: [Equation]

Original: We begin by bounding P (∣ µ i −

Original: ˆ r

Original: i ( s )∣ ≥ b

Original: i ( s )) for a ﬁxed i ∈

Original: [ K ] .

Original: Since action i is observed at most

Original: [Equation]

Original: times up to and including phase s , we can write

Original: [Equation]

Original: where we used the fact that for s ≥ 7 ,K ≥ 2 ,α ′ ≤ 512 the following inequality holds

Original: [Equation]

Original: A union bound over i ∈ [ K ] completes the proof.

Original: Lemma B.3.

Original: Under the same assumptions as in Lemma B.2, we have

Original: [Equation]

Original: Furthermore, if we let E upper = {∀ i ∈ [ K ] ,t ∈ [ s / 2 ,s ]

Original: ∶ ˆ∆ i ( t ) ≤ ∆ i ∨ ∆ t } then P (E upper ) ≥ 1 − ( ∆ s / 2 + 1 K )

Original: α − 2 .

Original: Proof.

Original: A union bound over Lemma B.2, together with picking α sufﬁciently large imply

Original: P (∃ i ∈

Original: [ K ] ,t ∈ [ s / 2 ,s ] ∶ ∣ µ i − ˆ r

Original: i

Original: t )

Original: ∣ ≥ b i

Original: [Equation]

Original: t

Original: [Equation]

Original: ∑

Original: [Equation]

Original: For the second part of the lemma, we assume WLOG ∆ i ≥ ∆ s / 2 .

Original: Thus, we have

Original: [Equation]

Original: On the event that {∀ i ∈ [ K ] ,t ∈ [ s / 2 ,s ] ∶ ∣ µ i − ˆ r

Original: i

Original: ( t )∣ ≤ b i ( t )} we have ˆ r i ∗ t ( t )

Original: − b i ∗ t ( t ) ≤ µ i ∗ t and ˆ r i ( t )

Original: + b i ( t ) ≥ µ i .

Original: This implies that w.p. 1 − ( ∆ s / 2 + 1 K ) α − 2 we have

Original: [Equation]

Original: for all i ∈ [ K ] and t ∈ [ s / 2 ,s ] .

Original: Lemma B.4.

Original: Let α ′

Original: [-= 
-]
Revised: [-= 
-]

Original: α in LP3.

Original: On the event E upper it holds that b i ( s ) ≤ ∆ i ∨ ∆ s, ∀ i ∈

Original: [ K ] .

Original: Thus, for any α ≥ 3 and any log ( K ) ∨ 10 ≤ s ≤ log ( T ) the following inequality holds

Original: [Equation]

Original: Proof.

Original: Recall that b

Original: i ( s ) =

Original: [Equation]

Original: n i ( s

Original: [Equation]

Original: so we are going to bound n i ( s ) from below.

Original: Assume that ∆ i ≥ ∆ s , the other case is handled similarly.

Original: Let s i be the phase at which ∆ s i =

Original: ∆ i .

Original: On the event

Original: E upper we know that ˆ∆

Original: i ( t ) ≤ ∆ i for all t ∈

Original: [ s i ,s ] ,i ∈

Original: [ K ] .

Original: The constraints in LP3 imply

Original: [Equation]

Original: i

Original: [Equation]

Original: The above implies that

Original: [Equation]

Original: For α ′ =

Original: 768 α the above implies b i ( s ) ≤ ∆ i 8 .

Original: Lemma B.5.

Original: Let E gap ( s )

Original: =

Original: {∀ i ∈

Original: [ k ] ∶ ∆ i 2 ∨ ∆ s ≤ ˆ∆ i ( s ) ≤ ∆ i ∨ ∆ s } .

Original: Under the assumptions of

Original: Lemma B.4 we have

Original: [Equation]

Original: Proof.

Original: If s is such that ∆ i ≤ ∆ s the statement of the lemma holds from Lemma B.3 and the deﬁnition of ˆ∆

Original: i ( t ) .

Original: We now consider the case ∆ i ≥ ∆ s and assume that E upper holds.

Original: Lemma B.4 now implies that b i ( s ) ≤ ∆ i 8 .

Original: Further, assume that ∣ µ i − ˆ r

Original: i ( s )∣ ≤ b i ( s ) , ∀ i ∈ [ K ] .

Original: We have ˆ∆ i ( s )

Original: =

Original: max

Original: [Equation]

Original: ∈[ K ] ˆ r j ( s ) − b j ( s ) − ˆ r

Original: i ( s ) − b i ( s ) ≥ ˆ r i ∗ ( s ) − b i ∗ ( s ) − ˆ r

Original: i ( s ) − b i ( s ) ≥ ∆ i − 2 ( b i ∗ ( s )

Original: + b i ( s )) ≥ ∆ i 2 .

Original: Our assumptions fail with probability at most ( ∆ s / 2 + 1 K ) α − 2 + 2 ( ∆ s + 1 K ) α − 1 .

Original: Lemma B.6.

Original: For any phase s ≥ log ( K ) ∨ 10 , it holds that P ( ˆΓ s + 1 /⊆ Γ s ) ≤ 3 ( ∆ s / 4 + 1 K )

Original: α − 2 .

Original: Proof.

Original: We have P ( ˆΓ s + 1 /⊆ Γ s ) ≤ ∑ i /∈ Γ s P ( i

Original: ∈ ˆΓ s + 1 ) .

Original: The fact i /∈ Γ s implies that ∆ i ≥ 2∆ s .

Original: The result now follows by Lemma B.5.

Original: Lemma B.7 (Lemma 5.1) .

Original: Let D LP 3 ( s ) be the value of LP3 at phase s and let D LP 4 ( s ) be the value of LP4 at phase s .

Original: For any s ≥ log ( K ) ∨ 10 the following inequality holds

Original: [Equation]

Original: with probability at least 1 − 3 ( ∆ s / 2 + 1 K ) α − 2 .

Original: Further, for any s ≥ log ( K ) ∨ 10 ∨ log ( ∣ I ∗ ∣ ∆ min ) we have that the regret incurred for playing according to LP3 is at most 16 α ′ c ∗ ( G,µ ) with probability at least 1 − 3 ( ∆ s / 2 + 1 K ) α − 2 .

Original: Proof of Lemma 5.1.

Original: For any s and all i ∈ [ K ] Lemma B.5 and Lemma B.6 imply that ˆΓ s + 1 ⊆ Γ s and ∆ s 2 ≤ ˆ∆ ( s ) ≤ ∆ s with probability at least 1 − 3 ( ∆ s / 2 + 1 K ) α − 2 .

Original: If we let x ∗ LP 4 ( s ) be a solution to

Original: LP4 at phase s , then these conditions imply that 4 x ∗ LP 4 ( s ) is feasible for LP3.

Original: This implies

Original: D LP

Original: [Equation]

Original: s

Original: [Equation]

Original: ≤

Original: [Equation]

Original: LP

Original: [Equation]

Original: ˆ∆

Original: [Equation]

Original: [Equation]

Original: Further, for s ≥ log ( 1 / ∆ min ) , Γ s consists only of I ∗ .

Original: Let x ∗ LP 3 be a solution to LP3, and let ˆ x ∗ be a solution to the LP dropping all constraints on I ∗ and its neighborhood.

Original: Note that ⟨ ˆ x ∗ , ˆ∆ ( s )⟩ ≤ 8 α ′ c ∗ ( G,µ ) under E( s ) .

Original: We show by contradiction that

Original: [Equation]

Original: which by ∆ i ≤ 2 ˆ∆ i ( s ) completes the proof.

Original: Assume the opposite is true, take a new x such that

Original: [Equation]

Original: x is a feasible solution of LP3.

Original: Next, we only consider s ≥ log (∣ I ∗ ∣/( 4∆ min )) which implies that

Original: [Equation]

Original: which is a contradiction to ⟨ x, ˆ∆ s ( s )⟩ ≥ D LP 3 ( s ) .

Original: Denote the value of LP2 at phase s as D LP 2 ( s ) and a solution to the LP as x ∗ LP 2 ( s ) .

Original: We note that for any s it holds that x LP 4 ( s ) =

Original: ( α log ( K / ∆ s + 1 ) ∨ α ′ )

Original: ∑ t ≤ s x ∗ LP 2 ( t ) is feasible for LP4.

Original: Further we have that for all t ≤ s it holds that ⟨ x, ∆ s ⟩ ≤ ⟨ x, ∆ t ⟩ .

Original: These two observations imply

Original: [Equation]

Original: Further, we have D LP 2 ( t ) ≤ D LP 4 ( s ) .

Original: We can assume that s ≥ 10 , otherwise the regret is O ( K ) .

Original: Thus we can characterize the optimality of Algorithm 1 up to factors of log 2 ( 1 / ∆ min ) as follows.

Original: Theorem B.8 (Theorem 5.2) .

Original: Let d ∗ ( G,µ )

Original: = max s ≤ log (∣ I ∗ ∣/ ∆ min ) D LP 2 ( s ) .

Original: The expected regret

Original: R ( T ) of playing according to Algorithm 2 with α = 4 and α ′ =

Original: 768 α is bounded as

Original: [Equation]

Original: Further, for any algorithm, there exists an environment on which the expected regret of the algorithm is at least Ω ( d ∗ ( G,µ )) .

Original: Proof.

Original: Lemma 5.1 implies that the regret bounds fail to hold at any phase s ≥ log ( K ) w.p.

Original: at most 3 ( 1 2 s / 2 + 1 K )

Original: α − 2 .

Original: Further the regret at phase s is always bounded by α ′ K 2 s log ( 1 / ∆ min )

Original: Choosing α = 4 implies expected regret of only O ( log ( 1 / ∆ min )) on the union bound of failure events.

Original: For the remainder of the proof we now have for s ≤ log (∣ I ∗ ∣/ ∆ min )

Original: [Equation]

Original: For s ≥ log (∣ I ∗ ∣/ ∆ min )

Original: we have that

Original: [Equation]

Original: Finally the regret incurred in the ﬁrst s ≤ log ( K ) phases is at most O ( γ ( G ) K log ( K ))

Original: as the algorithm plays the approximate solution corresponding to the minimum dominating set of G .

Original: Combining all of the above shows the instance dependent regret upper bound.

Original: To show the instance independent bound we again use the bound D LP 4 ( s ) ≤ ( α log ( K / ∆ s + 1 ) ∨ α ′ )

Original: ∑ t ≤ s D LP 2 ( t ) .

Original: Let I s be a maximum independent set of Γ s .

Original: If ∆ s ≥ √ ∣ I s ∣ T then by the argument in Section D.1 it holds that D LP 2 ( s ) ≤ √ α ( G ) T .

Original: Next, we consider ∆ s < √ ∣

Original: I s ∣ T .

Original: Let s be the smallest index for which ∆ s < √ ∣

Original: I s ∣ T so that ∆ s − 1 ≥ √ ∣ I s − 1 ∣ T .

Original: Since ∆ s = 12 ∆ s − 1 we have that

Original: D LP 2 ( s ) ≤ 4 D LP 2 ( s − 1 ) ≤ 4 √ α ( G ) T .

Original: Further, playing every arm in I s for 1∆ 2 s rounds will exceed the time horizon of the game and incur at most 4 √ α ( G ) T regret.

Original: Suppose that the solution of LP2 at epoch s is such that the game does not end if the solution is followed and consider playing according

Original: LP2 at epoch s

Original: + 1 .

Original: If the game does not end, then the incurred regret is no more than 2 √ α ( G ) T .

Original: This follows from the fact that

Original: [Equation]

Original: where the last inequality follows from the fact that x i = 4∆ 2 s

Original: [Equation]

Original: 1 ,i ∈ I s is feasible for LP2 at epoch s .

Original: Proceeding by induction, it holds that any epoch s at which the solution of LP 2 does not end the game must have regret at most 2 √ α ( G ) T .

Original: Suppose now that the game ends, then the regret of the ﬁnal epoch is at most twice the preceding epochs regret as playing 4 -times the optimal solution for the previous epoch is feasible for the current epoch, and so the total regret incurred for playing according to LP2 is at most 4 √ α ( G ) T .

Original: By the argument that D LP 4 ( s ) ≤

Original: [Equation]

Original: [Equation]

Original: we can construct a feasible solution for LP4 at any epoch s during which from the solutions to LP2.

Original: These feasible solutions have bounded regret for any epoch before the terminal epoch for LP 2 .

Original: Finally, we note that the terminal epoch for LP 2 must be no smaller than the terminal epoch for LP4, otherwise we can construct a feasible solution to LP2 from that of LP 4 with regret smaller than the value of LP 2 , which would be a contradiction.

Original: The construction sets all variables not in Γ s to 0 and rescales the non-zero variables normalizing with the additional logarithmic factors in the constraint of LP4.

Original: Thus, the total regret for playing according to LP 4 is at most O ( log 2 ( KT ) √ α ( G ) T ) .

Original: Further, any feasible solution for LP4 is feasible for LP2 and any feasible solution for LP2 must play more than T rounds

Original: The regret lower bound follows from Lemma 6.1.

Original: C Regret lower bounds C.1 Proof of Lemma 6.

Original: Lemma C.1 (Lemma 6.1) .

Original: Fix any instance µ s.t.

Original: µ i ≤ 1 − 2∆ s ,i ∈ I ∗ .

Original: Let Λ s ( µ ) be the set of problem instances with means µ ′ ∈ µ + [ 0 , 2∆ s ] k .

Original: Then for any algorithm, there exists an instance in Λ s ( µ ) such that the regret is lower bounded by LP2.

Original: Proof.

Original: We take as a base environment the instance with expected rewards vector µ and assume that the rewards follow a Gaussian with variance 1 √ 2 .

Original: Let τ ∈ R + ⋃{∞} be the time at which the following is satisﬁed

Original: [Equation]

Original: where the expectation is with respect to the randomness of the sampling of the rewards and A .

Original: First we argue that we can assume τ < ∞ .

Original: Consider τ = ∞ .

Original: Let

Original: [Equation]

Original: Fix a time horizon T and let x T be the vector of expected number of observations of A on environment µ and X T the random vector of actual observations.

Original: By the assumption that τ = ∞ and Markov’s inequality, we have that P [ X T,i ∗ ≥ 1∆ 2 s ] ≤ 12 .

Original: Consider the algorithm ¯

Original: A which after 1∆ 2 s observations of i ∗ switches to playing uniformly at random from [ K ] ∖ N i ∗ so that it never observes i ∗ again.

Original: Let µ ′ be the instance which changes the expected reward of µ i ∗ to µ ′ i ∗ =

Original: µ i ∗ + 2∆ s , and so µ ′ ∈ Λ s ( µ ) .

Original: The KL-divergence between the measures induced by playing A on these two instances for τ rounds is bounded as 4∆ 2 s x τ,i ∗ .

Original: If we let x ′ T denote the vector of expected number of observations under environment µ ′ then Pinsker’s inequality implies that

Original: [Equation]

Original: Thus, the expected regret of ¯ A under µ ′ is at least ( T − 14∆ 2 s ) ∆ s for any T ≥ 14∆ 2 s .

Original: Further, by

Original: Pinsker’s inequality the probability that X T,i ∗ ≥

Original: 1∆ 2 s under ¯ A in environment µ ′ is bounded by 34 .

Original: Since A and ¯ A act in the same way up to 1∆ 2 s observations of i ∗ it holds that the expected regret of A in environment µ ′ is at least 14 ( T − 14∆ 2 s ) ∆ s for any T ≥ 14∆ 2 s .

Original: Thus for T large enough, e.g.,

Original: T =

Original: Ω ( D LP 2 ( s ) /

Original: ∆ s + 14∆ 2 s ) the conclusion of the lemma holds.

Original: We now assume that τ ≤ ∞ .

Original: Let x τ,i =

Original: E [∑ τt = 1 P [ A t ∈ N i ]] be the expected number of observations of action i after τ rounds.

Original: Assume that ∑ i /∈ Γ s x τ,i ∆ i ≤ D LP 2 ( s ), otherwise we are done.

Original: The deﬁnition of τ with the above assumption imply that

Original: [Equation]

Original: [Equation]

Original: Let µ ′ be the instance which changes the expected reward of µ i ∗ to µ ′ i ∗ = µ i ∗ + 2∆ s , and so µ ′ ∈ Λ s ( µ ) .

Original: The KL-divergence between the measures induced by playing A on these two instances for τ rounds is bounded as 4∆ 2 s x τ,i ∗ .

Original: If we let x ′ τ denote the vector of expected number of observations under environment µ ′ then

Original: Pinsker’s inequality implies that

Original: [Equation]

Original: This implies

Original: [Equation]

Original: C.2 Proof of Theorem 4.

Original: Theorem C.2.

Original: There exists a feedback graph G , with K ≥ 32 vertices, such that for any algorithm

Original: A

Original: there exists an

Original: environment µ

Original: on which R ( T ) ≥ Ω ( K 1 / 8 c ∗ ( G,µ

Original: Proof.

Original: For any algorithm A , deﬁne the algorithm A as follows:

Original: If there have been more than K 78 64∆pulls of actions in N 2 , then commit to action N 3 until end of time.

Original: We call the random time-step where A and A deviate in trajectory as τ .

Original: Deﬁne the stopping times

Original: [Equation]

Original: Let n ∗ be the node in N 2 with the smallest number of expected observations at time T under algorithm

Original: A .

Original: Let N i denote the number of times an action in N i has been played by A .

Original: The total number of observations over all actions in N 2 is

Original: [Equation]

Original: Hence the number of observations of n ∗ is bounded by 1 32∆ 22 .

Original: Consider the environment E 1 , where all we change is increasing the reward of n ∗ by up to 2∆ 2 .

Original: By Pinsker’s inequality we have

Original: [Equation]

Original: as the largest difference in probability of any event under the two environments.

Original: We consider two possible cases below.

Original: Case 1 T < T 1 .

Original: Set the reward of n ∗ to ν + ∆ 2 .

Original: Deﬁne the following event:

Original: [Equation]

Original: In the ﬁrst environment, we have

Original: [Equation]

Original: Hence the probability of E is at least 18 in the changed environment.

Original: The regret of A is at least

Original: [Equation]

Original: However, the value of LP1 for this environment is Θ ( K 78 ∆ 2 ) =

Original: Θ ( K 58 ∆ ) .

Original: Case 2 T =

Original: T

Original: Set the reward of n ∗ to ν .

Original: Deﬁne the following event:

Original: [Equation]

Original: In the base environment, we have

Original: [Equation]

Original: Hence the probability of E is at least 18 in the changed environment.

Original: The regret of A is at least

Original: [Equation]

Original: However, the value of LP1 for this environment is Θ ( K 12 ∆ ) .

Original: We note that this also characterizes d ∗ =

Original: Θ ( K 1 / 8 c ∗ ) , as playing according to LP2 will result in Θ ( K 7 / 8 / ∆ 22 ) plays of arms in N 2 .

Original: Hence for any algorithm, there exist an environment and time step T

Original: =

Original: O

Original: ( K 12 ∆ 2 ) , such that the algorithm suffers a regret that is a factor K 18 larger than c ∗ .

Original: D Characterizing d ∗

Original: D.1 Improving on bound in Lykouris et al.

Original: We now show that d ∗ ( G,µ ) ≤ max I ∈

Original: I ( G )

Original: ∑ i ∈

Original: I 1∆ i :

Original: [Equation]

Original: [Equation]

Original: The ﬁrst inequality follows from the deﬁnition of the LP, the second inequality follows from the fact that the domination number is no larger than the independence number, the third inequality follows from the fact that for any i ∈

Original: Γ

Original: s we have ∆ s ≥ ∆ i , and the ﬁfth inequality holds by the fact that I ( Γ s ) ⊆

Original: I ( G ) .

Original: D.2 Bound on d ∗ for star-graphs

Original: Lemma D.1.

Original: For the star-graph G and any instance µ , the following inequality holds: c ∗ + ∣ I ∗ ∣ ∆ min ≥ d ∗ .

Original: Proof.

Original: Consider the dual of LP1 given below

Original: [Equation]

Original: Note that for any i ∈

Original: [ K ] we can take the intersection of N i with Γ s as no action j ∈ Γ C can increase the value of the objective of LP5.

Original: The analysis is split into two parts.

Original: First consider all phases s for which it holds that ∆ r ≤ ∆ s .

Original: We argue that the solution to LP2 for these phases is to just play the revealing vertex for 1∆ 2 s times.

Original: Indeed we can just set y r =

Original: ∆ s and observe that this is feasible for the dual LP with value 1∆ s .

Original: Further, setting x r = ∆ s in the primal also yields a value of 1∆ s .

Original: The fact that ∆ r ≥ ∆ min together with the lower bound of R ( T ) ≥ Ω ( 1∆ min ) for any strategy, implies that playing according to LP2 is optimal up to at least the phase at which ∆ r > ∆ s .

Original: Next, consider the setting of s s.t.

Original: ∆ r > ∆ s .

Original: The following is feasible for LP

Original: [Equation]

Original: Thus the value of LP2 is ∆ r ∆ 2 s in the ﬁrst case and ∣ Γ s ∣ ∆ s in the second as we can match these values in the primal by setting either x r = 1∆ 2 s or x i

Original: =

Original: 1∆ 2 s ,i ∈ Γ s in the primal.

Original: To show that both of these values are dominated by c ∗ consider the dual of LP1 below

Original: [Equation]

Original: ∀ i ∈

Original: [ K ] .

Original: [Equation]

Original: First consider the setting in which the value of LP2 equals ∆ r ∆ 2 s .

Original: Set all y

Original: i ∈

Original: Γ s ∖ I ∗ to y i =

Original: ∆ r ∣ Γ s ∣ and all other y i = 0 .

Original: This is feasible for LP6 and implies that

Original: [Equation]

Original: where the second inequality follows because ∆ s ≥ ∆ i .

Original: This is sufﬁcient to guarantee that c ∗ + ∣ I ∗ ∣ ∆ min ≥ ∆ r ∆ 2 s .

Original: Next consider the setting in which the value of LP2 equals ∣ Γ s ∣ ∆ s .

Original: Set all y i

Original: ∶ i ∈ Γ s ∖ I ∗ to y i =

Original: ∆ i and all other y i = 0 .

Original: This is again feasible for LP6 because ∑ i ∈ Γ s ∖ I ∗ ∆ i ≤ ∣ Γ s ∣ ∆ s ≤ ∆ r and further implies that

Original: [Equation]

Original: Again this is sufﬁcient to guarantee that c ∗ + ∣ I ∗ ∣ ∆ min ≥ ∣ Γ s ∣ ∆ s .

Original: D.3 Proof of Lemma 7.

Original: Proof.

Original: Again we assume that G consists only of a single connected component.

Original: First we argue that C ( G ) is a star-graph.

Original: Consider three vertices v 1 ,v 2 ,v 3 ∈ C ( G ) such that v 1 ,v 3 ∈ N v 2 .

Original: Assume that v 1 ∈ N v 3 .

Original: This implies that there exists a vertex u ∈ C ( G ) such that u ∈ N i but u /∈ N j for i ≠ j,i,j ∈ { 1 , 2 , 3 } , otherwise N v 1 = N v 2 =

Original: [-N v 
-]
Revised: [-N v 
-]

Original: and they collapse to a single vertex under C ( G ) .

Original: Assume that u ∈ N 1 but u /∈ N 2 .

Original: Then this implies there exists a path of length 3 between u and v 2 , given by ( u,v 1 ,v 3 ,v 2 ) .

Original: All other cases are symmetric and so this contradicts v 2 ∈ N v 3 .

Original: Further, it can not occur that there exists a neighbor u of v 2 or v 3 s.t. u /∈ N v 1 .

Original: The above two arguments show that for G every vertex must neighbor v 1 and no two vertices v 2 ,v 3 ≠ v 1 can be neighbors making C ( G ) a star graph.

Original: Next, we show that for any µ there exists a µ ′ deﬁned on C ( G ) s.t.

Original: c ∗ ( G,µ )

Original: =

Original: c ∗ ( C ( G ) ,µ ′ ) and d ∗ ( G,µ ) =

Original: d ∗ ( C ( G ) ,µ ′ ) .

Original: For any equivalence class [ v ] ∈ C ( G ) , deﬁne the expected reward of [ v ] as µ ′[ v ] = max u ∈[ v ] µ u .

Original: For the remainder of the proof we represent the equivalence class by the action v with maximum reward µ v = µ ′[ v ] .

Original: By construction, max v µ ′[ v ] = max u µ u , hence the gaps are also identical ∆ [ v ] = ∆ v .

Original: We ﬁrst show that we can drop the constraints for any u ∈ [ v ] ∖ { v } without changing the value of the LP.

Original: The LHS of all constraints for u ∈ [ v ] is identical since it depends only on N u .

Original: Hence, we can remove all but the largest constraint, which is obtained for the smallest gap, i.e. the constraint for v .

Original: Next we show that we can also remove x u for any u ∈ [ v ] ∖ { v } .

Original: Assume x u >is a feasible solution of the LP, then we obtain another feasible solution x ′ by x ′ u = 0 ,x ′ v = x u + x v , while leaving everything else unchanged.

Original: However, since ∆ v ≤ ∆ u , the objective value of x ′ is smaller or equal that of x .

Original: Hence there exists an optimal solution where all u ∈ [ v ] ∖ { v } are 0 and these variables can be dropped from the LP.

Original: The resulting LP after dropping constraints and variables for u ∈ [ v ] ∖ { v } is exactly given by C ( G ) ,µ ′ .

Original: This shows that c ∗ ( C ( G ) ,µ ′ ) =

Original: c ∗ ( G,µ ′ ) .

Original: The claim that d ∗ ( G,µ ) =

Original: d ∗ ( C ( G ) ,µ ′ ) follows analogously.