Original: Non-Stationary Bandits under Recharging Payoffs: Improved Planning with Sublinear Regret
Revised: Non-Stationary Bandits under Recharging Payoffs: Improved Planning with Sublinear Regret

Original: Abstract
Revised: Abstract

Original: The stochastic multi-armed bandit setting has been recently studied in the nonstationary regime, where the mean payoff of each action is a non-decreasing function of the number of rounds passed since it was last played.
Revised: The stochastic multi-armed bandit setting has been recently studied in the nonstationary regime, where the mean payoff of each action is a non-decreasing function of the number of rounds passed since it was last played.

Original: This model captures natural behavioral aspects of the users which crucially determine the performance of recommendation platforms, ad placement systems, and more.
Revised: This model captures natural behavioral aspects of the users which crucially determine the performance of recommendation platforms, ad placement systems, and more.

Original: Even assuming prior knowledge of the mean payoff functions, computing an optimal planning in the above model is NP-hard, while the state-of-the-art is a 1 / 4 -approximation algorithm for the case where at most one arm can be played per round.
Revised: Even assuming prior knowledge of the mean payoff functions, computing an optimal planning in the above model is NP-hard, while the state-of-the-art is a 1 / 4 -approximation algorithm for the case where at most one arm can be played per round.

Original: We first focus on the setting where the mean payoff functions are known.
Revised: We ﬁrst focus on the setting where the mean payoff functions are known.

Original: In this setting, we significantly improve the best-known guarantees for the planning problem by developing a polynomial-time (1 − 1 /e )
Revised: In this setting, we signiﬁcantly improve the best-known guarantees for the planning problem by developing a polynomial-time (1  1 /e )

Original: -approximation
Revised: -approximation

Original: algorithm (asymptotically and in expectation), based on a novel combination of randomized
Revised: algorithm (asymptotically and in expectation), based on a novel combination of randomized

Original: LP rounding and a time-correlated (interleaved) scheduling method.
Revised: LP rounding and a time-correlated (interleaved) scheduling method.

Original: Furthermore, our algorithm achieves improved guarantees – compared to prior work – for the case where more than one arm can be played at each round.
Revised: Furthermore, our algorithm achieves improved guarantees – compared to prior work – for the case where more than one arm can be played at each round.

Original: Moving to the bandit setting, when the mean payoff functions are initially unknown, we show how our algorithm can be transformed into a bandit algorithm with sublinear regret.
Revised: Moving to the bandit setting, when the mean payoff functions are initially unknown, we show how our algorithm can be transformed into a bandit algorithm with sublinear regret.

Original:  Introduction
Revised:  Introduction

Original: In the last two decades, the predominant rise of the social media industry has made the notion of a newsfeed an integral part of our lives.
Revised: In the last two decades, the predominant rise of the social media industry has made the notion of a newsfeed an integral part of our lives.

Original: In a newsfeed, a user observes a structured sequence of content items (posts, photos etc.) particularly selected by the platform according to her/his preferences.
Revised: In a newsfeed, a user observes a structured sequence of content items (posts, photos etc.) particularly selected by the platform according to her/his preferences.

Original: Apart from social media, an analogous idea – potentially relabeled – also appears in different domains as, for example, “frequently bought together” in e-commerce, “shuffling similar songs” in music recommendation, or “recommended articles” in scholarly literature indexing databases.
Revised: Apart from social media, an analogous idea – potentially relabeled – also appears in different domains as, for example, “frequently bought together” in e-commerce, “shufﬂing similar songs” in music recommendation, or “recommended articles” in scholarly literature indexing databases.

Original: Whether it is measured in terms of click-rate or time devoted, the high-level objective of newsfeeds is fairly well-known: to maximize the user’s engagement with the platform.
Revised: Whether it is measured in terms of click-rate or time devoted, the high-level objective of newsfeeds is fairly well-known: to maximize the user’s engagement with the platform.

Original: In many applications, however, achieving this objective is not as simple as identifying the user’s “favorite” content, given that her/his satisfaction can depend on the time passed since the same (or similar) content has been observed.
Revised: In many applications, however, achieving this objective is not as simple as identifying the user’s “favorite” content, given that her/his satisfaction can depend on the time passed since the same (or similar) content has been observed.

Original: As an example, a user’s engagement can worsen if a social media feed (resp., a music recommendation platform) constantly presents content from the same source (resp., same artist).
Revised: As an example, a user’s engagement can worsen if a social media feed (resp., a music recommendation platform) constantly presents content from the same source (resp., same artist).

Original: Motivated by such scenarios, researchers have recently studied online decision making problems capturing the notion of “recovering” payoffs, namely, scenarios where the payoff of an action drops (to zero) after each play and then slowly increases back to a baseline.
Revised: Motivated by such scenarios, researchers have recently studied online decision making problems capturing the notion of “recovering” payoffs, namely, scenarios where the payoff of an action drops (to zero) after each play and then slowly increases back to a baseline.

Original: In the context of online learning, these non-stationary models interpolate between multi-armed bandits, where the environment is assumed to be intact, and reinforcement learning, since the actions may now alter the future environment in a structured manner.
Revised: In the context of online learning, these non-stationary models interpolate between multi-armed bandits, where the environment is assumed to be intact, and reinforcement learning, since the actions may now alter the future environment in a structured manner.

Original: These models are wide enough to capture many real-life applications, yet special enough to accept efficiently computable (near-)optimal solutions.
Revised: These models are wide enough to capture many real-life applications, yet special enough to accept efﬁciently computable (near-)optimal solutions.

Original: In this direction, the following general model was first introduced (under slightly different assumptions) by
Revised: In this direction, the following general model was ﬁrst introduced (under slightly different assumptions) by

Original: Immorlica and Kleinberg [IK18] and has been recently studied by Simchi-Levi et al.
Revised: Immorlica and Kleinberg [IK18] and has been recently studied by Simchi-Levi et al.

Original: [SLZZ21]:
Revised: [SLZZ21]:

Original: Problem (Multi-armed bandits under recharging payoffs) .
Revised: Problem (Multi-armed bandits under recharging payoffs) .

Original: We consider a set of actions (or arms ), where each arm i is associated with a mean payoff function p i .
Revised: We consider a set of actions (or arms ), where each arm i is associated with a mean payoff function p i .

Original: For each τ , p i ( τ ) is the expected payoff collected for playing action i , when i has been played before exactly τ rounds (we refer to τ as the delay of an arm at a specific round).
Revised: For each ⌧ , p i ( ⌧ ) is the expected payoff collected for playing action i , when i has been played before exactly ⌧ rounds (we refer to ⌧ as the delay of an arm at a speciﬁc round).

Original: For each arm, we assume that its payoff function: (a) is monotone non-decreasing in τ , and (b) has a known finite recovery time , namely, a specific delay after which the function remains constant.
Revised: For each arm, we assume that its payoff function: (a) is monotone non-decreasing in ⌧ , and (b) has a known ﬁnite recovery time , namely, a speciﬁc delay after which the function remains constant.

Original: At each round, the player pulls a subset of at most k actions, observes the realized payoffs of the arms played (each computed with respect to the delay of the corresponding action), and collects their sum.
Revised: At each round, the player pulls a subset of at most k actions, observes the realized payoffs of the arms played (each computed with respect to the delay of the corresponding action), and collects their sum.

Original: The objective is to maximize the expected cumulative payoff collected within an unknown time horizon.
Revised: The objective is to maximize the expected cumulative payoff collected within an unknown time horizon.

Original: We denote by k -RB an instance of the problem where at most k arms can be played at each round.
Revised: We denote by k -RB an instance of the problem where at most k arms can be played at each round.

Original: Further, we distinguish between the planning setting, where the payoff functions are known to the player a priori (thus, payoffs can be considered deterministic), and the learning setting, where these are initially unknown and the user assumes semi-bandit feedback on the payoffs of the played actions.
Revised: Further, we distinguish between the planning setting, where the payoff functions are known to the player a priori (thus, payoffs can be considered deterministic), and the learning setting, where these are initially unknown and the user assumes semi-bandit feedback on the payoffs of the played actions.

Original: In the former case, the goal is to design an efficient algorithm which closely-approximates the optimal planning (since this is generally an NP-hard problem), while in the latter, the objective is to construct a bandit algorithm of sublinear regret, defined as the difference in expected payoff between a planning algorithm and its bandit variant due to the initial absence of information of the latter.
Revised: In the former case, the goal is to design an efﬁcient algorithm which closely-approximates the optimal planning (since this is generally an NP-hard problem), while in the latter, the objective is to construct a bandit algorithm of sublinear regret, deﬁned as the difference in expected payoff between a planning algorithm and its bandit variant due to the initial absence of information of the latter.

Original: 1.1 Background and Related Work
Revised: 1.1 Background and Related Work

Original: Immorlica and Kleinberg [IK18] first study the 1 -RB problem and provide a (1 − ϵ ) -approximation (asymptotically) for the planning setting, under the additional assumption that the payoff functions are weakly concave over the whole time horizon.
Revised: Immorlica and Kleinberg [IK18] ﬁrst study the 1 -RB problem and provide a (1  ✏ ) -approximation (asymptotically) for the planning setting, under the additional assumption that the payoff functions are weakly concave over the whole time horizon.

Original: On the other extreme, Basu et al.
Revised: On the other extreme, Basu et al.

Original: [BSSS19] provide a long-run (1 − 1 / e ) -approximation for the case where, after each play, an arm cannot be played again for a fixed number of rounds – a problem which can be cast as a special instance of 1 -RB using Heaviside step payoff functions.
Revised: [BSSS19] provide a long-run (1  1 / e ) -approximation for the case where, after each play, an arm cannot be played again for a ﬁxed number of rounds – a problem which can be cast as a special instance of 1 -RB using Heaviside step payoff functions.

Original: 1 Shortly after the introduction of the problem, a number of special cases and variations have been studied [CCB20, PC21, PBG19, BPCS21, BCMTT20,
Revised: 1 Shortly after the introduction of the problem, a number of special cases and variations have been studied [CCB20, PC21, PBG19, BPCS21, BCMTT20,

Original: CCB20, LCCBGB22, APB + 21] (we address the reader to Appendix A for an overview).
Revised: CCB20, LCCBGB22, APB + 21] (we address the reader to Appendix A for an overview).

Original: In their recent work, Simchi-Levi et al.
Revised: In their recent work, Simchi-Levi et al.

Original: [SLZZ21] study the k -RB problem and prove the first
Revised: [SLZZ21] study the k -RB problem and prove the ﬁrst

Original: O (1) -approximation guarantee under no additional assumptions.
Revised: O (1) -approximation guarantee under no additional assumptions.

Original: For large k , this result is the stateof-the-art.
Revised: For large k , this result is the stateof-the-art.

Original: However, for small k , understanding the magnitude of the constant becomes the primary theoretical question.
Revised: However, for small k , understanding the magnitude of the constant becomes the primary theoretical question.

Original: Specifically, the motivating case of newsfeeds (when, for instance, the content items are presented to a user sequentially) can be modeled as an instance of 1 -RB.
Revised: Speciﬁcally, the motivating case of newsfeeds (when, for instance, the content items are presented to a user sequentially) can be modeled as an instance of 1 -RB.

Original: In this case, the approximation guarantee of [SLZZ21] becomes 1 / 4 , which is significantly weaker compared to both the (1 − ϵ ) -approximation of [IK18] for concave functions and the (1 − 1 / e ) -approximation of [BSSS19] for the most “extreme” example of monotone convex functions, i.e., that of Heaviside step functions.
Revised: In this case, the approximation guarantee of [SLZZ21] becomes 1 / 4 , which is signiﬁcantly weaker compared to both the (1  ✏ ) -approximation of [IK18] for concave functions and the (1  1 / e ) -approximation of [BSSS19] for the most “extreme” example of monotone convex functions, i.e., that of Heaviside step functions.

Original: The above observation indicates that either the approximability status of the problem is not well-understood, or that the problem does not gradually become “easier” by increasing the concavity of the payoff functions.
Revised: The above observation indicates that either the approximability status of the problem is not well-understood, or that the problem does not gradually become “easier” by increasing the concavity of the payoff functions.

Original: 1.2 Technical Contributions
Revised: 1.2 Technical Contributions

Original: In our work, we resolve this discrepancy by designing a long-run (1 − 1 / e ) -approximation
Revised: In our work, we resolve this discrepancy by designing a long-run (1  1 / e ) -approximation

Original: algorithm (in expectation) for 1 -RB, which improves the state-of-the-art due to [SLZZ21].
Revised: algorithm (in expectation) for 1 -RB, which improves the state-of-the-art due to [SLZZ21].

Original: Simultaneously, our algorithm enjoys the same asymptotic guarantee of (1 − O ( 1 / √ k )) for the general case of k -RB as in Simchi-Levi et al.
Revised: Simultaneously, our algorithm enjoys the same asymptotic guarantee of (1  O ( 1 / p k )) for the general case of k -RB as in Simchi-Levi et al.

Original: [SLZZ21] with improved and explicit constants, as opposed to the guarantees of [SLZZ21] which do not come in a closed-form (see Table 1 for a summary of the best-known
Revised: [SLZZ21] with improved and explicit constants, as opposed to the guarantees of [SLZZ21] which do not come in a closed-form (see Table 1 for a summary of the best-known

Original: Payoff functions
Revised: Payoff functions

Original: State-of-the-art
Revised: State-of-the-art

Original: Reference
Revised: Reference

Original: This work
Revised: This work

Original: Non-decreasing, concave, Lipschitz (1 − ϵ ) -PTAS for k =[IK18] –
Revised: Non-decreasing, concave, Lipschitz (1  ✏ ) -PTAS for k =[IK18] [Table]

Original: Scaled/translated Heaviside step
Revised: Scaled/translated Heaviside step

Original:  [Equation] [Equation]
Revised: [Table] [Equation] [Table]

Original: Non-decreasing (finite recovery)
Revised: Non-decreasing (ﬁnite recovery)

Original:  [Equation] [Equation]
Revised: [Table] [Equation] [Table]

Original: approximation guarantees).
Revised: approximation guarantees).

Original: Our algorithm is based on a novel combination of linear programming (LP) rounding and a time-correlated (interleaved) scheduling method, and is significantly simpler to implement compared to prior work.
Revised: Our algorithm is based on a novel combination of linear programming (LP) rounding and a time-correlated (interleaved) scheduling method, and is signiﬁcantly simpler to implement compared to prior work.

Original: For the case where the mean payoff functions are initially unknown, we show how our algorithm can be transformed into a bandit algorithm with sublinear regret guarantees.
Revised: For the case where the mean payoff functions are initially unknown, we show how our algorithm can be transformed into a bandit algorithm with sublinear regret guarantees.

Original: 1.3 Key Ideas and Intuition
Revised: 1.3 Key Ideas and Intuition

Original: Henceforth, we refer to any instance where the payoff function of each action i has the form q i ( τ ) =
Revised: Henceforth, we refer to any instance where the payoff function of each action i has the form q i ( ⌧ ) =

Original: p i · H ( τ − d i ) , where p i ∈ [0 , 1] , d i ∈ N , and H ( · ) the Heaviside step function, as the Heaviside k -RB problem.
Revised: p i · H ( ⌧  d i ) , where p i 2 [0 , 1] , d i 2 N , and H ( · ) the Heaviside step function, as the Heaviside k -RB problem.

Original: The fact that our (1 − 1 / e ) -approximation for 1 -RB matches the state-of-theart bound obtained for Heaviside 1 -RB [BSSS19, PC21] is no coincidence.
Revised: The fact that our (  1 / e ) -approximation for 1 -RB matches the state-of-theart bound obtained for Heaviside 1 -RB [BSSS19, PC21] is no coincidence.

Original: Our solution technique shows that each k -RB problem is in fact (approximately) hiding a Heaviside k -RB problem.
Revised: Our solution technique shows that each k -RB problem is in fact (approximately) hiding a Heaviside k -RB problem.

Original: Structural characterization of a natural LP relaxation.
Revised: Structural characterization of a natural LP relaxation.

Original: A key idea in both [IK18] and [SLZZ21] is to construct a concave relaxation of the optimal solution.
Revised: A key idea in both [IK18] and [SLZZ21] is to construct a concave relaxation of the optimal solution.

Original: Instead, we take a more direct approach and construct a natural LP relaxation (see Section 3) of the optimal average payoff (our construction uses a uniform upper bound on the recovery time of any arm).
Revised: Instead, we take a more direct approach and construct a natural LP relaxation (see Section 3) of the optimal average payoff (our construction uses a uniform upper bound on the recovery time of any arm).

Original: By carefully analyzing the structure of our LP, we prove that its extreme point solutions follow a particularly interesting sparsity pattern : there exist unique delays { τ i } associated with the arms (which we refer to as “critical” delays), such that playing each arm i exactly once every τ i rounds matches the average payoff of the relaxation.
Revised: By carefully analyzing the structure of our LP, we prove that its extreme point solutions follow a particularly interesting sparsity pattern : there exist unique delays { ⌧ i } associated with the arms (which we refer to as “critical” delays), such that playing each arm i exactly once every ⌧ i rounds matches the average payoff of the relaxation.

Original: As we show, an exception to the above rule can be at most a single arm, which we refer to as irregular .
Revised: As we show, an exception to the above rule can be at most a single arm, which we refer to as irregular .

Original: The above observation already hints that our problem could potentially be reduced to an instance of
Revised: The above observation already hints that our problem could potentially be reduced to an instance of

Original: Heaviside k -RB for which better approximation guarantees (compared to [SLZZ21]) are known to exist (at least for small k )
Revised: Heaviside k -RB for which better approximation guarantees (compared to [SLZZ21]) are known to exist (at least for small k )

Original: [BSSS19, PC21].
Revised: [BSSS19, PC21].

Original: Improved approximation guarantees for the planning k -RB.
Revised: Improved approximation guarantees for the planning k -RB.

Original: Constructing a planning where each arm is played at a rate indicated by its critical delay (as described above) is generally infeasible, since these rates stem from a relaxation of the problem.
Revised: Constructing a planning where each arm is played at a rate indicated by its critical delay (as described above) is generally infeasible, since these rates stem from a relaxation of the problem.

Original: However, targeting these rates becomes the starting point of our approximation.
Revised: However, targeting these rates becomes the starting point of our approximation.

Original: In Section 4, we design a novel algorithm for the k -RB problem, called “Randomize-Then-Interleave”, which starts from computing an optimal solution to our LP relaxation.
Revised: In Section 4, we design a novel algorithm for the k -RB problem, called “Randomize-Then-Interleave”, which starts from computing an optimal solution to our LP relaxation.

Original: By combining the sparse structure of this solution with a randomized rounding step for determining a critical delay for the (possible) irregular arm, our algorithm produces a “proxy” instance of the Heaviside k -RB problem.
Revised: By combining the sparse structure of this solution with a randomized rounding step for determining a critical delay for the (possible) irregular arm, our algorithm produces a “proxy” instance of the Heaviside k -RB problem.

Original: Finally, the algorithm applies the interleaved scheduling method of [PC21] on the resulting Heaviside k -RB instance.
Revised: Finally, the algorithm applies the interleaved scheduling method of [PC21] on the resulting Heaviside k -RB instance.

Original: As we prove, the above algorithm achieves a long-run (1 − O ( 1 / √ k )) -approximation guarantee (in expectation), but with improved and explicit constants compared to [SLZZ21].
Revised: As we prove, the above algorithm achieves a long-run (1  O ( 1 / p k )) -approximation guarantee (in expectation), but with improved and explicit constants compared to [SLZZ21].

Original: Importantly, for the particular case of k -RB, our guarantee is stronger than the one provided in the original work of [PC21].
Revised: Importantly, for the particular case of k -RB, our guarantee is stronger than the one provided in the original work of [PC21].

Original: As a first step, in Section 4.1 we study the performance of our algorithm under the simplifying assumption that the solution returned by our LP relaxation does not contain an irregular arm, that is, every arm is associated with a unique critical delay.
Revised: As a ﬁrst step, in Section 4.1 we study the performance of our algorithm under the simplifying assumption that the solution returned by our LP relaxation does not contain an irregular arm, that is, every arm is associated with a unique critical delay.

Original: For this case, the approximation guarantee of our algorithm follows by critically leveraging existing results on the correlation gap of the weighted rank function of uniform matroids due to [Yan11] (see Section 2).
Revised: For this case, the approximation guarantee of our algorithm follows by critically leveraging existing results on the correlation gap of the weighted rank function of uniform matroids due to [Yan11] (see Section 2).

Original: In Section 4.2, we relax this simplifying assumption by carefully studying the contribution of the irregular arm to the produced solution.
Revised: In Section 4.2, we relax this simplifying assumption by carefully studying the contribution of the irregular arm to the produced solution.

Original: Our strategy relies on the novel idea of considering parallel (fictitious) copies of the irregular arm, one for each possible critical delay.
Revised: Our strategy relies on the novel idea of considering parallel (ﬁctitious) copies of the irregular arm, one for each possible critical delay.

Original: While the marginal probabilities of each copy being played under the corresponding delay at any round are consistent with the initial LP solution, these events are mutually exclusive .
Revised: While the marginal probabilities of each copy being played under the corresponding delay at any round are consistent with the initial LP solution, these events are mutually exclusive .

Original: Nevertheless, we show that expected payoff collected can only decrease by assuming that the parallel copies were instead independent.
Revised: Nevertheless, we show that expected payoff collected can only decrease by assuming that the parallel copies were instead independent.

Original: Having established that, the analysis reduces back to the “easier” case where an irregular arm does not exist.
Revised: Having established that, the analysis reduces back to the “easier” case where an irregular arm does not exist.

Original: A bandit adaptation with sublinear regret.
Revised: A bandit adaptation with sublinear regret.

Original: Finally, we turn our focus to the learning setting, where the mean payoff functions are initially unknown, and the player has semi-bandit feedback on the realized payoffs of the played arms of each round.
Revised: Finally, we turn our focus to the learning setting, where the mean payoff functions are initially unknown, and the player has semi-bandit feedback on the realized payoffs of the played arms of each round.

Original: In this setting, we begin by analyzing the robustness of our planning algorithm under small perturbations of the payoff functions.
Revised: In this setting, we begin by analyzing the robustness of our planning algorithm under small perturbations of the payoff functions.

Original: Then, we provide sample complexity results to bound the number of samples required to get accurate estimates of the mean payoff functions.
Revised: Then, we provide sample complexity results to bound the number of samples required to get accurate estimates of the mean payoff functions.

Original: By combining the above elements, we transform our planning algorithm into a bandit one, based on an Explore-then-Commit (ETC) scheme, and prove that the latter achieves sublinear regret relative to its planning counterpart.
Revised: By combining the above elements, we transform our planning algorithm into a bandit one, based on an Explore-then-Commit (ETC) scheme, and prove that the latter achieves sublinear regret relative to its planning counterpart.

Original: We present these results in Section 5.
Revised: We present these results in Section 5.

Original: All the omitted proofs of our results have been moved to the Appendix.
Revised: All the omitted proofs of our results have been moved to the Appendix.

Original:  Preliminaries
Revised:  Preliminaries

Original: Problem definition and notation.
Revised: Problem deﬁnition and notation.

Original: We consider a set
Revised: We consider a set

Original: A = { 1 , 2 , . . . , n } of n actions (or arms ), where each arm i ∈{+
+}
Revised: A = { 1 , 2 , . . . , n } of n actions (or arms ), where each arm i {+
+}

Original: A is associated with a (mean) payoff function p i : N → [0 , 1] .
Revised: A is associated with a (mean) payoff function p i : N !

Original: For each τ ∈ N , p i ( τ ) is the expected payoff for playing action i ∈ A , when i has been played before exactly τ rounds (i.e., τ = 1 if the action has been played in the previous round).
Revised: For each ⌧ 2 N , p i ( ⌧ ) is the expected payoff for playing action i 2 A , when i has been played before exactly ⌧ rounds (i.e., ⌧ = 1 if the action has been played in the previous round).

Original: We refer to the number of rounds since the last play of an arm as the delay .
Revised: We refer to the number of rounds since the last play of an arm as the delay .

Original: For each arm i ∈ A , we assume that the function p i ( τ ) : (a) is monotone non-decreasing in τ , and (b) has a polynomial recovery time τ max i such that p i ( τ ) =
Revised: For each arm i 2 A , we assume that the function p i ( ⌧ ) : (a) is monotone non-decreasing in ⌧ , and (b) has a polynomial recovery time ⌧ max i such that p i ( ⌧ ) =

Original: p i ( τ max i ) for every τ ≥ τ max i .
Revised: p i ( ⌧ max i ) for every ⌧  ⌧ max i .

Original: We assume knowledge of a universal upper bound τ max on the recovery time of all arms, i.e., τ max ≥ max i ∈A τ{+
 A ⌧+} max i .
Revised: We assume knowledge of a universal upper bound ⌧ max on the recovery time of all arms, i.e., ⌧ max  max i {+
 A ⌧+} max i .

Original: At each round, the player plays a subset of at most k < n actions and collects the sum of their associated payoffs (each computed with respect to the delay of the corresponding action at the time it is played).
Revised: At each round, the player plays a subset of at most k < n actions and collects the sum of their associated payoffs (each computed with respect to the delay of the corresponding action at the time it is played).

Original: The objective is to maximize the cumulative payoff collected within an (unknown) horizon of T rounds.
Revised: The objective is to maximize the cumulative payoff collected within an (unknown) horizon of T rounds.

Original: We refer to the above instance as k -RB.
Revised: We refer to the above instance as k -RB.

Original: As a convention, we assume that the delay of each arm i is initially equal to 1 (i.e., all arms are played at time t = 0 ).
Revised: As a convention, we assume that the delay of each arm i is initially equal to 1 (i.e., all arms are played at time t = 0 ).

Original: Note that, even if we assume that all arms start with delay τ max (i.e., the arms have never been played before), our results do not change qualitatively.
Revised: Note that, even if we assume that all arms start with delay ⌧ max (i.e., the arms have never been played before), our results do not change qualitatively.

Original: For any non-negative integer n ∈ N , we define [ n ] = { 1 , 2 , . . . , n } .
Revised: For any non-negative integer n 2 N , we deﬁne [ n ] = { 1 , 2 , . . . , n } .

Original: For any vector w ∈ R n and set S ⊆ [ n ] , we denote w ( S ) =
Revised: For any vector w 2 R n and set S ✓ [ n ] , we denote w ( S ) =

Original: (cid:80) i ∈ S w i .
Revised: P i 2 S w i .

Original: Finally, throughout this work we assume that, when comparing between different payoffs, ties are broken arbitrarily.
Revised: Finally, throughout this work we assume that, when comparing between different payoffs, ties are broken arbitrarily.

Original: Continuous extensions and the correlation gap.
Revised: Continuous extensions and the correlation gap.

Original: Let f : 2 [ n ] → [0 , ∞ ) be a set function defined over a finite set of n elements.
Revised: Let f : 2 [ n ] ! [0 , 1 ) be a set function deﬁned over a ﬁnite set of n elements.

Original: For any vector y ∈ [0 , 1] n , we denote by D ( y ) a distribution over 2 [ n ] with marginal probabilities y
Revised: For any vector y 2 [0 , 1] n , we denote by D ( y ) a distribution over 2 [ n ] with marginal probabilities y

Original: =
Revised: =

Original: ( y i ) i ∈ [ n ] .
Revised: ( y i ) i 2 [ n ] .

Original: We denote by S ∼ D ( y ) a random subset S ⊆ [ n ] sampled from the distribution D ( y ) .
Revised: We denote by S ⇠ D ( y ) a random subset S ✓ [ n ] sampled from the distribution D ( y ) .

Original: Further, we denote by S∼ I ( y ) a random subset S ⊆
Revised: Further, we denote by S

Original:  [ n ] which is constructed by adding each element i ∈ [ n ] to S , independently, with probability y i .
Revised: ⇠ I ( y ) a random subset S ✓ [ n ] which is constructed by adding each element i 2 [ n ] to S , independently, with probability y i .

Original: We recall two canonical continuous extensions of a set function (see [CCPV07, Sch03]):
Revised: We recall two canonical continuous extensions of a set function (see [CCPV07, Sch03]):

Original: Definition 2.1 (Multi-linear extension) .
Revised: Deﬁnition 2.1 (Multi-linear extension) .

Original: For any vector y ∈ [0 , 1] n , the multi-linear extension of a set function f is defined as
Revised: For any vector y 2 [0 , 1] n , the multi-linear extension of a set function f is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Definition 2.2 (Concave closure) .
Revised: Deﬁnition 2.2 (Concave closure) .

Original: For any vector y ∈ [0 , 1] n , the concave closure of a set function f is defined as
Revised: For any vector y 2 [0 , 1] n , the concave closure of a set function f is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where 1 S ∈ { 0 , 1 } n is an indicator vector such that ( 1 S )
Revised: where 1 S 2 { 0 , 1 } n is an indicator vector such that ( 1 S )

Original: i
Revised: i

Original: = 1 , if i ∈ S , and ( 1 S ) i = 0 , otherwise.
Revised: = 1 , if i 2 S , and ( 1 S ) i = 0 , otherwise.

Original: For any non-negative weight vector w ∈ [0 , ∞ ) n , of particular importance in the analysis of our algorithm is the function f w ,k : 2 [ n ] → [0 , ∞ ) , defined as
Revised: For any non-negative weight vector w 2 [0 , 1 ) n , of particular importance in the analysis of our algorithm is the function f w ,k : 2 [ n ] ! [0 , 1 ) , deﬁned as

Original: [Equation]
Revised: [Equation]

Original: We remark that f w ,k corresponds to the weighted rank function of the (rank- k ) uniform matroid under a weight vector w and, hence, is non-decreasing submodular [Sch03].
Revised: We remark that f w ,k corresponds to the weighted rank function of the (rank- k ) uniform matroid under a weight vector w and, hence, is non-decreasing submodular [Sch03].

Original: 3 We denote by F w ,k and f + w ,k the multi-linear extension and the concave closure of f w ,k , respectively.
Revised: 3 We denote by F w ,k and f + w ,k the multi-linear extension and the concave closure of f w ,k , respectively.

Original: The correlation gap [ADSY10] of a set function is defined as sup y ∈ [0 , 1] n f + ( y ) F ( y ) .
Revised: The correlation gap [ADSY10] of a set function is deﬁned as sup y 2 [0 , 1] n f + ( y ) F ( y ) .

Original: The following result due to [Yan11] provides an upper bound on the correlation gap of f w ,k ( · ) :
Revised: The following result due to [Yan11] provides an upper bound on the correlation gap of f w ,k ( · ) :

Original: Lemma 2.3 (Correlation gap [Yan11]) .
Revised: Lemma 2.3 (Correlation gap [Yan11]) .

Original: Let f w ,k : 2 [ n ] → [0 , ∞ ) be the weighted rank function of the rank- k uniform matroid.
Revised: Let f w ,k : 2 [ n ] ! [0 , 1 ) be the weighted rank function of the rank- k uniform matroid.

Original: Then, for any vector y ∈ [0 , 1] n we have
Revised: Then, for any vector y 2 [0 , 1] n we have

Original: [Equation]
Revised: [Equation]

Original:  Structural Properties of a Natural LP Relaxation
Revised:  Structural Properties of a Natural LP Relaxation

Original: We consider the following LP relaxation which applies to any planning instance of k -RB:
Revised: We consider the following LP relaxation which applies to any planning instance of k -RB:

Original: [Equation]
Revised: [Equation]

Original: In the above formulation, each variable x i,τ denotes the fraction of time where arm i is played under delay τ – namely, exactly τ time steps after it was played for the last time.
Revised: In the above formulation, each variable x i, ⌧ denotes the fraction of time where arm i is played under delay ⌧ – namely, exactly ⌧ time steps after it was played for the last time.

Original: In ( LP k ), constraint (C.1) follows by the fact that the total fraction of time where any action can be played (under any delay) cannot be more than k .
Revised: In ( LP k ), constraint (C.1) follows by the fact that the total fraction of time where any action can be played (under any delay) cannot be more than k .

Original: Further, constraints (C.2) are valid constraints for the fractional plays of any individual arm under different delays.
Revised: Further, constraints (C.2) are valid constraints for the fractional plays of any individual arm under different delays.

Original: We start by proving that the optimal solution of ( LP k ) is asymptotically an upper bound on the optimal average expected payoff:
Revised: We start by proving that the optimal solution of ( LP k ) is asymptotically an upper bound on the optimal average expected payoff:

Original: Lemma 3.1.
Revised: Lemma 3.1.

Original: For any instance of the k -RB problem, let V ∗ be the optimal value of ( LP k ) and OPT( T ) be the maximum payoff that can be collected in a time horizon of T rounds.
Revised: For any instance of the k -RB problem, let V ⇤ be the optimal value of ( LP k ) and OPT( T ) be the maximum payoff that can be collected in a time horizon of T rounds.

Original: Then T · V ∗ ≥ OPT( T ) .
Revised: Then T · V ⇤  OPT( T ) .

Original: Assuming knowledge of an upper bound τ max ≥ max i ∈A { τ max i } on the maximum recovery time of any action, an optimal extreme point solution to ( LP k ) can be computed in polynomial time:
Revised: Assuming knowledge of an upper bound ⌧ max  max i 2 A { ⌧ max i } on the maximum recovery time of any action, an optimal extreme point solution to ( LP k ) can be computed in polynomial time:

Original: Fact 3.2.
Revised: Fact 3.2.

Original: There exists an optimal solution to ( LP k ) which is supported in ( x i,τ ) i ∈A ,τ ∈ [ τ max i ] .
Revised: There exists an optimal solution to ( LP k ) which is supported in ( x i, ⌧ ) i 2 A , ⌧ 2 [ ⌧ max i ] .

Original: Further, given an upper bound τ max on the maximum recovery time of any action, an optimal extreme point solution to ( LP k ) can be computed in polynomial time.
Revised: Further, given an upper bound ⌧ max on the maximum recovery time of any action, an optimal extreme point solution to ( LP k ) can be computed in polynomial time.

Original: Let us introduce the notion of supported actions:
Revised: Let us introduce the notion of supported actions:

Original: Definition 3.3 (Supported actions) .
Revised: Deﬁnition 3.3 (Supported actions) .

Original: For any (potentially infeasible) solution x to ( LP k ) , we say that an action i ∈ A is supported in x , denoted by i∈
Revised: For any (potentially infeasible) solution x to ( LP k ) , we say that an action i 2 A is supported in x , denoted by i

Original:  A ( x ) , if there exists τ ∈ N such that x i,τ > 0 .
Revised: 2 A ( x ) , if there exists ⌧ 2 N such that x i, ⌧ > 0 .

Original: We introduce the following class of solutions:
Revised: We introduce the following class of solutions:

Original: Definition 3.4 (Delay-feasible solutions) .
Revised: Deﬁnition 3.4 (Delay-feasible solutions) .

Original: We say that a vector x = ( x i,τ ) i ∈A ,τ ∈ N is delay-feasible , if for each supported action i ∈ A ( x ) there exists at most one non-zero variable x i,τ i , and for this variable it holds x i,τ i = 1 / τ i .
Revised: We say that a vector x = ( x i, ⌧ ) i 2 A , ⌧ 2 N is delay-feasible , if for each supported action i 2 A ( x ) there exists at most one non-zero variable x i, ⌧ i , and for this variable it holds x i, ⌧ i = 1 / ⌧ i .

Original: Further, we say that x is almost-delay-feasible , if it is delay-feasible, possibly with the exception of a single arm ι ∈ A ( x ) , for which there exist at most two non-zero variables x ι,τ ι,a , x ι,τ ι,b , such that x ι,τ ι,a < 1 / τ ι,a and x ι,τ ι,b < 1 / τ ι,b .
Revised: Further, we say that x is almost-delay-feasible , if it is delay-feasible, possibly with the exception of a single arm ◆ 2 A ( x ) , for which there exist at most two non-zero variables x ◆ , ⌧ ◆ ,a , x ◆ , ⌧ ◆ ,b , such that x ◆ , ⌧ ◆ ,a < 1 / ⌧ ◆ ,a and x ◆ , ⌧ ◆ ,b < 1 / ⌧ ◆ ,b .

Original: In the case of almost-delay- feasible solutions, we refer to ι as the irregular arm.
Revised: In the case of almost-delay- feasible solutions, we refer to ◆ as the irregular arm.

Original: We remark that delay-feasible solutions subsume by definition almost-delay-feasible solutions and that a (almost-)delay-feasible solution is not necessarily a feasible solution to ( LP k ).
Revised: We remark that delay-feasible solutions subsume by deﬁnition almost-delay-feasible solutions and that a (almost-)delay-feasible solution is not necessarily a feasible solution to ( LP k ).

Original: As we prove in the next key-lemma, any extreme point solution of ( LP k ) is almost-delay-feasible:
Revised: As we prove in the next key-lemma, any extreme point solution of ( LP k ) is almost-delay-feasible:

Original: Lemma 3.5 (Sparsity pattern of extreme point solutions) .
Revised: Lemma 3.5 (Sparsity pattern of extreme point solutions) .

Original: Let x be any extreme point solution of ( LP k ) .
Revised: Let x be any extreme point solution of ( LP k ) .

Original: Then, x is almost-delay-feasible.
Revised: Then, x is almost-delay-feasible.

Original:  Improved Approximation Guarantees for Planning
Revised:  Improved Approximation Guarantees for Planning

Original: For any instance of k -RB, our algorithm (see Algorithm 1 below), called “Randomize-ThenInterleave”, constructs a feasible planning schedule without requiring knowledge of the time horizon.
Revised: For any instance of k -RB, our algorithm (see Algorithm 1 below), called “Randomize-ThenInterleave”, constructs a feasible planning schedule without requiring knowledge of the time horizon.

Original: We remark that for the planning setting of the problem, we treat the payoff of any arm under any possible delay as deterministic.
Revised: We remark that for the planning setting of the problem, we treat the payoff of any arm under any possible delay as deterministic.

Original: The algorithm starts from computing an optimal extreme point solution x ∗ to ( LP k ), and then uses this solution to determine a critical delay τ ∗ i for each supported arm
Revised: The algorithm starts from computing an optimal extreme point solution x ⇤ to ( LP k ), and then uses this solution to determine a critical delay ⌧

Original:  i ∈ A ( x ∗ ) .
Revised: ⇤ i for each supported arm i 2 A ( x ⇤ ) .

Original: By Lemma 3.5, we know that x ∗ is almost-delay-feasible, which means that – possibly with the exception of a single irregular arm ι – for every arm i ∈
Revised: By Lemma 3.5, we know that x ⇤ is almost-delay-feasible, which means that – possibly with the exception of a single irregular arm ◆ – for every arm i 2 A

Original: A ( x ∗ ) \ { ι } , there exists a unique τ i , such that x ∗ i,τ i = 1 / τ i > 0 .
Revised: ( x ⇤ ) \ { ◆ } , there exists a unique ⌧ i , such that x ⇤ i, ⌧ i = 1 / ⌧ i > 0 .

Original: Thus, Algorithm 1 sets this unique τ i to be the critical delay of each arm
Revised: Thus, Algorithm 1 sets this unique ⌧ i to be the critical delay of each arm

Original: i ∈{+
+}
Revised: i {+
+}

Original: A ( x ∗ ) \ { ι } .
Revised: A ( x ⇤ ) \ { ◆ } .

Original: In the case where an irregular arm ι ∈ A ( x ∗ ) exists, then, by Definition 3.4, there exist at most two distinct τ ι,a , τ ι,b with x ∗ ι,τ ι,a > 0 and x ∗ ι,τ ι,b ≥ 0 , in which case the critical delay τ ∗ ι of ι is set to τ ι,a or τ ι,b , with marginal probability τ ι,a x ∗ ι,τ ι,a and τ ι,b x ∗ ι,τ ι,b , respectively.
Revised: In the case where an irregular arm ◆ 2 A ( x ⇤ ) exists, then, by Deﬁnition 3.4, there exist at most two distinct ⌧ ◆ ,a , ⌧ ◆ ,b with x ⇤ ◆ , ⌧ ◆ ,a > 0 and x ⇤ ◆ , ⌧ ◆ ,b  0 , in which case the critical delay ⌧ ⇤ ◆ of ◆ is set to ⌧ ◆ ,a or ⌧ ◆ ,b , with marginal probability ⌧ ◆ ,a x ⇤ ◆ , ⌧ ◆ ,a and ⌧ ◆ , b x ⇤ ◆ , ⌧ ◆ ,b , respectively.

Original: Notice that by constraints (C.2) of ( LP k ), the above sampling process is well-defined, since τ ι,a x ∗ ι,τ ι,a + τ ι,b x ∗ ι,τ ι,b ≤ 1 .
Revised: Notice that by constraints (C.2) of ( LP k ), the above sampling process is well-deﬁned, since ⌧ ◆ ,a x ⇤ ◆ , ⌧ ◆ ,a + ⌧ ◆ ,b x ⇤ ◆ , ⌧ ◆ ,b  1 .

Original: In the case where no τ ∗ ι is sampled, the irregular arm is removed from A ( x ∗ ) .
Revised: In the case where no ⌧ ⇤ ◆ is sampled, the irregular arm is removed from A ( x ⇤ ) .

Original: After computing the critical delays, for each arm i ∈ A ( x ∗ ) the algorithm draws a random offset r i independently and uniformly from { 0 , 1 , . ..
Revised: After computing the critical delays, for each arm i 2 A ( x ⇤ ) the algorithm draws a random offset r i independently and uniformly from { 0 , 1 , . .

Original: , τ ∗ i − 1 } .
Revised: ⌧ ⇤ i  1 } .

Original: Algorithm
Revised: Algorithm

Original: /* Initialization phase */ Compute an optimal extreme point solution x ∗ to ( LP k ).
Revised: /* Initialization phase */ Compute an optimal extreme point solution x ⇤ to ( LP k ).

Original: Let ι ∈ A be the irregular arm (if such an arm exists).
Revised: Let ◆ 2 A be the irregular arm (if such an arm exists).

Original: for each arm i ∈ A ( x ∗ ) \ { ι } do 
Revised: for each arm i 2 A ( x ⇤ ) \ { ◆ } do Let ⌧

Original: Let τ ∗ i be the unique τ ∈ N which satisfies x ∗ i,τ = 1 / τ .
Revised: ⇤ i be the unique ⌧ 2 N which satisﬁes x ⇤ i, ⌧ = 1 / ⌧ .

Original: if an irregular arm ι ∈ A ( x ∗ ) exists then 
Revised: if an irregular arm ◆ 2 A ( x ⇤ ) exists then Let ⌧ ◆ ,a , ⌧ ◆ ,

Original: Let τ ι,a , τ ι,b ∈ N be such that x ∗ ι,τ ι,a > 0 , x ∗ ι,τ ι,b ≥ 0 , and τ ι,a ̸ = τ ι,b .
Revised: b 2 N be such that x ⇤ ◆ , ⌧ ◆ ,a > 0 , x ⇤ ◆ , ⌧ ◆ ,b  0 , and ⌧ ◆ ,a 6 = ⌧ ◆ ,b .

Original: Set p a ← τ ι,a x ∗ ι,τ ι,a and p b ← τ ι,b x ∗ ι,τ ι,b .
Revised: Set p a ⌧ ◆ ,a x ⇤ ◆ , ⌧ ◆ ,a and p b ⌧ ◆ ,b x ⇤ ◆ , ⌧ ◆ ,b .

Original: Sample τ ∗ ι from { τ ι,a , τ ι,b , ∞} with marginals p a , p b , and 1 − p a − p b , respectively.
Revised: Sample ⌧ ⇤ ◆ from { ⌧ ◆ ,a , ⌧ ◆ ,b , 1 } with marginals p a , p b , and 1  p a  p b , respectively.

Original: if τ ∗ ι = ∞ then 
Revised: if ⌧ ⇤ ◆ = 1 then Remove ◆ from

Original: Remove ι from A ( x ∗ ) .
Revised: A ( x ⇤ ) .

Original: for each arm i ∈ A ( x ∗ ) do Sample an offset r i uniformly at random from { 0 , 1 , . . 
Revised: for each arm i 2 A ( x ⇤ ) do Sample an offset r i uniformly at random from { 0 , 1 , . . . ,

Original: , τ ∗ i − 1 } .
Revised: ⌧ ⇤ i  1 } .

Original: /* Online phase */ for t = 1 , 2 , . . .
Revised: /* Online phase */ for t = 1 , 2 , . . .

Original: do
Revised: do

Original: Let C t ⊆ A be the subset of candidate arms, defined as C t=
Revised: Let C t ✓ A be the subset of candidate arms, deﬁned as C t

Original:  { i ∈ A ( x ∗ )
Revised: = { i 2 A ( x ⇤ )

Original: | t mod τ ∗ i ≡ r i } .
Revised: | t mod ⌧ ⇤ i ⌘ r i } .

Original: Play the maximum-payoff subset of k arms in C t , namely, A t = argmaxS
Revised: Play the maximum-payoff subset of k arms in C t , namely, A t = argmax

Original: ⊆ C t , | S |≤ k (cid:80) i ∈ S p
Revised: S ✓ C t , | S |  k P i 2 S p

Original: i ( σ i,t ) , where σ i,t ( actual delay ) is the time passed since arm i was last played before t .
Revised: i (  i,t ) , where  i,t ( actual delay ) is the time passed since arm i was last played before t .

Original: After initialization, Algorithm 1 proceeds to its online phase where, at each round t , it first computes a subset of candidate arms C t , namely, all arms i ∈
Revised: After initialization, Algorithm 1 proceeds to its online phase where, at each round t , it ﬁrst computes a subset of candidate arms C t , namely, all arms i 2 A ( x ⇤ ) which satisfy t mod ⌧

Original: A ( x ∗ ) which satisfy t mod τ ∗ i ≡ r i .
Revised: ⇤ i ⌘ r i .

Original: Then, the algorithm computes the maximum-payoff subset A t of at most k candidate arms, computed using their actual delays (the number of rounds passed since last play), and plays these arms.
Revised: Then, the algorithm computes the maximum-payoff subset A t of at most k candidate arms, computed using their actual delays (the number of rounds passed since last play), and plays these arms.

Original: 4.1 Approximation Analysis: Assuming Delay-Feasible LP Solutions
Revised: 4.1 Approximation Analysis: Assuming Delay-Feasible LP Solutions

Original: We now present an analysis of the approximation guarantee of Algorithm 1 for general k .
Revised: We now present an analysis of the approximation guarantee of Algorithm 1 for general k .

Original: In order to facilitate the presentation, we first focus on a special class of “easy” instances, where the optimal extreme point solution x ∗ of ( LP k ) is delay-feasible (i.e., there is no irregular arm).
Revised: In order to facilitate the presentation, we ﬁrst focus on a special class of “easy” instances, where the optimal extreme point solution x ⇤ of ( LP k ) is delay-feasible (i.e., there is no irregular arm).

Original: Consequently, we show how this assumption can be relaxed at the expense of additional technical effort.
Revised: Consequently, we show how this assumption can be relaxed at the expense of additional technical effort.

Original: Recall that we denote by A ( x ∗ ) the set of arms that are supported in the solution x ∗ .
Revised: Recall that we denote by A ( x ⇤ ) the set of arms that are supported in the solution x ⇤ .

Original: Notice that, since x ∗ is delay-feasible by assumption, the only source of randomness in Algorithm 1 is due to the random offsets.
Revised: Notice that, since x ⇤ is delay-feasible by assumption, the only source of randomness in Algorithm 1 is due to the random offsets.

Original: Since each offset r i is sampled independently and uniformly from { 0 , 1 , . ., τ ∗ i − 1 } and i ∈
Revised: Since each offset r i is sampled independently and uniformly from { 0 , 1 , . .

Original:  C t 
Revised: ⌧ ⇤ i  1 } and i 2 C t if and only if t mod ⌧

Original: if and only if t mod τ ∗ i ≡ r i , the following fact is immediate:
Revised: ⇤ i ⌘ r i , the following fact is immediate:

Original: Fact 4.1.
Revised: Fact 4.1.

Original: Let { τ ∗ i } i ∈A ( x ∗ ) be the critical delays of the supported arms.
Revised: Let { ⌧ ⇤ i } i 2 A ( x ⇤ ) be the critical delays of the supported arms.

Original: At any round t , each arm i ∈ A ( x ∗ ) belongs to C t independently with probability 1 / τ ∗ i .
Revised: At any round t , each arm i 2 A ( x ⇤ ) belongs to C t independently with probability 1 / ⌧ ⇤ i .

Original: Let us focus on the expected payoff collected by Algorithm 1 at any round t ≥ τ max .
Revised: Let us focus on the expected payoff collected by Algorithm 1 at any round t  ⌧ max .

Original: By our simplifying assumption that x ∗ is delay-feasible, for each supported arm i ∈ A ( x ∗ ) , there is a unique critical delay τ ∗ i , such that x ∗ i,τ ∗ i =
Revised: By our simplifying assumption that x ⇤ is delay-feasible, for each supported arm i 2 A ( x ⇤ ) , there is a unique critical delay ⌧ ⇤ i , such that x ⇤ i, ⌧

Original:  1 / τ ∗
Revised: ⇤ i = 1 / ⌧

Original:  i is the only non-zero variable of x ∗ corresponding to i .
Revised: ⇤ i is the only non-zero variable of x ⇤ corresponding to i .

Original: We construct a vector y =
Revised: We construct a vector y =

Original: [0 , 1] n , such that y i = 1 / τ ∗ i for each
Revised: [0 , 1] n , such that y i = 1 / ⌧

Original:  i ∈ A ( x ∗ ) , and y i = 0 , otherwise.
Revised: ⇤ i for each i 2 A ( x ⇤ ) , and y i = 0 , otherwise.

Original: Further, let us define a vector w ∈ [0 , ∞ ) n such that w i = p
Revised: Further, let us deﬁne a vector w 2 [0 , 1 ) n such that w i = p

Original: i ( τ ∗ i ) for each i ∈ A ( x ∗ ) , and w i = 0 , otherwise.
Revised: i ( ⌧ ⇤ i ) for each i 2 A ( x ⇤ ) , and w i = 0 , otherwise.

Original: Recall that for any fixed round t ≥ τ max , by definition of C t and the construction of the offsets, the events { i ∈ C t } i ∈A are independent.{+
 A+}
Revised: Recall that for any ﬁxed round t  ⌧ max , by deﬁnition of C t and the construction of the offsets, the events { i 2 C t } i {+
 A+}

Original:  Hence, we can think of C t as a random set, where each arm i ∈ A belongs to C t independently with probability y i , namely, C t ∼ I ( y ) .
Revised: are independent. Hence, we can think of C t as a random set, where each arm i 2 A belongs to C t independently with probability y i , namely, C t ⇠ I ( y ) .

Original: Let A t = argmax 
Revised: Let A t = argmax S

Original: [Equation]
Revised: [Equation]

Original: (cid:80) i ∈ S p 
Revised: p i [Equation]

Original: i ( τ ∗ i ) be a set of at most k arms in C t which maximizes the total payoff measured with respect to the critical delays.
Revised: ) be a set of at most k arms in C t which maximizes the total payoff measured with respect to the critical delays.

Original: By construction of C t , the actual delay σ i,t of each i ∈ A ( x ∗ ) is (deterministically) lower bounded by τ ∗ i , since any arm must be a candidate in order to be played (which happens exactly every τ ∗ i rounds).
Revised: By construction of C t , the actual delay  i,t of each i 2 A ( x ⇤ ) is (deterministically) lower bounded by ⌧ ⇤ i , since any arm must be a candidate in order to be played (which happens exactly every ⌧ ⇤ i rounds).

Original: Thus, for the expected collected payoff of any round t ≥ τ max we have
Revised: Thus, for the expected collected payoff of any round t  ⌧ max we have

Original: [Equation]
Revised: [Equation]

Original: Using Equation (1) in combination with Definitions 2.1 and 2.2, we get
Revised: Using Equation (1) in combination with Deﬁnitions 2.1 and 2.2, we get

Original: [Equation]
Revised: [Equation]

Original: where the last inequality follows by Lemma 2.3.
Revised: where the last inequality follows by Lemma 2.3.

Original: The following technical lemma relates f + w ,k ( y ) with optimal value of ( LP k ):
Revised: The following technical lemma relates f + w ,k ( y ) with optimal value of ( LP k ):

Original: Lemma 4.2.
Revised: Lemma 4.2.

Original: For any vectors w ∈ [0 , ∞ ) n and y ∈ [0 , 1] n , let f + w ,k
Revised: For any vectors w 2 [0 , 1 ) n and y 2 [0 , 1] n , let f + w ,k

Original: ( y ) be the concave closure of the weighted rank function of the rank- k uniform matroid with weights w , evaluated at y .
Revised: ( y ) be the concave closure of the weighted rank function of the rank- k uniform matroid with weights w , evaluated at y .

Original: Then, if y satisfies ∥ y ∥ 1 ≤ k , it is the case that f + w ,k ( y ) ≥ (cid:80) i ∈
Revised: Then, if y satisﬁes k y k 1  k , it is the case that f + w ,k ( y )  P [Equation]

Original:  [ n ] w i
Revised: 2 [ n ] w i

Original: y i .
Revised: y i .

Original: Let V ∗ be the optimal value of ( LP k ).
Revised: Let V ⇤ be the optimal value of ( LP k ).

Original: Since ∥ y ∥ 1 ≤ k , then by applying Lemma 4.2 with vectors w and y as constructed above, we have that
Revised: Since k y k 1  k , then by applying Lemma 4.2 with vectors w and y as constructed above, we have that

Original: [Equation]
Revised: [Equation]

Original: By combining Equations (2) to (4) with Lemma 3.1, it follows that, for any round t ≥ τ max , the expected payoff collected by Algorithm 1 is at least(cid:0) 1 − k k / e k k ! (cid:1) -times
Revised: By combining Equations (2) to (4) with Lemma 3.1, it follows that, for any round t  ⌧ max , the expected payoff collected by Algorithm 1 is at least

Original: the average optimal payoff.
Revised: [Equation]

Original: By using linearity of expectations together with the fact that all payoffs are bounded in [0 , 1] , we can prove a long-run (cid:0) 1 − k k / e k k !
Revised: By using linearity of expectations together with the fact that all payoffs are bounded in [0 , 1] , we can prove a long-run [Equation]

Original: (cid:1) -approximation guarantee (in expectation) for instances where x ∗ is delay-feasible.
Revised: approximation guarantee (in expectation) for instances where x ⇤ is delay-feasible.

Original: In the rest of this section, we show how the above analysis can be extended for the general case where x ∗ is almost-delay-feasible (i.e., an irregular arm exists).
Revised: In the rest of this section, we show how the above analysis can be extended for the general case where x ⇤ is almost-delay-feasible (i.e., an irregular arm exists).

Original: 4.2 Approximation Analysis: Handling the Irregular Arm
Revised: 4.2 Approximation Analysis: Handling the Irregular Arm

Original: We now extend the analysis of Section 4.1 and relax the assumption that the optimal extreme point solution of ( LP k ) is delay-feasible.
Revised: We now extend the analysis of Section 4.1 and relax the assumption that the optimal extreme point solution of ( LP k ) is delay-feasible.

Original: Let x ∗ be an optimal solution to ( LP k ) and let ι ∈ A ( x ∗ ) be the irregular arm with τ ∗ ι,a and τ ∗ ι,b being the two potential critical delays such that x ∗ ι,τ ι,a , x ∗ ι,τ ι,b ≥ 0 (with at least one of x ∗ ι,τ ι,a and x ∗ ι,τ ι,b being strictly positive, by Definition 3.4).
Revised: Let x ⇤ be an optimal solution to ( LP k ) and let ◆ 2 A ( x ⇤ ) be the irregular arm with ⌧ ⇤ ◆ ,a and ⌧ ⇤ ◆ , b being the two potential critical delays such that x ⇤ ◆ , ⌧ ◆ ,a , x ⇤ ◆ , ⌧ ◆ ,b  0 (with at least one of x ⇤ ◆ , ⌧ ◆ ,a and x ⇤ ◆ , ⌧ ◆ ,b being strictly positive, by Deﬁnition 3.4).

Original: Instead of conditioning on the sampled critical delay of the irregular arm ι , we construct two (fictitious) parallel copies ι a and ι b – one for each possible realization (and associated payoff).
Revised: Instead of conditioning on the sampled critical delay of the irregular arm ◆ , we construct two (ﬁctitious) parallel copies ◆ a and ◆ b – one for each possible realization (and associated payoff).

Original: Let us define A′ =
Revised: Let us deﬁne A =

Original: A ( x ∗ ) \ { ι } ∪ { ι a , ι b } to be the set of supported arms of x ∗ , where we replace the irregular arm ι with the two parallel copies ι a and ι b .
Revised: A ( x ⇤ ) \ { ◆ } [ { ◆ a , ◆ b } to be the set of supported arms of x ⇤ , where we replace the irregular arm ◆ with the two parallel copies ◆ a and ◆ b .

Original: In the next lemma, we characterize the marginal probability that any arm in A ′ is added to the set of candidate arms at any round.
Revised: In the next lemma, we characterize the marginal probability that any arm in A 0 is added to the set of candidate arms at any round.

Original: Notice that, as opposed to Fact 4.1, the randomness in this case is taken over both the sampling of a critical delay for the irregular arm as well as the random offsets.
Revised: Notice that, as opposed to Fact 4.1, the randomness in this case is taken over both the sampling of a critical delay for the irregular arm as well as the random offsets.

Original: Lemma 4.3.
Revised: Lemma 4.3.

Original: For any arm i ∈ A ′ and at any round t , we have that P [ i ∈ C t and τ ∗ i = τ ] = x∗ i,τ .
Revised: For any arm i 2 A 0 and at any round t , we have that P [ i 2 C t and ⌧ ⇤ i = ⌧ ] = x

Original:  Notice that, for the parallel copies of the irregular arm, the events { ι a ∈ C t } and { ι b ∈ C t } are now dependent and, in particular, mutually exclusive (since, eventually, at most one of them is realized).
Revised: ⇤ i, ⌧ . Notice that, for the parallel copies of the irregular arm, the events { ◆ a 2 C t } and { ◆ b 2 C t } are now dependent and, in particular, mutually exclusive (since, eventually, at most one of them is realized).

Original: However, we are able to show that the expected payoff collected at any round t ≥ τ max can only decrease if the two events were instead independent with the same marginal probabilities.
Revised: However, we are able to show that the expected payoff collected at any round t  ⌧ max can only decrease if the two events were instead independent with the same marginal probabilities.

Original: For convenience, we relabel the arms in A ′ in non-increasing order of p i ( τ ∗ i ) (breaking ties arbitrarily).
Revised: For convenience, we relabel the arms in A 0 in non-increasing order of p i ( ⌧ ⇤ i ) (breaking ties arbitrarily).

Original: Note that by exchanging ι with the two copies ι a and ι b the cardinality of A ′ can be at most n + 1 .
Revised: Note that by exchanging ◆ with the two copies ◆ a and ◆ b the cardinality of A 0 can be at most n + 1 .

Original: Let us construct a ( n + 1)
Revised: Let us construct a ( n + 1)

Original: -dimensional vector y ∈ [0 , 1] n +1 , such that y i =
Revised: -dimensional vector y 2 [0 , 1] n +1 , such that y i =

Original: x 
Revised: x ⇤ i, ⌧

Original: ∗ i,τ ∗ i for any i ∈ A ′ , and y i = 0 , otherwise.
Revised: ⇤ i for any i 2 A 0 , and y i = 0 , otherwise.

Original: Further, we construct a weight vector w ∈ [0 , ∞ ) n +1 , such that w i = p
Revised: Further, we construct a weight vector w 2 [0 , 1 ) n +1 , such that w i = p

Original: i ( τ ∗ i ) for anyi ∈
Revised: i ( ⌧ ⇤ i ) for any

Original:  A ′ , and w i = 0 , otherwise.
Revised: i 2 A 0 , and w i = 0 , otherwise.

Original: Let D ( y ) be a distribution over subsets of A ′ , where every arm 
Revised: Let D ( y ) be a distribution over subsets of A 0 , where every arm i 2 A 0 \{ ◆ a , ◆ b } (except for ◆ a and ◆ b ) is added to a set S ⇠ D ( y ) , independently, with probability x ⇤ i, ⌧

Original:  i ∈
Revised: ⇤ i .

Original: A ′ \{ ι a , ι b } (except for ι a and ι b ) is added to a set S ∼ D ( y ) , independently, with probability x ∗ i,τ ∗ i . The arms ι a and ι b are also added to S ∼ D ( y ) with marginal probabilities x ∗ ι a,τ ∗ ιa and x ∗ ι b,τ ∗ ιb , respectively, yet not independently (since they are mutually exclusive).
Revised: The arms ◆ a and ◆ b are also added to S ⇠ D ( y ) with marginal probabilities x ⇤ ◆ a , ⌧ ⇤ ◆ a and x ⇤ ◆ b , ⌧ ⇤ ◆ b , respectively, yet not independently (since they are mutually exclusive).

Original: Therefore, by the above definitions and Lemma 4.3, the set of candidate arms at any fixed round t ≥ τ max is distributed as
Revised: Therefore, by the above deﬁnitions and Lemma 4.3, the set of candidate arms at any ﬁxed round t  ⌧ max is distributed as

Original: C t ∼ D ( y ) .
Revised: C t ⇠ D ( y ) .

Original: Let us fix any round t ≥ τ max .
Revised: Let us ﬁx any round t  ⌧ max .

Original: Similarly to Section 4.1, by using the fact that the actual delay σ i,τ of each arm i ∈ A ′ is deterministically lower bounded by its critical delay τ ∗ i , we can lower bound the expected payoff collected by our algorithm at time t as
Revised: Similarly to Section 4.1, by using the fact that the actual delay  i, ⌧ of each arm i 2 A 0 is deterministically lower bounded by its critical delay ⌧ ⇤ i , we can lower bound the expected payoff collected by our algorithm at time t as

Original: [Equation]
Revised: [Equation]

Original: where D ( y ) is defined as described above.
Revised: where D ( y ) is deﬁned as described above.

Original: As we show in the following lemma, the RHS of the above inequality can only decrease, if we replace
Revised: As we show in the following lemma, the RHS of the above inequality can only decrease, if we replace

Original: C t ∼ D ( y ) with C t ∼ I ( y ) , namely, if we assume that the two copies ι a and ι b are added to C t independently, but with the same marginals ( y ι a and y ι b , respectively).
Revised: C t ⇠ D ( y ) with C t ⇠ I ( y ) , namely, if we assume that the two copies ◆ a and ◆ b are added to C t independently, but with the same marginals ( y ◆ a and y ◆ b , respectively).

Original: As we show in the following lemma, the above property in fact holds for any submodular function (including f w ,k ):
Revised: As we show in the following lemma, the above property in fact holds for any submodular function (including f w ,k ):

Original: Lemma 4.4.
Revised: Lemma 4.4.

Original: Let f : 2 [ n ] → R be a submodular function over a ground set of n elements, where n ≥ 2 .
Revised: Let f : 2 [ n ] ! R be a submodular function over a ground set of n elements, where n  2 .

Original: Let I ( y ) be a distribution over 2 [ n ] , where each element i is added in set S∼ I ( y ) , independently, with probability
Revised: Let I ( y ) be a distribution over 2 [ n ] , where each element i is added in set S

Original:  y i .
Revised: ⇠ I ( y ) , independently, with probability y i .

Original: Further, let D ( y ) be a distribution over 2 [ n ] , where each element i ∈ [ n ] \ { ι a , ι b } is added to a set S ∼ D ( y ) , independently, with probability y i , except for two distinct elements ι a , ι b , whose addition to S is mutually exclusive but with the same marginals y ι a and y ι b , respectively.
Revised: Further, let D ( y ) be a distribution over 2 [ n ] , where each element i 2 [ n ] \ { ◆ a , ◆ b } is added to a set S ⇠ D ( y ) , independently, with probability y i , except for two distinct elements ◆ a , ◆ b , whose addition to S is mutually exclusive but with the same marginals y ◆ a and y ◆ b , respectively.

Original: In this case, we have
Revised: In this case, we have

Original: [Equation]
Revised: [Equation]

Original: By combining inequality (5) with Lemma 4.4 for n + 1 elements (recall, the function f w ,k is submodular), the expected reward collected at any round t ≥ τ max can be lower bounded as
Revised: By combining inequality (5) with Lemma 4.4 for n + 1 elements (recall, the function f w ,k is submodular), the expected reward collected at any round t  ⌧ max can be lower bounded as

Original: [Equation]
Revised: [Equation]

Original: In addition, by combining Lemma 2.3 with Lemma 4.2 for the above choice of vectors w and y (again, it holds ∥ y ∥ 1 ≤ k ), for any t ≥ τ max , it follows that
Revised: In addition, by combining Lemma 2.3 with Lemma 4.2 for the above choice of vectors w and y (again, it holds k y k 1  k ), for any t  ⌧ max , it follows that

Original: [Equation]
Revised: [Equation]

Original: where V ∗ is the optimal value of ( LP k ).
Revised: where V ⇤ is the optimal value of ( LP k ).

Original: Finally, using the fact that V ∗ ≥ OPT( T ) / T by Lemma 3.1 and applying linearity of expectations over all rounds t , the following result follows directly:
Revised: Finally, using the fact that V ⇤  OPT( T ) / T by Lemma 3.1 and applying linearity of expectations over all rounds t , the following result follows directly:

Original: Theorem 4.5.
Revised: Theorem 4.5.

Original: For any instance of k -RB, where OPT( T ) is the optimal expected payoff that can be collected in T rounds, the total expected payoff collected by Algorithm 1 is at least
Revised: For any instance of k -RB, where OPT( T ) is the optimal expected payoff that can be collected in T rounds, the total expected payoff collected by Algorithm 1 is at least

Original: [Equation]
Revised: [Equation]

Original: We remark that, although the asymptotic approximation guarantee of Algorithm 1 is the same as the one in [SLZZ21], the constants of our algorithm are significantly better.
Revised: We remark that, although the asymptotic approximation guarantee of Algorithm 1 is the same as the one in [SLZZ21], the constants of our algorithm are signiﬁcantly better.

Original: In particular, for k ∈ { 1 , 2 , 3 , 4 , 5 , 10 } , the guarantees of [SLZZ21] and Algorithm 1 are { 0 . 25 , 0 .
Revised: In particular, for k 2 { 1 , 2 , 3 , 4 , 5 , 10 } , the guarantees of [SLZZ21] and Algorithm 1 are { 0 . 25 , 0 .

Original: 57 } and { 0 . 63 , 0 .
Revised: 57 } and { 0 . 63 , 0 .

Original: 82 , 0 . 87 } , respectively.
Revised: 82 , 0 . 87 } , respectively.

Original:  Online Learning with Sublinear Regret
Revised:  Online Learning with Sublinear Regret

Original: In this section, we consider the learning setting of k -RB where the payoff functions are initially unknown.
Revised: In this section, we consider the learning setting of k -RB where the payoff functions are initially unknown.

Original: At each round, the player plays a subset of at most k arms and observes the realized payoff of each individual arm played (semi-bandit feedback).
Revised: At each round, the player plays a subset of at most k arms and observes the realized payoff of each individual arm played (semi-bandit feedback).

Original: Every time an arm i ∈ A is played under delay τ , the realized payoff is drawn independently from a distribution of mean p i ( τ ) , bounded in [0 , 1] .
Revised: Every time an arm i 2 A is played under delay ⌧ , the realized payoff is drawn independently from a distribution of mean p i ( ⌧ ) , bounded in [0 , 1] .

Original: For this setting, our goal is to show that there exists a bandit variant of Algorithm 1 with sublinear γ k -approximate regret guarantee for k -RB, defined as
Revised: For this setting, our goal is to show that there exists a bandit variant of Algorithm 1 with sublinear  k -approximate regret guarantee for k -RB, deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where γ k = 1 − k k / e k k ! is the (multiplicative) approximation guarantee of Algorithm 1 and R ( T ) is the total expected payoff collected by the bandit algorithm in T rounds.
Revised: where  k = 1  k k / e k k ! is the (multiplicative) approximation guarantee of Algorithm 1 and R ( T ) is the total expected payoff collected by the bandit algorithm in T rounds.

Original: Robustness of Algorithm 1.
Revised: Robustness of Algorithm 1.

Original: As a first step, we study the robustness of Algorithm 1 under small perturbations of the payoff functions; in particular, we analyze its approximation guarantee when it runs using estimates { (cid:98) p i } i ∈A instead of the actual payoffs { p i } i ∈A .
Revised: As a ﬁrst step, we study the robustness of Algorithm 1 under small perturbations of the payoff functions; in particular, we analyze its approximation guarantee when it runs using estimates { b p i } i 2 A instead of the actual payoffs { p i } i 2 A .

Original: We remark that the monotonicity of the (estimated) payoff functions is not required for the correctness of Algorithm 1, but only for proving its approximation guarantee.
Revised: We remark that the monotonicity of the (estimated) payoff functions is not required for the correctness of Algorithm 1, but only for proving its approximation guarantee.

Original: Lemma 5.1.
Revised: Lemma 5.1.

Original: Let (cid:98) p i be an estimate of the payoff function p i for each i ∈ A , such that | (cid:98) p
Revised: Let b p i be an estimate of the payoff function p i for each i 2 A , such that | [Equation] ✏

Original: i ( τ ) − p
Revised: [Equation]

Original: i ( τ ) | ≤ ϵ, ∀ τ ∈ N for some ϵ ∈ (0 , 1) .
Revised: 8 ⌧ 2 N for some ✏ 2 (0 , 1) .

Original: The expected payoff collected in T rounds by Algorithm 1, when it uses the estimates instead of the actual payoff functions, is lower bounded by
Revised: The expected payoff collected in T rounds by Algorithm 1, when it uses the estimates instead of the actual payoff functions, is lower bounded by

Original: [Equation]
Revised: [Equation]

Original: Estimation of the payoff functions.
Revised: Estimation of the payoff functions.

Original: By using standard concentration results [Hoe63], we bound the number of samples required for each arm-delay pair in order to get an ϵ -estimate (cid:98) p i for the mean payoff function p i of every arm i ∈ A .
Revised: By using standard concentration results [Hoe63], we bound the number of samples required for each arm-delay pair in order to get an ✏ -estimate b p i for the mean payoff function p i of every arm i 2 A .

Original: Lemma 5.2.
Revised: Lemma 5.2.

Original: For any ϵ, δ ∈ (0 , 1) , let (cid:98) p
Revised: For any ✏ ,  2 (0 , 1) , let b p

Original: i ( τ ) be the empirical average of samples drawn from p i ( τ ) for every arm i ∈
Revised: i ( ⌧ ) be the empirical average of samples drawn from p i ( ⌧ ) for every arm i 2 A and delay ⌧ , using m = 12 ✏ 2 ln [Equation]

Original: A and delay τ , using m = 12 ϵ 2 ln (cid:0) 2 τ max n δ (cid:1) samples, where n is the number of arms.
Revised:  samples, where n is the number of arms.

Original: Then, with probability 1 − δ , it holds |(cid:98) p
Revised: Then, with probability 1   , it holds |

Original:  i ( τ ) − p
Revised: [Equation] )  p i ( ⌧ ) |  ✏

Original: i ( τ ) | ≤ ϵ for every i ∈ A and τ ∈ [ τ max ] .
Revised: for every i 2 A and ⌧ 2 [ ⌧ max ] .

Original: The following observation provides an upper bound on the number of rounds required in order to get m samples for each arm-delay pair in any k -RB instance:
Revised: The following observation provides an upper bound on the number of rounds required in order to get m samples for each arm-delay pair in any k -RB instance:

Original: Fact 5.3.
Revised: Fact 5.3.

Original: In any k -RB instance, the number of rounds required to collect m independent samples from p i ( τ ) , for each arm i ∈ A and delay τ ∈ [ τ max ] , is upper-bounded by O ( nm ( τ max ) 2 / k ) .
Revised: In any k -RB instance, the number of rounds required to collect m independent samples from p i ( ⌧ ) , for each arm i 2 A and delay ⌧ 2 [ ⌧ max ] , is upper-bounded by O ( nm ( ⌧ max ) 2 / k ) .

Original: Bandit algorithm with sublinear regret.
Revised: Bandit algorithm with sublinear regret.

Original: By combining the above elements, it is easy to design an
Revised: By combining the above elements, it is easy to design an

Original: Explore-then-Commit (ETC) variant of Algorithm 1, achieving sublinear regret in the bandit setting.
Revised: Explore-then-Commit (ETC) variant of Algorithm 1, achieving sublinear regret in the bandit setting.

Original: For simplicity, we assume here that the time horizon T is known to the player – an assumption that can be relaxed using the doubling trick method [LS20].
Revised: For simplicity, we assume here that the time horizon T is known to the player – an assumption that can be relaxed using the doubling trick method [LS20].

Original: Let m = 1 / 2 ϵ 2 ln ( 2 τ max n / δ ) for some ϵ, δ ∈ (0 , 1) .
Revised: Let m = 1 / 2 ✏ 2 ln ( 2 ⌧ max n /  ) for some ✏ ,  2 (0 , 1) .

Original: In the first O ( nm ( τ max ) 2 / k ) rounds, the bandit algorithm collects m independent samples from p i ( τ ) , for each i ∈ A and τ ∈ [ τ max ] .
Revised: In the ﬁrst O ( nm ( ⌧ max ) 2 / k ) rounds, the bandit algorithm collects m independent samples from p i ( ⌧ ) , for each i 2 A and ⌧ 2 [ ⌧ max ] .

Original: Then, by taking the empirical average of these samples, it constructs estimates (cid:98) p i ( τ ) for each i and τ .
Revised: Then, by taking the empirical average of these samples, it constructs estimates b p i ( ⌧ ) for each i and ⌧ .

Original: For the rest of the time horizon, the algorithm simulates Algorithm 1, using the empirical estimates (cid:98) p i ( τ ) in place of the (initially unknown) payoff functions.
Revised: For the rest of the time horizon, the algorithm simulates Algorithm 1, using the empirical estimates b p i ( ⌧ ) in place of the (initially unknown) payoff functions.

Original: By setting the parameters ϵ, δ accordingly and combining Lemmas 5.1 and 5.2 with Fact 5.3, we prove the following result:
Revised: By setting the parameters ✏ ,  accordingly and combining Lemmas 5.1 and 5.2 with Fact 5.3, we prove the following result:

Original: Theorem 5.4.
Revised: Theorem 5.4.

Original: There exists a learning adaptation of Algorithm 1 in the semi-bandit setting, for which the γ k -approximate regret for T rounds can be upper-bounded as
Revised: There exists a learning adaptation of Algorithm 1 in the semi-bandit setting, for which the  k -approximate regret for T rounds can be upper-bounded as

Original: [Equation]
Revised: [Equation]

Original: Conclusion and Further Directions
Revised: Conclusion and Further Directions

Original: Following the recent line of work on non-stationary bandits with recharging payoffs, we revisited one of the most general formulations studied in the area and significantly improved the state-of-the-art for the planning setting.
Revised: Following the recent line of work on non-stationary bandits with recharging payoffs, we revisited one of the most general formulations studied in the area and signiﬁcantly improved the state-of-the-art for the planning setting.

Original: In particular, when at most one arm is played per round, we designed an algorithm that collects at least 63% (asymptotically and in expectation) of the optimal payoff, improving the best-known 25% , due to prior work.
Revised: In particular, when at most one arm is played per round, we designed an algorithm that collects at least 63% (asymptotically and in expectation) of the optimal payoff, improving the best-known 25% , due to prior work.

Original: By providing results on the robustness and sample complexity of our algorithm, we transformed the latter into a bandit algorithm with sublinear regret.
Revised: By providing results on the robustness and sample complexity of our algorithm, we transformed the latter into a bandit algorithm with sublinear regret.

Original: Our work leaves a number of interesting open questions.
Revised: Our work leaves a number of interesting open questions.

Original: An approximation guarantee of (1 − 1 / e ) has been proved to be the best achievable – under standard complexity assumptions – for a number combinatorial problems.
Revised: An approximation guarantee of (1  1 / e ) has been proved to be the best achievable – under standard complexity assumptions – for a number combinatorial problems.

Original: Therefore, an immediate future direction would be to either provide a matching hardness result for the planning setting of the problem, or to further improve on the best-known approximation.
Revised: Therefore, an immediate future direction would be to either provide a matching hardness result for the planning setting of the problem, or to further improve on the best-known approximation.

Original: Another interesting question would be to explore whether one can design a bandit adaptation of our algorithm with improved regret guarantees.
Revised: Another interesting question would be to explore whether one can design a bandit adaptation of our algorithm with improved regret guarantees.

Original: Finally, the empirical evaluation of our algorithm on real (or artificial) data is another interesting direction.
Revised: Finally, the empirical evaluation of our algorithm on real (or artiﬁcial) data is another interesting direction.

Original: Acknowledgements
Revised: Acknowledgements

Original: This work was partially funded by the NSF IFML Institute (NSF 2019844), the NSF AI-EDGEInstitute (NSF 2112471), and NSF 1704778. A Related Work
Revised: This work was partially funded by the NSF IFML Institute (NSF 2019844), the NSF AI-EDGE

Original: Since the work of Thompson [Tho33] and later [LR + 85], the (stochastic) multi-armed bandits (MAB) problem in stationary environments has been thoroughly studied (see [BCB12, LS20, ACBF02] and references therein).

Original: In an attempt to capture non-stationary environments, where the payoff distributions change over time, a number of generalizations of MAB – which are also special cases of reinforcement learning (RL) – have been proposed.

Original: Two particularly interesting classes are rested and restless bandits [TL12], where each action is associated with a state-machine and has a different payoff distribution depending on its current state.

Original: In rested bandits [Git79, TL12], the state of each action (hence, its payoff distribution) changes stochastically only when the arm is played, while in restless bandits [Whi88, TL12] the state changes at every round, independently of the player’s actions.

Original: Optimal algorithms for rested and restless bandits essentially result from solving Bellman’s equations [Ber00] – an approach which requires an exponentially large state space in the number of arms.

Original: In fact, Papadimitriou and Tsitsiklis [PT99] show that it is PSPACE-hard to optimally solve (or even approximate) restless bandits, even in the simpler case where state transitions are deterministic.

Original: One of the first attempts to provide constant approximation algorithms for special cases of restless bandits is due to Guha et al.

Original: [GMS10], who give a 2 -approximation for a special case of restless bandits, which they call “monotone” bandits.

Original: Another interesting class of non-stationary bandits is that of sleeping bandits [KNMS10, SGV20], which generalizes the standard MAB setting, given that only an adversarially chosen subset of actions can be played at each round.

Original: Finally, we address the reader to [BGZ14, LCM17, KZ17, AGO19] for additional non-stationary bandit models.

Original: Shifting our focus to the case of “recharging” (or “recovering”) payoffs, Immorlica and Kleinberg [IK18] first consider the 1 -RB problem, under the additional assumption that the payoff functions are weakly concave and weakly increasing over the whole time horizon.

Original: For this problem, the authors provide a PTAS that achieves a (1 − ϵ ) -approximation (asymptotically) to the optimal planning, and show how it can be adapted into a learning algorithm with sublinear regret guarantees.

Original: To achieve their approximation, they first compute an “optimal” playing frequency for each arm via a concave relaxation, and then construct an approximate planning schedule, in which the rate of playing each arm closely approximates this frequency.

Original: In this direction, they authors combine enumeration techniques with the novel technique of interleaved rounding .

Original: The main idea behind the latter is that each arm is first associated with a particular (randomly perturbed) sequence of real numbers and, then, the arms are played sequentially according to the order they appear on the real line.

Original: Basu et al.

Original: [BSSS19] study a variant of stochastic multi-armed bandits where, after each play, an arm becomes unavailable for a known number of rounds (blocking time).

Original: As they show, for the case of one arm per round, a simple greedy heuristic, which plays, at each round, the arm of highest expected payoff among the non-blocked arms, yields (asymptotically) a (1 − 1 / e ) -approximation.

Original: In order to prove this guarantee, the authors compare the payoff collected by their algorithm with the optimal solution of an LP-relaxation, by carefully characterizing the latter via analyzing its optimality conditions.

Original: Although not stated explicitly in [BSSS19], the “hard” constraint of not pulling a blocked arm is equivalent to choosing the payoff function of each arm to be a Heaviside step function parameterized by its blocking time and scaled by its baseline mean payoff.

Original: In other words, the payoff drops to zero for a fixed number of rounds (which, in this case coincides with the recovery time), and then rises up to a baseline.

Original: Since pulling a zero-payoff arm can only harm the overall solution, the two problems are (without loss of generality) equivalent, and [BSSS19] becomes a special case of 1 -RB.

Original: Notice, however, that the greedy approach of [BSSS19] fails to provide any non-trivial approximation guarantee for the 1 -RB setting.

Original: Shortly after its introduction, the model of Basu et al.

Original: [BSSS19] has been also studied in contextual [BPCS21], adversarial [BCMTT20], and combinatorial environments [APB + 21, PC21].

Original: In [PC21],

Original: Papadigenopoulos and Caramanis study the blocking setting in the regime where more than one arms can be played at each round, subject to matroid constraints.

Original: For the planning setting, they provide a (1 − 1 / e ) -approximation algorithm (asymptotically and in expectation) for any matroid.

Original: [-Their technique, called interleaved scheduling – which is similar in spirit with the interleaved rounding method of [IK18] – is essentially a time-correlated arm-sampling scheme, which provides the following two guarantees: (i) at any fixed round, each arm i is sampled with probability 
-]
Revised: [-Their technique, called interleaved scheduling – which is similar in spirit with the interleaved rounding method of [IK18] – is essentially a time-correlated arm-sampling scheme, which provides the following two guarantees: (i) at any fixed round, each arm i is sampled with probability 
-]

Original: / τ i (the maximum possible in any optimal solution), where τ i is the blocking time of the arm.

Original: (ii)

Original: Every time an arm is sampled, its payoff must have returned to its baseline (i.e., is never blocked).

Original: Notice that the algorithm of [PC21], which essentially chooses a maximum-payoff independent set among the sampled arms of each round, also provides an alternative method for reaching the (1 − 1 / e ) -approximation of [BSSS19] for the special case of one arm per round.

Original: Cella and Cesa-Bianchi [CCB20] consider a similar model to 1 -RB, where they assume that all the actions have identical payoff functions up to different scaling (i.e., baseline payoff), but different and unknown recovery times.

Original: For the planning problem, by focusing on a simple class of periodic ranking policies , the authors develop an algorithm, accompanied by a worst-case approximation guarantee that depends on characteristics of the input (payoff functions, recovery times, and baselines).

Original: Finally, other special cases and variations of the k -RB problem include payoff functions described by linear dynamics [LKKLM21], sampled by Gaussian processes [PBG19], and more [LCCBGB22].

Original: Simchi-Levi et al.

Original: [SLZZ21] provide the first O (1) -approximation the general case of the k -RB problem.

Original: Specifically, by focusing on a particular class of policies, which they call purely periodic , the authors provide a long-run (1 − O ( 1 / √ k )) -approximation for any instance of k -RB.

Original: A key-element in both Immorlica and Kleinberg [IK18] and, subsequently, Simchi-Levi et al.

Original: [SLZZ21] is the construction of a concave relaxation of the optimal solution, based on the critical notion of rate of return : a piecewise linear function, which denotes the maximum payoff that can be achieved asymptotically by playing an arm at most a specific fraction of rounds.

Original: The authors in [IK18] provide an FPTAS for computing an (1 − ϵ ) -approximate solution to their relaxation (for the case where k = 1 ), while, in [SLZZ21], such a solution is efficiently computed assuming finite recovery times.

Original: B Structural Properties of a Natural LP Relaxation: Omitted Proofs

Original: Lemma 3.1.

Original: For any instance of the k -RB problem, let V ∗ be the optimal value of ( LP k ) and OPT( T ) be the maximum payoff that can be collected in a time horizon of T rounds.

Original: Then T · V ∗ ≥ OPT( T ) .

Original: Proof.

Original: Let n i,τ be the number of times arm i is played under delay τ ∈ N in an optimal solution.

Original: By fixing x i,τ = n i,τ T , we have that

Original: [Equation]

Original: Further, note that in any instance where at most k arms can be played at each round, the total number of arm plays (independently of the delay) cannot be more than k · T .

Original: Thus, we get

Original: [Equation]

Original: Finally, consider any action i ∈

Original: A which is played at time t under delay τ .

Original: The number of rounds that are occupied for the above specific play belong to the interval I = { t − τ + 1 , t } , with | I | =

Original: τ (note that for τ = 1 only round t is occupied).

Original: Recalling our assumption that all actions start with delay 1 at round t = 1 , then for any such interval we have that I ⊆

Original: [ T ] .

Original: Combining the above with the fact that, for any action i ∈ A , these intervals cannot overlap, we get that

Original: [Equation]

Original: Therefore, since x is a feasible solution to ( LP k ) with value equal to OPT( T ) / T , the proof follows directly.

Original: Fact 3.2.

Original: There exists an optimal solution to ( LP k ) which is supported in ( x i,τ ) i ∈A ,τ ∈ [ τ max i ] .

Original: Further, given an upper bound τ max on the maximum recovery time of any action, an optimal extreme point solution to ( LP k ) can be computed in polynomial time.

Original: Proof.

Original: Clearly, proving that any optimal solution to ( LP k ) is without loss of generality supported in ( x i,τ ) i ∈A ,τ ∈ [ τ max i ] implies that we can restrict ourselves to a polynomial-size version of ( LP k ), which is constructed by dropping all variables except for ( x i,τ ) i ∈A ,τ ∈ [ τ max i ] .

Original: In this way, an optimal extreme point solution to this restricted LP can be computed efficiently using a standard LP solver.

Original: Let x be any optimal solution to ( LP k ) such that there exists an arm i and delay τ ′ >

Original: τ max i satisfying x i,τ ′ > 0 .

Original: Note that by definition of the recovery time, it must hold p i ( τ ′ )

Original: =

Original: p i ( τ max i ) .

Original: Thus, we can construct a solution x ′ which is identical to x , except for the fact that x ′ i,τ ′ = 0 and x ′ i,τ max i =

Original: x i,

Original: [Equation]

Original: + x i,τ ′ .

Original: Since p i ( τ ′ )

Original: =

Original: p i ( τ max i ) the solutions x ′ and x have exactly the same value, while x ′ trivially satisfies constraints (C.1).

Original: In addition, constraints (C.2) are still satisfied by x ′ for any arm different than i .

Original: For arm i , we have

Original: [Equation]

Original: where the first inequality follows by the fact that τ ′ >

Original: τ max i and the second by feasibility of x .

Original: By repeating the above process for every arm i and delay τ > τ max i with x i,τ ′ > 0 , we can transform any solution to ( LP k ) to an equivalent solution of the desired form.

Original: Lemma 3.5 (Sparsity pattern of extreme point solutions) .

Original: Let x be any extreme point solution of ( LP k ) .

Original: Then, x is almost-delay-feasible.

Original: Proof.

Original: Let A ( x ) ⊆

Original: A be the set of supported actions in an extreme point solution x of ( LP k ), and let | A ( x ) | = n ′ .

Original: Let LP ′ be the LP that results by dropping from ( LP k ) all the variables and constraints that correspond to the non-supported actions in A \ A ( x ) .

Original: Then, the solution x ′ =

Original: ( x i,τ ) i ∈A ( x ) ,τ ∈ N (i.e., the solution x restricted to the arms of A ( x ) ) is an extreme point solution of this reduced LP.

Original: In order to see that, let us assume that x ′ is not an extreme point and, hence, x ′ =

Original: λ x ′ 1 +(1 − λ )

Original: x ′for some λ ∈ (0 , 1) and x ′ 1 , x ′ 2 feasible solutions to LP ′ .

Original: Then, by padding x ′ 1 and x ′ 2 with zero variables for any i ∈

Original: A \ A ( x ) and τ ∈ N , the resulting vectors, let x 1 and x 2 , respectively, are feasible solutions to ( LP k ).

Original: However, it also holds that x = λ x 1 +(1 − λ ) x 2 , contradicting the fact that x is an extreme point of ( LP k ).

Original: Now let us focus on the reduced formulation LP ′ .

Original: By construction, since the actions in A ( x ′ ) are all supported in x ′ , it has to be that ∥ x ′ ∥ 0 ≥ n ′ .

Original: Let u be the number of variables of LP ′ .

Original: By counting constraint (C.1), constraints (C.2) (one for each

Original: i ∈

Original: A ( x ′ ) ), and u non-negativity constraints, we can see that the number of constraints in LP ′ is u + n ′ + 1 .

Original: Hence, since x ′ is an extreme point solution of LP ′ , it must contain at most n ′ + 1 non-zero variables (since at least u − n ′ − 1 of the inequalities which are tight in x ′ must correspond to non-negativity constraints).

Original: This gives ∥ x ′ ∥ 0 ≤ n ′ + 1 and, thus, we can conclude that ∥ x ′ ∥ 0 ∈ { n ′ , n ′ + 1 } .

Original: We now distinguish between two cases:

Original: (a) If ∥ x ′ ∥ 0 = n ′ , then

Original: since

Original: every arm

Original: in the solution

Original: x ′ is supported and A ( x ′ = n ′
Revised: Institute (NSF 2112471), and NSF 1704778.

Original: , by pigeonhole principle each arm i ∈

Original: A ( x ′ ) has to be supported by a unique non-zero variable x

Original: ′ i,τ i > 0 for some

Original: τ

Original: i

Original: ∈

Original: N .

Original: Further, since exactly n ′ out of the u non-negativity constraints are not tight, then n ′ out of the n ′ + 1 inequalities in (C.1) and (C.2) have to be met with equality.

Original: (i)

Original: In the case where these n ′ constraints come from (C.2), then for each i ∈

Original: A ( x ′ ) , there exists a unique delay τ i and variable x

Original: ′ i,τ i , such that x

Original: ′ i,τ i = 1 / τ i , implying that x ′ and, thus, x is delay-feasible.

Original: (ii)

Original: In the case where only n ′ − 1 from the tight constraints in (C.1) and (C.2) come from (C.2), then for each corresponding arm i ∈

Original: A ( x ′ ) , there exists a unique delay τ i and variable x

Original: ′ i,τ i , such that x

Original: ′ i,τ i = 1 / τ i .

Original: Therefore, the arm ι ∈ A ( x ′ ) which is associated with the non-tight constraint in (C.2) has to be an arm which is supported by a single variable 0 < x ι,τ ι < 1 / τ ι .

Original: In this case, x ′ and, thus, x is almost-delay-feasible, and ι corresponds to the irregular arm.

Original: [-(b) In the case where ∥ x ′ ∥ 
-]
Revised: [-(b) In the case where ∥ x ′ ∥ 
-]

Original: = n ′ + 1 , by a similar argument as above, each arm

Original: i ∈

Original: A ( x ′ ) has to be supported by a unique non-zero variable x

Original: ′ i,τ i > 0 for some τ i ∈ N , with the exception of a single arm ι ∈ A ( x ′ ) , which is supported by two non-zero variables x ι,τ ι,a , x ι,τ ι,b > 0 .

Original: Further, since ∥ x ′ ∥ 0 = n ′ + 1 , it has to be that all the constraints in (C.1) and (C.2) are met with equality.

Original: Therefore, for any arm i ∈

Original: A ( x ′ ) \ { ι } , there exists a unique delay τ i and variable x

Original: ′ i,τ i , such that x

Original: ′ i,τ i = 1 / τ i .

Original: Again, the solution x of ( LP k ) in this case is almost-delay-feasible, and ι corresponds to the irregular arm.

Original: Since delay-feasible solutions subsume almost-delay-feasible by definition, it follows that, in any case, x is almost-delay-feasible.

Original: C Improved Approximation Guarantees for Planning: Omitted Proofs

Original: Lemma 4.2.

Original: For any vectors w ∈ [0 , ∞ ) n and y ∈ [0 , 1] n , let f + w ,k

Original: ( y ) be the concave closure of the weighted rank function of the rank- k uniform matroid with weights w , evaluated at y .

Original: Then, if y satisfies ∥ y ∥ 1 ≤ k , it is the case that f + w ,k ( y ) ≥ (cid:80) i ∈

Original: [ n ] w i

Original: y i .

Original: Proof.

Original: Let us consider the intersection of the n -dimensional hypercube [0 , 1] n with the halfspace

Original: H = { z ∈ R n | (cid:80) i ∈ [ n ] z i ≤ k } .

Original: It is easy to verify that every vertex v of the resulting polytope lies in { 0 , 1 } n and satisfies ∥ v ∥ 0 ≤ k .

Original: Then, since y ∈ [0 , 1] n ∩ H , by standard arguments in polyhedral combinatorics, y can be expressed as a convex combination of characteristic vectors of vertices of [0 , 1] n ∩ H , each corresponding to a subset of [ n ] of cardinality at most k .

Original: This convex combination induces a probability distribution D ′

Original: ( y ) over subsets of cardinality at most k with marginals y .

Original: Therefore, by definition of f + w ,k , we have

Original: [Equation]

Original: where the last equality follows by the fact that D ′ ( y ) is only supported by sets of cardinality at most k .

Original: Finally, using the fact that D ′ ( y ) respects the marginals y , we can conclude

Original: [Equation]

Original: Lemma 4.3.

Original: For any arm i ∈

Original: A ′ and at any round t , we have that P [ i ∈ C t and τ ∗ i = τ ] = x ∗ i,τ .

Original: Proof.

Original: Fix any arm

Original: i ∈

Original: A ′ and round t .

Original: By the sampling process of critical delays using the solution x ∗ , we have that P [ τ ∗ i = τ ] =

Original: τ · x ∗ i,τ .

Original: Notice that, when arm i is not the irregular arm, by

Original: Definition 3.4 this probability is identically one.

Original: Since the offset r i is sampled uniformly at random from { 0 , 1 , . . .

Original: , τ ∗ i − 1 } and i ∈

Original: C t

Original: if and only if t mod τ ∗ i =

Original: r i , we get that P [ i ∈ C t | τ ∗ i = τ ] = 1 / τ .

Original: Hence, we can conclude that

Original: [Equation]

Original: Lemma 4.4.

Original: Let f : 2 [ n ] → R be a submodular function over a ground set of n elements, where n ≥ 2 .

Original: Let I ( y ) be a distribution over 2 [ n ] , where each element i is added in set S ∼ I ( y ) , independently, with probability

Original: y i .

Original: Further, let D ( y ) be a distribution over 2 [ n ] , where each element i ∈ [ n ] \ { ι a , ι b } is added to a set S ∼ D ( y ) , independently, with probability y i , except for two distinct elements ι a , ι b , whose addition to S is mutually exclusive but with the same marginals y ι a and y ι b , respectively.

Original: In this case, we have

Original: [Equation]

Original: Proof.

Original: By definition of the distribution D ( y ) , since the events { ι a ∈ S } and { ι b ∈ S } are mutually exclusive, we have that

Original: [Equation]

Original: and

Original: [Equation]

Original: Further

Original: [Equation]

Original: we have that

Original: [Equation]

Original: Using the above

Original: [Equation]

Original: we can rewrite the LHS of the desired inequality as

Original: [Equation]

Original: Now

Original: [Equation]

Original: for any fixed set S ′ ⊆

Original: [ n ] \ { ι a , ι b }

Original: [Equation]

Original: the following identities can be easily verified:

Original: [Equation]

Original: By combining the above identities

Original: [Equation]

Original: we get that

Original: [Equation]

Original: S ′ + ι b ) + y ι a y ι b f ( S ′ + ι a + ι b ) , where, in the inequality, we use that f ( S ′ )

Original: + f ( S ′ + ι a + ι b ) ≤ f ( S ′ + ι a ) + f ( S ′ + ι b ) , by submodularity of f .

Original: By applying the above inequality for each S ′ ⊆

Original: [ n ] \ { ι a , ι b } , we can conclude that

Original: [Equation]

Original: thus proving the lemma.

Original: D Online Learning with Sublinear Regret: Omitted Proofs

Original: Lemma 5.1.

Original: Let (cid:98) p i be an estimate of the payoff function p i for each i ∈ A , such that | (cid:98) p

Original: i ( τ ) − p

Original: i ( τ ) | ≤ ϵ, ∀ τ ∈ N for some ϵ ∈ (0 , 1) .

Original: The expected payoff collected in T rounds by Algorithm 1, when it uses the estimates instead of the actual payoff functions, is lower bounded by

Original: [Equation]

Original: Proof.

Original: Let x ∗ and (cid:98) x be the optimal solution of ( LP k ) in the case where the payoff functions in the objective are given by { p i } i ∈A and { (cid:98) p i } i ∈A , respectively.

Original: Let also V ∗ and (cid:98) V be the optimal value of ( LP k ) in each of the aforementioned cases.

Original: We have

Original: [Equation]

Original: where in the first inequality we use the fact that (cid:98) x is an optimal extreme point solution of ( LP k ), when the estimates are used in the objective.

Original: In the second inequality, we use that | (cid:98) p i ( τ ) − p

Original: i ( τ ) | ≤ ϵ, ∀ i ∈

Original: A , τ ∈ N , while the third follows by constraint (C.1) of ( LP k ).

Original: We observe that Algorithm 1 can produce a feasible planning using any given extreme point solution of ( LP k ) (not necessarily the optimal).

Original: Let (cid:98)

Original: A t be the set of arms played at some round t ≥ τ max by Algorithm 1 (using the estimates) and let (cid:98) C t be the corresponding set of candidate arms.

Original: By construction of the algorithm, it is the case that

Original: [Equation]

Original: where (cid:98) σ i,t is the actual delay of arm i at time t in a run of Algorithm 1 (using the estimates).

Original: Let (cid:98) τ ∗

Original: i be the (sampled) critical delay of arm i in a run of Algorithm 1 (using the estimates).

Original: By working along the lines of Theorem 4.5, using monotonicity of p i ( τ ) and the fact that (cid:98) σ i,t ≥ (cid:98) τ ∗ i , for the expected payoff collected at any round t ≥ τ max (over the randomness of the payoff realizations), we have

Original: [Equation]

Original: where in the last inequality we use the fact that (cid:98) p i is ϵ -close to p i for each arm

Original: i ∈ A .

Original: Let γ k be the multiplicative approximation ratio of Algorithm 1 for k -RB (given in Theorem 4.5).

Original: By the analysis of Theorem 4.5, it follows that

Original: [Equation]

Original: where the last inequality follows by Lemma 3.1.

Original: Therefore, for any round t ≥ τ max , the expected payoff collected by Algorithm 1 (using the estimates) satisfies

Original: [Equation]

Original: The proof follows by summing the above inequalities for every round t ∈ [ τ max , T ] and using linearity of expectation together with the fact that OPT( T ) / T ≤ k .

Original: Lemma 5.2.

Original: For any ϵ, δ ∈ (0 , 1) , let (cid:98) p

Original: i ( τ ) be the empirical average of samples drawn from p

Original: i ( τ ) for every arm i ∈

Original: A and delay τ , using m = 12 ϵ 2 ln (cid:0) 2 τ max n δ (cid:1) samples, where n is the number of arms.

Original: Then, with probability 1 − δ , it holds | (cid:98) p

Original: i ( τ ) − p

Original: i ( τ ) | ≤ ϵ for every i ∈ A and τ ∈ [ τ max ] .

Original: Proof.

Original: For any arm i ∈ A and delay τ , let (cid:98) p

Original: i ( τ ) be the empirical average of m samples drawn independently from a distribution of mean p i ( τ ) (and bounded in [0 , 1] ).

Original: By Hoeffding inequality [Hoe63], we have that

Original: [Equation]

Original: By setting m = 12 ϵ 2 ln( 2 τ max n δ ) , we get that | (cid:98) p i ( τ ) − p

Original: i ( τ ) | ≤ ϵ with probability 1 − δ τ max n .

Original: Therefore, by a simple union bound over all arms i ∈ A and delays τ ∈ [ τ max ] , we get

Original: P [ ∃ i ∈ A , τ ∈ [ τ max ] : | (cid:98) p i ( τ ) − p i ( τ ) | > ϵ ] ≤ δ, thus, conlcuding the proof.

Original: Fact 5.3.

Original: In any k -RB instance, the number of rounds required to collect m independent samples from p

Original: i ( τ ) , for each arm i ∈ A and delay τ ∈ [ τ max ] , is upper-bounded by O ( nm ( τ max ) 2 / k ) .

Original: Proof.

Original: For any arm i ∈ A and delay τ , we can collect m independent samples from the distribution of mean p i ( τ ) , by playing arm i for m times – once every τ rounds – which can be achieved in

Original: O ( mτ ) time steps.

Original: In order to collect m samples for any delay τ ∈ [ τ max ] for a fixed arm i , we thus need O (cid:16) (cid:80) τ ∈

Original: [ τ max ] mτ (cid:17) =

Original: O (cid:0) m ( τ max ) 2 (cid:1) rounds in the worst case.

Original: Assuming that we can play at most one arm per round, in order to collect m samples for each arm/delay pair, we thus need O ( nm ( τ max ) 2 ) rounds in total.

Original: In a k -RB instance, where we can play at most k < n arms per round, we can parallelize and, thus, expedite the above sampling process by a factor of k , leading to a total of O ( nm ( τ max ) 2 /k ) exploration rounds.

Original: Theorem 5.4.

Original: There exists a learning adaptation of Algorithm 1 in the semi-bandit setting, for which the γ k -approximate regret for T rounds can be upper-bounded as

Original: [Equation]

Original: Proof.

Original: Let R ( T ) be the expected payoff collected by our ETC-based bandit algorithm in T rounds.

Original: Recall that the γ k -approximate regret for T rounds is defined as

Original: [Equation]

Original: For ϵ, δ ∈ (0 , 1) (to be specified later), let m =

Original: 12 ϵ 2 ln ( 2 τ max n / δ ) .

Original: By Fact 5.3, we can collect m samples from the distribution of each arm-delay pair in the first O (cid:16) n ( τ max ) 2 ϵ 2 k

Original: [Equation]

Original: rounds

Original: [Equation]

Original: Since the mean payoffs are all bounded in [0 , 1] and we can play at most k arms per round, the regret accumulated for each of the exploration rounds is at most k , which implies that the total regret due to exploration can be upper bounded by O (cid:16) n ( τ max ) 2 ϵ

Original: [Equation]

Original: Moving on to the exploitation phase, for every arm i ∈ A and delay τ ∈ [ τ max ] , let (cid:98) p

Original: i ( τ ) be the empirical estimate of p

Original: i ( τ ) using m samples collected in the exploration phase.

Original: For the rest

Original: T − O ( n ( τ max ) 2 ϵ 2 k

Original: ln

Original: ( τ max n δ )) ≤ T rounds, by Lemma 5.2, we get that, with probability 1 − δ , it holds | (cid:98) p

Original: i ( τ ) − p

Original: i ( τ ) | ≤ ϵ for every arm i and delay τ .

Original: We refer to the above event as a “nice” sampling.

Original: Thus, the total regret accumulated in the exploitation phase in the case where sampling is not nice (which happens with probability δ ) is at most T · k · δ .

Original: In the case where the sampling is nice, then

Original: Lemma 5.1 suggests that for the remaining T ′ rounds of the exploitation phase, we have

Original: [Equation]

Original: which implies that the total regret accumulated in the exploitation phase can be upper bounded by

Original: O ( k · ϵ · T + k · τ max + T · k · δ ) .

Original: By the above analysis

Original: [Equation]

Original: it follows that for any ϵ, δ ∈ (0 , 1) , the total regret of our bandit algorithm satisfies

Original: [Equation]

Original: By setting ϵ =

Original: O

Original: [Equation]

Original: and δ = 1 T

Original: [Equation]

Original: we can upper bound the regret as

Original: [Equation]

Original: , thus proving that our algorithm achieves a sublinear (in the time horizon) regret guarantee.

Original: Societal Impact and Limitations

Original: This is a theoretical work and, thus, any (positive or negative) societal impact depends on the application.

Original: A possible limitation of our work is the assumption that an upper bound on the recovery times of all arms is known a priori.

Original: Notice, however, this assumption (also made in the state-of-the-art prior work [SLZZ21]) is natural for settings where the payoff functions have a non-parametric form.