When Combinatorial Thompson Sampling meets Approximation Regret
Abstract
We study the Combinatorial Thompson Sampling policy ( CTS ) for combinatorial multi-armed bandit problems (CMAB), within an approximation regret setting.
Although CTS has attracted a lot of interest, it has a drawback that other usual CMAB policies do not have when considering non-exact oracles: for some oracles, CTS has a poor approximation regret (scaling linearly with the time horizon T )
[Wang and Chen, 2018].
A study is then necessary to discriminate the oracles on which CTS could learn.
This study was started by Kong et al.
[2021]: they gave the ﬁrst approximation regret analysis of CTS for the greedy oracle , obtaining an upper bound of order O (cid:0) log( T ) / ∆ 2min (cid:1) , where ∆ min is some minimal reward gap.
In this paper, our objective is to push this study further than the simple case of the greedy oracle.
We provide the ﬁrst O (log( T ) / ∆ min ) approximation regret upper bound for CTS , obtained under a speciﬁc condition on the approximation oracle, allowing a reduction to the exact oracle analysis.
We thus term this condition REDUCE 2 EXACT , and observe that it is satisﬁed in many concrete examples.
Moreover, it can be extended to the probabilistically triggered arms setting, thus capturing even more problems, such as online inﬂuence maximization .
Introduction
Stochastic multi-armed bandits (MAB) Robbins [1952], Berry and Fristedt [1985], Lai and Robbins [1985] are decision-making problems in which an agent acts sequentially in an uncertain environment.
At each round t ∈ N ∗ , the agent must select one arm from a ﬁxed set of n arms, denoted by [ n ] (cid:44) { 1 , . . . , n } , using a policy , based on the feedback from the previous rounds.
Then it gets as feedback an outcome X i t ∈ R — a random variable sampled from P X i , independently from previous rounds — where i is the selected arm and P X i is a probability distribution — unknown to the agent — of mean µ ∗ i .
The goal for the agent is to maximize the cumulative reward over a total of
T rounds ( T is the time horizon and may be unknown).
The performance metric of a policy is its regret R T , which is the expectation of the difference over T rounds between the cumulative reward of the policy that always picked the arm with the highest expected reward and the cumulative reward of the learning policy.
MAB models the so called dilemma between exploration and exploitation, i.e., whether to continue exploring arms to obtain more information (and thus strengthen the conﬁdence in the estimates of the distributions P X i ), or to use the information gathered by playing the best arm according to the observations so far.
In this paper, we study stochastic combinatorial multi-armed bandit (CMAB), with semi-bandit feedback , a.k.a.
stochastic semi
-bandit (still abbreviated as CMAB in this paper), an extension of MAB where the agent plays an action (also called super-arm )
A t ∈ A ⊂ P ([ n ]) at each round t , where A is ﬁxed and called action space .
The feedback includes the outcomes of all base arms in the played super-arm.
The expected reward, given A t , is assumed to be in the form 2 r
( A t , µ ∗ ) , where µ ∗ ∈ R n is the unknown vector of expectations (traditionally, the reward is linear and equal to e T A t µ ∗ ).
In recent years, CMAB has attracted a lot of interest (see e.g. Cesa-Bianchi and Lugosi [2012], Gai et al.
[2012], Chen et al.
[2013, 2016], Kveton et al.
[2015], Wang and Chen [2017], Perrault [2020]), particularly due to its wide applications in network routing, online advertising, recommender system, inﬂuence marketing, etc.
Many CMAB policies are based on the Upper Conﬁdence Bound (UCB) approach, extending the classical UCB policy [Auer et al., 2002] from MAB to CMAB.
This type of approach uses an optimistic estimate µ t of µ ∗ (i.e., for which the reward function is overestimated), lying in a wellchosen conﬁdence region.
Then, the action is chosen by plugging µ t inside an oracle (typically,
Oracle( µ ∗ ) is a maximizer of the reward function A 7→ r ( A, µ ∗ ) ).
An example of such policy is Combinatorial Upper Conﬁdence Bound ( CUCB ) [Chen et al., 2013, Kveton et al., 2015], that uses a Cartesian product of the individual conﬁdence intervals of each arm as a conﬁdence region.
For mutually independent arms, Combes et al.
[2015] provided the UCB-style policy Efﬁcient
Sampling for Combinatorial Bandit ( ESCB ), building a tighter axis-aligned ellipsoidal conﬁdence region around the empirical mean, which helps to better restrict the exploration.
Degenne and Perchet [2016] provided a policy called OLS - UCB , leveraging a sub-Gaussianity assumption on the arms to generalize the ESCB approach.
These policies have been further extended to more general settings afterwards [Perrault et al., 2020c,d, 2019a].
Although improving CUCB , all these generalizations are inefﬁcient in terms of computation time.
Another paradigm that has recently gained interest (and which will be our focus in this paper) is to rely on Thompson Sampling ( TS ) instead of UCB , still targetting frequentist regret .
Although introduced much earlier by Thompson [1933], the theoretical analysis of TS for MAB is quite recent:
Kaufmann et al.
[2012], Agrawal and Goyal [2012] gave a regret bound matching the UCB policy theoretically.
Moreover, TS often performs better than UCB in practice, making TS an attractive policy for further investigations.
For CMAB, TS extends to Combinatorial Thompson Sampling ( CTS ).
In CTS , the unknown mean µ ∗ is associated with a belief (a prior distribution, that could be e.g. a product of Beta or Gaussian distributions) updated to a posterior with the Bayes’rule, each time a feedback is received.
In order to choose an action at round t , CTS draws a sample θ t from the current belief, and plays the action given by Oracle( θ t ) .
CTS is an attractive policy because it has similar advantages to the previously mentioned policies working with ellipsoidal conﬁdence regions while being, like CUCB , computationally efﬁcient.
Indeed, recently, for mutually independent arms and sub-Gaussian arms respectively, Wang and Chen [2018], Perrault et al.
[2020a] proposed tight analyses of CTS .
Unlike UCB-based policies, the analysis of CTS is valid only when Oracle is exact, i.e., when
[Equation]
Although this holds true for many combinatorial problem described by the pair ( r, A ) (we recall that r is usually linear), there exist some problems where the requirement on Oracle has to be relaxed in order to make it tractable.
This is usually done considering an α - approximation oracle [Chen et al., 2013, 2016, Wen et al., 2016], for α ∈ (0 , 1) :
[Equation]
Under an α -approximation oracle, the benchmark cumulative reward is the α -fraction of the optimal reward, leading to the notion of approximation regret [Kakade et al., 2009, Streeter and Golovin, 2009, Chen et al., 2016].
Aware of the limitation of their CTS analysis (that works only with exact oracles), Wang and Chen [2018] also proved, in their Theorem 2, that this limitation is not a technical artifact.
More precisely, they provided a speciﬁc CMAB instance with an associated approximation oracle, such that CTS on this instance and with this oracle must have a regret scaling linearly in T .
Although this negative result is of great interest to the research community, some concerns limit its consideration.
Indeed, not only the CMAB instance provided by Wang and Chen [2018] is actually a MAB one (meaning that there is an efﬁcient oracle that simply enumerates the arms), so the use of an approximation regret is not justiﬁed, but above all, both the oracle and the instance are uncommon and designed for the proof.
Interested in the question of whether the example provided by Wang and Chen [2018] is pathological or generalizable, Kong et al.
[2021] recently initiated a study, which revealed that linear approximation regret for CTS seems to be pathological.
More precisely, they derived a O (cid:0) log( T ) / ∆ 2min (cid:1) bound for the speciﬁc case of a greedy oracle 3 , where ∆ min is some reward gap.
This result is obtained by bounding the approximation regret by a greedy regret , that simply replaces α max A ∈A r ( A, µ ∗ ) with r (Oracle( µ ∗ ) , µ ∗ ) , using equation (1).
They also gave a tight lower bound on the greedy regret.
In this paper, we want to explore another class of oracles covering more problems in practice.
Our goal is to demonstrate that although there are instrumental examples of problems where CTS has a linear regret, the majority of concrete problems do not follow this regime, and are in fact similar to the exact oracle case.
Contributions With a speciﬁc condition on Oracle , we describe a general set of CMAB problems where the approximation regret of CTS has a O (log( T ) / ∆ min ) bound, improving by a factor 1 / ∆ min over the bound of Kong et al.
This does not contradict their lower bound, which is focused on the greedy regret.
We call this set of CMAB problems REDUCE 2 EXACT , because, as we will see, a REDUCE 2 EXACT problem can be approximated using a reduction to sub-problems that can be solved exactly. Our main result is the approximation regret guaranty for REDUCE 2 EXACT problems, provided in Theorem 2.
The REDUCE 2 EXACT condition on Oracle is structural and applies notably to greedy algorithms for submodular maximization.
In particular, it allows to deal with problems such as probabilistic maximum coverage . We note that REDUCE 2 EXACT is compatible with the probabilistically triggered arms setting, which allows us to capture even more problems, such as online inﬂuence maximization [Wen et al., 2017, Wang and Chen, 2017].
As we want to focus on concrete problems, we provide several other examples that belongs to REDUCE 2 EXACT : Metric k-center , Vertex cover , Max-Cut and Travelling salesman problem .
Further related work We refer the reader to Wang and Chen [2018] for more related work on TS for combinatorial bandits.
Brieﬂy, one can mention Gopalan et al.
[2014], that gave a frequentist high-probability regret bounds for TS with a general action space and feedback model — Komiyama et al.
[2015], that studied TS for the m -sets action space — Wen et al.
[2015], that studied TS for contextual CMAB problems, using the Bayesian regret metric (see also Russo and Van Roy [2016]).
Other known limitations of CTS
Apart from the limitation related to the approximation regret that interests us in this paper, there are some other existing limitations of CTS highlighted in the literature, which we review here: The CTS policy has an exponential constant term in its regret upper bound [Wang and Chen, 2018, Perrault et al., 2020a], and Wang and Chen [2018] proved in their Theoremthat this is unavoidable.
A similar behavior have been demonstrated in Zhang and Combes [2021], where it is shown that CTS does not scale polynomially in the ambient dimension n in general. In addition, Zhang and Combes [2021] also proved that CTS is not minimax optimal. Actually, they even proved that in high dimensions, the minimax regret of CTS is almost linear in T .
The strengths of CTS
Despite the weaknesses mentioned above, CTS remains a widely used policy, mainly because of its empirical performance.
Indeed, CTS generally outperforms other policies such as CUCB and ESCB
[Wang and Chen, 2018, Perrault et al., 2020a].
Moreover, it is relatively simple to implement, and is computationally efﬁcient (just like CUCB ).
On the theory side, another advantage is that for an exact oracle, CTS is asymptotically quasi-optimal 4 for many settings where CUCB is not, and where ESCB is computationally inefﬁcient [Perrault et al., 2020a].
It would be desirable that these advantages also apply to the case of approximation regret, thus motivating our investigations.
Model and deﬁnitions
For more generality, we consider the probabilistically triggered arms extension of CMAB [Chen et al., 2016, Wang and Chen, 2017], abbreviated to CMAB-T.
In this context, the action A ∈ A selected is not necessarily equal to the triggered super-arm S .
More precisely, the action space
A is no longer necessarily a subset of P ([ n ]) .
At round t , the agent selects A t ∈ A , based on the history of observations H t (cid:44) σ (cid:0) X 1 (cid:12) e S 1 , . . .
, X t − 1 (cid:12) e S t − 1 (cid:1) and a possible extra source of randomness (we denote by F t the ﬁltration containing H t and the extra randomness of round t — in particular, A t is F t -measurable).
Then, an independent sample X t ∼ P X , X t ∈ R n is drawn and a random subset S t ∈ S ⊂ P ([ n ]) of arms are triggered ( S is called super-arm space or subset space ).
We assume that S t is drawn independently from a distribution D trig ( A t , X t ) .
For the feedback, the outcome of each triggered arm is observed, i.e., e S t (cid:12) X t is observed.
The expected reward is of the form r ( A t , µ ∗ ) , where r is a function deﬁned on a domain A × M , with M ⊂ R n .
The objects A , D trig , r are known to the agent.
In the following, we give the deﬁnition of the probability that an arm i ∈ [ n ] is triggered (and thus that a feedback from i is obtained) by having played a certain action A ∈ A .
Deﬁnition 1 (Triggering probabilities) .
The triggering probabilities are deﬁned for all i ∈ [ n ] and
A ∈ A as p i ( A ) (cid:44) P [ i ∈ S ] , where S ∼ D trig ( A, X ) , X ∼ P X . Under an α -approximation Oracle , we use the approximation regret to evaluate the performance of a policy π , deﬁned as follows.
Deﬁnition 2 (Approximation regret) .
The T -round α -approximation regret of a learning policy π that selects action A t ∈ A at round t is deﬁned as follows, where the approximation gap is deﬁned as ∆ t = ∆( A t ) (cid:44) {+
+} ∨ ( αr ( A ∗ , µ ∗ ) − r ( A t , µ ∗ )) , with A ∗ ∈ arg max A ∈A r ( A, µ ∗ ) . R T,α ( π ) (cid:44) E
[Equation]
∈ [ T ]
[Equation]
t [Equation] To approach the problem of minimizing R T,α , we consider the following standard assumptions [Wang and Chen, 2017].
Assumption 1 (Approximation oracle) .
The agent has access to an Oracle such that for any mean vector µ ∈ M ,
[Equation] Assumption 2 (1-norm triggering probability modulated bounded smoothness) .
There exists B ∈ R n + such that for all A ∈ A , for all µ , µ 0 ∈ M ,
[Equation]
[-[ n ] p i ( A ) B i | µ i − µ 
-]
Assumption 3 (Sub-Gaussianity of the outcome distribution) .
P X is such that ∀ λ ∈ R n ,
[Equation]
For example, P X = ⊗ i ∈
[ n ] P X i , and X i a.s. ∈ [0 , 1] (from Hoeffding’s Lemma [Hoeffding, 1963]).
Deﬁnition 3 (Other deﬁnitions) .
We deﬁne, for i ∈ [ n ] , the minimal gap of an action containing i as
[Equation]
i
[Equation]
The minimal and maximal gaps are deﬁned as
[Equation]
min i ∈ [ n ] ∆ i, min and ∆ max (cid:44) max A ∈A ∆( A ) .
For A ∈ A , we let T ( A ) (cid:44) { i ∈ [ n ] : p
i ( A ) > 0 } be the set of arms that are triggerable by selecting action A .
We ﬁnally deﬁne
[Equation]
Algorithm 1 CTS -
Initialization :
For each arm i , let γ i
= δ i = 1 .
For all t ≥ 1 :
Draw θ t ∼ ⊗ i ∈
[ n ] Beta ( γ i , δ i ) .
Play A t =
Oracle( θ t ) .
Get the observation X t (cid:12) e S t , and draw Y t ∼ ⊗ i ∈ S t Bernoulli( X i,t ) .
For all i ∈ S t update γ i ← γ i + Y
i,t and δ i ← δ i + 1 − Y i,t .
Algorithm
Input : β > 1 .
Initialization : Play each arm once (if the agent knows that µ ∗ ∈ [ a, b ] n , this might be skipped)
For every subsequent round t :
Draw θ t ∼ ⊗ i ∈
[ n ] N (cid:0) µ i,t − 1 , N − 1 i,t − 1 β/ 4 (cid:1) ( θ i,t ∼ U [ a, b ] if N i,t − 1 = 0 ).
Play A t =
Oracle( θ t ) .
Get the observations X t (cid:12) e S t and let Y t =
X t .
Update µ t − 1 and counters accordingly.
Combinatorial Thompson Sampling and exact oracle analysis
In this section, we present the CTS policy, focusing on two versions, one working with a Beta prior, and the other with a Gaussian prior.
Then, we present an associated analysis for the exact oracle case (i.e., with α = 1 ).
Based on the above assumptions, we focus on two versions of CTS . The ﬁrst version is CTS - BETA
[Wang and Chen, 2018] (Algorithm 1), working when we assume furthermore that P X = ⊗ i ∈
[ n ] P X i and X a.s. ∈ [0 , 1] n (note that this actually covers the case of bounded X , by adjusting the parameter
B ).
For each arm i ∈ [ n ] , CTS - BETA maintains a Beta prior distribution with parameters γ i and δ i (initialized to 1 ). At each round t , for each arm i , the algorithm sample θ i,t from the corresponding prior, representing the current estimate of µ ∗ i . Then the oracle outputs the action A t to play according to the input vector θ t . Based on the observation feedback, the algorithm then updates the corresponding Beta distributions.
The second version is CTS - GAUSSIAN [Perrault et al., 2020a] (Algorithm 2), that works under the more general Assumption 3. It is essentially the same as
CTS - BETA , except that the prior distributions are Gaussian.
For both Algorithm 1 and Algorithm 2, and an arm i ∈ [ n ] , we deﬁne the number of time i has been triggered at the beginning of round t , called counter of arm i , as [Equation] We also deﬁne the empirical mean at the beginning of round t as
[Equation] 3.2 Analysis of CTS : the α = 1 case
Although this is close to some known results in the current literature [Huyuk and Tekin, 2019, Perrault et al., 2020a], there is no proof for the classical O (cid:16) log( m ) log( T ) P i ∈ [ n ] B 2 i / ∆ i, min (cid:17) regret bound under the above assumptions in the CMAB-T setting, either for CTS - BETA (Algorithm 1) or CTSGAUSSIAN (Algorithm 2).
We thus provide such a result in Theorem 1 (the proof is postponed to Appendix A).
We can notice a difference with the work of Huyuk and Tekin [2019] concerning the Assumption 2, where the triggering probabilities do not appear (and are present in the main term of their ﬁnal regret bound).
The main difference with Perrault et al.
[2020a] is that they do not consider probabilistically triggered arms.
Theorem 1.
For α = 1 , the policy π described in Algorithm 1 (under Assumptions 1, 2 and P X = ⊗ i ∈
[ n ] P X i , X a.s. ∈ [0 , 1] n ) or Algorithm 2 (under Assumptions 1, 2 and 3) has regret of order R T,
[Equation] [Equation]
) = O   X i ∈ [ n ] B 2 i log( m ) log( T ) ∆ i, min   · In addition to being a new result in itself, Theorem 1 will be useful for the α < 1 case.
Concerning the bound, after changing the CMAB setting to CMAB-T, it should be noticed that the T -independent additive constant has a 1 /p ∗ dependence, the cause being the use of the Lipschitz condition (Assumption 2) without weighting by probabilities.
We think that this dependence should be avoidable, but it seems that another technique has to be considered (maybe using the triggering probability groups from the CUCB analysis of Wang and Chen [2017]).
Finally, we remark that this kind of bound can be usually transformed into a gap-independent √ T bound [Chen et al., 2013], however, for CTS , achieving this transformation is impossible since CTS is not minimax optimal (as we mentioned in the paragraph "Other known limitations of CTS").
The α < 1 case for REDUCE 2 EXACT problems
We will now look at the α < 1 case.
Our strategy is based on the following observation: many approximation algorithms involve a relaxation, or a reduction to one or more problems that can be solved exactly (in this paper, we use the terminology sub-problems to refer to them).
The approximation guarantee for such an approximation algorithm is thus obtained by linking the original problem to those sub-problems. Let’s give a simple abstract example to illustrate this idea. Let’s say we want to maximize a function f on some set F , using an α -approximation algorithm. Assume there exist two other functions g and h deﬁned on sets G and H respectively, such that G × H ⊂ F and such that g and h can be maximized exactly on G and H respectively.
Finally, assume that for all ( x, y ) ∈ G × H , [Equation] This means that an α -approximation algorithm to maximize f can simply output the feasible solution (arg max G g, arg max H h ) ∈ F . This example may seem very basic and artifactual at ﬁrst glance, but it turns out that many approximation algorithms rely on the same principle, as we will see in subsection 4.2.
To see how this can be exploited for approximation regret minimization, we can notice that the LHS of (2) is an approximation gap (as deﬁned in Deﬁnition 2, taking the max withand considering that the choice ( x, y ) ∈ F is that of a policy at a given round, with F playing the role of the action space) and that the RHS is the sum of two gaps, with G (respectively H ) playing the role of the action space, but this time without approximation factor.
Summing over the rounds, we ﬁnd that the corresponding approximation regret is bounded by the sum of two "classical" regrets (in this paper, we use the terminology sub-regrets).
Thus, the bounds we obtain on these two sub-regrets using CTS with the corresponding exact oracles translate into an bound on the approximation regret.
Avoiding the "mismatch" phenomenon Kong et al.
[2021] identiﬁed the reason why their regret bound has a ∆ 2min in the denominator, while the usual CMAB algorithms only have a ∆ min . They term it a "mismatch" between the estimated gaps that need to be eliminated by exploration and the actual regret the algorithm needs to pay. We argue that, in fact, this mismatch phenomenon should exist in principle, even for non-approximation regret. It is usually avoided using a smoothness assumption like Assumption 2, linking our estimations (here the arms that generate the feedback) and what is paid.
In the above example, although we are in an approximation context, the situation is fundamentally no different.
We see that (2) links the paid approximation regret with two exact sub-regrets which are themselves assumed to be related to our estimates through a smoothness-like property. There is thus an indirect link between the approximation regret and the outcomes, which enables one of the ∆ min present in the denominator of the exploration term log( T ) / ∆ 2min to be cancelled out by the actual regret paid, thus avoiding the mismatch phenomenon.
To summarize, just as the approximation relation is obtained by linking the original problem to sub-problems that can be solved exactly, we link the approximation regret to several sub-regrets each satisfying the appropriate properties for CTS , namely a smoothness relationship and the availability of an exact oracle.
We formalize this idea in the following assumption. Assumption 4 ( REDUCE 2 EXACT ) .
There exist ‘ ∈ N ∗ , c ∈ R ‘ + and B j ∈ R n + for all j ∈ [ ‘ ] such that the following is true. Oracle is of the form Oracle = Oracle 2 ◦ Oracle 1 , where Oracle 1 and Oracle 2 are described as follows.
• Oracle 1 : For µ ∈ M , Oracle 1 ( µ ) must output a sequence ( E 1 , . . .
, E ‘ ) described as follows. For each j ∈ [ ‘ ] , let E j =
E j
( E 1 , . . . , E j − 1 ) be a sub-action space which may depend on E 1 , . . . , E j − 1 and let r j ( · , µ ) :
E j → R be a reward sub-function. Then, we require that E j ∈ arg max E ∈E j r j ( E, µ ) .
• Oracle 2 : For an input E 1 ∈ E 1 , . . .
, E ‘ ∈ E ‘ , Oracle 2 ( E 1 , . . .
, E ‘ ) must output an action in A such that:
[Equation]
where for all j ∈ [ ‘ ] , E ∗ j ∈ arg max E ∈E j r j ( E, µ ∗ ) . • Finally, in addition to the above constraints on Oracle 1 and Oracle 2 , for each j ∈ [ ‘ ] , we require that the reward sub-function r j satisﬁes Assumption 2 with the constants B j and with the triggering probabilities p
i (Oracle 2 ( E 1 , . . .
, E ‘ )) , i ∈ [ n ] .
Informally, in the above Assumption 4, for µ ∈ M , Oracle 1 ( µ ) exactly solves a ﬁnite sequence of recursively deﬁned optimization sub-problems and Oracle 2 builds an action in A for the original approximation problem using the intermediate solutions provided by Oracle 1 .
At ﬁrst sight, Assumption 4 seems very speciﬁc and rather difﬁcult to fulﬁll.
However, we will see in subsection 4.2 that many concrete problems satisfy it.
4.1 Analysis
In this subsection, we give in Theorem 2 the main result of this paper.
It basically states that under
Assumptions 4, the CTS policy have a regret bound comparable to the exact oracle case.
The proof is postponed to Appendix B.
Theorem 2.
The policy π described in Algorithm 1 (under Assumption 4 and P X = ⊗ i ∈
[ n ] P X i ,
X a.s. ∈ [0 , 1] n ) or Algorithm 2 (under Assumptions 3 and 4) has regret of order
[Equation]
The idea of the proof is quite simple once Assumption 4 has been made.
We can see that the approximation regret can be decomposed into ‘ sub-regrets, according to equation (3).
Then, one must focus on the fact that the sub-regrets may dependent on each other and that the right gap (deﬁned with the original reward function) must be obtained in the denominator of the ﬁnal bound.
4.2 Examples of REDUCE 2 EXACT problems
Here, we present several problems belonging to REDUCE 2 EXACT .
Each time, after a quick introduction of the problem, we translate it into our CMAB-T context, and ﬁnally show how it satisﬁes the
REDUCE 2 EXACT criteria.
The Travelling salesman problem (TSP) is treated in Appendix C. Submodular maximization (e.g., probabilistic maximum coverage (PMC)) Here, we only expose the PMC example, noting that the same derivation can be applied to a monotone submodularfunction. PMC is one of the main examples proposed by Kong et al.
Given a weighted bipartite graph G =
( L, R, E ) , with weights µ ∗ (cid:44) ( µ ∗ ( u,v ) )
( u,v ) ∈ E (notice there are thus n = | E | arms, recalling that for us an arm is only something that produces an outcome, not something we can choose as an action, explaining why arms are not indexed by vertices here), the goal is to ﬁnd an action A ∈ A (cid:44) { A ⊂ L : | A | = k } , k ∈ N ∗ , maximizing the expected number of inﬂuenced nodes in R , where each node v ∈ R can be independently inﬂuenced by u ∈ A with probability µ ∗ ( u,v ) , i.e., maximizing f ( A, µ ∗ ) (cid:44) P v ∈ R (cid:16)
1 − Q u ∈ A : ( u,v ) ∈ E (1 − µ ∗ ( u,v ) ) (cid:17) .
This problem can be applied to the semi-bandit framework called the ad placement problem , where L are the web pages, R are the users and µ ∗ ( u,v ) is the probability that user v clicks on the ad on web page u .
In this application, the user’s click probabilities are unknown and must be learned as the rounds progress.
The Greedy oracle can provide an approximate solution with approximation ratio α = 1 − 1 /e
[Nemhauser et al., 1978].
This setting ﬁts REDUCE 2 EXACT as follows:
• Oracle 1 ( µ ) :
For i ∈ [ k ] , let E i (cid:44) { ( a 1 , . . .
, a i ) :
( a 1 , . .
, a i − 1 )
=
E i − 1 , a i ∈ L \ E
i − 1 } and r i (( a 1 , . .
. , a i ) , µ ) (cid:44) f
( { a 1 , . .
. , a i } , µ ) .
• Oracle 2 ( E 1 , . . .
, E k ) : Let ( a 1 , . . .
, a k ) (cid:44) E k .
Oracle 2 returns A
= { a 1 , . .
. , a k } .
Let A i
= { a 1 , . .
. , a i } for some i ∈ [ k ] and by abuse of notation, let f = f ( · , µ ∗ ) .
Informally, we see that in the above decomposition of the oracle, at each step, Oracle 1 maximizes f
( A i ) with A i −ﬁxed, i.e., Oracle 1 optimizes only on a i (we thus recover the greedy algorithm).
It is precisely these sub-problems of ﬁnding a i that can be solved exactly.
Then, we see that Oracle 2 simply returns the last A i constructed.
We will now prove the relation (3). The following is true using that f is monotone submodular (this is actually the way Nemhauser et al.
[1978] proved the approximation guarantee, and is true for any monotone submodular function):
[Equation]
Once the above relation is obtained, we can actually continue the original proof from Nemhauser et al.
[1978], skipping each step where we would need to use the property of Oracle 1 , thus leaving a term in the right-hand side for each time we skipped.
[Equation]
Finally, since (cid:0) 1 − 1 k (cid:1) k ≤ e − 1 , we get that ∆(Oracle 2 ( E 1 , . . . , E k ))
= (1 − e − 1 ) f ( A ∗ ) − f ( A ) is bounded by k X
[Equation]
Note in passing that we recover the classical approximation if the right-hand-side was equal to 0 .
It is easy to see that the reward function[-satisﬁes Assumption 
 [Chen et al., 2016], which directly impliesthat each-] r j satisﬁes Assumption 2 with B j = e [ n ] . We thus ﬁnally get our Assumption 4.
Online inﬂuence maximization (OIM)
As the analysis mentioned above only uses submodularity, it can be extended to the problem of online inﬂuence maximization in a social network.
A social network is modeled as a directed graph G =
( V, E ) , with nodes V representing users and edges E representing connections.
For a node i ∈ V , a subset A ⊂ V , and a vector x ∈ { 0 , 1 } E , let the predicate
A x (cid:32) i hold if, in the graph deﬁned by G x (cid:44) ( V, { ij ∈ E, x ij = 1 } ) , there is a forward path from a node in A to the node i .
If it holds, we say that i is inﬂuenced by A under x .
The goal is to ﬁnd an action A ∈ A (cid:44) { A ⊂ V : | A | = k } maximizing the inﬂuence spread σ ( A, µ ∗ ) (cid:44) E h(cid:12)(cid:12)(cid:12)n i ∈ V, A X (cid:32) i o(cid:12)(cid:12)(cid:12)i , where X ∼ ⊗ ( u,v )
∈
E Bernoulli( µ ∗ ( u,v ) ) .
This model is called the independent cascade model [Kempe et al., 2003, 2015].
A notable property to use the greedy oracle is that σ is monotone submodular.
As the exact calculation of σ is prohibitive, it is estimated by simulating the diffusion process, resulting in an approximation factor α = 1 − e − 1 − ε in the above greedy oracle analysis [Kempe et al., 2015, Feige, 1998, Chen et al., 2010], with ε > 0 .
In OIM, arms may be probabilistically triggered, and Assumption 2 holds with the constants being all equal to max u ∈ V (cid:12)(cid:12)(cid:12)(cid:12)(cid:26) v ∈ V, { u } ( I { µ ∗ e > 0 } ) e ∈ E (cid:32) v (cid:27)(cid:12)(cid:12)(cid:12)(cid:12) , which is the largest number of nodes any node can reach [Wang and Chen, 2017]. As previously, we thus get Assumption 4 with B j being e [ n ] times this constant.
Metric k -center This example and the following ones are less common for CMAB, but allow to well illustrate REDUCE 2 EXACT . Given a set of cities, one wants to build k warehouses in different cities and minimize the maximum distance of a city to a warehouse.
Formally, given a complete undirected weighted graph G =
( V, E ) whose distances d ( v i , v j ) satisfy the triangle inequality, the goal is to ﬁnd an action A ∈ A (cid:44) { A ⊂ V : | A | = k } that minimizes max v ∈ V d ( v, A ) .
We can consider the semi-bandit setting where the set of base arms is E , µ ∗ (cid:44) ( d ( v i , v j ))
( v i ,v j ) ∈ E and the feedback set S includes the edges of the graph induced by the chosen action.
We can target an approximation regret with α = 1 / 2 using the following oracle (which is simply the standard greedy algorithm for this problem).
• Oracle 1 ( µ ) :
For i ∈ [ k ] , let E i (cid:44) { ( a 1 , . . .
, a i ) :
( a 1 , . .
, a i − 1 )
=
E i − 1 , a i ∈ V \ E
i − 1 } and r i (( a 1 , . . .
, a i ) , µ ) (cid:44)
min j ∈
[ i − 1] µ a i ,a j (with r 1 = 0 ). One
can notice we thus have E ∗ i ∈ arg max ( a 1 ,...,a i ) ∈E
[Equation]
d ( a i , { a 1 , . . .
, a i − 1 } )
[Equation]
arg
[Equation] ( a 1 ,...,a i ) ∈E i d ( a i , E i − 1 ) . • Oracle 2 ( E 1 , . . . , E k ) : Let ( a 1 , . . .
, a k ) (cid:44) E k .
Oracle 2 returns A
= { a 1 , . .
. , a k } .
Let w [Equation] v, [Equation]
We thus have:
[Equation]
Where the ﬁrst inequality is deduced as follows: the map f : v 7→ arg min a ∗ ∈
A ∗ d ( a ∗ , v ) deﬁnes a partition of V into k =
| A ∗ | clusters. By the the pigeonhole principle, one cluster contains 2 different points a, a 0 ∈ A ∪ { w } (simply because its size is k + 1 ). We can assume a 0 ∈ A without loss of generality.
Thus, since f ( a ) = f ( a 0 ) , we get d ( a, a 0 ) ≤ d ( a, f ( a ))
+ d ( a 0 , f ( a 0 ))
=
d ( a, A ∗ )
+ d ( a 0 , A ∗ ) ≤ 2 max v ∈ V d ( v, A ∗ ) .
Vertex cover The problem consists, given an undirected graph G =
( V, E ) , in ﬁnding a set of vertices with minimal cost to cover all the edges of E .
Formally, with µ ∗ ∈ R V + , the goal is to ﬁnd an action A ∈ A (cid:44) { A ⊂ V : ∀ ( u, v ) ∈ E, u ∈ A or v ∈ A } that minimizes e T A µ ∗ . The semi-bandit feedback is deﬁned directly as S = A . We can target an approximation regret with α = 1 / 2 using the following linear programming (LP) relaxation oracle.
• Oracle 1 ( µ ) : Let 6 E 1 (cid:44) n x ∈ { 0 , 1 / 2 , 1 } V : ∀ ( u, v ) ∈ E, x u + x v ≥ 1 o and r 1 ( x , µ ) (cid:44) − x T µ . • Oracle 2 ( E 1 ) : Let x (cid:44) E 1 .
Oracle 2 returns A = { v ∈ V : x v ≥ 1 / 2 } ∈ A (since ∀ ( u, v ) ∈ E, x u + x v ≥ 1 , so x u ≥ 1 / 2 or x v ≥ 1 / 2 , so u ∈ A or v ∈ A ). Let x ∗ (cid:44) E ∗ 1 .
We have: [Equation] Notice, r 1 satisﬁes Assumption 2 with the constants being 1 , using x ∈ { 0 , 1 / 2 , 1 } V , so that x ≤ e A .
Max-Cut Given an undirected weighted graph G =
( V, E ) , with weights µ ∗ (cid:44) ( µ ∗ ( u,v ) ) ( u,v ) ∈ E , the goal is to ﬁnd an action A ∈ A (cid:44) { A ⊂ V } maximizing the total weight of the edges between A and its complement, i.e., 12 P ( u,v ) ∈ E µ ∗ ( u,v ) (1 − y u y v ) , where y (cid:44) e A − e V \ A . We consider the semi-bandit context where E is the set of arms, and the feedback set includes the edges between A and its complement
, i.e., S = { ( u, v ) ∈ E, y u y v = − 1 } . To include randomization within the oracle, we can extend the action space A to the set of probability measures on { A ⊂ V } , replacing ∆( A ) by its expectation on A , as a function of the distribution of A . The polynomial-time approximation algorithm for Max-Cut with the best known approximation ratio [Goemans and Williamson, 1995] uses semideﬁnite programming and randomized rounding, and achieves an approximation ratio of α = 2 π min 0 ≤ θ ≤ π θ 1 − cos θ ≈ 0 . In our context, it can be deﬁned as follows.
• Oracle 1 ( µ ) : We deﬁne E(cid:44) n ( v u ) u ∈ V ∈ (cid:8) v ∈ R V : k v k 2 = 1 (cid:9) V o and r 1 (( v u ) u ∈ V , µ ) (cid:44) 12 P ( u,v ) ∈
E µ ( u,v ) (1 − v T u v v ) . • Oracle 2 ( E 1 ) : Let ( v u ) u ∈ V (cid:44) E 1 . Oracle 2 returns the distribution of A = { u ∈ V, v T u Z ≥ 0 } , where Z ∼ U (cid:0)(cid:8) [Equation]
The following is proved by Goemans and Williamson [1995]: [Equation] Thus, r 1 satisﬁes Assumption 2 with the constants being 1 /α .
We also get, with y ∗ (cid:44) e A ∗ − e V \ A ∗ : ∆(Oracle [Equation]
= α X
[Equation]
Conclusion
In this article, our main objective is to further expand the scope of the CTS policy.
We not only expand
CTS to probabilistically triggered arms (which was one of the open questions by Kong et al.
but we also consider a broader class of oracle compatible with CTS .
More precisely, we propose a condition, REDUCE 2 EXACT , which may seem unnatural at ﬁrst, but which in fact simply expresses that sub-problems that can be solved exactly must be hidden in the original approximation problem, and that the approximation oracle exploit them to output the ﬁnal solution.
Knowing that the majority of approximation algorithms use one or more relaxations to an exact problem (e.g., solving a convex programming relaxation to obtain a fractional solution and then rounding this fractional solution to get a feasible solution), our assumption falls within the range of many CMAB-T settings.
From this reduction, we naturally obtain the standard tight regret bound O (log( T ) / ∆ min ) .
This is the ﬁrst tight bound for the approximation regret on non-exact oracles.
As future work, it may be interesting to explore other types of assumption leading to a similar bound.
We also think our setting should be generalizable to budgeted regret (see Perrault et al.
[2019b, 2020c,b] for examples with an approximation oracle).
Finally, we have that the approximation regret is in some way concervative compared to the greedy regret.
For a given oracle, we can easily consider the equivalent of the greedy regret for that oracle.
An interesting investigation would then be to extend the work of Kong et al.
[2021] in this direction, considering other types of oracle.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] See Abstract and section 1.
(b) Did you describe the limitations of your work?
[Yes] See the paragraph "Other known limitations of CTS" and the second paragraph in section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
This work does not have any potential negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See section 2.
(b) Did you include complete proofs of all theoretical results?
[Yes] See all sections in Appendix. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Proof of Theorem
In the following proof, we will treat both algorithms at the same time, detailing the steps where there is a difference.
Thus, for Algorithm 1, we use the convention β = 1 in the following.
We let
B (cid:44) k B k ∞ .
Let 0 < ε < ∆ min / (2 B ( m ∗ 2 + 1)) .
For the two algorithms, we consider the following events for any time step t ∈ N ∗ :
• Z t (cid:44) { ∆ t > 0 } , • B t (cid:44) (cid:8)(cid:13)(cid:13) p ( A t ) (cid:12) B (cid:12) (cid:0) µ t − 1 − µ ∗ (cid:1)(cid:13)(cid:13) 1 > ∆ min / 2 − B ( m ∗ 2 + 1) ε (cid:9) , • C t (cid:44) (cid:8) k p
A t ) (cid:12) B (cid:12) ( θ t − µ ∗ ) k 1 > ∆ t − B (cid:0) m ∗ 2 + 1 (cid:1) ε (cid:9) , • D t (cid:44) n(cid:13)(cid:13) p ( A t ) (cid:12) B (cid:12) (cid:0) θ t − µ t − 1 (cid:1)(cid:13)(cid:13) 1 ≥ q 0 .
5 · log( |A| 2 n T ) P i ∈ [ n ] p i ( A t ) 2 B 2 i β/N i,t − 1 o .
We decompose the regret analysis into several steps, each step corresponding to a ﬁltration of the regret against a combination of these events.
Step 1: bound under Z t ∧ ¬ C t
The ﬁltered regret bound
[Equation]
is deduced from the following two lemmas, where we consider the following events for a subset
Z ⊂ [ n ] :
[Equation]
n Z ⊂ T
[Equation]
A [Equation] A =
Oracle (cid:0)
[Equation]
(cid:13)(cid:13)
[Equation]
(cid:12) [Equation] (cid:12) (cid:0) [Equation]
(cid:1)(cid:13)(cid:13)
[Equation]
A
[Equation]
[Equation]
[Equation]
o
[Equation]
S
[Equation]
(cid:13)(cid:13)(cid:0) µ ∗
[Equation]
θ 0 (cid:1) (cid:12) e Z (cid:13)(cid:13) ∞ ≤ ε, R ( θ 0 (cid:12) e Z
+ θ t (cid:12) e Z c , Z ) holds (cid:9) ,
T t
[Equation]
Lemma 1.
Z t , ¬ C t ⇒ ∃ Z ⊂ T ( A ∗ ) , Z 6 = ∅ s.t.
the event S t ( Z ) ∧ T t ( Z ) holds.
Lemma 2.
There are two constants c, c 0 such that
[Equation]
Proof of Lemma 1.
It is sufﬁcient to prove that
[Equation]
(4) because ¬ C t and S t ( Z ) together imply T t ( Z ) .
Then, we get (4) in a similar way as Lemma 2 from
Huyuk and Tekin [2019], which is possible as their Assumption 3 is implied by our Assumption 2.
More precisely, the only places where we use our Assumption 2 instead of their Assumption 3 are in cases 1 a, 2 a, ... , when we show that R ( θ 0 (cid:12) e Z + θ t (cid:12) e Z c , Z ) holds.
The detailed proof is given in the following.
We ﬁrst consider the choice Z = Z 1 = T ( A ∗ ) .
Two cases can be distinguished:
1a) ∀ θ 0 s.t.
(cid:13)(cid:13)(cid:0) µ ∗ − θ 0 (cid:1) (cid:12) e Z 1 (cid:13)(cid:13) ∞
≤ ε , we have Z 1 ⊂ T (cid:0) Oracle (cid:0) θ 0 (cid:12)
e Z 1 + θ t (cid:12) e Z 1 c (cid:1)(cid:1) .
1b)
∃ θ 0 s.t.
(cid:13)(cid:13)(cid:0) µ ∗ − θ 0 (cid:1) (cid:12) e Z 1 (cid:13)(cid:13) ∞
≤ ε such that Z 1 6⊂ T (cid:0) Oracle (cid:0) θ 0 (cid:12)
e Z 1 + θ t (cid:12) e Z 1 c (cid:1)(cid:1) .
1a)
For the ﬁrst case, consider any vector θ 0 such that (cid:13)(cid:13)(cid:0) µ ∗ − θ 0 (cid:1) (cid:12) e Z 1 (cid:13)(cid:13) ∞ (5) ≤ ε and let A (6) =
Oracle (cid:0) θ 0 (cid:12) e Z 1 + θ t (cid:12) e
Z 1 c (cid:1) .
We can write r (cid:0) A, θ 0 (cid:12) e
Z 1 + θ t (cid:12) e
{+Z 1 c (cid:1) (7) ≥ r (cid:0) A ∗ , θ 
+}
(cid:12) e Z 1 + θ t (cid:12) e
Z 1 c (cid:1) (8) ≥ r ( A ∗ , µ ∗ ) − Bm ∗ ε,
where (7) is from (6), and (8) is from (5).
This rewrites as
[Equation]
so R t ( θ 0 (cid:12) e Z 1 + θ t (cid:12) e Z 1 c , Z 1 ) holds.
Therefore, we have proved that S t ( Z 1 ) holds.
1b)
For the second case, we have some vector θ 0 such that (cid:13)(cid:13)(cid:0) µ ∗ − θ 0 (cid:1) (cid:12) e Z 1 (cid:13)(cid:13) ∞ (9) ≤ ε , and some action A =
Oracle (cid:0) θ 0 (cid:12) e Z 1 + θ t (cid:12) e Z 1 c (cid:1) such that Z 1 6⊂ T ( A ) .
We consider Z 2 =
Z 1 ∩ T ( A ) 6 =
Z 1 .
We ﬁrst prove that Z 2 6 = ∅ by showing that if an action A 0 is such that Z 1 ∩ T ( A 0 ) (10) = ∅ , then
A 0 6 =
A as it has a lower reward value than that of A ∗ :
[Equation]
where (11) is from (10), (12) is from the deﬁnition of A t , (13) is from ¬ C t and (14) is from (9).
Now, we again distinguish two cases:
{+2a) ∀ θ 
+}
s.t.
(cid:13)(cid:13)(cid:0) µ ∗ − θ 00 (cid:1) (cid:12) e Z 2 (cid:13)(cid:13) ∞
≤ ε , we have Z 2 ⊂ T (cid:0) Oracle (cid:0) θ 00 (cid:12)
e Z 2 + θ t (cid:12) e Z 2 c (cid:1)(cid:1) .
{+2b) ∃ θ 
+}
s
t. (cid:13)(cid:13)(cid:0) µ ∗ − θ 00 (cid:1) (cid:12) e Z 2 (cid:13)(cid:13) ∞ ≤ ε such that Z 2 6⊂ T (cid:0) Oracle (cid:0) θ 0 (cid:12) e Z 2 + θ t (cid:12) e Z 2 c (cid:1)(cid:1) .
Notice that when (cid:13)(cid:13)(cid:0) µ ∗ − θ 00 (cid:1) (cid:12) e Z 2 (cid:13)(cid:13) ∞ (15) ≤ ε , then
[Equation]
Indeed, (16) is a consequence of
[Equation]
≤
[Equation]
m
[Equation]
−
[Equation]
where we used (15), (9) and that Z 2 is strictly included in Z 1 .
2a)
For the ﬁrst case, considering any vector θ 00 such that (cid:13) (cid:13) (cid:0) µ ∗ − θ 00 (cid:1) (cid:12) e Z 2 (cid:13) (cid:13) ∞ ≤ ε , we have with e A =
Oracle (cid:0) θ 00 (cid:12) e Z 2 + θ t (cid:12) e
Z 2 c (cid:1)
that
[Equation]
r
[Equation]
µ
[Equation]
) − B ( m ∗
[Equation]
+ 1) ε, where (17) uses (16) and (18) uses (9).
{+Therefore, R t ( θ 0 (cid:12) e Z 
+}
+ θ t (cid:12) e
Z 2 c , Z 2 ) holds, and we proved that S t ( Z 2 ) holds. 2b)
For the second case, we have a vector θ 00 such that (cid:13)(cid:13)(cid:0) µ ∗ − θ 00 (cid:1) (cid:12) e Z 2 (cid:13)(cid:13) ∞ ≤ ε and an action e A
= Oracle (cid:0) θ 00 (cid:12) e
Z 2 + θ t (cid:12) e
Z 2 c (cid:1) such that Z 2 6⊂ T (cid:16) e A (cid:17) .
We consider Z
=
Z 2 ∩ T (cid:16) e A (cid:17) .
Again, Z 3 6 = ∅ because for any A 00 such that T ( A 00 ) ∩ Z 2 = ∅ , we have
[Equation]
where the last inequality is obtained in the same way as in inequalities from (17) to (18).
We could repeat the above argument and each time the size Z i is decreased by at least 1 .
Thus, after at most m ∗ − 1 steps, since m ∗ + 2( m ∗ − 1) + 2( m ∗ − 2) + · · · + 2 = m ∗ 2 < m ∗ 2 + 1 , we could reach the end and ﬁnd a Z i 6 = ∅ such that S t ( Z i ) holds.
Proof of Lemma 2.
We use the same proof as Lemma 3 from Huyuk and Tekin [2019], except that we replace their use of their Fact 3 with the following (where τ j and η q,k are deﬁned in Huyuk and
Tekin [2019]):
[Equation]
,k +
Y i ∈
Z
P [ | θ i,τ − µ ∗ i | ≤ ε |H
τ ] # − 1 , which is true by the same method as the proof of Lemma 4 in Perrault et al.
[2020a].
From there, we use the same derivations as in Perrault et al.
[2020a], either the Appendix A, Step 4 (if we consider
Algorithm 1) or the Appendix C, Step 4 (if we consider Algorithm 2).
In both cases, the end of the proof of Lemma 3 from Huyuk and Tekin [2019] leads to the desired result.
Step 2: bound under Z t ∧ B t
The ﬁltered regret bound
[Equation]
is obtained as follows.
Let t ≥ 1 .
First, note that B t implies
[Equation]
Then, ﬁxing i ∈ [ n ] , we can ensure that i ∈ S t in the event, using that p
i ( A t ) =
P [ i ∈ S t |F t ] :
[Equation]
1) ε o # = E "
E " T X t =  p
i ( A t ) I n
i ∈ S t , nB i (cid:12)(cid:12) µ i,t − 1 − µ ∗ i (cid:12)(cid:12) >
∆ min / 2 − B ( m ∗ 2 + 1) ε o(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) F t ## = E " T X t =  p
i ( A t ) I n
i ∈ S t , nB i (cid:12)(cid:12) µ i,t − 1 − µ ∗ i (cid:12)(cid:12) >
∆ min / 2 − B ( m ∗ 2 + 1) ε
o # ≤ E " T X t =  p ∗ I n i ∈ S t , nB i (cid:12)(cid:12) µ i,t − 1 − µ ∗ i (cid:12)(cid:12) >
∆ min / 2 − B ( m ∗ 2 + 1) ε
o # ≤p ∗
X t ≥1 ∧ 2 exp − 2 t (cid:18) ∆ min 2 nB i −
B ( m ∗ 2 + 1) ε
nB i (cid:19) 2 !!
[Equation]
 
[Equation]
Step 3: bound under Z t ∧ D t
The ﬁltered regret bound
[Equation]
follows from the following Lemma 3.
Lemma 3.
In Algorithm 1 and 2, for all round t ≥ 1 , we have
[Equation]
Proof.
We rely on the fact that conditionally on the history, the sample θ t is either a Gaussian random vector of mean µ t − 1 and of diagonal covariance given by βN − 1 i,t − 1 / 4 (for Algorithm 2), or a product of Beta random variable, that is sub-Gaussian with the same covariance matrix [Marchal et al., 2017] (for Algorithm 1).
We thus deﬁne the functions
[Equation]
, and λ t ( A
[Equation]
(cid:44)
[Equation]
t
[Equation]
A
[Equation]
i ∈
A βB 2 i p i ( A
[Equation]
N
[Equation]
,t [Equation] , we have P (cid:2)(cid:13)(cid:13)
[Equation]
≤ X A ∈A [Equation] ≤ X
[Equation]
≤ X
A ∈A e [Equation] Y [Equation]
≤ X
A
[Equation]
−
[Equation]
λ t ( A ) p i ( A ) B i ( µ i
[Equation]
t − 1 − θ i,t ) (cid:12)(cid:12)(cid:12)
[Equation]
t
[Equation]
≤
[Equation]
Step 4: bound under Z t ∧ C t ∧ ¬ B t ∧ ¬ D t
We get the ﬁltered regret bound
E
[Equation]
from the following derivations.
Let t ≥ 1 .
Then, under Z t ∧ C t ∧ ¬ B t ∧ ¬ D t , we have ∆
[Equation]
C t
[Equation]
Thus
[Equation]
the following event holds
[Equation]
and we can apply Lemma 4 to get the bound
E
[Equation]
X
[Equation]
Lemma 4 (Adapted from Wang and Chen [2018]) .
Let’s ﬁx the time horizon T .
For all i ∈ [ n ] , let β
i
T ∈ R
+ .
For t ≥ 1 , consider the event
[Equation]
Let
[Equation]
Then, we have
[Equation]
Proof.
We use the regret allocation method from Wang and Chen [2018].
Speciﬁcally, we want to prove that for any time step t where A t holds, we have the following allocation of the regret to each arm i ∈
[ n ] :
[Equation]
where the allocation functions are deﬁned for all i ∈ [ n ] as
[Equation]
Indeed, we can already see that such an allocation produces the bound we are looking for.
Notably the following derivation uses, for i ∈ [ n ] , the equality p
i ( A t ) =
E
[ I { N i,t = N i,t − 1 + 1 }|F t ] , so that for all function f , a sum of the form ≥ ≥ 0 f [Equation]
[Equation]
[Equation]


[Equation]
[Equation]
t 1 / 
[Equation]
E  [Equation] X i ∈ [ n ] L i, 1 X t =
L i, 2 + β i,T tδ i, min  
[Equation]
i
[Equation]
(cid:19)(cid:19)
[Equation]
Therefore, we prove the lemma if we show that A t implies (19).
Let t ≥ 1 be such that A t holds.
We can assume that ∆ t > 0 (otherwise, the inequality (19) is trivial).
We ﬁrst observe that
[Equation]
This will be useful to prove that the allocation on all arms i such that N i,t − 1 > L i, 1 can be 0 .
Then we distinguish the following cases: • If there exists i 0 ∈ A t such that N i 0 ,t − 1 = 0 , then ∆ t ≤ ∆ [Equation]
• If there exists i 0 ∈
{+A t such that 0 < N i 0 ,t − 1 ≤ p i 
+}
( A t ) 2 β
i 0 ,T / ∆ 2 t , then N i 0 ,t − 1 ≤ L i 0 , 2 and we have
[Equation]
• If for all i ∈ [ n ] , N i,t − 1 > p i ( A t ) 2 β i,T / ∆ 2 t , then, [Equation] On the other hand, using the event A t [Equation]
we have
[Equation] Now, using [Equation] we get ∆ t ≤
[Equation] We can therefore end the proof in the following way, using (21), ∆ t ≤ X i ∈ [ n ] , L i, 1 ≥ N i,t −p i ( A t ) 2 β i,T ∆ t N i,t −= X i ∈ [ n ] , L i, 2 ≥ N i,t −p i ( A t ) 2 β i,T ∆ t N i,t − 1 +
X i ∈ [ n ] , L i, 1 ≥ N i,t − 1 > L i,p i ( A t ) 2 β i,T ∆ t N i,t −≤ X i ∈ [ n ] , L i, 2 ≥ N i,t −g i ( N i,t − 1 ) +
X i ∈ [ n ] , L i, 1 ≥ N i,t − 1 > L i,p i ( A t ) β i,T δ i, min N i,t −= X i ∈ [ n ] , L i, 2 ≥ N i,t −g i ( N i,t − 1 ) +
X i ∈ [ n ] , L i, 1 ≥ N i,t − 1 > L i,g i ( N i,t − 1 ) = X i [Equation]
[ n ] [Equation] i ( N i,t [Equation]
As for Theorem 1, in the following proof, we will treat both algorithms at the same time. Let’s ﬁrst remark that we can apply steps 1,2 and 3 from Theorem 1, but taking r 1 instead of the true reward function. Indeed, letting E j,t ∈ arg max E ∈E j,t r j ( E, θ t ) and E ∗ j,t ∈ arg max E ∈E j,t r j ( E, µ ∗ ) for all t ≥ 1 , j ∈ [ ‘ ] , we see that the sub-policy playing E 1 ,t at round t is actually minimizing the regret with respect to r 1 using CTS with an exact oracle.
Since r 1 satisﬁes the assumptions required for Theorem 1, we can apply the steps. On the other hand, in steps 1,2 and 3, the presence of ∆ max each time in the upper bound tells us that the considered events are sufﬁcient to obtain the bounds. In other words, the ∆ t suffered does not come into play and is bounded by its max value.
Thus, we can apply these steps by replacing the reward function by r 1 , noting that the ∆ min in B t and the step 2 bound are replaced by a minimum gap with respect to r 1 . To summarize, we can place ourselves under the events (for r 1 ) of step 4, obtaining in parallel 3 constant terms. Our goal now is to do the same for the other reward functions r j .
However, since E j,t can depend on E 1 ,t , . . . , E j − 1 ,t , we deﬁne the following ﬁltration [Equation]
Let us suppose that we have treated the r 1 , . . . , r j − 1 cases, then, we have at our disposal a ﬁltered approximation regret against events Y 1 ,t , . . . , Y j − 1 ,t that r 1 , . . . , r j − 1 are in the situation of the step 4, respectively.
We can write the ﬁltered approximation regret in the following way, by conditioning the expectation with this ﬁltration to get rid of the randomness carried by E j,t :
[Equation]
We can now apply the same procedure as described above with the reward function r j , on the inner conditional expectation, thus obtaining 3 additional T -independent terms and the new ﬁltered approximation regret [Equation] X t ∈ [ T ] ∆ t I
Y j,t } (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) G j −  I { Y 1 ,t , . . . Y j − 1 ,t }   =
E   X t ∈ [ T ] ∆ t I { Y 1 ,t , . . .
, Y j,t }   . Therefore, in the end, we have 3 j constant terms and a ﬁltered approximation regret where all reward functions r j are in the situation of step 4. Now, we place ourselves under this event to derive the dominant term of the bound on the approximation regret. From the derivation in step 4 applied to the function r j , we have
[Equation]
We can therefore use equation (3) to obtain [Equation]
From there, we can repeat the end of the proof of Theorem 1 with the weight in front of each arm i being P j ∈ [ ‘ ] B ij c j . C Travelling salesman problem (TSP) Here we give another example of a problem that falls into the REDUCE 2 EXACT setting, namely the
Travelling salesman problem (TSP).
We will see that unlike the examples mentioned before, here we need to modify the algorithm a bit to fully fall within the REDUCE 2 EXACT setting.
Given a complete undirected weighted graph G =
( V, E ) whose distances µ ∗ (cid:44) ( d ( u, v ))
( u,v ) ∈
E have to satisfy the triangle inequality, the goal is to ﬁnd an Hamiltonian cycle
A ∈ A (cid:44) (cid:8)(cid:8) ( v 0 , v 1 ) , . .
, ( v | V |− 1 , v | V | ) (cid:9) : (cid:8) v 1 , . . .
, v | V | = v 0 (cid:9) =
V (cid:9) of minimum cost P i ∈ [ | V | ]
d ( v i − 1 , v i ) .
We consider the following oracle from the Christoﬁdes [1976] algorithm ( α = 2 / 3 ).
• Oracle 1 ( µ ) :
The algorithm of Christoﬁdes combines E 1 (cid:44) { spanning trees of G } and E 2 (cid:44) { perfect matchings of the subgraph G 0 of G induced by the vertices of odd order in E 1 } , with r 1 being the weight of the spanning tree and r 2 the weight of the perfect matching.
• Oracle 2 ( E 1 , E 2 ) :
Oracle 2 combines the edges of E 1 and E 2 to form a connected multigraph e G = (cid:16)e V , e E (cid:17) in which all vertices have even degree (so it is Eulerian), forms an Eulerian circuit in this multigraph, and ﬁnally, outputs the Hamiltonian cycle obtained by skipping repeated vertices (shortcutting).
Thanks to the triangle inequality, shortcutting does not increase the weight, so we have
[Equation]
Let’s now deal with A ∗ .
Removing an edge from A ∗ produces a spanning tree, so e T A ∗ µ ∗ ≥ r 1 ( E ∗ 1 , µ ∗ ) .
On the other hand, by the triangle inequality, the weight of the optimal TSP solution for
G 0 is lower than e T A ∗ µ ∗ (visiting more nodes does not, in any case, reduce the total cost).
Taking every second edge of this cycle (which is of even length since all graphs have an even number of vertices of odd degree) we obtain a matching that has a weight less than half the weight of the cycle (if this is not the case we can take the complementary), so e T A ∗ µ ∗ [Equation] ≥ [Equation]
[Equation]
E
[Equation] 2 , µ [Equation] To summarize, we have
[Equation]
From the above, we can see that all the criteria of REDUCE 2 EXACT are satisﬁed, except Assumption 2.
Indeed, we assume that we receive feedback from A , while we would need feedback from the set e E .
We could probably have foreseen that the TSP would pose a difﬁculty in our assumption: indeed, among the many operations performed by the oracle to build the ﬁnal solution, shortcutting is the one that does not imply an optimization, but rather makes the solution feasible (so it does not represent a sub-problem as we have deﬁned it in this paper).
In other words, if we allowed the tour to pass over the same vertex several times, then the TSP would belong to REDUCE 2 EXACT by skipping the shortcutting step.
Yet, we note that a workaround is possible by by taking a closer look at the shortcutting step: in this step, we have an Eulerian circuit that we follow by skipping some edges.
Even if the skipped edges are replaced by new ones such that the distance traveled decreases, it is precisely the absence of feedback on the skipped edges that poses an issue.
Therefore, for a given edge of the Eulerian circuit, it would be helpful to be able to guarantee that some feedback is obtained on this edge, i.e., that it has a chance to belong to the ﬁnal Hamiltonian cycle.
The trick is then to notice that the shortcutting step depends on the edge from which we start.
In particular, this edge is guaranteed to be in the ﬁnal Hamiltonian cycle.
This choice of the ﬁrst edge is generally presupposed to be arbitrary and has no inﬂuence on the guarantees obtained previously, but for us, it can be used to force an edge to produce a feedback.
More precisely, to choose this ﬁrst edge, we can use a uniform randomization on e E , meaning that for each edge e ∈ e E , the probability q e that e belongs to A is such that
[Equation]
Notice that since e E is a multigraph edge set, for a "true" edge e ∈ E , we can have two edges e 1 , e 2 ∈ e E representing it and {+in that case q e 1 + q e 
+}
=
p
e ( A ) .
We thus get:
[Equation]
We thus get our Assumption 4.