When Combinatorial Thompson Sampling meets Approximation Regret
Abstract
We study the Combinatorial Thompson Sampling policy ( CTS ) for combinatorial multi-armed bandit problems (CMAB), within an approximation regret setting.
Although CTS has attracted a lot of interest, it has a drawback that other usual
CMAB policies do not have when considering non-exact oracles: for some oracles,
CTS has a poor approximation regret (scaling linearly with the time horizon T )
[Wang and Chen, 2018].
A study is then necessary to discriminate the oracles on which CTS could learn.
This study was started by Kong et al.
[2021]: they gave the ﬁrst approximation regret analysis of CTS for the greedy oracle , obtaining an upper bound of order O (cid:0) log( T ) / ∆ 2 (cid:1) , where ∆ is some minimal reward gap.
In this paper, our objective is to push this study further than the simple case of the greedy oracle.
We provide the ﬁrst O (log( T ) / ∆) approximation regret upper bound for CTS , obtained under a speciﬁc condition on the approximation oracle, allowing a reduction to the exact oracle analysis.
We thus term this condition REDUCE 2 EXACT , and observe that it is satisﬁed in many concrete examples.
Moreover, it can be extended to the probabilistically triggered arms setting, thus capturing even more problems, such as online inﬂuence maximization .
Introduction
Stochastic multi-armed bandits (MAB) [Robbins, 1952, Berry and Fristedt, 1985, Lai and Robbins, 1985] are decision-making problems in which an agent acts sequentially in an uncertain environment.
At each round t ∈ N ∗ , the agent must select one arm from a ﬁxed set of n arms, denoted by [ n ] (cid:44) { 1 , . . . , n } , using a policy , based on the feedback from the previous rounds.
Then it gets as feedback an outcome
X i,t ∈ R — a random variable sampled from P X i , independently from previous rounds — where i is the selected arm and P X i is a probability distribution — unknown to the agent — of mean µ ∗ i .
The goal for the agent is to maximize the cumulative reward over a total of
T rounds ( T is the time horizon and may be unknown).
The performance metric of a policy is its regret R T , which is the expectation of the difference over T rounds between the cumulative reward of the policy that always picked the arm with the highest expected reward and the cumulative reward of the learning policy.
MAB models the so called dilemma between exploration and exploitation, i.e., whether to continue exploring arms to obtain more information (and thus strengthen the conﬁdence in the estimates of the distributions P X i ), or to use the information gathered by playing the best arm according to the observations so far.
In this paper, we study stochastic combinatorial multi-armed bandit (CMAB), with semi-bandit feedback , a.k.a.
stochastic semi
-bandit (still abbreviated as CMAB in this paper), an extension of
MAB where the agent plays an action (also called super-arm )
A t ∈ A ⊂ P ([ n ]) at each round t , where A is ﬁxed and called action space .
The feedback includes the outcomes of all base arms in the played super-arm.
The expected reward, given A t , is assumed to be in the form 2 r
( A t , µ ∗ ) , where µ ∗ ∈ R n is the unknown vector of expectations (traditionally, the reward is linear and equal to e T A t µ ∗ ).
In recent years, CMAB has attracted a lot of interest (see e.g. Cesa-Bianchi and Lugosi [2012], Gai et al.
[2012], Chen et al.
[2013, 2016], Kveton et al.
[2015], Wang and Chen [2017],
Perrault [2020]), particularly due to its wide applications in network routing, online advertising, recommender system, inﬂuence marketing, etc.
Many CMAB policies are based on the Upper Conﬁdence Bound (UCB) approach, extending the classical UCB policy [Auer et al., 2002] from MAB to CMAB.
This type of approach uses an optimistic estimate µ t of µ ∗ (i.e., for which the reward function is overestimated), lying in a wellchosen conﬁdence region.
Then, the action is chosen by plugging µ t inside an oracle (typically,
Oracle( µ ∗ ) is a maximizer of the reward function A 7→ r ( A, µ ∗ ) ).
An example of such policy is Combinatorial Upper Conﬁdence Bound ( CUCB )
[Chen et al., 2013, Kveton et al., 2015], that uses a Cartesian product of the individual conﬁdence intervals of each arm as a conﬁdence region.
For mutually independent arms, Combes et al.
[2015] provided the UCB-style policy Efﬁcient
Sampling for Combinatorial Bandit ( ESCB ), building a tighter axis-aligned ellipsoidal conﬁdence region around the empirical mean, which helps to better restrict the exploration.
Degenne and Perchet [2016] provided a policy called OLS - UCB , leveraging a sub-Gaussianity assumption on the arms to generalize the ESCB approach.
These policies have been further extended to more general settings afterwards [Perrault et al., 2020c,d, 2019a].
Although improving CUCB , all these generalizations are inefﬁcient in terms of computation time.
Another paradigm that has recently gained interest (and which will be our focus in this paper) is to rely on Thompson Sampling ( TS ) instead of UCB , still targetting frequentist regret .
Although introduced much earlier by Thompson [1933], the theoretical analysis of TS for MAB is quite recent:
Kaufmann et al.
[2012], Agrawal and Goyal [2012] gave a regret bound matching the UCB policy theoretically.
Moreover, TS often performs better than UCB in practice, making TS an attractive policy for further investigations.
For CMAB, TS extends to Combinatorial Thompson Sampling ( CTS ).
In CTS , the unknown mean µ ∗ is associated with a belief (a prior distribution, that could be e.g. a product of
Beta or Gaussian distributions) updated to a posterior with the Bayes’rule, each time a feedback is received.
In order to choose an action at round t , CTS draws a sample θ t from the current belief, and plays the action given by Oracle( θ t ) .
CTS is an attractive policy because it has similar advantages to the previously mentioned policies working with ellipsoidal conﬁdence regions while being, like
CUCB , computationally efﬁcient.
Indeed, recently, for mutually independent arms and sub-Gaussian arms respectively, Wang and Chen [2018], Perrault et al.
[2020a] proposed tight analyses of CTS .
Unlike UCB-based policies, the analysis of CTS is valid only when Oracle is exact, i.e., when
[Equation]
Although this holds true for many combinatorial problem described by the pair ( r, A ) (we recall that r is usually linear), there exist some problems where the requirement on Oracle has to be relaxed in order to make it tractable.
This is usually done considering an α - approximation oracle [Chen et al., 2013, 2016, Wen et al., 2016], for α ∈ (0 , 1) :
[Equation]
Under an α -approximation oracle, the benchmark cumulative reward is the α -fraction of the optimal reward, leading to the notion of approximation regret [Kakade et al., 2009, Streeter and Golovin, 2009, Chen et al., 2016].
Aware of the limitation of their CTS analysis (that works only with exact oracles), Wang and Chen [2018] also proved, in their Theorem 2, that this limitation is not a technical artifact.
More precisely, they provided a speciﬁc CMAB instance with an associated approximation oracle, such that CTS on this instance and with this oracle must have a regret scaling linearly in T .
Although this negative result is of great interest to the research community, some concerns limit its consideration.
Indeed, not only the CMAB instance provided by Wang and Chen [2018] is actually a MAB one (meaning that there is an efﬁcient oracle that simply enumerates the arms), so the use of an approximation regret is not justiﬁed, but above all, both the oracle and the instance are uncommon and designed for the proof.
Interested in the question of whether the example provided by Wang and Chen [2018] is pathological or generalizable, Kong et al.
[2021] recently initiated a study, which revealed that linear approximation regret for CTS seems to be pathological.
More precisely, they derived a O (cid:0) log( T ) / ∆ 2 (cid:1) bound for the speciﬁc case of a greedy oracle 3 , where ∆ is some reward gap.
This result is obtained by bounding the approximation regret by a greedy regret , that simply replaces α max A ∈A r ( A, µ ∗ ) with r (Oracle( µ ∗ ) , µ ∗ ) , using equation (1).
They also gave a tight lower bound on the greedy regret.
In this paper, we want to explore another class of oracles covering more problems in practice.
Our goal is to demonstrate that although there are instrumental examples of problems where CTS has a linear regret, the majority of concrete problems do not follow this regime, and are in fact similar to the exact oracle case.
Contributions With a speciﬁc condition on Oracle , we describe a general set of CMAB problems where the approximation regret of CTS has a O (log( T ) / ∆) bound, improving by a factor 1 / ∆ over the bound of Kong et al.
This does not contradict their lower bound, which is focused on the greedy regret.
We call this set of CMAB problems REDUCE 2 EXACT , because, as we will see, a REDUCE 2 EXACT problem can be approximated using a reduction to sub-problems that can be solved exactly.
Our main result is the approximation regret guaranty for REDUCE 2 EXACT problems, provided in Theorem 2.
The REDUCE 2 EXACT condition on Oracle is structural and applies notably to greedy algorithms for submodular maximization.
In particular, it allows to deal with problems such as probabilistic maximum coverage .
We note that REDUCE 2 EXACT is compatible with the probabilistically triggered arms setting, which allows us to capture even more problems, such as online inﬂuence maximization [Wen et al., 2017, Wang and Chen, 2017].
As we want to focus on concrete problems, we provide several other examples that belongs to REDUCE 2 EXACT : Metric k-center , Vertex cover , Max-Cut and Travelling salesman problem .
Further related work We refer the reader to Wang and Chen [2018] for more related work on TS for combinatorial bandits.
Brieﬂy, one can mention Gopalan et al.
[2014], that gave a frequentist high-probability regret bounds for TS with a general action space and feedback model — Komiyama et al.
[2015], that studied TS for the m -sets action space — Wen et al.
[2015], that studied TS for contextual CMAB problems, using the Bayesian regret metric (see also Russo and Van Roy [2016]).
Other known limitations of CTS
Apart from the limitation related to the approximation regret that interests us in this paper, there are some other existing limitations of CTS highlighted in the literature, which we review here: The CTS policy has an exponential constant term in its regret upper bound [Wang and Chen, 2018, Perrault et al., 2020a], and Wang and Chen [2018] proved in their Theoremthat this is unavoidable.
A similar behavior have been demonstrated in Zhang and Combes [2021], where it is shown that CTS does not scale polynomially in the ambient dimension n in general.
In addition, Zhang and Combes [2021] also proved that CTS is not minimax optimal.
Actually, they even proved that in high dimensions, the minimax regret of CTS is almost linear in T .
The strengths of CTS
Despite the weaknesses mentioned above, CTS remains a widely used policy, mainly because of its empirical performance.
Indeed, CTS generally outperforms other policies such as CUCB and ESCB
[Wang and Chen, 2018, Perrault et al., 2020a].
Moreover, it is relatively simple to implement, and is computationally efﬁcient (just like CUCB ).
On the theory side, another advantage is that for an exact oracle, CTS is asymptotically quasi-optimal 4 for many settings where CUCB is not, and where ESCB is computationally inefﬁcient [Perrault et al., 2020a].
It would be desirable that these advantages also apply to the case of approximation regret, thus motivating our investigations.
Model and deﬁnitions
For more generality, we consider the probabilistically triggered arms extension of CMAB [Chen et al., 2016, Wang and Chen, 2017], abbreviated to CMAB-T.
In this context, the action A ∈ A selected is not necessarily equal to the triggered super-arm S .
More precisely, the action space
A is no longer necessarily a subset of P ([ n ]) and can be inﬁnite.
At round t , the agent selects A t ∈ A , based on the history of observations H t (cid:44) σ (cid:0) X 1 (cid:12) e S 1 , . . .
, X t − 1 (cid:12) e S t − 1 (cid:1) and a possible extra source of randomness (we denote by F t the ﬁltration containing H t and the extra randomness of round t — in particular, A t is F t -measurable).
Then, an independent sample X t ∼ P X , X t ∈ R n is drawn and a random subset S t ∈ S ⊂ P ([ n ]) of arms are triggered ( S is called super-arm space or subset space ).
We assume that S t is drawn independently from a distribution D trig ( A t , X t ) and that the outcome of an arm does not depend on whether it is triggered. In addition, if we don’t have P X = ⊗ i ∈ [ n ] P X i , we assume that D trig doesn’t depend on X t .
For the feedback, the outcome of each triggered arm is observed, i.e., e S t (cid:12) X t is observed.
The expected reward is of the form r ( A t , µ ∗ ) , where r is a function deﬁned on a domain A × M , with M ⊂ R n .
The objects A , D trig , r are known to the agent.
We assume that r ( · , µ ∗ ) admits a maximum r ( A ∗ , µ ∗ ) on A .
In the following, we give the deﬁnition of the probability that an arm i ∈ [ n ] is triggered (and thus that a feedback from i is obtained) by having played a certain action A ∈ A .
Deﬁnition 1 (Triggering probabilities) .
The triggering probabilities are deﬁned for all i ∈ [ n ] and
A ∈ A as p i
( A )
(cid:44) P [ i ∈ S ] , where S ∼ D trig ( A, X ) , X ∼ P X .
Under an α -approximation Oracle , we use the approximation regret to evaluate the performance of a policy π , deﬁned as follows.
Deﬁnition 2 (Approximation regret) .
The T -round α -approximation regret of a learning policy π that selects action A t ∈ A at round t is deﬁned as follows, where the approximation gap is deﬁned as ∆ t = ∆( A t ) (cid:44)
∨ ( αr ( A ∗ , µ ∗ ) − r
( A t , µ ∗ )) , with A ∗ ∈ arg max A ∈A r ( A, µ ∗ ) .
[Equation]
To approach the problem of minimizing R T,α , we consider the following standard assumptions
[Wang and Chen, 2017].
Assumption 1 (Approximation oracle) .
The agent has access to an Oracle such that for any mean vector µ ∈ M ,
[Equation]
Assumption 2 (1-norm triggering probability modulated bounded smoothness) .
There exists B ∈ R n + such that for all A ∈ A , for all µ , µ 0 ∈ M ,
[Equation]
Assumption 3 (Sub-Gaussianity of the outcome distribution) .
P X is such that ∀ λ ∈ R n ,
[Equation]
For example, P X = ⊗ i ∈
[ n ] P X i , and X i a.s. ∈ [0 , 1] (from Hoeffding’s Lemma [Hoeffding, 1963]).
Deﬁnition 3 (Other deﬁnitions) .
We deﬁne, for i ∈ [ n ] , the minimal gap of an action containing i as
[Equation]
The minimal and maximal gaps are deﬁned as
[Equation]
For A ∈ A , we let T ( A ) (cid:44) { i ∈ [ n ] : p
i ( A ) > 0 } be the set of arms that are triggerable by selecting action A .
We ﬁnally deﬁne
[Equation]
, and p ∗ (cid:44) inf i ∈ [ n ] , A
[Equation]
: p i ( A
[Equation]
> 0 p i ( A ) .
Algorithm 1 CTS -
Initialization :
For each arm i , let γ i
= δ i = 1 .
For all t ≥ 1 :
Draw θ t ∼ ⊗ i ∈
[ n ] Beta ( γ i , δ i ) .
Play A t =
Oracle( θ t ) .
Get the observation X t (cid:12) e S t , and draw Y t ∼ ⊗ i ∈ S t Bernoulli( X i,t ) .
For all i ∈ S t update γ i ← γ i + Y
i,t and δ i ← δ i + 1 − Y i,t .
Algorithm
Input : β > 1 .
Initialization : Play each arm once (if the agent knows that µ ∗ ∈ [ a, b ] n , this might be skipped)
For every subsequent round t :
Draw θ t ∼ ⊗ i ∈
[ n ] N (cid:0) µ i,t − 1 , N − 1 i,t − 1 β/ 4 (cid:1) ( θ i,t ∼ U [ a, b ] if N i,t − 1 = 0 ).
Play A t =
Oracle( θ t ) .
Get the observations X t (cid:12) e S t and let Y t =
X t .
Update µ t − 1 and counters accordingly.
Combinatorial Thompson Sampling and exact oracle analysis
In this section, we present the CTS policy, focusing on two versions, one working with a Beta prior, and the other with a Gaussian prior.
Then, we present an associated analysis for the exact oracle case (i.e., with α = 1 ).
3.1 Algorithms
Based on the above assumptions, we focus on two versions of CTS .
The ﬁrst version is CTS - BETA
[Wang and Chen, 2018] (Algorithm 1), working when we assume furthermore that P X = ⊗ i ∈
[ n ] P X i and X a.s. ∈ [0 , 1] n (note that this actually covers the case of bounded X , by adjusting the parameter
B ).
For each arm i ∈ [ n ] , CTS - BETA maintains a Beta prior distribution with parameters γ i and δ i (initialized to 1 ).
At each round t , for each arm i , the algorithm sample θ i,t from the corresponding prior, representing the current estimate of µ ∗ i .
Then the oracle outputs the action A t to play according to the input vector θ t .
Based on the observation feedback, the algorithm then updates the corresponding Beta distributions.
The second version is CTS - GAUSSIAN
[Perrault et al., 2020a]
(Algorithm 2), that works under the more general Assumption 3.
It is essentially the same as
CTS - BETA , except that the prior distributions are Gaussian.
For both Algorithm 1 and Algorithm 2, and an arm i ∈ [ n ] , we deﬁne the number of time i has been triggered at the beginning of round t , called counter of arm i , as
[Equation]
We also deﬁne the empirical mean at the beginning of round t as
[Equation]
3.2 Analysis of CTS : the α = 1 case
Although this is close to some known results in the current literature [Huyuk and Tekin, 2019, Perrault et al., 2020a], there is no proof for the classical O (cid:16) log( m ) log( T ) P i ∈ [ n ] B 2 i / ∆ i, min (cid:17) regret bound under the above assumptions in the CMAB-T setting, either for CTS - BETA (Algorithm 1) or CTSGAUSSIAN (Algorithm 2).
We thus provide such a result in Theorem 1 (the proof is postponed to
Appendix A).
We can notice a difference with the work of Huyuk and Tekin [2019] concerning the
Assumption 2, where the triggering probabilities do not appear (and are present in the main term of their ﬁnal regret bound).
The main difference with Perrault et al.
[2020a] is that they do not consider probabilistically triggered arms.
Theorem 1.
If ∆ min > 0 and p ∗ > 0 , the policy π described in Algorithm 1 (under Assumptions 1,and P X = ⊗ i ∈ [ n ] P X i , X a.s. ∈ [0 , 1] n ) or Algorithm 2 (under Assumptions 1, 2 and 3) has a regret of order
[Equation]
In addition to being a new result in itself, Theorem 1 will be useful for the α < 1 case.
Concerning the bound, after changing the CMAB setting to CMAB-T, it should be noticed that the T -independent additive constant depends on 1 /p ∗ , the cause being the use of the Lipschitz condition (Assumption 2) without weighting by probabilities.
We think that this dependence should be avoidable, but it seems that another technique has to be considered.
Finally, we remark that this kind of bound can be usually transformed into a gap-independent √ T bound [Chen et al., 2013], however, for CTS , achieving this transformation is impossible since CTS is not minimax optimal (as we mentioned in the paragraph "Other known limitations of CTS").
The α < 1 case for REDUCE 2 EXACT problems
We will now look at the α < 1 case.
Our strategy is based on the following observation: many approximation algorithms involve a relaxation, or a reduction to one or more problems that can be solved exactly (in this paper, we use the terminology sub-problems to refer to them).
The approximation guarantee for such an approximation algorithm is thus obtained by linking the original problem to those sub-problems.
We give a simple abstract example to illustrate this idea.
Let’s say we want to maximize a function f on some set F , using an α -approximation algorithm.
Assume there exist two other functions g and h deﬁned on sets G and H respectively, such that G × H ⊂ F and such that g and h can be maximized exactly on G and H respectively.
Finally, assume that for all ( x, y ) ∈ G × H ,
[Equation]
This means that an α -approximation algorithm to maximize f can simply output the feasible solution (arg max G g, arg max H h ) ∈ F .
This example may seem very basic and artifactual at ﬁrst glance, but it turns out that many approximation algorithms rely on the same principle, as we will see in subsection 4.2.
To see how this can be exploited for approximation regret minimization, we can notice that the LHS of (2) is an approximation gap (as deﬁned in Deﬁnition 2, taking the max withand considering that the choice ( x, y ) ∈ F is that of a policy at a given round, with F playing the role of the action space) and that the RHS is the sum of two gaps, with G (respectively H ) playing the role of the action space, but this time without approximation factor.
Summing over the rounds, we ﬁnd that the corresponding approximation regret is bounded by the sum of two "classical" regrets (in this paper, we use the terminology sub-regrets).
Thus, the bounds we obtain on these two sub-regrets using CTS with the corresponding exact oracles translate into an bound on the approximation regret.
Avoiding the "mismatch" phenomenon Kong et al.
[2021] identiﬁed the reason why their regret bound has a ∆ 2 in the denominator, while the usual CMAB algorithms only have a ∆ .
They term it a "mismatch" between the estimated gaps that need to be eliminated by exploration and the actual regret the algorithm needs to pay.
We argue that, in fact, this mismatch phenomenon should exist in principle, even for non-approximation regret.
It is usually avoided using a smoothness assumption like Assumption 2, linking our estimations (here the arms that generate the feedback) and what is paid.
In the above example, although we are in an approximation context, the situation is fundamentally no different.
We see that (2) links the paid approximation regret with two exact sub-regrets which are themselves assumed to be related to our estimates through a smoothness-like property.
There is thus an indirect link between the approximation regret and the outcomes, which enables one of the ∆ present in the denominator of the exploration term log( T ) / ∆ 2 to be cancelled out by the actual regret paid, thus avoiding the mismatch phenomenon.
To summarize, just as the approximation relation is obtained by linking the original problem to sub-problems that can be solved exactly, the idea behind REDUCE 2 EXACT problems is to link the approximation regret to several sub-regrets, each satisfying the appropriate properties for CTS , namely a smoothness relationship and the availability of an exact oracle.
We formalize this in the following assumption.
Assumption 4 ( REDUCE 2 EXACT ) .
There exist ‘ ∈ N ∗ , c ∈ R ‘ + and B j ∈ R n + for all j ∈ [ ‘ ] such that the following is true.
Oracle is of the form Oracle =
Oracle 2 ◦ Oracle 1 , where Oracle 1 and
Oracle 2 are described as follows.
• Oracle 1 : For µ ∈ M , Oracle 1 ( µ ) must output a sequence ( E 1 , . . .
, E ‘ ) described as follows.
For each j ∈ [ ‘ ] , let E j =
E
j
( E 1 , . . .
, E j − 1 ) be a sub-action space which may depend on E 1 , . . .
, E j − 1 and let r j ( · , µ ) :
E j → R be a reward sub-function.
Then, we require that E j ∈ arg max E ∈E j r j ( E, µ ) .
• Oracle 2 :
For an input E 1 ∈ E 1 , . . .
, E ‘ ∈ E ‘ , Oracle 2 ( E 1 , . . .
, E ‘ ) must output an action in A such that:
[Equation]
where for all j ∈ [ ‘ ] , E ∗ j ∈ arg max E ∈E j r j ( E, µ ∗ ) .
• Finally, in addition to the above constraints on Oracle 1 and Oracle 2 , for each j ∈ [ ‘ ] , we require that the reward sub-function r j satisﬁes Assumption 2 with the constants B j and with the triggering probabilities p
i (Oracle 2 ( E 1 , . . .
, E ‘ )) , i ∈ [ n ] .
Informally, in the above Assumption 4, for µ ∈ M , Oracle 1 ( µ ) exactly solves a ﬁnite sequence of recursively deﬁned optimization sub-problems and Oracle 2 builds an action in A for the original approximation problem using the intermediate solutions provided by Oracle 1 .
At ﬁrst sight, Assumption 4 seems very speciﬁc and rather difﬁcult to fulﬁll.
However, we will see in subsection 4.2 that many concrete problems satisfy it.
4.1 Analysis
In this subsection, we give in Theorem 2 the main result of this paper.
It basically states that under
Assumptions 4, the CTS policy have a regret bound comparable to the exact oracle case.
The proof is postponed to Appendix B.
Theorem 2.
If ∆ min > 0 and p ∗ > 0 , the policy π described in Algorithm 1 (under Assumptionand P X = ⊗ i ∈ [ n ] P X i , X a.s. ∈ [0 , 1] n ) or Algorithm 2 (under Assumptions 3 and 4) has regret of order
[Equation]
The idea of the proof is quite simple once Assumption 4 has been made.
We can see that the approximation regret can be decomposed into ‘ sub-regrets, according to equation (3).
Then, one must focus on the fact that the sub-regrets may dependent on each other and that the right gap (deﬁned with the original reward function) must be obtained in the denominator of the ﬁnal bound.
4.2 Examples of REDUCE 2 EXACT problems
Here, we present several problems belonging to REDUCE 2 EXACT .
Each time, after a quick introduction of the problem, we translate it into our CMAB-T context, and ﬁnally show how it satisﬁes the
REDUCE 2 EXACT criteria.
The Travelling salesman problem (TSP) is treated in Appendix C.
Submodular maximization (e.g., probabilistic maximum coverage (PMC))
Here, we only expose the PMC example, noting that the same derivation can be applied to a monotone submodularfunction.
PMC is one of the main examples proposed by Kong et al.
Given a weighted bipartite graph G =
( L, R, E ) , with weights µ ∗ (cid:44) ( µ ∗ ( u,v ) )
( u,v ) ∈ E (notice there are thus n = | E | arms, recalling that for us an arm is only something that produces an outcome, not something we can choose as an action, explaining why arms are not indexed by vertices here), the goal is to ﬁnd an action A ∈ A (cid:44) { A ⊂ L : | A | = k } , k ∈ N ∗ , maximizing the expected number of inﬂuenced nodes in R , where each node v ∈ R can be independently inﬂuenced by u ∈ A with probability µ ∗ ( u,v ) , i.e., maximizing f ( A, µ ∗ ) (cid:44) P v ∈ R (cid:16)
1 − Q u ∈ A : ( u,v ) ∈ E (1 − µ ∗ ( u,v ) ) (cid:17) .
This problem can be applied to the semi-bandit framework called the ad placement problem , where L are the web pages, R are the users and µ ∗ ( u,v ) is the probability that user v clicks on the ad on web page u .
In this application, the user’s click probabilities are unknown and must be learned as the rounds progress.
The Greedy oracle can provide an approximate solution with approximation ratio α = 1 − 1 /e
[Nemhauser et al., 1978].
This setting ﬁts REDUCE 2 EXACT as follows:
• Oracle 1 ( µ ) :
For i ∈ [ k ] , let E i (cid:44) { ( a 1 , . . .
, a i ) :
( a 1 , . .
, a i − 1 )
=
E i − 1 , a i ∈ L \ E
i − 1 } and r i (( a 1 , . .
. , a i ) , µ ) (cid:44) f
( { a 1 , . .
. , a i } , µ ) .
• Oracle 2 ( E 1 , . . .
, E k ) : Let ( a 1 , . . .
, a k ) (cid:44) E k .
Oracle 2 returns A
= { a 1 , . .
. , a k } .
Let A i
= { a 1 , . .
. , a i } for some i ∈ [ k ] and by abuse of notation, let f = f ( · , µ ∗ ) .
Informally, we see that in the above decomposition of the oracle, at each step, Oracle 1 maximizes f
( A i ) with A i −ﬁxed, i.e., Oracle 1 optimizes only on a i (we thus recover the greedy algorithm).
It is precisely these sub-problems of ﬁnding a i that can be solved exactly.
Then, we see that Oracle 2 simply returns the last A i constructed.
We will now prove the relation (3).
The following is true using that f is monotone submodular (this is actually the way Nemhauser et al.
[1978] proved the approximation guarantee, and is true for any monotone submodular function):
[Equation]
Once the above relation is obtained, we can actually continue the original proof from Nemhauser et al.
[1978], skipping each step where we would need to use the property of Oracle 1 , thus leaving a term in the right-hand side for each time we skipped.
[Equation]
Finally, since (cid:0) 1 − 1 k (cid:1) k ≤ e − 1 , we get that ∆(Oracle 2 ( E 1 , . . .
, E k ))
= (1 − e − 1 ) f
( A ∗ )
− f
( A ) is bounded by k X
[Equation]
Note in passing that we recover the classical approximation if the right-hand-side was equal to 0 .
It is easy to see that the reward function r j satisﬁes Assumption 2 with B j = e [ n ] .
We thus ﬁnally get our
Assumption 4.
Online inﬂuence maximization (OIM)
As the analysis mentioned above only uses submodularity, it can be extended to the problem of online inﬂuence maximization in a social network.
A social network is modeled as a directed graph G =
( V, E ) , with nodes V representing users and edges
E representing connections.
For a node i ∈ V , a subset A ⊂ V , and a vector x ∈ { 0 , 1 } E , let the predicate
A x (cid:32) i hold if, in the graph deﬁned by G x (cid:44) ( V, { ij ∈ E, x ij = 1 } ) , there is a forward path from a node in A to the node
i .
If it holds, we say that i is inﬂuenced by A under x .
The goal is to ﬁnd an action A ∈ A (cid:44) { A ⊂ V : | A | = k } maximizing the inﬂuence spread σ ( A, µ ∗ ) (cid:44) E h(cid:12)(cid:12)(cid:12)n i ∈ V, A X (cid:32) i o(cid:12)(cid:12)(cid:12)i , where X ∼ ⊗ ( u,v )
∈
E Bernoulli( µ ∗ ( u,v ) ) .
This model is called the independent cascade model [Kempe et al., 2003, 2015].
A notable property to use the greedy oracle is that σ is monotone submodular.
As the exact calculation of σ is prohibitive, it is estimated by simulating the diffusion process, resulting in an approximation factor α = 1 − e − 1 − ε in the above greedy oracle analysis [Kempe et al., 2015, Feige, 1998, Chen et al., 2010], with ε > 0 .
In
OIM, arms may be probabilistically triggered, and Assumption 2 holds with the constants being all equal to max u ∈ V (cid:12)(cid:12)(cid:12)(cid:12)(cid:26) v ∈ V, { u } ( I { µ ∗ e > 0 } )
e ∈ E (cid:32) v (cid:27)(cid:12)(cid:12)(cid:12)(cid:12) , which is the largest number of nodes any node can reach [Wang and Chen, 2017].
As previously, we thus get Assumption 4 with B j being e [ n ] times this constant.
Metric k -center
This example and the following ones are less common for CMAB, but allow to well illustrate REDUCE 2 EXACT .
Given a set of cities, one wants to build k warehouses in different cities and minimize the maximum distance of a city to a warehouse.
Formally, given a complete undirected weighted graph G =
( V, E ) whose distances d ( v i , v j ) satisfy the triangle inequality, the goal is to ﬁnd an action A ∈ A (cid:44) { A ⊂ V : | A | = k } that minimizes max v ∈ V d ( v, A ) .
We can consider the semi-bandit setting where the set of base arms is E , µ ∗ (cid:44) ( d ( v i , v j ))
( v i ,v j ) ∈ E and the feedback set S includes the edges of the graph induced by the chosen action.
We can target an approximation regret with α = 1 / 2 using the following oracle (which is simply the standard greedy algorithm for this problem).
• Oracle 1 ( µ ) :
For i ∈ [ k ] , let E i (cid:44) { ( a 1 , . . .
, a i ) :
( a 1 , . .
, a i − 1 )
=
E i − 1 , a i ∈ V \ E
i − 1 } and r i (( a 1 , . . .
, a i ) , µ ) (cid:44)
min j ∈
[ i − 1] µ a i ,a j (with r 1 = 0 ).
One
can notice we thus have
E ∗ i ∈ arg max ( a 1 ,...,a i ) ∈E
[Equation]
d ( a i , { a 1 , . . .
, a i − 1 } )
[Equation]
arg
[Equation]
( a 1 ,...,a i ) ∈E i d ( a i , E i − 1 ) .
• Oracle 2 ( E 1 , . . .
, E k ) : Let ( a 1 , . . .
, a k ) (cid:44) E k .
Oracle 2 returns A
= { a 1 , . .
. , a k } .
Let w
[Equation]
v,
[Equation]
We thus have:
[Equation]
Where the ﬁrst inequality is deduced as follows: the map f : v 7→ arg min a ∗ ∈
A ∗ d ( a ∗ , v ) deﬁnes a partition of V into k =
| A ∗ | clusters.
By the the pigeonhole principle, one cluster contains 2 different points a, a 0 ∈ A ∪ { w } (simply because its size is k + 1 ).
We can assume a 0 ∈ A without loss of generality.
Thus, since f ( a ) = f ( a 0 ) , we get d ( a, a 0 ) ≤ d ( a, f ( a ))
+ d ( a 0 , f ( a 0 ))
=
d ( a, A ∗ )
+ d ( a 0 , A ∗ ) ≤ 2 max v ∈ V d ( v, A ∗ ) .
Vertex cover The problem consists, given an undirected graph G =
( V, E ) , in ﬁnding a set of vertices with minimal cost to cover all the edges of E .
Formally, with µ ∗ ∈ R V + , the goal is to ﬁnd an action A ∈ A (cid:44) { A ⊂ V : ∀ ( u, v ) ∈ E, u ∈ A or v ∈ A } that minimizes e T A µ ∗ .
The semi-bandit feedback is deﬁned directly as S =
A .
We can target an approximation regret with α = 1 / 2 using the following linear programming (LP) relaxation oracle.
• Oracle 1 ( µ ) : Let 6 E 1 (cid:44) n x ∈ { 0 , 1 / 2 , 1 } V : ∀ ( u, v ) ∈
E, x u + x v ≥ 1 o and r 1 ( x , µ ) (cid:44) − x T µ .
• Oracle 2 ( E 1 ) : Let x (cid:44) E 1 .
Oracle 2 returns A
= { v ∈ V : x v ≥ 1 / 2 } ∈ A (since ∀ ( u, v ) ∈
E, x u + x v ≥ 1 , so x u ≥ 1 / 2 or x v ≥ 1 / 2 , so u ∈ A or v ∈ A ).
Let x ∗ (cid:44) E ∗ 1 .
We have:
[Equation]
Notice, r 1 satisﬁes Assumption 2 with the constants being 1 , using x ∈ { 0 , 1 / 2 , 1 } V , so that x ≤ e A .
Max-Cut Given an undirected weighted graph G =
( V, E ) , with weights µ ∗ (cid:44) ( µ ∗ ( u,v ) )
( u,v ) ∈ E , the goal is to ﬁnd an action A ∈ A (cid:44) { A ⊂ V } maximizing the total weight of the edges between
A and its complement, i.e., 12 P ( u,v ) ∈ E µ ∗ ( u,v ) (1 − y u y v ) , where y (cid:44) e A − e V \ A .
We consider the semi-bandit context where E is the set of arms, and the feedback set includes the edges between A and its complement
, i.e., S = { ( u, v ) ∈ E, y u y v = − 1 } .
To include randomization within the oracle, we can extend the action space A to the set of probability measures on { A ⊂ V } , replacing ∆( A ) by its expectation on A , as a function of the distribution of A .
The polynomial-time approximation algorithm for Max-Cut with the best known approximation ratio [Goemans and Williamson, 1995] uses semideﬁnite programming and randomized rounding, and achieves an approximation ratio of α = 2 π min 0 ≤ θ ≤ π θ 1 − cos θ ≈ 0 .
In our context, it can be deﬁned as follows.
• Oracle 1 ( µ ) : We deﬁne E(cid:44) n ( v u ) u ∈ V ∈
(cid:8) v ∈ R V : k v k 2 = 1 (cid:9) V o and r 1 (( v u ) u ∈ V , µ ) (cid:44) 12 P ( u,v ) ∈
E µ ( u,v ) (1 − v T u v v ) .
• Oracle 2 ( E 1 ) : Let ( v u ) u ∈ V (cid:44) E 1 .
Oracle 2 returns the distribution of A = { u ∈ V, v T u Z ≥ 0 } , where Z ∼ U (cid:0)(cid:8)
[Equation]
The following is proved by Goemans and Williamson [1995]:
[Equation]
Thus, r 1 satisﬁes Assumption 2 with the constants being 1 /α .
We also get, with y ∗ (cid:44)
e A ∗ − e V \
A ∗ : ∆
[Equation]
Oracle
[Equation]
Conclusion
In this article, our main objective is to further expand the "approximation regret scope" of the CTS policy.
We not only expand it to probabilistically triggered arms (which was one of the open questions by Kong et al.
but we also consider a broader class of oracles compatible with CTS .
More precisely, we propose a condition, REDUCE 2 EXACT , which may seem unnatural at ﬁrst, but which in fact simply expresses that sub-problems that can be solved exactly must be hidden in the original approximation problem, and that the approximation oracle exploit them to output the ﬁnal solution.
Knowing that the majority of approximation algorithms use one or more relaxations to an exact problem (e.g., solving a convex programming relaxation to obtain a fractional solution and then rounding this fractional solution to get a feasible solution), our assumption falls within the range of many CMAB-T settings.
From this reduction, we naturally obtain the standard tight regret bound O (log( T ) / ∆ min ) .
This is the ﬁrst tight bound for the approximation regret on non-exact oracles.
As future work, it may be interesting to explore other CMAB-T problems where the REDUCE 2 EXACT condition does (or doesn’t) hold.
We also think our setting should be generalizable to the budgeted regret setting without much difﬁculty (see Perrault et al.
[2019b, 2020c,b] for examples with an approximation oracle).
Finally, we have that the approximation regret is in some way conservative compared to the greedy regret.
For a given oracle, we can easily consider the equivalent of the greedy regret for that oracle.
An interesting investigation would then be to extend the work of Kong et al.
[2021] in this direction, considering other types of oracle.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] See Abstract and section 1.
(b) Did you describe the limitations of your work?
[Yes] See the paragraph "Other known limitations of CTS" and the second paragraph in section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
This work does not have any potential negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See section 2.
(b) Did you include complete proofs of all theoretical results?
[Yes] See all sections in Appendix. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[-Z 1 c (cid:1) (7) ≥ r (cid:0) A ∗ , θ 
-]
[-2a) ∀ θ 
-]
[-2b) ∃ θ 
-]
[-Therefore, R t ( θ 0 (cid:12) e Z 
-]
[N/A]
[-A t such that 0 < N i 0 ,t − 1 ≤ p i 
-]
[-Notice that since e E is a multigraph edge set, for a "true" edge e ∈ E , we can have two edges e 1 , e 2 ∈ e E representing it and in that case q e 1 + q e 
-]