Lipschitz Bandits with Batched Feedback
Anonymous Author(s)
AffiliationAddress email
Abstract
In this paper, we study Lipschitz bandit problems with batched feedback, where theexpected reward is Lipschitz and the reward observations are communicated to theplayer in batches.
We introduce a novel landscape-aware algorithm, called Batched
Lipschitz Narrowing (BLiN), that optimally solves this problem.
Specifically, weshow that for a T -step problem with Lipschitz reward of zooming dimension d z ,our algorithm achieves theoretically optimal regret rate (cid:101) O (cid:16) T dz +1 dz +2 (cid:17) using only
O (log log T ) batches.
We also provide complexity analysis for this problem.
Ourtheoretical lower bound implies that Ω(log log T ) batches are necessary for anyalgorithm to achieve the optimal regret.
Thus, BLiN achieves optimal regret rateusing minimal communication.
Introduction
Multi-Armed Bandit (MAB) algorithms aim to exploit the good options while explore the decisionspace.
These algorithms and methodologies find successful applications in artificial intelligenceand reinforcement learning [e.g., 36].
While the classic MAB setting assumes that the rewards areimmediately observed after each arm pull, real-world data often arrives in different patterns.
Forexample, observations from clinical trials are often be collected in a batched fashion [33].
Anotherexample is from online advertising, where strategies are tested on multiple customers at the sametime [10].
In such cases, any observation-dependent decision-making should comply with thisdata-arriving pattern, including MAB algorithms.
In this paper, we study the Lipschitz bandit problem with batched feedback – a MAB problem wherethe expected reward is Lipschitz and the reward observations are communicated to the player inbatches.
In such settings, rewards are communicated only at the end of the batches, and the algorithmcan only make decisions based on information up to the previous batch.
Existing Lipschitz banditalgorithms heavily rely on timely access to the reward samples, since the partition of arm spacemay change at any time.
Therefore, they can not solve the batched feedback setting.
To addressthis difficulty, we present a novel adaptive algorithm for Lipschitz bandits with communicationconstraints, named Batched Lipschitz Narrowing (BLiN).
BLiN learns the landscape of the reward byadaptively narrowing the arm set, so that regions of high reward are more frequently played.
Also,
BLiN determines the data collection procedure adaptively, so that only very few data communicationsare needed.
The above BLiN procedure achieves optimal regret rate (cid:101) O (cid:16) T
dz +1 dz +2 (cid:17) ( d z is the zooming dimension[25, 14]), and can be implemented in a clean and easy-to-implement form.
In addition to achievingthe optimal regret rate, BLiN is also optimal in the following senses:
• BLiN’s communication complexity is optimal.
BLiN only needs O (log log T ) rounds of commu- nications to achieve the optimal regret rate (Theorem 2), and no algorithm can achieve this rate with fewer than Ω(log log T ) rounds of communications (Corollary 2). • BLiN’s time complexity is optimal (Remark 1): if the arithmetic operations and sampling are ofcomplexity O (1) , then the time complexity of BLiN is O ( T ) , which improve the best known timecomplexity O
( T log T ) for Lipschitz bandit problems [14].
1.1 Settings & Preliminaries
For a Lipschitz bandit problem (with communication constraints), the arm set is a compact doublingmetric space ( A , d A ) .
The expected reward µ : A → R is 1 -Lipschitz with respect to the metric d A ,that is, | µ ( x 1 ) − µ ( x 2 ) | ≤ d A ( x 1 , x 2 ) for any x 1 , x 2 ∈ A .
At time t ≤ T , the learning agent pulls an arm x t ∈ A that yields a reward sample y t = µ ( x t ) + ϵ t ,where ϵ t is a mean-zero independent sub-Gaussian noise.
Without loss of generality, we assume thatϵ t ∼ N (0 , 1) , since generalizations to other sub-Gaussian noises are not hard.
Similar to most bandit learning problems, the agent seeks to minimize regret in the batched feedbackenvironment.
The regret is defined as R ( T ) =
(cid:80) Tt =1 ( µ ∗ − µ ( x t )) , where µ ∗ denotes max x ∈A µ ( x ) .
For simplicity, we define ∆ x = µ ∗ − µ ( x ) (called optimality gap of x ) for all x ∈ A .
1.1.1 Doubling Metric Spaces and the ([0 , 1] d , ∥ · ∥ ∞ ) Metric Space
By the Assouad’s embedding theorem [6], the (compact) doubling metric space ( A , d A ) can beembedded into a Euclidean space with some distortion of the metric; See [42] for more discussions ina machine learning context.
Due to existence of such embedding, the metric space ([0 , 1] d , ∥ · ∥ ∞ ) ,where metric balls are hypercubes, is sufficient for the purpose of our paper.
For the rest of the paper,we will use hypercubes in algorithm design for simplicity, while our algorithmic idea generalizes toother doubling metric spaces.
1.1.2 Zooming Numbers and Zooming Dimensions
An important concept for bandit problems in metric spaces is the zooming number and the zoomingdimension [25, 13, 38], which we discuss now.
Define the set of r -optimal arms as S ( r ) = { x ∈ A : ∆ x ≤ r } .
For any r = 2 − i , the decision space[0 , 1] d can be equally divided into 2 di cubes with edge length r , which we call standard cubes (alsoreferred to as dyadic cubes).
The r -zooming number is defined as
N r :=
# { C : C is a standard cube with edge length r and C ⊂ S (16 r ) } .
In words, N r is the r -packing number of the set S (16 r ) in terms of standard cubes.
The zoomingdimension is then defined as
[Equation]
Moreover, we define the zooming constant C z as the minimal a to make the above inequality true ford z , C z = min { a > 0 : N r ≤ ar − d z , ∀ r = 2 − i for some i ∈ N } .
Zooming dimension d z can be significantly smaller than ambient dimension d and can be zero.
For asimple example, consider a problem with ambient dimension d = 1 and expected reward functionµ ( x ) =
x for 0 ≤ x ≤ 1 .
Then for any r = 2 − i with i ≥ 4 , we have S (16 r ) =
[1 − 16 r, 1] and
N r = 16 .
Therefore, for this problem the zooming dimension equals to 0 , with zooming constant
C z = 16 .
1.2 Batched feedback pattern and our results
In the batched feedback setting, for a T -step game, the player determines a grid T
= { t 0 , · · · , t B }adaptively , where
0 =
t 0 < t 1 < · · · < t B = T and B ≪ T .
During the game, reward observationsare communicated to the player only at the grid points t 1 , · · · , t B .
As a consequence, for any timet in the j -th batch, that is, t j − 1 < t ≤ t j , the reward y t cannot be observed until time t j , and thedecision made at time t depends only on rewards up to time t j − 1 .
The determination of the grid Tis adaptive in the sense that the player chooses each grid point t j ∈ T based on the operations andobservations up to the previous point t j − 1 .
In this work, we present BLiN algorithm to solve Lipschitz bandits under batched feedback.
Duringthe learning procedure, BLiN detects and eliminates the ‘bad area’ of the arm set in batches and partition the remaining area according to an approporiate edge-length sequence .
Our first theoreticalupper bound is that with simple Doubling Edge-length Sequence r m = 2 − m +1 , BLiN achievesoptimal regret rate (cid:101) O (cid:16) T
dz +1 dz +2 (cid:17) by using O (log T ) batches.
Theorem 1.
With probability exceeding 1 − 2 T 6 , the T -step total regret R ( T ) of BLiN with Doubling
Edge-length Sequence (D-BLiN) satisfies
[Equation]
where d z is the zooming dimension of the problem instance.
In addition, D-BLiN only needs no morethan O
(log T ) rounds of communications to achieve this regret rate.
Here and henceforth, ≲ onlyomits constants.
While D-BLiN is efficient for batched Lipschitz bandits, its communication complexity is not optimal.
We then propose a new edge-length sequence, which we call Appropriate Combined Edge-length
Sequence (ACE Sequence) to improve the algorithm.
The idea behind this sequence is that byappropriately combining some batches, the algorithm can achieve better communication boundwithout incurring increased regret.
As we shall see, BLiN with ACE Sequence (A-BLiN) achievesregret rate (cid:101) O (cid:16) T dz +1 dz +2 (cid:17) with only O (log log T ) batches.
Theorem 2.
With probability exceeding 1 − 2 T 6 , the T -step total regret R ( T ) of A-BLiN satisfies
[Equation]
where d z is the zooming dimension of the problem instance.
In addition, Algorithm 1 only needs nomore than O (log log T ) rounds of communications to achieve this regret rate.
As a comparison, seminal works [25, 38, 14] show that the optimal regret bound for Lipschitz banditswithout communications constraints, where the reward observations are immediately observable aftereach arm pull, is R ( T ) ≲ T dz +1 dz +2 · (log T ) 1 dz +2 in terms of zooming dimension, and
R ( T ) ≲ R z ( T ) ≜ inf r  r 0 T + (cid:88)
r =2 − i ,r ≥ r
N r r log T   (1) in terms of zooming number.
Therefore, A-BLiN achieves optimal regret rate of Lipschitz bandits byusing very few batches.
Moreover, our lower-bound analysis shows that Ω(log log T ) batches are necessary for any algorithmto achieve the optimal regret rate.
Thus, BLiN is optimal in terms of both regret and communication.
1.3 Related Works
The history of the Multi-Armed Bandit (MAB) problem can date back to Thompson [41].
Solvers forthis problems include the UCB algorithms [27, 4, 7], the arm elimination method [19, 31, 35], theϵ -greedy strategy [7, 39], the exponential weights and mirror descent framework [8].
Recently, with the prevalence of distributed computing and large-scale field experiments, the settingof batched feedback has captured attention [e.g., 16].
Perchet et al.
[32] mainly consider batchedbandit with two arms, and a matching lower bound for static grid is proved.
It was then generalizedby Gao et al.
[20] to finite-armed bandit problems.
In their work, the authors designed an eliminationmethod for finite-armed bandit problem and proved matching lower bounds for both static andadaptive grid.
Soon afterwards, Zhang et al.
[44] studies inference for batched bandits.
Esfandiari etal.
[18] studies batched linear bandits and batched adversarial bandits.
Han et al.
[21] and Ruan etal. [34] provide solutions for batched contextual linear bandits.
Li and Scarlett [28] studies batched
Gaussian process bandits.
Batched dueling bandits have also been studied by [2].
Parallel to theregret control regime, best arm identification with limited number of batches was studied in [1] and[22].
Top-
k arm identification in the collaborative learning framework is also closely related to thebatched setting, where the goal is to minimize the number of iterations (or communication steps)between agents.
In this setting, tight bounds have been obtained in the recent works [40, 23].
Yet theproblem of Lipschitz bandit with communication constraints remains unsolved.
The Lipschitz bandit problem is important in its own stand.
The Lipschitz bandit problem wasintroduced as “continuum-armed bandits” [3], where the arm space is a compact interval.
Along thisline, bandits that are Lipschitz (or Hölder) continuous have been studied.
For this problem, Kleinberg[24] proves a Ω( T 2 / 3 )
lower bound and introduced a matching algorithm.
Under extra conditionson top of Lipschitzness, regret rate of (cid:101)
O
( T 1 / 2 ) was achieved [9, 17].
For general (doubling) metricspaces, the Zooming bandit algorithm [25] and the Hierarchical Optimistic Optimization (HOO)algorithm [14] were developed.
In more recent years, some attention has been focused on Lipschitzbandit problems with certain extra structures.
To name a few, Bubeck et al.
[15] study Lipschitzbandits for differentiable rewards, which enables algorithms to run without explicitly knowing the
Lipschitz constants.
Magureanu et al.
[30] derive a new concentration inequality and study discrete
Lipschitz bandits.
The idea of robust mean estimators [11, 5, 12] was applied to the Lipschitz banditproblem to cope with heavy-tail rewards, leading to the development of a near-optimal algorithm for
Lipschitz bandit with heavy-tailed rewards [29].
Lipschitz bandits where a clustering is used to inferthe underlying metric, has been studied by [43].
Contextual Lipschitz bandits have also been studiedby [38] and [26].
Yet all of the existing works for Lipschitz bandits assume that the reward sample isimmediately observed after each arm pull, and none of them solve the Lipschitz bandit problem withcommunication constraints.
This paper is organized as follows.
In section 2, we introduce the BLiN algorithm and give a visualillustration of the algorithm procedure.
In section 3, we prove that BLiN with ACE Sequence achievesthe optimal regret rate using only O (log log T ) rounds of communications.
Section 4 providesinformation-theoretical lower bounds for Lipschitz bandits with communication constraints, whichshows that BLiN is optimal in terms of both regret and rounds of communications.
Experimentalresults are presented in Section 5.
Algorithm
With communication constraints, the agent’s knowledge about the environment does not accumulatewithin each batch.
This characteristic of the problem suggests a ‘uniform’ type algorithm – weshall treat each step within the same batch equally.
Following this intuition, in each batch, weuniformly play the remaining arms, and then eliminate arms of low reward after the observations arecommunicated.
Next we describe the uniform play rule and the arm elimination rule.
Uniform Play Rule:
At the beginning of each batch m , a collection of subsets of the arm space
A m =
{ C m, 1 , C m, 2 , · · · , C m, |A
m | } is constructed.
This collection of subset A m consists of standardcubes, and all cubes in A m have the same edge length r m .
We will detail the construction of A mwhen we describe the arm elimination rule.
We refer to cubes in A m as active cubes of batch m .
During batch m , each cube in A m is played n m :=
16log T r 2 m times, where T is the total time horizon.
More specifically, within each C ∈
A m , arms x C, 1 , x C, 2 , · · · , x C,n m ∈ C are played.
The rewardsamples { y C, 1 , y C, 2 , · · · , y C,n m } C ∈A m corresponding to { x C, 1 , x C, 2 , · · · , x C,n m } C ∈A m will becollected at the end of the this batch.
Arm Elimination Rule: At the end of batch m , information from the arm pulls is collected, andwe estimate the reward of each C ∈
A m by (cid:98) µ m ( C ) =
1 n m (cid:80) n m i =1 y C,i .
Cubes of low estimatedrewards are then eliminated, according to the following rule: a cube C ∈
A m is eliminated if(cid:98) µ max m − (cid:98) µ m ( C ) ≥ 4 r m , where (cid:98) µ max m := max C ∈A m (cid:98) µ m ( C ) .
After necessary removal of “bad cubes”,each cube in A m that survives the elimination is equally partitioned into (cid:16) r m r m +1 (cid:17) d subcubes of edgelength r m +1 , where r m +1 is predetermined.
These cubes (of edge length r m +1 ) are collected toconstruct A m +1 , and the learning process moves on to the next batch.
Appropriate rounding may berequired to ensure the ratio r m r m +1 is an integer.
See Remark 2 for more details.
The learning process is summarized in Algorithm 1.
The following theorem gives regret and communication upper bound of BLiN with Doubling Edgelength Sequence r m = 2 − m +1 (see Appendix B for proof).
Note that this result implies Theorem1.
T
Theorem 3.
With probability exceeding 1 − 2 T 6 , the T -step total regret R ( T ) of BLiN with Doubling
Edge-length Sequence (D-BLiN) satisfies
[Equation]
where d z is the zooming dimension of the problem instance.
In addition, D-BLiN only needs no morethan log
T − loglog T d z + + 2 rounds of communications to achieve this regret rate.
Although D-BLiN efficiently solves batched Lipschitz bandits, its simple partition strategy leadsto suboptimal communication complexity.
Now we show that by approporiately combining somebatches, BLiN achieves the optimal communication bound, without incurring increasing regret.
Specifically, we introduce the following edge-length sequence, which we call ACE Sequence.
Definition 1.
For a problem with ambient dimension d , zooming dimension d z and time horizon T ,we denote c 1 = d z +1 ( d +2)
( d z +2) log T log T and c i +1 =
ηc i for any i ≥ 1 , where η = d +1 − d z d +2 .
Then
the
Appropriate Combined Edge-length Sequence { r m } is defined by r m = 2 − (cid:80) mi =1 c i for any m ≥ 1 .
We show that BLiN with ACE Sequence (A-BLiN) obtains an improved communication complexity,thus proves Theorem 2.
Theorem 4.
With probability exceeding 1 − 2 T 6 , the T -step total regret R ( T ) of Algorithm 1 satisfies
[Equation]
where d z is the zooming dimension of the problem instance.
In addition, Algorithm 1 only needs nomore than loglog T log d +2 d +1 − dz + 1 rounds of communications to achieve this regret rate.
The partition and elimination process of a real A-BLiN run is in Figure 1.
In the i -th subgraph, thewhite cubes are those remaining after the ( i − 1) -th batch.
In this experiment, we set A = [0 , 1] 2 ,and the optimal arm is x ∗ =
Note that x ∗ is not eliminated during the game.
More detailsof this experiment are in Section 5.
Remark 1 (Time and space complexity) .
The time complexity of our algorithm is O ( T ) , which isbetter than the state of the art
O
( T log T ) in [14].
This is because that the running time of a batch jis of order O
( l j ) , where l j = t j − t j − 1 is number of samples in batch j .
Since (cid:80) j l j = T , the timecomplexity of BLiN is O ( T ) .
Besides, the space complexity of BLiN is also improved, because we donot need to store information of cubes in previous batches.
Regret Analysis of A-BLiN
In this section, we provide regret analysis for A-BLiN.
The highlight of the finding is that O (log log T )batches are sufficient to achieve optimal regret rate of (cid:101) O (cid:16) T
dz +1 dz +2 (cid:17) , as summarized in Theorem 4.
To prove Theorem 4, we first show that the estimator (cid:98) µ is concentrated to the true expected reward µ(Lemma 1), and the optimal arm survives all eliminations with high probability (Lemma 2).
In thefollowing analysis, we let B stop be the total number of batches of the BLiN run.
Lemma 1.
Define
[Equation]
It holds that P ( E ) ≥ 1 − 2 T − 6 .
Lemma 2.
Under event E , the optimal arm x ∗ = arg max µ ( x ) is not eliminated after the first
B stop − 1 batches.
Based on these results, we show the cubes that survive elimination are of high reward.
Lemma 3.
Under event E , for any 1 ≤ m ≤ B stop , any C ∈ A m and any x ∈ C , ∆ x satisfies∆ x ≤ 8 r m − 1 .
[Equation]
The proofs of Lemma 1-3 are in Appendix A.
We are now ready to prove the theorem.
Proof of Theorem 4.
Let R m denote regret of the m -th batch.
Fixing any positive number B , thetotal regret R ( T ) can be divided into two parts: R ( T ) =
(cid:80) m ≤ B R m + (cid:80) m>B R m .
In the following,we bound these two parts separately and then determine B to obtain the upper bound of the totalregret.
Moreover, we show A-BLiN uses only O (log log T ) rounds of communications to achieve theoptimal regret.
Recall that A m is set of the active cubes in the m -th batch.
According to Lemma 3, for anyx ∈ ∪ C ∈A m C , we have ∆ x ≤ 8 r m − 1 .
Let A + m be set of cubes not eliminated in batch m .
Then anycube in A + m − 1 is a subset of S (8 r m − 1 ) , and thus
[Equation]
By definition, r m = r m − 1 2 − c m , so
[Equation]
The total regret of the m -th batch is
[Equation]
where (i) follows from (5), (ii) follows from (4), and (iii) follows from the definition of ACE Sequence.
Define C m =
( (cid:80) m − 1 i =1 c i )( d z + 1)
+ c m ( d + 2) .
Since c m = c m − 1 · d +1 − d z d +2 , calculation shows that
C m =
( (cid:80) m − 2 i =
c
i )( d z + 1)
+ c m − 1 ( d + 2) + c m − 1 ( d z + 1 − d − 2)
+ c m ( d + 2) =
C m − 1 .
Thusfor any m , we have C m =
C 1 =
c 1 ( d + 2) .
Hence,
[Equation]
The inequality (6) holds even if the m -th batch does not exist (where we let R m = 0 ) or is notcompleted.
Thus we obtain the first upper bound (cid:80) m ≤ B [Equation] m
[Equation]
Lemma 3 implies that any arm x played after the first B batches satisfies ∆ x ≤ 8 r B , so the totalregret after B batches is bounded by(cid:88)
[Equation]
Therefore, the total regret R ( T ) satisfies
[Equation]
This inequality holds for any positive B .
Then by choosing B ∗ = loglog T − log( d z +2) log 1 η =loglog T − log( d z +2) log
d + d +1 − dz , we have η B ∗ d z +2 = 1 log T and
[Equation]
The above analysis implies that we can achieve the optimal regret rate (cid:101) O (cid:16) T
dz +1 dz +2 (cid:17) by letting thefor-loop run B ∗ times and finishing the remaining rounds in the Cleanup step.
In other words, B ∗ + 1rounds of communications are sufficient for A-BLiN to achieve the regret bound (2).
Remark 2.
The quantity r m r m +1 in line 9 of Algorithm 1 may not be integers for some m .
Thus,in practice we denote α n =
⌊ (cid:80) ni
=1 c i ⌋ , β n = ⌈ (cid:80) ni
=1 c i ⌉ , and define rounded ACE Sequence{ (cid:101)
r m } m ∈ N by (cid:101) r m = 2 − α k for m = 2 k − 1 and (cid:101) r m = 2 − β k for m = 2 k .
Then the total regret canbe divided as R ( T ) =
(cid:80) 1 ≤ k ≤ B ∗ R 2 k − 1 + (cid:80) 1 ≤ k ≤ B ∗ R 2 k + (cid:80) m> 2 B ∗ R m .
For the first part wehave (cid:101) r 2 k − 2 ≤ r k − 1 and (cid:101) r 2 k − 1 ≥ r k , while for the second part we have (cid:101) r 2 k − 1 (cid:101) r 2 k = 2 .
Therefore, bysimilar argument to the proof of Theorem 4, we can bound these three parts separately, and concludethat BLiN with rounded ACE sequence achieves the optimal regret bound (cid:101)
O ( T
dz +1 dz +2 ) by using only
O (log log T ) rounds of communications.
The details are in Appendix C.
Lower Bounds
In this section, we present lower bounds for Lipschitz bandits with batched feedback, which in turngives communication lower bounds for all Lipschitz bandit algorithms.
Our lower bounds depend onthe rounds of communications B .
When B is sufficiently large, our results match the lower boundfor the vanilla Lipschitz bandit problem (cid:101) Θ( R z ( T ))
( R z ( T ) is defined in Eq. 1).
More importantly,this dependency on B gives the minimal rounds of communications needed to achieve optimal regretbound for all Lipschitz bandit algorithms, which is summarized in Corollary 2.
Since this lowerbound matches the upper bound presented in Theorem 4, BLiN optimally solves Lipschitz banditswith minimal communication.
4.1 Proof Outline
Similar to most lower bound proofs, we need to construct problem instances that are difficult todifferentiate.
What’s different is that we need to carefully integrate batched feedback pattern [32] withthe Lipschitz payoff reward [38, 29].
To capture the adaptivity in grid determination, we construct“static reference communication grids” to remove the stochasticity in grid selection [1, 20].
Below, wefirst consider the static grid case, where the grid is predetermined.
This static grid case will provideintuition for the adaptive and more general case.
The expected reward functions of these instances are constructed as follows: we choose some‘position’ and ‘height’, such that the expected reward function obtains local maximum of the specified‘height’ at the specified ‘position’.
We will use the word ‘peak’ to refer to the local maxima.
Thefollowing theorem presents the lower bound for the static grid case.
Theorem 5 (Lower Bound for Static Grid) .
Consider Lipschitz bandit problems with time horizon Tsuch that the grid of reward communication T is static and determined before the game.
If B roundsof communications are allowed, then for any policy π , there exists a problem instance such that
[Equation]
where c > 0 is a numerical constant independent of B , T , π and T ,
R z ( T ) is defined in (1), and d isthe dimension of the arm space.
To prove Theorem 5, we first show that for any k > 1 there exists an instance such that E [ R T ( π )] ≥t k t 1 d +2 k − 1 .
Fixing k > 1 , we let r k =
[Equation]
1 d +2 k
[Equation]
1 and M k :=
t k − 1 r 2 k = 1 r dk .
Then we construct a set of probleminstances
I k
= { I k, 1 , · · · , I k,M k } , such that the gap between the highest peak and the second highestpeak is about r k for every instance in I k .
Based on this construction, we prove that no algorithm can distinguish instances in I k from oneanother in the first ( k − 1) batches, so the worst-case regret is at least r k t k , which gives the inequalitywe need.
For the first batch (0 , t 1 ] , we can easily construct a set of instances where the worst-caseregret is at least t 1 , since no information is available during this time.
Thus, there exists a probleminstance such that E [ R T ( π )] ≳ max (cid:40) t 1 , tt 1 d + , · · · , t B t 1 d +2 B −(cid:41) .
Since 0 < t 1 < · · · < t B = T , theinequality in Theorem 5 follows.
As a result of Theorem 5, we can derive the minimum rounds of communications needed to achieveoptimal regret bound for Lipschitz bandit problem, which is stated in Corollary 1.
Corollary 1.
Any Lipschitz bandit algorithm needs Ω(log log T ) rounds of communications to achievethe optimal regret rate, for the case that the times of reward communication are predetermined andstatic.
The detailed proof of Theorem 5 and Corollary 1 are deferred to Appendix D and E.
4.2 Communication Lower Bound for BLiN
So far we have derived lower bounds for the static grid case.
Yet there is a gap between the static andthe adaptive case.
We will close this gap in the following Theorem.
Theorem 6 (Lower Bound for Adaptive Grid) .
Consider Lipschitz bandit problems with time horizon
T such that the grid of reward communication T is adaptively determined by the player.
If B roundsof communications are allowed, then for any policy π , there exists an instance such that
[Equation]
where c > 0 is a numerical constant independent of B , T , π and T ,
R z ( T ) is defined in (1), and d isthe dimension of the arm space.
To prove Theorem 6, we consider a reference static grid T r =
{ T 0 , T 1 , · · · , T B } , where T j = T 1 − εj 1 − εBfor ε = 1 d
+
We construct a series of ‘worlds’, denoted by I 1 , · · · , I B .
Each world is a set of problem instances, and each problem instance in world I j is defined by peak location set U j andbasic height r j , where the sets U j and quantities r j for 1 ≤ j ≤ B are presented in Appendix F.
Based on these constructions, we first prove that for any adaptive grid and policy, there exists anindex j such that the event A j = { t j − 1 < T j − 1 , t j ≥ T j } happens with sufficiently high probabilityin world
I j .
Then similar to Theorem 5, we prove that in world I j there exists a set of probleminstances that is difficult to differentiate in the first j − 1 batches.
In addition, event A j implies thatt j ≥ T j , so the worst-case regret is at least r j T j , which gives the lower bound we need.
The proof of Theorem 6 is deferred to Appendix F. Similar to Corollary 1, we can prove that at leastΩ(log log T ) rounds of communications are needed to achieve optimal regret bound.
This result isformally summarized in Corollary 2.
Corollary 2.
Any Lipschitz bandit algorithm 2 needs Ω(log log T ) rounds of communications toachieve the optimal regret rate.
Experiments
In this section, we present numerical studies of A-BLiN.
In the experiments, we use the arm space
A = [0 , 1] 2 and the expected reward function µ ( x ) = 1 − 12 ∥ x − x 1 ∥ 2 − 310 ∥ x − x 2 ∥ 2 , wherex 1 = (0 . 8 , 0 . 7) and x 2 = (0 . 1 , 0 . 1) .
The landscape of µ and the resulting partition is shown in
Figure 2(a).
As can be seen, the partition is finer in the area closer to the optimal arm x ∗ = (0 . 8 , 0 . 7) .
[Figure]
show accumulated regret of A-BLiN and zooming algorithm [25].
In the figure, different background colors represent different batches of A-BLiN.
For the total time horizon T = 80000 , A-BLiN needsrounds of communications.
We let the time horizon T = 80000 , and report the accumulated regret in Figure 2(b).
The regretcurve is sublinear, which agrees with the regret bound (2).
Besides, different background colorsin Figure 2(b) represent different batches.
For the total time horizon T = 80000 , A-BLiN onlyneeds 4 rounds of communications.
We also present regret curve of zooming algorithm
[25] forcomparison.
Different from zooming algorithm, regret curve of A-BLiN is approximately piecewiselinear, which is because the strategy of BLiN does not change within each batch.
Results of morerepeated experiments are in Appendix G, as well as experimental results of D-BLiN.
Conclusion
In this paper, we study Lipschitz bandits with communication constraints, and propose the BLiNalgorithm as a solution.
We prove that BLiN only need O (log log T ) rounds of communicationsto achieve the optimal regret rate of best previous Lipschitz bandit algorithms [25, 13] that need
T batches.
This improvement in number of the batches significantly saves data communicationcosts.
We also provide complexity analysis for this problem.
We show that Ω(log log T ) roundsof communications are necessary for any algorithm to optimally solve Lipschitz bandit problems.
Hence, BLiN algorithm is optimal.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes](c)
Did you discuss any potential negative societal impacts of your work?
[N/A](d)
Have you read the ethics review guidelines and ensured that your paper conforms to them?[Yes]
If you are including theoretical results
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental
results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e g., data splits, hyperparameters, how they werechosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experimentsmultiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e g., type of GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board
(IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent onparticipant compensation?
[N/A]