Lipschitz Bandits with Batched Feedback
Abstract
In this paper, we study Lipschitz bandit problems with batched feedback, where the expected reward is Lipschitz and the reward observations are communicated to the player in batches.
We introduce a novel landscape-aware algorithm, called Batched
Lipschitz Narrowing (BLiN), that optimally solves this problem.
Specifically, we show that for a T -step problem with Lipschitz reward of zooming dimension d z , our algorithm achieves theoretically optimal (up to logarithmic factors) regret rate (cid:101) O (cid:16) T
dz +1 dz +2 (cid:17) using only O (log log T ) batches.
We also provide complexity analysis for this problem.
Our theoretical lower bound implies that Ω(log log T ) batches are necessary for any algorithm to achieve the optimal regret.
Thus, BLiN achieves optimal regret rate using minimal communication.
Introduction
Multi-Armed Bandit (MAB) algorithms aim to exploit the good options while explore the decision space.
These algorithms and methodologies find successful applications in artificial intelligence and reinforcement learning [e g., 37].
While the classic MAB setting assumes that the rewards are immediately observed after each arm pull, real-world data often arrives in different patterns.
For example, observations from clinical trials are often be collected in a batched fashion [34].
Another example is from online advertising, where strategies are tested on multiple customers at the same time [10].
In such cases, any observation-dependent decision-making should comply with this data-arriving pattern, including MAB algorithms.
In this paper we study the Lipschitz bandit problem with batched feedback – a MAB problem where the expected reward is Lipschitz and the reward observations are communicated to the player in batches.
In such settings, rewards are communicated only at the end of the batches and the algorithm can only make decisions based on information up to the previous batch.
Existing Lipschitz bandit algorithms heavily rely on timely access to the reward samples, since the partition of arm space may change at any time.
Therefore, they can not solve the batched feedback setting.
To address this difficulty, we present a novel adaptive algorithm for Lipschitz bandits with communication constraints, named Batched Lipschitz Narrowing (BLiN).
BLiN learns the landscape of the reward by adaptively narrowing the arm set, so that regions of high reward are more frequently played.
Also,
BLiN determines the data collection procedure adaptively, so that only very few data communications are needed.
The above BLiN procedure achieves optimal regret rate (cid:101) O (cid:16) T
dz +1 dz +2 (cid:17) ( d z is the zooming dimension [26, 15]), and can be implemented in a clean and easy-to-implement form.
In addition to achieving the optimal regret rate, BLiN is also optimal in the following senses:
• BLiN’s communication complexity is optimal.
BLiN only needs O (log log T ) rounds of communications to achieve the optimal regret rate (Theorem 2), and no algorithm can achieve this rate with fewer than Ω(log log T ) rounds of communications (Corollary 2).
BLiN’s time complexity is optimal (Remark 1): if the arithmetic operations and sampling are of complexity O (1) , then the time complexity of BLiN is O ( T ) , which improve the best known time complexity O
( T log T ) for Lipschitz bandit problems [15].
1.1 Settings & Preliminaries
For a Lipschitz bandit problem (with communication constraints), the arm set is a compact doubling metric space ( A , d A ) .
The expected reward µ : A → R is 1 -Lipschitz with respect to the metric d A , that is, | µ ( x 1 ) − µ ( x 2 ) | ≤ d A ( x 1 , x 2 ) for any x 1 , x 2 ∈ A .
At time t ≤ T , the learning agent pulls an arm x t ∈ A that yields a reward sample y t = µ ( x t ) + ϵ t , where ϵ t is a mean-zero independent sub-Gaussian noise.
Without loss of generality, we assume that ϵ t ∼ N (0 , 1) , since generalizations to other sub-Gaussian noises are not hard.
Similar to most bandit learning problems, the agent seeks to minimize regret in the batched feedback environment.
The regret is defined as R ( T ) =
(cid:80) Tt =1 ( µ ∗ − µ ( x t )) , where µ ∗ denotes max x ∈A µ ( x ) .
For simplicity, we define ∆ x = µ ∗ − µ ( x ) (called optimality gap of x ) for all x ∈ A .
1.1.1 Doubling Metric Spaces and
the ([0 , 1] d , ∥ · ∥ ∞ ) Metric Space
By the Assouad’s embedding theorem [6], the (compact) doubling metric space ( A , d A ) can be embedded into a Euclidean space with some distortion of the metric; See [43] for more discussions in a machine learning context.
Due to existence of such embedding, the metric space ([0 , 1] d , ∥ · ∥ ∞ ) , where metric balls are hypercubes, is sufficient for the purpose of our paper.
For the rest of the paper, we will use hypercubes in algorithm design for simplicity, while our algorithmic idea generalizes to other doubling metric spaces.
1.1.2 Zooming Numbers and Zooming Dimensions
An important concept for bandit problems in metric spaces is the zooming number and the zooming dimension [26, 14, 39], which we discuss now.
Define the set of r -optimal arms as S ( r ) = { x ∈ A : ∆ x ≤ r } .
For any r = 2 − i , the decision space [0 , 1] d can be equally divided into 2 di cubes with edge length r , which we call standard cubes (also referred to as dyadic cubes).
The r -zooming number is defined as
N r :=
# { C : C is a standard cube with edge length r and C ⊂ S (16 r ) } .
In words, N r is the r -packing number of the set S (16 r ) in terms of standard cubes.
The zooming dimension is then defined as
[Equation]
some
[Equation] ∈ N [Equation] Moreover, we define the zooming constant C z as the minimal a to make the above inequality true for d z , C z = min { a > 0 : N r ≤ ar − d z , ∀ r = 2 − i for some i ∈ N } .
Zooming dimension d z can be significantly smaller than ambient dimension d and can be zero.
For a simple example, consider a problem with ambient dimension d = 1 and expected reward function µ ( x ) =
x for 0 ≤ x ≤ 1 .
Then for any r = 2 − i with i ≥ 4 , we have S (16 r ) =
[1 − 16 r, 1] and
N r = 16 .
Therefore, for this problem the zooming dimension equals to 0 , with zooming constant
C z = 16 .
1.2 Batched feedback pattern and our results
In the batched feedback setting, for a T -step game, the player determines a grid T
= { t 0 , · · · , t B } adaptively , where
0 =
t 0 < t 1 < · · · < t B = T and B ≪ T .
During the game, reward observations are communicated to the player only at the grid points t 1 , · · · , t B .
As a consequence, for any time t in the j -th batch, that is, t j − 1 < t ≤ t j , the reward y t cannot be observed until time t j , and the decision made at time t depends only on rewards up to time t j − 1 .
The determination of the grid T is adaptive in the sense that the player chooses each grid point t j ∈ T based on the operations and observations up to the previous point t j − 1 .
In this work, we present BLiN algorithm to solve Lipschitz bandits under batched feedback.
During the learning procedure, BLiN detects and eliminates the ‘bad area’ of the arm set in batches and partition the remaining area according to an approporiate edge-length sequence .
Our first theoretical upper bound is that with simple Doubling Edge-length Sequence r m = 2 − m +1 , BLiN achieves optimal regret rate (cid:101) O (cid:16) T
dz +1 dz +2 (cid:17) by using O (log T ) batches.
Theorem 1.
With probability exceeding 1 − 2 T 6 , the T -step total regret R ( T ) of BLiN with Doubling
Edge-length Sequence (D-BLiN) satisfies
[Equation]
where d z is the zooming dimension of the problem instance.
In addition, D-BLiN only needs no more than O (log T ) rounds of communications to achieve this regret rate.
Here and henceforth, ≲ only omits constants.
While D-BLiN is efficient for batched Lipschitz bandits, its communication complexity is not optimal.
We then propose a new edge-length sequence, which we call Appropriate Combined Edge-length
Sequence (ACE Sequence) to improve the algorithm.
The idea behind this sequence is that by appropriately combining some batches, the algorithm can achieve better communication bound without incurring increased regret.
As we shall see, BLiN with ACE Sequence (A-BLiN) achieves regret rate (cid:101) O (cid:16) T dz +1 dz +2 (cid:17) with only O (log log T ) batches.
Theorem 2.
With probability exceeding 1 − 2 T 6 , the T -step total regret R ( T ) of A-BLiN satisfies
[Equation]
where d z is the zooming dimension of the problem instance.
In addition, Algorithm 1 only needs no more than O (log log T ) rounds of communications to achieve this regret rate.
As a comparison, seminal works [26, 39, 15] show that the optimal regret bound for Lipschitz bandits without communications constraints, where the reward observations are immediately observable after each arm pull, is R ( T ) ≲ T dz +1 dz +2 · (log T ) 1 dz +2 in terms of zooming dimension, and
[Equation]
in terms of zooming number.
Therefore, A-BLiN achieves optimal regret rate of Lipschitz bandits by using very few batches.
Moreover, our lower-bound analysis shows that Ω(log log T ) batches are necessary for any algorithm to achieve the optimal regret rate.
Thus, BLiN is optimal in terms of both regret and communication.
1.3 Related Works
The history of the Multi-Armed Bandit (MAB) problem can date back to Thompson [42].
Solvers for this problems include the UCB algorithms [28, 4, 7], the arm elimination method [20, 32, 36], the ϵ -greedy strategy [7, 40], the exponential weights and mirror descent framework [8].
Recently, with the prevalence of distributed computing and large-scale field experiments, the setting of batched feedback has captured attention [e.g., 17].
Perchet et al.
[33] mainly consider batched bandit with two arms, and a matching lower bound for static grid is proved.
It was then generalized by Gao et al.
[21] to finite-armed bandit problems.
In their work, the authors designed an elimination method for finite-armed bandit problem and proved matching lower bounds for both static and adaptive grid.
Soon afterwards, Zhang et al.
[46] studies inference for batched bandits.
Esfandiari et al.
[19] studies batched linear bandits and batched adversarial bandits.
Han et al.
[22] and Ruan et al.
[35] provide solutions for batched contextual linear bandits.
Li and Scarlett [29] studies batched
Gaussian process bandits.
Batched dueling bandits have also been studied by [2].
Parallel to the regret control regime, best arm identification with limited number of batches was studied in [1] and [23].
Top-
k arm identification in the collaborative learning framework is also closely related to the batched setting, where the goal is to minimize the number of iterations (or communication steps) between agents.
In this setting, tight bounds have been obtained in the recent works [41, 24].
Yet the problem of Lipschitz bandit with communication constraints remains unsolved.
The Lipschitz bandit problem is important in its own stand.
The Lipschitz bandit problem was introduced as “continuum-armed bandits” [3], where the arm space is a compact interval.
Along this line, bandits that are Lipschitz (or Hölder) continuous have been studied.
For this problem, Kleinberg [25] proves a Ω( T 2 / 3 )
lower bound and introduced a matching algorithm.
Under extra conditions on top of Lipschitzness, regret rate of (cid:101)
O
( T 1 / 2 ) was achieved [9, 18].
For general (doubling) metric spaces, the Zooming bandit algorithm [26] and the Hierarchical Optimistic Optimization (HOO) algorithm [15] were developed.
In more recent years, some attention has been focused on Lipschitz bandit problems with certain extra structures.
To name a few, Bubeck et al.
[16] study Lipschitz bandits for differentiable rewards, which enables algorithms to run without explicitly knowing
the
Lipschitz constants.
Wang et al.
[44] studied discretization-based Lipschitz bandit algorithms from a Gaussian process perspective.
Magureanu et al.
[31] derive a new concentration inequality and study discrete Lipschitz bandits.
The idea of robust mean estimators [11, 5, 13] was applied to the Lipschitz bandit problem to cope with heavy-tail rewards, leading to the development of a nearoptimal algorithm for Lipschitz bandit with heavy-tailed rewards [30].
Lipschitz bandits where a clustering is used to infer the underlying metric, has been studied by [45].
Contextual Lipschitz bandits have also been studied by [39] and [27].
Yet all of the existing works for Lipschitz bandits assume that the reward sample is immediately observed after each arm pull, and none of them solve the Lipschitz bandit problem with communication constraints.
This paper is organized as follows.
In section 2, we introduce the BLiN algorithm and give a visual illustration of the algorithm procedure.
In section 3, we prove that BLiN with ACE Sequence achieves the optimal regret rate using only O (log log T ) rounds of communications.
Section 4 provides information-theoretical lower bounds for Lipschitz bandits with communication constraints, which shows that BLiN is optimal in terms of both regret and rounds of communications.
Experimental results are presented in Section 5.
Algorithm
With communication constraints, the agent’s knowledge about the environment does not accumulate within each batch.
This characteristic of the problem suggests a ‘uniform’ type algorithm – we shall treat each step within the same batch equally.
Following this intuition, in each batch, we uniformly play the remaining arms, and then eliminate arms of low reward after the observations are communicated.
Next we describe the uniform play rule and the arm elimination rule.
Uniform Play Rule:
At the beginning of each batch m , a collection of subsets of the arm space
A m =
{ C m, 1 , C m, 2 , · · · , C m, |A
m | } is constructed.
This collection of subset A m consists of standard cubes, and all cubes in A m have the same edge length r m .
We will detail the construction of A m when we describe the arm elimination rule.
We refer to cubes in A m as active cubes of batch m .
During batch m , each cube in A m is played n m :=
16log T r 2 m times, where T is the total time horizon.
More specifically, within each C ∈
A m , arms x C, 1 , x C, 2 , · · · , x C,n m ∈ C are played.
The reward samples { y C, 1 , y C, 2 , · · · , y C,n m } C ∈A m corresponding to { x C, 1 , x C, 2 , · · · , x C,n m } C ∈A m will be collected at the end of the this batch.
Arm Elimination Rule: At the end of batch m , information from the arm pulls is collected, and we estimate the reward of each C ∈
A m by (cid:98) µ m ( C ) =
1 n m (cid:80) n m i =1 y C,i .
Cubes of low estimated rewards are then eliminated, according to the following rule: a cube C ∈
A m is eliminated if (cid:98) µ max m − (cid:98) µ m ( C ) ≥ 4 r m , where (cid:98) µ max m := max C ∈A m (cid:98) µ m ( C ) .
After necessary removal of “bad cubes”, each cube in A m that survives the elimination is equally partitioned into (cid:16) r m r m +1 (cid:17) d subcubes of edge length r m +1 , where r m +1 is predetermined.
These cubes (of edge length r m +1 ) are collected to construct A m +1 , and the learning process moves on to the next batch.
Appropriate rounding may be required to ensure the ratio r m r m +1 is an integer.
See Remark 2 for more details.
The learning process is summarized in Algorithm 1.
1: Input.
Arm set A =
[0 , 1] d ; time horizon T .
2: Initialization Number of batches B ; Edge-length sequence { r m } B +1 m =1 ; The first grid point t 0 = 0 ;
Equally partition A to r d 1 subcubes and define A 1 as the collection of these subcubes.
3: Compute n m = 16log T r 2 m for m = 1 , · · · , B + 1 . 4: for m = 1 , 2 , · · · , B do 5: For each cube C ∈ A m , play arms x C, 1 , · · · x C,n m from C .
Collect the rewards of all pulls up to t m .
Compute the average payoff (cid:98) µ m ( C ) =
(cid:80) nmi
=1 y C,i n m for each cube C ∈
A m .
Find (cid:98) µ maxm = max C ∈A m (cid:98) µ ( C ) .
For each cube C ∈
A m , eliminate C if (cid:98) µ maxm − (cid:98) µ m ( C ) > 4 r m .
Let A + m be set of cubes not eliminated.
{+Compute t m +
+}
=
t m + ( r m /r m +1 ) d · |A + m | · n m +1 . If t m +1 ≥ T or m = B then break .
Equally partition each cube in A + m into ( r m /r m +1 ) d subcubes and define A m +1 as the collection of these subcubes.
/* See Remark 2 for more details on cases where ( r m /r m +1 ) d is not an integer. */ 10: end for 11: Cleanup: Arbitrarily play the remaining arms until all T steps are used. The following theorem gives regret and communication upper bound of BLiN with Doubling Edgelength Sequence r m = 2 − m +1 (see Appendix B for proof).
Note that this result implies Theorem 1.
Theorem 3.
With probability exceeding 1 − 2 T 6 , the T -step total regret R ( T ) of BLiN with Doubling
Edge-length Sequence (D-BLiN) satisfies
[Equation]
where d z is the zooming dimension of the problem instance.
In addition, D-BLiN only needs no more than log
T − loglog T d z + + 2 rounds of communications to achieve this regret rate.
Although D-BLiN efficiently solves batched Lipschitz bandits, its simple partition strategy leads to suboptimal communication complexity.
Now we show that by approporiately combining some batches, BLiN achieves the optimal communication bound, without incurring increasing regret.
Specifically, we introduce the following edge-length sequence, which we call ACE Sequence.
Definition 1.
For a problem with ambient dimension d , zooming dimension d z and time horizon T , we denote c 1 = d z +1 ( d +2)
( d z +2) log T log T and c i +1 =
ηc i for any i ≥ 1 , where η = d +1 − d z d +2 .
Then
the
Appropriate Combined Edge-length Sequence { r m } is defined by r m = 2 − (cid:80) mi =1 c i for any m ≥ 1 .
We show that BLiN with ACE Sequence (A-BLiN) obtains an improved communication complexity, thus proves Theorem 2.
Theorem 4.
With probability exceeding 1 − 2 T 6 , the T -step total regret R ( T ) of Algorithm 1 satisfies
[Equation]
where d z is the zooming dimension of the problem instance.
In addition, Algorithm 1 only needs no more than loglog T log d +2 d +1 − dz + 1 rounds of communications to achieve this regret rate.
The partition and elimination process of a real A-BLiN run is in Figure 1.
In the i -th subgraph, the white cubes are those remaining after the ( i − 1) -th batch.
In this experiment, we set A = [0 , 1] 2 , and the optimal arm is x ∗ =
Note that x ∗ is not eliminated during the game.
More details of this experiment are in Section 5.
Remark 1 (Time and space complexity) .
The time complexity of our algorithm is O ( T ) , which is better than the state of the art O
( T log T ) in [15].
This is because that the running time of a batch j is of order O
( l j ) , where l j = t j − t j − 1 is number of samples in batch j .
Since (cid:80) j l j = T , the time complexity of BLiN is O ( T ) .
Besides, the space complexity of BLiN is also improved, because we do not need to store information of cubes in previous batches.
Regret Analysis of A-BLiN
In this section, we provide regret analysis for A-BLiN.
The highlight of the finding is that O (log log T ) batches are sufficient to achieve optimal regret rate of (cid:101) O (cid:16) T
dz +1 dz +2 (cid:17) , as summarized in Theorem 4.
To prove Theorem 4, we first show that the estimator (cid:98) µ is concentrated to the true expected reward µ (Lemma 1), and the optimal arm survives all eliminations with high probability (Lemma 2).
In the following analysis, we let B stop be the total number of batches of the BLiN run.
Lemma 1.
Define
[Equation]
∈ C (cid:41) .
It holds that P ( E ) ≥ 1 − 2 T − 6 .
Lemma 2.
Under event E , the optimal arm x ∗ = arg max µ ( x ) is not eliminated after the first
B stop − 1 batches.
Based on these results, we show the cubes that survive elimination are of high reward.
Lemma 3.
Under event E , for any 1 ≤ m ≤ B stop , any C ∈ A m and any x ∈ C , ∆ x satisfies
[Equation]
The proofs of Lemma 1-3 are in Appendix A.
We are now ready to prove the theorem.
Proof of Theorem 4.
Let R m denote regret of the m -th batch.
Fixing any positive number B , the total regret R ( T ) can be divided into two parts: R ( T ) =
(cid:80) m ≤ B R m + (cid:80) m>B R m .
In the following, we bound these two parts separately and then determine B to obtain the upper bound of the total regret.
Moreover, we show A-BLiN uses only O (log log T ) rounds of communications to achieve the optimal regret.
Recall that A m is set of the active cubes in the m -th batch.
According to Lemma 3, for any x ∈ ∪ C ∈A m C , we have ∆ x ≤ 8 r m − 1 .
Let A + m be set of cubes not eliminated in batch m .
Then any cube in A + m − 1 is a subset of S (8 r m − 1 ) , and thus
[Equation]
By definition, r m = r m − 1 2 − c m , so
[Equation]
The total regret of the m -th batch is
[Equation]
where (i) follows from (5), (ii) follows from (4), and (iii) follows from the definition of ACE Sequence.
Define C m =
( (cid:80) m − 1 i =1 c i )( d z + 1)
+ c m ( d + 2) .
Since c m = c m − 1 · d +1 − d z d +2 , calculation shows that
C m =
( (cid:80) m − 2 i =
c
i )( d z + 1)
+ c m − 1 ( d + 2) + c m − 1 ( d z + 1 − d − 2)
+ c m ( d + 2) =
C m − 1 .
Thus for any m , we have C m =
C 1 =
c 1 ( d + 2) .
Hence,
[Equation]
The inequality (6) holds even if the m -th batch does not exist (where we let R m = 0 ) or is not completed.
Thus we obtain the first upper bound (cid:80) m ≤ B R m ≤ T dz +1 dz +2 · 128 C z · B (log T ) 1 dz +2 .
Lemma 3 implies that any arm x played after the first B batches satisfies ∆ x ≤ 8 r B , so the total regret after B batches is bounded by
[Equation]
Therefore, the total regret R ( T ) satisfies
[Equation]
This inequality holds for any positive B .
Then by choosing B ∗ = loglog T − log( d z +2) log 1 η = loglog T − log( d z +2) log
d + d +1 − dz , we have η B ∗ d z +2 = 1 log T and R ( T ) ≤ [Equation] log
[Equation]
The above analysis implies that we can achieve the optimal regret rate (cid:101) O (cid:16) T
dz +1 dz +2 (cid:17) by letting the for-loop run B ∗ times and finishing the remaining rounds in the Cleanup step.
In other words, B ∗ +rounds of communications are sufficient for A-BLiN to achieve the regret bound (2).
Remark 2.
The quantity r m r m +1 in line 9 of Algorithm 1 may not be integers for some m .
Thus, in practice we denote α n =
⌊ (cid:80) ni
=1 c i ⌋ , β n = ⌈ (cid:80) ni
=1 c i ⌉ , and define rounded ACE Sequence { (cid:101)
r m } m ∈ N by (cid:101) r m = 2 − α k for m = 2 k − 1 and (cid:101) r m = 2 − β k for m = 2 k .
Then the total regret can be divided as R ( T ) =
(cid:80) 1 ≤ k ≤ B ∗ R 2 k − 1 + (cid:80) 1 ≤ k ≤ B ∗ R 2 k + (cid:80) m> 2 B ∗ R m .
For the first part we have (cid:101) r 2 k − 2 ≤ r k − 1 and (cid:101) r 2 k − 1 ≥ r k , while for the second part we have (cid:101) r 2 k − 1 (cid:101) r 2 k = 2 .
Therefore, by similar argument to the proof of Theorem 4, we can bound these three parts separately, and conclude that BLiN with rounded ACE sequence achieves the optimal regret bound (cid:101)
O ( T
dz +1 dz +2 ) by using only
O (log log T ) rounds of communications.
The details are in Appendix C.
Lower Bounds
In this section, we present lower bounds for Lipschitz bandits with batched feedback, which in turn gives communication lower bounds for all Lipschitz bandit algorithms.
Our lower bounds depend on the rounds of communications B .
When B is sufficiently large, our results match the lower bound for the vanilla Lipschitz bandit problem (cid:101) Θ( R z ( T ))
( R z ( T ) is defined in Eq. 1).
More importantly, this dependency on B gives the minimal rounds of communications needed to achieve optimal regret bound for all Lipschitz bandit algorithms, which is summarized in Corollary 2.
Since this lower bound matches the upper bound presented in Theorem 4, BLiN optimally solves Lipschitz bandits with minimal communication.
4.1 Proof Outline
Similar to most lower bound proofs, we need to construct problem instances that are difficult to differentiate.
What’s different is that we need to carefully integrate batched feedback pattern [33] with the Lipschitz payoff reward [39, 30].
To capture the adaptivity in grid determination, we construct “static reference communication grids” to remove the stochasticity in grid selection [1, 21].
Below, we first consider the static grid case, where the grid is predetermined.
This static grid case will provide intuition for the adaptive and more general case.
The expected reward functions of these instances are constructed as follows: we choose some ‘position’ and ‘height’, such that the expected reward function obtains local maximum of the specified ‘height’ at the specified ‘position’.
We will use the word ‘peak’ to refer to the local maxima.
The following theorem presents the lower bound for the static grid case.
Theorem 5 (Lower Bound for Static Grid) .
Consider Lipschitz bandit problems with time horizon T such that the grid of reward communication T is static and determined before the game.
If B rounds of communications are allowed, then for any policy π , there exists a problem instance such that
[Equation]
where c > 0 is a numerical constant independent of B , T , π and T ,
R z ( T ) is defined in (1), and d is the dimension of the arm space.
To prove Theorem 5, we first show that for any k > 1 there exists an instance such that E [ R T ( π )] ≥ t k t 1 d +2 k − 1 .
Fixing k > 1 , we let r k = t 1 d
{++
+}
k − 1 and M k : =
t k − 1 r 2 k = 1 r dk .
Then we construct a set of problem instances
I k
= { I k, 1 , · · · , I k,M k } , such that the gap between the highest peak and the second highest peak is about r k for every instance in I k .
Based on this construction, we prove that no algorithm can distinguish instances in I k from one another in the first ( k − 1) batches, so the worst-case regret is at least r k t k , which gives the inequality we need.
For the first batch (0 , t 1 ] , we can easily construct a set of instances where the worst-case regret is at least t 1 , since no information is available during this time.
Thus, there exists a problem instance such that E [ R T ( π )] ≳ max [Equation] [Equation]
Since 0 < t 1 < · · · < t B = T , the inequality in Theorem 5 follows.
As a result of Theorem 5, we can derive the minimum rounds of communications needed to achieve optimal regret bound for Lipschitz bandit problem, which is stated in Corollary 1.
Corollary 1.
Any Lipschitz bandit algorithm needs Ω(log log T ) rounds of communications to achieve the optimal regret rate, for the case that the times of reward communication are predetermined and static.
The detailed proof of Theorem 5 and Corollary 1 are deferred to Appendix D and E.
4.2 Communication Lower Bound for BLiN
So far we have derived lower bounds for the static grid case.
Yet there is a gap between the static and the adaptive case.
We will close this gap in the following Theorem.
Theorem 6 (Lower Bound for Adaptive Grid) .
Consider Lipschitz bandit problems with time horizon
T such that the grid of reward communication T is adaptively determined by the player.
If B rounds of communications are allowed, then for any policy π , there exists an instance such that
[Equation]
where c > 0 is a numerical constant independent of B , T , π and T ,
R z ( T ) is defined in (1), and d is the dimension of the arm space.
To prove Theorem 6, we consider a reference static grid T r =
{ T 0 , T 1 , · · · , T B } , where T j = T 1 − εj 1 − εB for ε = 1 d [-+
-]+2 .
We construct a series of ‘worlds’, denoted by I 1 , · · · , I B .
Each world is a set of problem instances, and each problem instance in world I j is defined by peak location set U j and basic height r j , where the sets U j and quantities r j for 1 ≤ j ≤ B are presented in Appendix F.
Based on these constructions, we first prove that for any adaptive grid and policy, there exists an index j such that the event A j = { t j − 1 < T j − 1 , t j ≥ T j } happens with sufficiently high probability in world I j .
Then similar to Theorem 5, we prove that in world I j there exists a set of problem instances that is difficult to differentiate in the first j − 1 batches.
In addition, event A j implies that t j ≥ T j , so the worst-case regret is at least r j T j , which gives the lower bound we need.
The proof of Theorem 6 is deferred to Appendix F. Similar to Corollary 1, we can prove that at least Ω(log log T ) rounds of communications are needed to achieve optimal regret bound.
This result is formally summarized in Corollary 2.
Corollary 2.
Any Lipschitz bandit algorithm 3 needs Ω(log log T ) rounds of communications to achieve the optimal regret rate.
Experiments
In this section, we present numerical studies of A-BLiN.
In the experiments, we use the arm space
A = [0 , 1] 2 and the expected reward function µ ( x ) = 1 − 12 ∥ x − x 1 ∥ 2 − 310 ∥ x − x 2 ∥ 2 , where x 1 = (0 . 8 , 0 . 7) and x 2 = (0 . 1 , 0 . 1) .
The landscape of µ and the resulting partition is shown in
Figure 2(a).
As can be seen, the partition is finer in the area closer to the optimal arm x ∗ = (0 . 8 , 0 . 7) .
[Figure]
We let the time horizon T = 80000 , and report the accumulated regret in Figure 2(b).
The regret curve is sublinear, which agrees with the regret bound (2).
Besides, different background colors in Figure 2(b) represent different batches.
For the total time horizon T = 80000 , A-BLiN only needs 4 rounds of communications.
We also present regret curve of zooming algorithm
[26] for comparison.
Different from zooming algorithm, regret curve of A-BLiN is approximately piecewise linear, which is because the strategy of BLiN does not change within each batch.
Results of more repeated experiments are in Appendix G, as well as experimental results of D-BLiN.
Our code is available at https://github.com/FengYasong-fifol/Batched-Lipschitz-Narrowing .
Conclusion
In this paper, we study Lipschitz bandits with communication constraints, and propose the BLiN algorithm as a solution.
We prove that BLiN only need O (log log T ) rounds of communications to achieve the optimal regret rate of best previous Lipschitz bandit algorithms [26, 14] that need
T batches.
This improvement in number of the batches significantly saves data communication costs.
We also provide complexity analysis for this problem.
We show that Ω(log log T ) rounds of communications are necessary for any algorithm to optimally solve Lipschitz bandit problems.
Hence, BLiN algorithm is optimal.
Acknowledgments and Disclosure of Funding
This work was partly supported by the National Key Research and Development Program of China (2020AAA0107600).
Checklist 1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]