Active Learning with Safety Constraints
Abstract
Active learning methods have shown great promise in reducing the number of samples necessary for learning.
As automated learning systems are adopted into real-time, real-world decision-making pipelines, it is increasingly important that such algorithms are designed with safety in mind.
In this work we investigate the complexity of learning the best safe decision in interactive environments.
We reduce this problem to a constrained linear bandits problem, where our goal is to ﬁnd the best arm satisfying certain (unknown) safety constraints.
We propose an adaptive experimental design-based algorithm, which we show efﬁciently trades off between the difﬁculty of showing an arm is unsafe vs suboptimal.
To our knowledge, our results are the ﬁrst on best-arm identiﬁcation in linear bandits with safety constraints.
In practice, we demonstrate that this approach performs well on synthetic and real world datasets.
Introduction
In many problems in online decision-making, the goal of the learner is to take measurements in such a way as to learn a near-optimal policy.
Oftentimes, though the space of policies may be large, the set of feasible, or safe policies could be much smaller, effectively constraining the search space of the learner.
Furthermore, these constraints may themselves depend on unknown problem parameters.
For example, consider the problem of bidding sequentially in a series of auctions where the bidder bids a price w t , the value of winning an item t is denoted v t , and the utility of winning that item and paying price p t is v t  p t .
The goal of the bidder is to choose an optimal strategy amongst bidding strategies s 2 S, s : R ! R .
When a bidder is deciding how to choose these strategies, they often face constraints: they may have a budget B they must abide to; they may wish to have those auctions they win be well-distributed across time (e.g. in the case of advertising campaigns); they may want to ensure the set of items they win satisfy some other property (e.g. for advertisements, they might want to ensure they are not over-targeting any demographic group).
As another example, inventory management systems may face similar issues of deciding amongst strategies, where there is some objective function (such as revenue) and a variety of constraints at play in this choice (e.g. capacity of a set of warehouses, employee scheduling constraints, or limits on the duration of delivery lag).
They also operate in markets with changing demand and other uncertainties, leading to uncertainty about which strategies are feasible or safe (satisfy constraints) and uncertainty about the revenue they generate.
Both of these scenarios motivate understanding the sample complexity of selecting an action or strategy which approximately maximizes an objective while also satisfying some constraints, where samples are needed to both learn the objective value of actions and whether or not they satisfy said constraints.
In this work, we study the active sample complexity of this task—if the learner can choose which examples to observe and have labeled, how many fewer samples might they need compared to the number needed in a passive setting?
We pose this as a best-arm identiﬁcation problem in the setting of linear bandits with safety constraints, where the goal is to estimate the best arm, subject to it meeting certain (initially unknown) safety constraints.
We propose an experiment design-based algorithm which efﬁciently learns the best safe decision, and show the efﬁcacy of this 36th on Information Systems (NeurIPS approach in practice through several experimental examples.
To the best of our knowledge, ours is the ﬁrst approach to handle best-arm identiﬁcation in linear bandits with safety constraints.
1.1 Linear Bandits with Safety Constraints
Let  2 (0 , 1) be a conﬁdence parameter, X , Z ✓ R d be ﬁnite known sets of vectors, and assume there exists ✓ ⇤ 2 R d , µ ⇤ 2 R m ⇥ d unknown to the learner.
For simplicity, we assume that k ✓ ⇤ k 2  1 , and k µ ⇤ ,i k 2  1 , i 2 [ m ] and k x k 2  1 , k z k 2  1 , 8 x 2 X , z 2 Z .
The learner plays according to the following protocol: at each time step t the learner chooses some action x t 2 X , observes ( r t , { s t,i } mi =1 ) where r t = ✓ >⇤ x t + w ✓ t and s t,i = µ > ⇤ ,i x t + w µt,i for all i 2 [ m ] , where w ✓ t , w µt,i are i.i.d.
mean zero 1-subGaussian noise.
The choice of action x t is measurable with respect to the history F t = { ( x j , r j , { s j,i } mi =1 ) } t  1 j =1 .
The learner stops at a stopping time ⌧  which is measurable with respect to the ﬁltration generated by F t  ⌧ , and returns b z ⌧ 2 Z .
In general, when referring to any expectation E or probability P , the underlying measure will be with respect to the actions, observed rewards, and internal randomness of the algorithm.
We are interested in the safe transductive best-arm identiﬁcation problem ( STBAI ), where the goal of the learner is to identify
[Equation]
for some (known) threshold  .
In words, our goal is to identify the best safe arm in Z , z ⇤ , where we say an arm z is safe if it satisﬁes every linear constraint: z > µ ⇤ ,i   , 8 i 2 [ m ] .
We are interested in obtaining learners that take the fewest number of samples possible to accomplish this.
In practice, we will consider a slightly easier objective.
Fix some tolerance ✏ > 0 and let
[Equation]
Then our goal is to obtain an ( ✏ ,  ) -PAC learner deﬁned as follows:
Deﬁnition 1 ( ( ✏ ,  ) -PAC Learner) .
A learner is ( ✏ ,  ) -PAC if for any instance it returns b z ⌧ such that
[Equation]
We deﬁne the optimality gap for any z 2 Z as  ( z ) :
= ✓ >⇤ ( z ⇤  z ) , and the safety gap for constraint i as  i safe ( z ) :=
µ > ⇤ ,i z .
Note that either  ( z ) or  i safe ( z ) can be negative.
If  ( z ) < 0 , it follows that z has larger value — z > ✓ ⇤ —than the best safe arm z ⇤ , which implies it must be unsafe.
If  i safe ( z ) < 0 for some i , then arm z is unsafe.
We also deﬁne the ✏ - safe optimality gap as:
[Equation]
✏ ( z ) is then the gap in value between arm z and the best arm with minimum safety gap at least ✏ .
Mathematical Notation.
Let k x k 2 A = x > Ax and p ( x ) :
= max { x, 0 } .
e O ( · ) hides factors that are logarithmic in the arguments. .
denotes inequality up to constants.
We denote the simplex as [Equation]
Safe Best-Arm Identiﬁcation in Linear Bandits 2.1 Algorithm Deﬁnition
The main challenge in algorithm design for the safe best-arm identiﬁcation problem is ensuring that we are efﬁciently balancing our exploration between reﬁning our estimates of both the safety gaps, as well as the optimality gaps.
Our approach is given in Algorithm 1, B ESIDE .
B ESIDE relies on a round-based adaptive experimental design approach.
In each round B ESIDE consists of three phases.
In the ﬁrst phase, it solves an experimental design over  ` 2 4 X , with the goal of reﬁning our estimates of the safety gaps.
It then takes ⌧ ` samples from  ` .
In the second phase these samples are used to estimate the safety constraints, b µ i, ` , and the safety gaps of each arm, b  i, ` safe ( z ) .
Finally, in Phase 3, an additional experimental design is solved which now aims to reﬁne our estimates of the optimality gaps, and the estimates of the optimality gaps b  ` ( z ) for each z 2 Z are then computed.
We encapsulate Phase 3 in a subroutine, R AGE ✏ , which we outline in the following.
We now carefully describe each phase—we begin with Phase 2 to explain how our estimator works.
1: input: tolerance ✏ , conﬁdence  2: ◆ ✏ d log [Equation] [Equation] i
[Equation]
safe ( z )
[Equation]
[Equation]
( z ) 0 for all z 2 Z 3: for ` = 1 , 2 , . .
. , ◆ ✏ do 4: ✏ ` 20 · 2  ` // Phase 1:
Solve design to reduce uncertainty
Deﬁne
[Equation]
Let ⌧ ` be the minimal value of ⌧ 2 R + which is greater than 4 log 4 m |Z| ` 2  such that the objective to the following is no greater than ✏ ` / 100 , and  ` the corresponding optimal distribution inf
[Equation]
Sample x t ⇠  ` , collect ⌧ ` observations { ( x t , r t , s t, 1 , . .
, s t,m ) } ⌧ ` t = // Phase 2:
Estimate safety constraints 8:
[Equation]
9: b
[Equation]
// Phase 3:
Refine
[Equation]
⌘ // Perform final round of exploration to ensure we find ✏ -good arm 11: Y end
[Equation] 13: return [Equation]
Phase 2:
In Phase 2 the algorithm would like to use the ⌧ ` samples drawn from the design  ` to estimate the constraints for each z 2 Z : z > µ ⇤ ,i for each i 2 [ m ] .
Past works using adaptive experimental design in the linear bandits literature have utilized the least-squares estimator along with complicated rounding schemes [13] which may require an additional poly ( d ) samples each round (this poly( d ) factor could be prohibitively large—for example, in active classiﬁcation problems, d is the total number of data points).
We instead utilize the RIPS estimator of [6] which gives us a guarantee of the form: with probability greater than 1   , for all z 2 Z ,
[Equation]
We describe the RIPS estimator in more detail in Appendix B.
Phase 1:
By our deﬁnition of the experimental design on Line 6, our safety gap estimation error bound in (2) satisﬁes, for each z 2 Z :
[Equation]
Note that our design chooses an allocation that minimizes the variance in our estimate of each safety constraint (up to some tolerance), which scales as k z k 2 A (  )  1 .
This can be thought of as a form of
XY -design —a design of the form inf  24 X max y 2 Y k y k 2 A (  )  1 —where here Y Z is chosen to reduce our uncertainty in estimating the safety value for each z 2 Z .
We refer to such a design objective henceforth as XY safe .
Assume that at round `  1 , we can guarantee
[Equation]
Then combining the above inequalities, we see that the experiment design on Line 6 aims to minimize the uncertainty in our estimate of z > µ ⇤ ,i up to a tolerance that scales as the maximum of the four terms in (4).
It follows that if any of these terms is large, we will only allocate a small number of samples to reﬁning our estimate of arm z .
Each one of these terms can be intuitively motivated by thinking through what is needed to prove that an arm z {+
+} =
z ⇤ .
• z has small safety gap min j |  j safe ( z ) | : if this term is large, it implies that minimum safety gap for z is large.
To show an arm is safe or unsafe, it sufﬁces to learn each safety gap up to a tolerance a constant factor from its value—regularizing by this term ensures we do just that.
• z fails some safety constraint max j p (   j safe ( z )) : if this term is large, it implies that arm z is very unsafe for some constraint.
In this case, we can easily determine z is unsafe, and therefore do not need to reduce our uncertainty in the safety gap any more.
• z is sub-optimal p (  ✏ `  1 ( z )) : if this term is large, it implies that z is very suboptimal compared to some safe arm with safety gap at least ✏ `  1 .
In this case, we do not need to estimate z ’s safety gap, as we will have already eliminated it.
It remains to ensure that (4) holds.
As we show in Appendix D through a careful inductive argument, combining (3) with our guarantee on the estimates of the optimality gaps obtained in Phase 3, b  ` ( z ) , is sufﬁcient to guarantee (4) holds.
In particular, if any gap is greater than ✏ ` it is estimated up to a constant factor, and otherwise it is estimated up to O ( ✏ ` ) .
This ensures that our gaps are estimated at the correct rate while guaranteeing we do not collect too many samples in each round.
Phase 3:
In this phase we estimate the suboptimality gaps using R AGE ✏ .
R AGE ✏ is inspired by the R AGE algorithm of [13] for best-arm identiﬁcation.
In the interest of space, we defer the full deﬁnition of R AGE ✏ to Appendix C but provide some intuition here.
After Phase 2, by (3) the set of arms Y ` := { z 2 Z : c s ( z ) . b  i,s ( z ) , 8 i 2 [ m ] } for s  ` are precisely the ones that we can certify are safe (note that we do not need to ever explicitly construct such a set—we can instead maintain an implicit deﬁnition through the constraints).
R AGE ✏ uses an adaptive experimental design procedure to sample in such a way as to optimally estimate the gaps ( z  b y ) > ✓ ⇤ , 8 z 2 Z and some b y 2 Y ` up to some (sufﬁcient) tolerance.
In particular, it also solves an XY -design, but now on the set
Y { z  b y : z 2 Z} .
Thus, rather than minimizing k z k 2 A (  )  1 , we minimize k z  b y k 2 A (  )  1 .
This design reduces uncertainty on the differences between arms, which allows us to reﬁne our estimates of their optimality gaps.
Henceforth we refer to such a design as XY di ↵ .
We describe the importance of the choice of design in more detail in Section 2.4.
Ultimately, if an arm z has value within a factor of ✏ ` of the best safe arm in Y ` , and if we have not yet shown arm z is unsafe, then we will estimate its optimality gap up to a constant factor of ✏ ` .
If we were maintaining arm sets explicitly (similar to the original RAGE algorithm of [13]) we would eliminate arms at this point.
Remark 1 (Computational Complexity) .
The main computational challenge in B ESIDE and R AGE ✏ is the calculation of the experimental designs (i.e. Line 6 and the corresponding design in R AGE ✏ ).
In general, the presence of the square root implies that the resulting optimization problem may not be convex in  .
To handle this issue we note that 2 p xy = min ↵ > 0 ↵ x + y ↵ —thus we can replace the existing design with inf  24 X max z 2 Z min ↵ > 0  1100 ( c ` ( z ) + ✏ ` )+ ↵ k z k 2 A (  )  1 +log( 4 m |Z| ` 2  ) /
( ↵⌧ ) .
By appropriately discretizing the space we search over for ⌧ and ↵ we can then apply the Frank-Wolfe algorithm to minimize over  .
While computationally efﬁcient in theory, this procedure is quite complicated and impractical for large problems.
In the experiments section we provide a practical heuristic that is motivated by the above algorithm and is computationally efﬁcient for larger problems.
2.2 Main Result
B ESIDE achieves the following complexity.
Theorem 1.
B ESIDE is ( ✏ ,  ) -PAC.
In other words, with probability at least 1   , B ESIDE returns an arm b z 2 Z such that
[Equation]
and terminates after collecting at most
[Equation]
[Equation] optimality
[Equation]
samples for some C
= poly log( 1 ✏ ) and C
=
poly log( 1 ✏ , |Z| ) · log 1  .
The complexity bound given in Theorem 1 may, at ﬁrst glance, appear rather opaque, yet it in fact yields a very intuitive interpretation.
The ﬁrst term in the complexity, the safety term, is the complexity needed to show each arm is safe or unsafe, if they have not otherwise been eliminated .
As described in the previous section, if p (  e ✏ ( z )) is large, this implies we have found an arm better than z , so learning its safety value is irrelevant.
The second term in the complexity, the optimality term, corresponds to the difﬁculty of showing an arm is worse than the best arm we can guarantee is safe .
Note that we can only guarantee an arm is suboptimal if we can ﬁnd a safe arm with higher value.
Recall the deﬁnition of  e ✏ ( z ) given in (1).
Intuitively,  e ✏ ( z ) denotes the gap in value between arm z and the best arm with safety gap at least e ✏ .
As we make e ✏ smaller, we can show additional arms are safe, which increases  e ✏ ( z ) .
While this makes it easier to show z is suboptimal, it comes at a cost—the extra samples necessary to decrease our safety tolerance, given by the ﬁrst term in the complexity.
B ESIDE trades off between optimizing for each of these terms—gradually decreasing its tolerance on both the safety and optimality terms to more easily eliminate suboptimal arms, while not allocating too many samples to guarantee safety.
To help illustrate this complexity, we consider a simple example with orthogonal arms, i.e. a multi-armed bandit example.
Example 1 (B ESIDE on Multi-Armed Bandits) .
In the multi-armed bandit setting, we have X =
Z = { e 1 , . . .
, e d } .
Let m = 1 , d = 3 , and consider the settings of ✓ ⇤ and µ ⇤ given in Figure 1.
Here we see that arm e 1 is safe and has value much higher than any other arm, so z ⇤ = e 1 , and can be shown to be safe relatively easily; arm e 2 has near-optimal value but is very unsafe; and arm e 3 is unsafe with very small safety gap, but has the smallest value.
[Figure]
Showing e 2 is Suboptimal.
As e 2 has near-optimal value,  ( e 2 ) is very small and it is very difﬁcult to show e 2 is suboptimal.
However,   safe ( e 2 ) =
O (1) , so it is very easy to show e 2 is unsafe.
It follows that p (   safe ( e 2 ))
=
O (1) so both denominators in our complexity will always be O (1) for z = e 2 — B ESIDE does not attempt to show e 2 is suboptimal, but instead shows it is unsafe, and therefore does not pay for the small optimality gap of  ( e 2 ) in the complexity.
Showing e 3 is Suboptimal.
Recall the deﬁnition of  ✏ ( z ) =
max z 0 :  safe ( z 0 )  ✏ ✓ >⇤ ( z 0  z ) .
In this case, for ✏ =
O (1) , we will have  safe ( e 1 )  ✏ , which implies that  ✏ ( e 3 ) =
✓ >⇤ ( e 1  e 2 ) =
( e 3 ) =
O (1) .
To show e 3 is suboptimal, we could either show it is unsafe (which is very difﬁcult) or suboptimal (which is very easy).
Observing the sample complexity of Theorem 1, we see that the denominator of both terms will always be O (1) for z = e 3 since  ✏ ( e 2 )
=
O (1) — B ESIDE never pays for the small safety gap of e 3 , it instead takes advantage of the fact that e 3 can easily be shown to be suboptimal, and uses this to eliminate it.
In both of these cases we see that B ESIDE does the “right” thing, always using the easier of the two criteria—either showing an arm is unsafe or suboptimal—to show that
z 6 = z ⇤ .
Combining the above observations, for ✏ ⇡ min {  ( e 3 ) ,   safe ( e 2 ) ,  safe ( e 1 ) } , it follows that on this example the total sample complexity of B ESIDE given by Theorem 1 scales as:
[Equation]
where the 1 /  safe ( e 1 ) 2 arises because we must also show e 1 is safe.
2.3 Optimality of B ESIDE
Optimality in Best-Arm Identiﬁcation.
Consider applying B ESIDE to a problem instance where m = 1 , µ ⇤ , 1 = 0 , and  = 1 .
In this case, every arm is safe, and the safety constraints are essentially vacuous—every arm can easily be shown safe.
We can therefore think of this as simply an instance of the best-arm identiﬁcation problem.
In this setting, we obtain the following corollary.
Corollary 1.
Consider running B ESIDE on a problem instance where m = 1 , µ ⇤ , 1 = 0 , and  = 1 , and set ✏ = 12 max z {+
+}
= z ⇤ ✓ >⇤ ( z ⇤  z ) .
Then with probability at least 1   , B ESIDE returns z ⇤ and has sample complexity bounded by:
[Equation]
Up to lower-order terms, this exactly matches the lower bound on best-arm identiﬁcation given in [13].
Thus, in settings where the safety constraint is vacuous, B ESIDE hits the optimal rate.
Worst-Case Performance of B ESIDE .
We next consider the worst-case performance of B ESIDE in settings when X = Z .
We have the following result.
Corollary 2.
Assume that X =
Z .
Then for any ✓ ⇤ and ( µ ⇤ ,i ) mi =1 , the sample complexity of B ESIDE necessary to return an ✏ -good and ✏ -safe arm is bounded as
[Equation]
Theorem 2 of [38] shows a worst-case lower bound of ⌦ ( d 2 / ✏ 2 ) on the sample complexity of identifying an ✏ -optimal arm in the standard linear bandit setting.
Safe best-arm identiﬁcation problems in which the safety constraint is vacuous are at least as hard as the standard best-arm identiﬁcation problem, since at minimum we need to ﬁnd the best arm out of every safe arm.
Thus, ⌦ ( d 2 / ✏ 2 ) is also a worst-case lower bound for the safe best-arm identiﬁcation problem.
The hard instance of [38] has |X| =
O (2 d ) , so it follows that on this instance, B ESIDE achieves a complexity of e
O ( d ✏ 2 · ( d + log 1  )) , and therefore B ESIDE has optimal dimensionality dependence.
In addition, this also implies that safe best-arm identiﬁcation, in the worst-case, is no harder than the standard best-arm identiﬁcation problem—it is no harder to ﬁnd the best safe arm, regardless of the number of safety constraints, than to ﬁnd the best arm, ignoring safety constraints.
2.4 The Role of Experiment Design
We can think of the safe best-arm identiﬁcation problem, in some sense, as an interpolation of the standard best-arm identiﬁcation problem, as well as the level-set estimation problem, where the goal is to identify z 2 Z satisfying z > µ ⇤   [29].
In the former problem, [13] shows that the instance-optimal rate can be attained by running a round-based algorithm and at every round solving an instance of the XY di ↵ experiment design, as deﬁned in Section 2.1.
In the latter problem, [29] also show that a round-based algorithm can hit the instance-optimal rate, but instead solving the
XY safe problem at each round.
It is natural to ask whether either of these strategies could be applied to the safe best-arm identiﬁcation problem directly, or if it is necessary to alternate between them.
The following results show that, on their own, each of these designs is unable to hit the optimal rate.
Proposition 2.
Fix some small enough ✏ > 0 .
Then there exist instances of the safe best-arm identiﬁcation problem, I i =
( ✓ i ⇤ , µ i ⇤ , X i , Z i ) , i = 1 , 2 , with d = |X i | =
|Z i | = 2 , m = , such that
: • On I 1 , any ( ✏ ,  ) -PAC algorithm which plays only allocations minimizing XY di ↵ must have E [ ⌧  ]  ⌦  1 ✏ 3 · log 1   , while B ESIDE identiﬁes an ✏ -optimal arm after e O ( 1 ✏ 2 · log 1 /  ) samples.
• On I 2 , any ( ✏ ,  ) -PAC algorithm which plays only allocations minimizing XY safe must have E [ ⌧  ] [-≥ Ω (cid:0) 
 ϵ-] ⌦  1 ✏ 3 / 2 · log 1   , while B ESIDE identiﬁes an ✏ -optimal arm after e O ( 1 ✏ · log 1 /  ) samples.
Proposition 2 implies that, to solve the safe best-arm identiﬁcation problem optimally, more care must be taken in exploring than either standard experiment design induces—we must trade off between
XY di ↵ and XY safe as B ESIDE does.
We remark brieﬂy on the instance I 1 .
On this instance we have
X = { e 1 , e 2 } and Z = { z 1 , z 2 } with z 1 = [1 / 4 , 1 / 2] and z 2 =
[3 / 4 , 1 / 2 + ↵ ] .
We set ✓ 1 ⇤ =
[1 , 0] , µ⇤ =
[0 , 1] , and  = 1 / 2 + ↵ / 2 .
Here z 2 is unsafe while z 1 is safe, so it follows that z ⇤ =
z 1 .
As z > 2 ✓ 1 ⇤ > z > 1 ✓ 1 ⇤ , to show z 2 6 =
z ⇤ , we must show it is unsafe.
However, if we solve the design XY di ↵ , we see that it places nearly all of the mass on the ﬁrst coordinate.
While this would be optimal if both z 1 and z 2 were safe and we simply wished to determine which has a higher value, to show z 2 is unsafe, the optimal strategy places (roughly) the same mass on each coordinate, since each coordinate could contribute to the safety value.
This is precisely the allocation B ESIDE will play, so it is able to show that z 2 is unsafe much more efﬁciently than a naive XY di ↵ approach.
Experiments for Safe Best Arm Identiﬁcation in Linear Bandits
We next present experimental results on B ESIDE to demonstrate the advantage of experimental design—especially combining XY di ↵ and XY safe designs.
As there are no existing algorithms that consider safe best-arm identiﬁcation, as a benchmark we consider the naive adaptive approach
B ASELINE that ﬁrst solves the problem of ﬁnding the safe arms up to a desired tolerance, and then solves the problem of ﬁnding the best (safe) arm among the arms that were found to be safe.
We ﬁrst describe instances on which we test B ESIDE .
Our experimental details and precise implementation of B ESIDE using elimination are described in Section F.
Multi-Armed Bandit.
We consider a best-arm identiﬁcation problem in which every arm is safe, but the arm with highest value is very difﬁcult to identify as safe, while the second-best arm can easily be shown safe.
We vary the total number of arms and run B ESIDE and B ASELINE with ✏ = 0 .and
= 0 .
From Figure 2, we observe that the sample complexity of B ESIDE is smaller (up to about two times for 100 arms) than the sample complexity of its baseline.
Linear Response Model.
Random Instance : We also consider the more general setup where
X , Z ⇢ R d , ✓ 2 R d and µ 2 R d are randomly generated from independent Gaussian random variables with mean 0 and variance 1 .
We set |X| =
50 and vary the size of |Z| .
In Figure 3, we see again that B ESIDE signiﬁcantly outperforms the baseline.
Hard Instance : We last consider the instance of Proposition 2 and benchmark against the strategy playing only allocations minimizing XY di ↵ .
In Figure 4, we see again that B ESIDE signiﬁcantly outperforms this baseline, corroborating the theoretical result of Proposition 2.
3.1 Practical Algorithms for Active Classiﬁcation Under Constraints
Next, we provide an application of the above ideas to pool-based active classiﬁcation with constraints— namely, adaptive sampling to learn the highest accuracy classiﬁer with a constraint on the false discovery rate (FDR).
We ﬁrst explain how this problem maps to the linear bandit setting.
Precisely, let
X be the example space and Y = { 0 , 1 } the label space.
Fix a hypothesis class H such that each h
H is a classiﬁer h : X ! Y .
We represent each h with an associated indicator vector z h 2 { 0 , 1 } |X| where z h ( x ) = 1 () h ( x ) = 1 .
Similarly, let ⌘ 2 [0 , 1] |X| represent the label distribution, i.e.
⌘ ( x ) =
P ( Y = 1 | X = x ) .
Then the risk of a classiﬁer R ( h ) :
= E x ⇠ Unif ( X ) ,Y ⇠ Ber ( ⌘ ( x ))
[ [ h ( x ) 6 =
Y ]]
= z > h (2 ⌘  1) and the FDR is deﬁned as FDR( h ) :
= ( 1  ⌘ ) > z/ 1 > z .
In the case when ⌘ 2 { 0 , 1 } |X| , FDR( h ) is the proportion of examples that h incorrectly labels as 1 out of all examples h labels as 1 .
Our goal is to solve the following constrained best arm identiﬁcation problem:
[Equation]
The main challenge in running B ESIDE on this problem directly is a potentially high computational cost from computing a design over an extremely large hypothesis class H (e.g. neural networks of a bounded width).
In this section we provide an alternative approach motivated by B ESIDE .
Algorithm 2 follows a similar design as B ESIDE and relies on an oracle, CERM , that can solve (5), i.e. given a dataset it returns the highest accuracy classiﬁer under an FDR constraint.
Such oracles are available in, for example in [1, 10].
In each round of Algorithm 2 we perform randomized exploration by perturbing the labels on our existing dataset with mean zero Gaussian noise, and then training k classiﬁers b h i , i 2 [ k ] , on the resulting datasets.
Implicitly, we are making the assumption that the loss function in the training of ERM can handle continuous labels, such as the MLE of logistic regression.
As described in [25], randomized exploration emulates sampling from a posterior distribution on our possible set of classiﬁers.
We then use the labels generated from these classiﬁers to compute safe classiﬁers
h i , i 2 [ k ] .
Finally, mimicking the strategy of B ESIDE , we compute XY safe and XY di ↵ designs on these k safe classiﬁers and repeat (note that the designs computed on Line 5 are equivalent to XY safe and XY di ↵ in the classiﬁcation setting).
Require: Batch size n , initial (labeled) data x (0)1 , . . .
, x (0) n , number of rounds L , number of classiﬁers per round k , perturbation variance  1: for ` = 1 , . . .
, L do 2: for i = 1 , . . .
, k do 3: b h
i = ERM ( { ( x ( ` ) t , y ( ` ) t + ✏ ( i ) t ) } nt =1 ) , where { ✏ ( i ) t } 1  t  n i.i.d. ⇠ N (0 ,  2 ) 4: h i = CERM ( { ( x, [Equation]
h i
[Equation] [Equation] Compute designs: [Equation]
Sample x ( ` ) 1 , . . .
, x ( ` ) n from a uniform mixture of  safe ,  diff 7:
Observe corresponding labels y ( ` ) 1 , . . .
, y ( ` ) n return
[Equation]
To validate Algorithm 2, we experiment against a passive baseline that selects points uniformly at randoms from the pool of examples X , retrains the model using the same Constrained Empirical Risk
Minimization oracle ( CERM ) as Algorithm 2 on its current samples, and report the accuracy and FDR.
We evaluate on two real world datasets and on one synthetic dataset next and provide an additional details on the experiments in Section F.
Adult dataset. We evaluate on the adult income data set [27] (48,842 examples) where the goal is to predict whether someone’s income is above $50k per year.
We set the constraint to be FDR < 0 . 15 and report in Figure 5 the accuracy and the FDR obtained when varying the number of labels given to each method (batch size is set to 25 and initial number of queried labels is 50).
We observe that for any desired accuracy Algorithm 2 allows us to provide a classiﬁer with lower FDR.
Also, for any chosen number of total labels—such as 500 , 750 , 2000 as reported in Figure 5—the Algorithm 2 gives a classiﬁer with higher accuracy and lower FDR.
In general we found that the active method needed half the number of samples as the passive sampling to achieve a given FDR.
This demonstrates the effectiveness of Algorithm 2 to learn simultaneously the objective (risk) and the constraint (FDR), in a similar favorable way as characterized by our theoretical ﬁndings. German Credit dataset. We consider the German
Credit Dataset originally from the Staﬂog Project
Databases [24].
The goal is to predict whether someone’s credit is ’bad’ or ’good’.
We report in Figure 6 the recall (TPR) and the precision ( 1  FDR) obtained when varying the number of labels given to each method. We observe that for any desired precision Algorithm 2 allows us to provide a classiﬁer with higher recall. Also, for any chosen number of total labels—such as 170 , 270 , 330 , 450 , 600 as reported in Figure 6—the Algorithm 2 gives a classiﬁer with higher precision and higher recall. As for the Adult dataset we found that the active method needed half the number of samples as the passive sampling to achieve a given precision.
Half circle dataset.
We consider a two-dimensional half circle dataset, visualized on Figure 10.
We report in Figures 11 and 12 the precision and (respectively)
the recall obtained when varying the number of labels given to each method.
The conﬁdence intervals are obtained over 25 repetitions.
We observe that Algorithm 2 allows us to provide a classiﬁer satisfying a given recall or precision in far fewer queries.
This is in line with the results of [16] on One Dimensional Thresholds, where the sample complexity of the active strategy is O ( log ( n )) while the sample complexity of the passive strategy is at least of order n .
Related works
Constrained Bandits.
A growing body of work seeks to address the question of safe learning in interactive environments.
In particular, the majority of such works have considered the problem of regret minimization in linear bandits with linear safety constraints.
Here, the goal is to maximize online reward, x > t ✓ ⇤ , by choosing actions x t 2 X ✓ R d , while ensuring a safety constraint of the form x > t µ ⇤   is met at all times (either in expectation or with high probability).
A variety of algorithms have been proposed, including UCB-style [23, 2, 32], and Thompson Sampling [30, 31].
While these works show that p T regret is attainable, they only provide worst-case bounds (while we obtain instance-dependent bounds) and do not study the pure-exploration best-arm identiﬁcation problem.
To our knowledge, the only work to offer instance-dependent guarantees is [9], yet they focus exclusively on the regret setting, and offer a relatively coarse notion of instance-dependence — analogous to O ( d · poly log T/  min ) bounds in the unconstrained linear bandits setting — in contrast to the more ﬁne-grained notion of instance-dependence we provide.
To our knowledge, only several existing works consider the question of best-arm identiﬁcation with safety constraints [36, 37, 39, 28].
The most related to ours is [28] which focuses on the easier problem of safe best arm identiﬁcation with known rewards and unknown constraints.
Since the reward is known, the main challenge in the setting of [28] is to learn constraints via G-optimal designs.
The key and novel challenge of our framework is to carefully balance between G and XY designs: naively spending enough budget to either learn the reward model (via a XY design) or to learn the safety constraints (via a G design) will fail catastrophically (see Example 2.1 and Proposition 1).
[36, 37] consider a general constrained optimization setting where the goal of the learner is to minimize some function f ( x ) over a domain x 2 D , while only having access to noisy samples of f ( x ) , f ( x t ) + w t , and guaranteeing that a safety constraint g ( x t )  h is met for every query point x t .
While they do provide a sample complexity upper bound, they give no lower bound, and, as shown in [39], their approach can be very suboptimal.
[39] considers the setting of best-arm identiﬁcation in multi-armed bandits.
In their setting, at every step t they query a value a t 2 A for a particular coordinate i t , and their goal is to identify the coordinate i ⇤ such that a ⇤ i ⇤ ✓ i ⇤  max i a ⇤ i ✓ i , where a ⇤ i is the largest value respecting the safety constraint: a ⇤ i =
arg max a 2 A a ✓ i s.t.
aµ i   .
Similar to [36, 37], they require that the safety constraint a t µ i t   must be met while learning.
Though they do show matching upper and lower bounds, and in addition consider a slightly more general setting that allows for nonlinear (but monotonic) response functions, they treat every coordinate as independent, and do not allow for information-sharing between coordinates—the key generalization the linear bandit setting targets.
We remark as well that in our setting, unlike these works, we allow the learner to query unsafe points during exploration, and only require that they output a safe decision at termination.
Best-Arm Identiﬁcation in Linear Bandits.
The best-arm identiﬁcation problem in multi-armed bandits (without safety constraints) is a classical and well-studied problem [3, 33, 12, 5], and nearoptimal algorithms exist [18, 22].
More recently, there has been a growing interest in understanding the sample complexity of best-arm identiﬁcation in linear bandits [35, 19, 40, 13, 20, 11].
We highlight in particular the work of [13] which proposes an experiment-design based algorithm, R AGE , that our approach takes inspiration from.
While much progress has been made in understanding best-arm identiﬁcation in linear bandits, to our knowledge, no existing works consider the setting of best-arm identiﬁcation in linear bandits with safety constraints, the setting of this work.
Active Classiﬁcation under FDR constraints We ﬁnally mention one other related body of work— the problem of actively sampling to ﬁnd a classiﬁer with high accuracy or recall under precision constraints.
Motivated by the experimental design approach of our main algorithm, B ESIDE , we provide a heuristic algorithm for this problem with good empirical performance in Section 3.1.
There is an extensive body of work on active learning (see the survey [14]) but only recently have works made the connection between best-arm identiﬁcation for linear bandits and classiﬁcation [21, 16, 7].
Precision constraints has been less studied in the adaptive context, we only know of [16, 4].
Conclusion
In this work we have shown that it is possible to efﬁciently ﬁnd the best safe arm in linear bandits with a carefully designed adaptive experiment design-based approach.
Our results open up several interesting directions for future work.
Instance Optimality.
While B ESIDE is worst-case optimal, in Appendix A we show an instancedependent lower bound which B ESIDE does not, in general, seem to hit.
We conjecture that this lower bound may be loose—addressing this discrepancy and showing matching instance-dependent upper and lower bounds is an exciting direction for future work.
Safety During Exploration.
Though there are many interesting applications where we may not require safety during exploration (i.e. only querying safe arms), in other cases we may need to ensure safety is met during exploration.
Extending our work to this setting is an interesting open problem.
Potential Impacts.
As with any algorithm making stochastic assumptions, if assumptions are not met we can not guarantee the performance.
In this case, one limitation is that if the underlying environment is changing (i.e. the constraints vary over time) the algorithm could have unexpected behavior with unintended consequences.
Such a situation could lead to harmful results in examples such as the online advertising bidding example from the introduction.
To mitigate this limitation of our setting, practitioners are encouraged to monitor many metrics, both short and long-term. Acknowledgements
The work of AW was supported by an NSF GFRP Fellowship DGE-1762114.
The work of JM was supported by an NSF Career award, and NSF AI institute (IFML) and the Simons collaborative grant on the foundations of fairness.
The work of KJ was funded in part by the AFRL and NSF TRIPODS 2023166.