L EARNING TO A CTIVELY L EARN : A R OBUST A PPROACH
Anonymous authors
Paper under double-blind review
A BSTRACT
This work proposes a procedure for designing algorithms for speciﬁc adaptive data collection tasks like active learning and pure-exploration multi-armed bandits.
Unlike the design of traditional adaptive algorithms that rely on concentration of measure and careful analysis to justify the correctness and sample complexity of the procedure, our adaptive algorithm is learned via adversarial training over equivalence classes of problems derived from information theoretic lower bounds.
In particular, a single adaptive learning algorithm is learned that competes with the best adaptive algorithm learned for each equivalence class.
Our procedure takes as input just the available queries, set of hypotheses, loss function and total query budget.
This is in contrast to existing meta-learning work that learns an adaptive algorithm relative to an explicit, user-deﬁned subset or prior distribution over problems which can be challenging to deﬁne and be mismatched to the instance encountered at test time.
This work is particularly focused on the regime when the total query budget is very small, such as a few dozen, which is much smaller than those budgets typically considered by theoretically derived algorithms.
We perform synthetic experiments to justify the stability and effectiveness of the training procedure and then evaluate the method on tasks derived from real data including a noisy 20 Questions game and a joke recommendation task
I NTRODUCTION
Closed-loop learning algorithms use previous observations to inform what measurements to take next in a closed-loop in order to accomplish inference tasks far faster than any ﬁxed measurement plan set in advance.
For example, active learning algorithms for binary classiﬁcation have been proposed that under favorable conditions require exponentially fewer labels than passive, random sampling to identify the optimal classiﬁer (Hanneke et al., 2014).
And in the multi-armed bandits literature, adaptive sampling techniques have demonstrated the ability to identify the “best arm” that optimizes some metric with far fewer experiments than a ﬁxed design (Garivier & Kaufmann, 2016;
Fiez et al., 2019).
Unfortunately, such guarantees often either require simplifying assumptions that limit robustness and applicability, or appeal to concentration inequalities that are very loose unless the number of samples is very large (e.g., web-scale).
The aim of this work is a framework that achieves the best of both worlds: algorithms that learn through simulated experience to be as effective as possible with a tiny measurement budget (e.g., 20 queries), while remaining robust due to adversarial training.
Our work ﬁts into a recent trend sometimes referred to as learning to actively learn (Konyushkova et al., 2017; Bachman et al., 2017;
Fang et al., 2017; Boutilier et al., 2020; Kveton et al., 2020) which tunes existing algorithms or learns entirely new active learning algorithms by policy optimization.
Previous works in this area learn a policy by optimizing with respect to data observed through prior experience (e.g., metalearning or transfer learning) or an assumed explicit prior distribution of problem parameters (e.g. the true weight vector for linear regression).
In contrast, our approach makes no assumptions about what parameters are likely to be encountered at test time, and therefore produces algorithms that do not suffer from a potential mismatch of priors.
Instead, our method learns a policy that attempts to mirror the guarantees of frequentist algorithms with instance dependent sample complexities : if the problem is hard you will suffer a large loss, if it is easy you will suffer little.
The learning framework is general enough to be applied to many active learning settings of interest and is intended to be used to produce novel and robust high performing algorithms.
The difference is that instead of hand-crafting hard instances that witness the difﬁculty of the problem, we use adversarial training inspired by the robust reinforcement learning literature to automatically train minimax policies.
Embracing the use of a simulator allows our learned policies to be very aggressive while maintaining robustness.
Indeed, this work is particularly useful in the setting where relatively few rounds of querying can be made, where concentration inequalities of existing algorithms are vacuous.
To demonstrate the efﬁcacy of our approach we implement the framework for the (transductive) linear bandit problem.
This paradigm includes pure-exploration combinatorial bandits (e.g., shortest path, matchings) as a special case which itself reduces to active binary classiﬁcation.
We empirically validate our framework on a simple synthetic experiment before turning our attention to datasets derived from real data including a noisy 20 questions game and a joke recommendation task.
P ROPOSED F RAMEWORK FOR R OBUST L EARNING TO A CTIVELY L EARN
Whether learned or deﬁned by an expert, any algorithm for active learning can be thought of as a policy from the perspective of reinforcement learning.
At time t , based on an internal state s t , the policy takes action x t and receives observation y t , which then updates the state and the process repeats.
In our work, at time t the state s t ∈ S is a function of the history { ( x i , y i ) } t − 1 i =1 such as its sufﬁcient statistics.
Without loss of generality, a policy π takes a state as input and deﬁnes a probability distribution over X so that at time t we have x t ∼ π ( s t ) .
Fix a horizon T .
For t = 1 , 2 , . . .
, T
• state s t ∈ S is a function of the history, { ( x i , y i ) } t − 1 i =1 , • action x t ∈ X is drawn at random from the distribution π ( s t ) deﬁned over X , and • next state s t +1 ∈ S is constructed by taking action x t in state s t and observing y t ∼ f ( ·| θ ∗ , s t , x t ) until the game terminates at time t = T and the policy receives loss L T .
Note that L T is a random variable that depends on the tuple ( π, { ( x i , y i ) }
Ti =1 , θ ∗ ) .
We assume that f is a distribution of known parameteric form to the policy (e.g., f ( ·| θ, s, x ) ≡ N ( (cid:104)
x, θ (cid:105) , 1) )
but the parameter θ is unknown to the policy.
Let P π,θ , E π,θ denote the probability and expectation under the probability law induced by executing policy π in the game with θ ∗ =
θ to completion.
Note that P π,θ includes any internal randomness of the policy π and the random observations y t ∼ f ( ·| θ, s t , x t ) .
Thus, P π,θ assigns a probability to any trajectory { ( x i , y i ) }
Ti =1 .
For a given policy π and θ ∗ = θ , the metric of interest we wish to minimize is the expected loss (cid:96) ( π, θ ) :
=
E π,θ [ L T ] where L T as deﬁned above is the loss observed at the end of the episode.
For a ﬁxed policy π , (cid:96) ( π, θ ) deﬁnes a loss surface over all possible values of θ .
This loss surface captures the fact that some values of θ are just intrinsically harder than others, but also that a policy may be better suited for some values of θ versus others.
Example : In active binary classiﬁcation, T is a label budget, X could be a set of images such that we can query the label of example image x t ∈ X , y t ∈ {− 1 , 1 } is the requested binary label, and the loss L T is the classiﬁcation error of a trained classiﬁer on these collected labels.
maximum gap to this rdependent baseline policy.
Finally, θ x = p ( y = 1 | x ) for all x ∈ X .
More examples can be found in Appendix A.
2. I NSTANCE DEPENDENT PERFORMANCE METRIC
We now deﬁne the sense in which we wish to evaluate a particular policy.
For any ﬁxed value of θ one could clearly design an algorithm that would maximize performance on θ , but then it might have very poor performance on some other value θ (cid:48) (cid:54)
= θ .
Thus, we would ideally like π to perform uniformly well over a set of θ ’s that are all equivalent in a certain sense.
Deﬁne a positive function
C : Θ → (0 , ∞ ) that assigns a score to each θ ∈ Θ that intuitively captures the “difﬁculty” of a particular θ , and can be used as a partial ordering of Θ .
Ideally, C ( θ ) is a monotonic transformation of (cid:96) ( (cid:101)
π, θ ) for some “best” policy (cid:101) π that we will deﬁne shortly.
We give the explicit C ( θ ) for the active binary classiﬁcation example in Section 3, further description of C in Section 2.2, and more examples in Appendix A. For any set of problem instances Θ deﬁne
[Equation]
And for any r ≥ 0 , deﬁne
[Equation]
The quantity (cid:96) ( π, Θ ( r ) ) − inf π (cid:48) (cid:96) ( π (cid:48) , Θ ( r ) ) is then a function of r that describes the sub-optimality gap of a given policy π relative to an r -dependent baseline policy trained speciﬁcally for each r .
For a ﬁxed r k > 0 , a policy π that aims to minimize just (cid:96) ( π, Θ ( r ) ) might focus just on the hard instances (i.e., those with C ( θ ) close to r ) and there may exist a different policy π (cid:48) that performs far better than π on easier instances (i.e., those with C ( θ ) (cid:28) r ).
To avoid this, assuming sup r (cid:96) ( π, Θ ( r ) ) − inf π (cid:48) (cid:96) ( π (cid:48) , Θ ( r ) ) < ∞ , we deﬁne
[Equation]
(1) as the policy that minimizes the worst case sub-optimality gap over all r > 0 .
Figure 1 illustrates these deﬁnitions.
Instead of computing inf π (cid:48)
(cid:96) ( π (cid:48) , Θ ( r ) ) for all r , in practice we deﬁne a grid with an increasing sequence { r k } Kk =1 , to ﬁnd an approximation to π ∗ .
We are now ready to state the goal of this work:
Objective: Given an increasing sequence r 1 < · · · < r K that indexes nested sets of problem instances of increasing difﬁculty, Θ ( r 1 ) ⊂
Θ ( r 2 ) ⊂ · · · ⊂ Θ ( r K ) , we wish to identify a policy (cid:98) π that minimizes the maximum sub-optimality gap with respect to this sequence.
Explicitly, we seek to learn
[Equation]
where (cid:96) ( π, Θ) : = sup θ ∈ Θ (cid:96) ( π, θ ) and (cid:96) ( π, θ ) is the expected loss incurred by policy π on instance θ . Note that as K → ∞ and sup k r
k
{++
 r+}
k → 1 , (1) and (2) are essentially equivalent under benign smoothness conditions on C ( θ ) , in which case (cid:98) π
→ π ∗ . In practice, we choose a ﬁnite K where Θ r K contains all problems that can be solved within the budget T relatively accurately, and a small (cid:15) > 0 , where max k r k +1 r k = 1 + (cid:15) . Furthermore, the objective in (2) is equivalent with
[Equation]
where π k ∈ arg inf
[Equation]
We can efﬁciently solve this objective by ﬁrst computing π k for all k ∈ [ K ] to obtain (cid:96) ( π k , Θ ( r k ) ) as benchmarks, and then use these benchmarks to train (cid:98) π .
2. P ICKING THE COMPLEXITY FUNCTION C ( θ )
We have deﬁned an optimal policy in terms of a function C ( θ ) that determines a partial ordering over instances θ .
This function can come from a heuristic that intuitively captures the difﬁculty of an instance. Or it can be deﬁned and motivated from information theoretic lower bounds that often describe a general ordering, but are typically very loose relative to empirical performance.
For example, consider the standard multi-armed bandit game where an agent has access to K distributions and in each round t ∈ [ T ] she chooses a distribution
I t ∈
[ K ] and observes a random variable in [0 , 1] with mean θ I t .
If her strategy is described by a policy π , once t reaches T she receives loss
L T = max i ∈
{+[ K ] (cid:80) Tt =
 θ i − θ I t with expectation (cid:96) ( π, θ ) = E [ L T ] where the expectation is taken with respect to the randomness+} in the observations, and potentially any randomness of the policy.
Under benign conditions, it is known that any policy must suffer (cid:96) ( π, θ ) (cid:38) min {√ KT, (cid:80) i (cid:54) =
∗ ( θ ∗ − θ i ) − 1 } where θ ∗ = max i ∈ [ k ] θ i (Lattimore & Szepesv´ari, 2018).
Such a lower bound is an ideal candidate for C ( θ ) .
We deﬁne a different C ( θ ) for our particular experiments of interest, and others are described in Appendix A.
The bottom line is that any function C ( θ ) works, but if it happens to correspond to an information theoretic lower bound, the resulting policy will match the lower bound if it is achievable.
2. D IFFERENTIABLE POLICY OPTIMIZATION
The ﬁrst step in learning the policy (cid:98) π deﬁned in Equation 2 is to learn each π k :=
inf π sup θ : C ( θ ) ≤ r k (cid:96) ( π, θ ) for all k = 1 , . .
, K .
Once all π k are deﬁned, (cid:98) π of (2) is an optimization of the same form after shifting the loss by the scalar (cid:96) ( π k , Θ ( r k ) ) .
Consequently, to learn (cid:98) π it sufﬁces to develop a training procedure to solve inf π sup θ ∈ Ω (cid:96) (cid:48) ( π, θ ) for an arbitrary set Ω and generic loss function (cid:96) (cid:48) ( π, θ ) .
To make the optimization problem inf π sup θ ∈ Ω (cid:96) (cid:48) ( π, θ ) tractable, we parameterize it as follows. First, to compute the suprema over Θ , we consider a ﬁnite set (cid:101) Θ := { (cid:101) θ i } Ni =1 ⊂ Ω , weighted by SOFTMAX ( w ) where w ∈ R N . In addition, instead of optimizing over all possible policies, we restrict the policy as the class of neural networks that take state representation as input and output a probability distribution over actions, parameterized by weights ψ .
Mathematically, it could be stated as the following:
[Equation]
optimal arginf π sup θ ∈ Ω (cid:96) ( π, θ ) =
arginf
π sup θ ∈ Ω E π,θ [ L T ] .
do
Collect rollouts of play: indices
I I M i.i.d. ∼ SOFTMAX ( w ) do Collect L independent rollout trajectories, denoted as τ m, 1: L , by the policy π ψ for problem instant θ I m and observe losses ∀ 1 ≤ l ≤ L
L T π ψ , τ m l θ I m )
end generating ascending gradient estimates:
[Equation] where L barrier is a differentiable barrier loss that heavily penalizes the (cid:101) θ I m ’s outside Ω . Optimize policy
: Update the policy by taking descending step on gradient estimate:
[Equation] 14 end Note that the objectives in (3) and (4) are indeed equivalent as (cid:101) θ 1: N are free parameters we optimize over rather than taking ﬁxed values. Now, to motivate (5), starting from the left hand side of (3), observe that a small change in π may result in a large change in argsup θ ∈ Ω (cid:96) ( π, θ ) .
Therefore, with the goal of covering the entire Ω , we optimize the N points so that when π changes a bit, there is at least one (cid:101) θ i close to the optimal argsup .
In addition, to covering the entire space of Ω , N is expected to be very large in practice.
However, to optimize the objective effectively, we can only evaluate on M of (cid:101) θ ’s ( M (cid:28) N ) in each iteration.
Therefore, instead of naively sampling M points uniformly at random from the N points, in (4), we optimize an extra multinomial distribution, SOFTMAX ( w ) , over the N points so that the points around the argsup are sampled more often.
The ﬁnal approximation in (5) comes from parameterizing the policy by a neural network.
To solve the saddle point optimization problem in (5), we use an instance of the Gradient Descent
Ascent (GDA) algorithm as shown in Algorithm 1.
The gradient estimates are unbiased estimates of the true gradients with respect to ψ , w and
(cid:101) Θ (shown in Appendix B).
We choose N large enough to avoid mode collapse , and M, L as large as possible to reduce variance in gradient estimates while ﬁtting the memory constraint.
We use Adam optimization (Kingma & Ba, 2014) in taking gradient updates and regularize some of the parameters (an example will be presented in the next section).
Note the decomposition for log( P π ψ ,θ (cid:48) ( τ )) in (7) and (8), where rollout τ = { ( x t , y t ) }
Tt =1 , and
[Equation]
=
[Equation]
x t ) (cid:17) .
Here π ψ and f are only dependent on ψ and (cid:101) Θ respectively.
During evaluation of a ﬁxed policy π , we are interested in solving sup θ ∈ Ω (cid:96) ( π, θ ) by gradient ascent updates like (7).
The decoupling of π ψ and f thus enables us to optimize the objective without differentiating through a policy π , which could be non-differentiable policies like deterministic algorithms.
Finally, we make a few remarks on the parameterization of (5).
As given in (5), we represent the generating distribution P as a simple ﬁnite number of weighted particles, analogous to a particle ﬁlter.
Our policy parameterization π ψ could be modelled by multi-layer perceptrons, recurrent neural networks, etc.
We note that when using alternative generator parameterization like GANs (Goodfellow et al., 2014), an unbiased gradient can also be derived similarly.
I MPLEMENTATION FOR L INEAR B ANDITS AND C LASSIFICATION
We now apply the general framework of the previous section to a speciﬁc problem: transductive linear bandits.
As described in Sections 5 and Appendix A this setting generalizes standard multiarmed bandits, linear bandits, and all of binary classiﬁcation through a simple reduction to combinatorial bandits.
We are particularly motivated to look at classiﬁcation because the existing agnostic active learning algorithms are very inefﬁcient (see Section 5).
Indeed when applied to our setting of
T = 20 they never get past their ﬁrst stage of uniform random sampling.
Consider the game:
Input : Policy π , X ⊂ R d , Z ⊂ R d , time horizon T ∈ T Initialization : Nature chooses θ ∗ ∈ R d (hidden from policy) for t = 1 , 2 , . . .
, T • Policy π selects x t ∈ X using history { ( x s , y s ) } t − 1 s = •
Nature reveals y t ∼ f ( ·| θ ∗ , x t ) with E [ y t | θ ∗ , x t ] = (cid:104) x
t , θ ∗ (cid:105) Output : Policy
π
recommends (cid:98)
z ∈ Z as an
estimate for z (cid:63) ( θ ∗ ) : = argmax z ∈Z (cid:104) z, θ ∗ (cid:105) and suffers loss L T =
[Equation]
The observation distribution f ( ·| θ, x ) is domain speciﬁc but typically taken to be either a Bernoulli distribution for binary data, or Gaussian for real-valued data.
We are generally interested in two objectives: B EST I DENTIFICATION which attempts to exactly identify the vector z (cid:63) ∈
Z that is most aligned with θ ∗ , and S IMPLE R EGRET which settles for an approximate maximizer.
Deﬁning C ( θ ) Recalling the discussion of Section 2.1, C ( θ ) should ideally be monotonically increasing in the intrinsic difﬁculty of minimizing the loss with respect to a particular θ .
For arbitrary
X ⊂ R d and Z ⊂ R d , it is shown in Fiez et al.
that the sample complexity of identifying z (cid:63) ( θ ) =
arg max z ∈Z (cid:104)
z, θ (cid:105) with high probability is proportional to a quantity ρ (cid:63) ( θ ) , the value obtained by an optimization program.
Another complexity term that appears in the combinatorial bandits literature (Cao & Krishnamurthy, 2017)
where
X = { e i : i ∈ [ d ] } and Z ⊂ { 0 , 1 } d is
[Equation]
One can show ρ (cid:63) ( θ ) ≤ (cid:101) ρ ( θ ) and in many cases track each other.
Because (cid:101) ρ ( θ ) can be computed much more efﬁciently compared to ρ (cid:63) ( θ ) , we use C ( θ )
= (cid:101) ρ ( θ ) in our experiments.
// In this case, π
Algorithm
Ω = Θ ( r K ) objective in (2), i.e., L T = (
cid:104) z (cid:63) ( θ ) − (cid:98) z, θ (cid:105) − (cid:96) ( (cid:101) π k ( θ ) , Θ ( r k ( θ ) ) )
7 else if obj is B EST I DENTIFICATION then
For each k ∈ [ K ] , warm start π k = (cid:101)
π k ; optimize π k by Algorithm 1 with Ω = Θ ( r k ) and B EST I DENTIFICATION loss; compute (cid:96) ( π k , Θ ( r k ) )
Warm start (cid:101) π = π (cid:98) K/ 2 (cid:99) ; optimize (cid:101) π by Algorithm 1 with Ω = Θ
( r K ) and objective in (2), i.e., L T = {+
 } z (cid:63) ( θ ) {+(cid:54) = (cid:98) z } − (cid:96) ( π k ( θ ) , Θ ( r k ( θ ) ) )
10 end 11 Output : (cid:98) π (an approximate solution to (2))
Training.
When training our policies, we follow the following procedure in Algorithm 2.
Note that even when we are training for B EST I DENTIFICATION , we still warm start the training with optimizing S IMPLE R EGRET .
This is because a random initialized policy performs so poorly that
B EST I DENTIFICATION is nearly always 1, making it difﬁcult to improve the policy.
In addition, our generating distribution parameterizations exactly follows from Section 2.3, while detailed state representations, policy parametrization and hyperparamters can be found in Appendix C, D.
Loss functions.
Instead of optimizing the approximated quantity from (5) directly, we add regularizers to the losses for both the policy and generator.
First, we choose the L barrier in (7) to be λ barrier · max { 0 , log( C ( X , Z , θ ))
− log( r k ) } , for some large constant λ barrier .
To discourage the policy from over committing to a certain action and/or the generating distribution from covering only a small subset of particles (i.e., mode collapse), we also add negative entropy penalties to both policy’s output distributions and SOFTMAX ( w ) with scaling factors λ Pol-reg and λ Gen-reg .
E XPERIMENTS
We now evaluate the approach described in the previous section for combinatorial bandits with
X = { e i : i ∈ [ d ] } and Z ⊂ { 0 , 1 } d .
We stress that the framework implemented here can be applied to any X , Z ⊂ R d and any appropriate f –just plug and play to learn a new policy.
In our experiments we take particular instances of combinatorial bandits with Bernoulli observations.
We evaluated based on two criterion: instance-dependent worst-case and average-case.
For instancedependent worst-case, we measure, for each r k and policy π , (cid:96) ( π, Θ
( r k ) ) : = max θ ∈ Θ ( rk ) (cid:96) ( π, θ ) and plot this value as a function of r k . We note that our algorithm is designed to optimize for such metric. For the secondary average-case metric, we instead measure, for policy π and some collected set Θ , 1 | Θ | (cid:80) θ ∈ Θ (cid:96) ( π, θ ) .
Performances of instance-dependent worst-case metric are reported in
Figures 2, 3, 4, 6, and 7 below while the average case performances are reported in the tables and
Figure 5.
Full scale of the ﬁgures can also be found in Appendix F.
Algorithms.
We compare against a number of baseline active learning algorithms (see Sectionfor a review).
U NCERTAINTY SAMPLING at time t computes the empirical maximizer of (cid:104)
z, (cid:98) θ (cid:105) and the runner-up, and samples an index uniformly from their symmetric difference; if either are not unique, an index is sampled from the region of disagreement of the winners (see Appendix G for details).
The greedy methods are represented by soft generalized binary search (SGBS) (Nowak, 2011) which maintains a posterior distribution over Z and samples to maximize information gain.
A hyperparameter β ∈ (0 , 1 / 2) of SGBS determines the strength of the likelihood update.
We plot or report a range of performance over β ∈ { . 01 , .
The agnostic algorithms for classiﬁcation (Dasgupta, 2006; Hanneke, 2007b;a; Dasgupta et al., 2008; Huang et al., 2015; Jain &
Jamieson, 2019) or combinatorial bandits (Chen et al., 2014; Gabillon et al., 2016; Chen et al., 2017;
Cao & Krishnamurthy, 2017; Fiez et al., 2019; Jain & Jamieson, 2019) are so conservative that given just T = 20 samples, they are all exactly equivalent to uniform sampling and hence represented by
U NIFORM .
To represent a policy based on learning to actively learn (LAL), we employ the method of Kveton et al.
(2020) with a ﬁxed prior (cid:101) P constructed by drawing a z uniformly at random from
Z and deﬁning θ = 2 z − 1 ∈ [ − 1 , 1] d (details in Appendix H).
When evaluating each policy, we use the successive halving algorithm
(Li et al., 2017; 2018) for optimizing our non-convex objective with randomly initialized gradient descent and restarts (details in Appendix E). [Figure] [Figure] [Figure]
, lower is better
Thresholds
We begin with a very simple instance to demonstrate the instance-dependent performance achieved by our learned policy.
For d = 25 let X =
e i : i ∈
[ d ] } , Z = { 0 + (cid:80) ki =1 e i : k = 0 , 1 , . . .
, d } , and f ( ·| θ, x ) is a Bernoulli distribution over {− 1 , 1 } with mean (cid:104)
x, θ (cid:105) ∈ [ − 1 , 1] .
Note this is a binary classiﬁcation task in one-dimension where the set of classiﬁers are thresholds on a line.
We trained baseline policies { π k } 9 k =1 for the B EST I DENTIFICATION metric with C ( θ ) =
(cid:101) ρ ( X , Z , θ ) and r k = 2 3+ i/ 2 for i ∈ { 0 , . . .
First we compare the base policies π k to (cid:98) π .
Figure 2 presents (cid:96) (
[Equation]
sup
[Equation]
) as a function of r for our base policies { π k } k and the global policy π ∗ , each as an individual curve.
Figure 3 plots the same information in terms of gap: (cid:96) ( π, Θ ( r ) )
− min k : r k − 1 <r ≤ r k (cid:96) ( π k , Θ ( r k ) ) .
We observe that each π k performs best in a particular region and π ∗ performs almost as well as the r -dependent baseline policies over the range of r .
This plot conﬁrms that our optimization objective of (2) was successful.
Under the same conditions as Figure 2, Figure 4 compares the performance of π ∗ to the algorithm benchmarks.
Since SGBS and LAL are deterministic, the adversarial training ﬁnds a θ that tricks them into catastrophic failure.
Figure 5 trades adversarial evaluation for evaluating with respect to a parameterized prior:
For each h ∈ { 0 . 5 , 0 . 6 , . . . , 1 } , θ ∼ P h is deﬁned by drawing a z uniformly at random from Z and then setting θ i =
(2 z i − 1)(
α
i − 1) where α i ∼ Bernoulli ( h ) .
Thus, each sign of 2 z − 1 is ﬂipped with probability h .
We then compute E θ [Equation]
h
[Equation]
P
[Equation]
[Equation]
While SGBS now performs much better than uniform and uncertainty sampling, our policy π ∗ is still superior to these policies.
However, LAL is best overall which is expected since the support of P h is basically a rescaled version of the prior used in LAL.
20 Questions.
We now address an instance constructed from the real data of Hu et al.
Summarizing how we used the data from Hu et al.
(see Appendix I for details), 100 yes/no questions were considered for 1000 celebrities.
Each question i ∈ [100] for each person j ∈ [1000] was answered by several annotators to construct an empirical probability ¯ p ( j ) i ∈
[0 , 1] denoting the proportion of annotators that answered “yes.”
To construct our instance, we take X =
{ e i : i ∈ [100] } and Z = { z ( j ) :
[ z ( j ) i ] = 1 { ¯ p ( j ) i > 1 / 2 }} ⊂ { 0 , 1 } 1000 .
Just as before, we trained { π k } 4 k = for the B EST I DENTIFICATION metric with C ( θ ) = (cid:101) ρ ( X , Z , θ ) and r i = 2 3+ i/ 2 for i ∈ { 1 , . . .
[Table] Figure 6 is analogous to Figure 4 but for this 20 questions instance.
Uncertainty sampling performs remarkably well on this instance.
A potential explanation is that on a noiseless instance (e.g., θ =
z − 1 for some z ∈ Z ), our implementation of uncertainty sampling is equivalent to CAL (Cohn et al., 1994) and is known to have near-optimal sample complexity (Hanneke et al., 2014).
Uncertainty sampling even outperforms our r -dependent baseline by a bit which in theory should not occur–we conjecture this is due to insufﬁcient convergence of our policies or local minima.
Our second experiment constructs a distribution (cid:98) P based on the dataset: to draw a θ ∼ (cid:98) P we uniformly at random select a j ∈ [1000] and sets θ i =
2¯ p
( j ) i − 1 for all
i ∈
[ d ] .
As shown in Table 1, SGBS and π ∗ are the winners.
LAL performs much worse in this case, potentially because of the distribution shift from (cid:101) P (prior we train on) to (cid:98) P (prior at test time).
The strong performance of SGBS may be due to the fact that sign ( θ i ) = 2 z (cid:63) ( θ ) i − 1 for all i and θ ∼ (cid:98) P , a realizability condition under which SGBS has strong guarantees (Nowak, 2011).
[Table]
Jester Joke Recommendation
We now turn our attention away from best identiﬁcation of the last two experiments (cid:96) ( π, θ ) =
P π,θ ( (cid:98)
z
(cid:54) =
z (cid:63) ( θ )) , to simple regret (cid:96) ( π, θ ) =
E π,θ [ (cid:104)
z (cid:63) ( θ ) − (cid:98)
z, θ (cid:105) ] .
We consider the Jester jokes dataset of Goldberg et al.
(2001) that contains jokes ranging innocent puns to grossly offensive jokes.
We ﬁlter the dataset to only contain users that rated all 100 jokes, resulting inusers.
A rating of each joke was provided on a [ − 10 , 10] scale which was rescaled to [ − 1 , 1] and observations were simulated as Bernoulli’s like above.
We then clustered the ratings of these users (see Appendix J for details) to 10 groups to obtain Z
= { z ( k ) : k ∈ [10] , z ( k ) ∈ { 0 , 1 } 100 } where z ( k ) i
= 1 corresponds to recommending the i th joke in user cluster z
( k ) ∈ Z .
Figure 7 shows the same style plot as Figures 4,6 but for this jokes dataset, with our policy alone nearly achieving the r -dependent baseline for all r .
Mirroring the construction of the 20Q prior, we construct (cid:98) P by uniformly sampling a user and employing their θ to answer queries.
Table 2 shows that despite our policy not being trained for this setting, its performance is still among the top.
R ELATED WORK
Learning to actively learn.
Previous works vary in how the parameterize the policy, ranging from parameterized mixtures of existing expertly designed active learning algorithms (Baram et al., 2004;
Hsu & Lin, 2015; Agarwal et al., 2016), parameterizing hyperparameters (e.g., learning rate, rate of forced exploration, etc.) in an existing popular algorithm (e.g, EXP3) (Konyushkova et al., 2017;
Bachman et al., 2017; Cella et al., 2020), and the most ambitious, policies parameterized end-to-end like in this work (Boutilier et al., 2020; Kveton et al., 2020; Sharaf & Daum´e III, 2019; Fang et al., 2017; Woodward & Finn, 2017).
These works take an approach of deﬁning a prior distribution either through past experience (meta-learning) or expert created (e.g., θ ∼ N (0 , Σ) ), and then evaluate their policy with respect to this prior distribution.
Deﬁning this prior can be difﬁcult, and moreover, if the θ encountered at test time did not follow this prior distribution, performance could suffer signiﬁcantly.
Our approach, on the other hand, takes an adversarial training approach and can be interpreted as learning a parameterized least favorable prior (Wasserman, 2013), thus gaining a much more robust policy as an end result.
Robust and Safe Reinforcement Learning:
Our work is also highly related to the ﬁeld of robust and safe reinforcement learning, where our objective can be considered as an instance of minimax criterion under parameter uncertainty (Garcıa & Fern´andez, 2015).
Widely applied in applications such as robotics (Mordatch et al., 2015; Rajeswaran et al., 2016), these methods train a policy in a simulator like Mujoco (Todorov et al., 2012) to minimize a deﬁned loss objective while remaining robust to uncertainties and perturbations to the environment (Mordatch et al., 2015; Rajeswaran et al., 2016).
Ranges of these uncertainty parameters are chosen based on potential values that could be encountered when deploying the robot in the real world.
In our setting, however, deﬁning the set of environments is far less straightforward and is overcome by the adoption of the C ( θ ) function.
Active Binary Classiﬁcation Algorithms.
The literature on active learning algorithms can be partitioned into model-based heuristics like uncertainty sampling, query by committee, or modelchange sampling (Settles, 2009), greedy binary-search like algorithms that typically rely on a form of bounded noise for correctness (Dasgupta, 2005; K¨a¨ari¨ainen, 2006; Golovin & Krause, 2011;
Nowak, 2011), and agnostic algorithms that make no assumptions on the probabilistic model (Dasgupta, 2006; Hanneke, 2007b;a; Dasgupta et al., 2008; Huang et al., 2015; Jain & Jamieson, 2019).
Though the heuristics and greedy methods can perform very well for some problems, it is typically easy to construct counter-examples (e.g., outside the assumptions) in which they catastrophically fail (as demonstrated in our experiments).
The agnostic algorithms have strong robustness guarantees but rely on concentration inequalities, and consequently require at least hundreds of labels to observe any deviation from random sampling (see Huang et al.
(2015) for comparison).
Therefore, they were not included in our experiments explicitly but were represented by uniform.
Pure-exploration Multi-armed Bandit Algorithms.
In the linear structure setting, for sets
X , Z ⊂ R d known to the player, pulling an “arm” x ∈ X results in an observation (cid:104) x, θ ∗ (cid:105) + zeromean noise, and the objective is to identify arg max z ∈Z (cid:104)
z, θ ∗ (cid:105) for a vector θ ∗ unknown to the player (Soare et al., 2014; Karnin, 2016;
Tao et al., 2018; Xu et al., 2017; Fiez et al., 2019).
A special case of linear bandits is combinatorial bandits
where X = { e i : i ∈ [ d ] } and Z ⊂ { 0 , 1 } d (Chen et al., 2014; Gabillon et al., 2016; Chen et al., 2017; Cao & Krishnamurthy, 2017; Fiez et al., 2019; Jain & Jamieson, 2019).
Active binary classiﬁcation is a special case of combinatorial pure-exploration multi-armed bandits (Jain & Jamieson, 2019), which we exploit in the threshold experiments.
While the above works have made great theoretical advances in deriving algorithms and information theoretic lower bounds that match up to constants, the constants are so large that these algorithms only behave well when the number of measurements is very large.
When applied to the instances of our paper (only 20 queries are made), these algorithms behave no differently than random sampling.
D ISCUSSION AND F UTURE D IRECTIONS
We see this work as an exciting but preliminary step towards realizing the full potential of this general approach.
From a practical perspective, training a π ∗ can take many hours of computational resources for even these small instances.
Scaling these methods to larger instances is an important next step.
While training time scales linearly with the horizon length T , we note that one can take multiple samples per time step with minimal computational overhead enabling problems that require larger sample complexities.
In our implementation we hard-coded the decision rule for (cid:98)
z given s T , while it could also be learned as in (Luedtke et al., 2020).
Likewise, the parameterization of the policy and generator worked well for our purposes but was chosen somewhat arbitrarily–are there more natural choices?
Finally, while we focused on stochastic settings, this work naturally extends to constrained fully adaptive adversarial sequences which is an interesting direction of future work.
F UNDING DISCLOSURE
Removed for anonymization purposes.
A CKNOWLEDGEMENT
Removed for anonymization purposes.
A I NSTANCE DEPENDENT SAMPLE COMPLEXITY
Identifying forms of C ( θ ) is not as difﬁcult a task as one might think due to the proliferation of tools for proving lower bounds for active learning (Mannor & Tsitsiklis, 2004; Tsybakov, 2008; Garivier & Kaufmann, 2016; Carpentier & Locatelli, 2016; Simchowitz et al., 2017; Chen et al., 2014).
One can directly extract values of C ( θ ) from the literature for regret minimization of linear or other structured bandits (Lattimore & Szepesvari, 2016; Van Parys & Golrezaei, 2020), contextual bandits (Hao et al., 2019), and tabular as well as structured MDPs (Simchowitz & Jamieson, 2019;
Ok et al., 2018).
Moreover, we believe that even reasonable surrogates of C ( θ ) should result in a high quality policy π ∗ .
We review some canonical examples:
• Multi-armed bandits.
In the best-arm identiﬁcation problem, there are d ∈ N Gaussian distributions where the i th distribution has mean θ i ∈ R for i = 1 , . . .
, d .
In the above formulation, this problem is encoded as action x t = i t results in observation y t ∼ Bernoulli
( θ i t ) and the loss (cid:96) ( π, θ ) :
= E π,θ [ 1 { (cid:98) i (cid:54) =
i (cid:63) ( θ ) } ] where (cid:98) i is π ’s recommended index and i (cid:63) ( θ ) =
arg max i
θ i .
It’s been shown that there exists a constant c 0 > 0 such that for any sufﬁciently large ν > 0 we have inf π [Equation] [Equation] exp
[Equation]
0 T/ν ) where
[Equation]
− θ i ) −
Moreover, for any θ ∈ R d there exists a policy (cid:101) π that achieves (cid:96) ( (cid:101) π, θ ) ≤ c 1 exp( − c 2 T/ C MAB ( θ )) where c 1 , c 2 capture constant and low-order terms (Carpentier & Locatelli, 2016; Karnin et al., 2013; Simchowitz et al., 2017; Garivier & Kaufmann, 2016).
The above correspondence between the lower bound and the upper bound suggests that C MAB ( θ ) plays a critical role in determining the difﬁcult of identifying i (cid:63) ( θ ) for any θ .
This exercise extends to more structured settings as well:
• Content recommendation / active search.
Consider n items (e.g., movies, proteins) where the i th item is represented by a feature vector x i ∈
X
⊂ R d and a measurement x t = x i (e g., preference rating, binding afﬁnity to a target) is modeled as a linear response model such that y t ∼ N ( (cid:104) x
i , θ (cid:105) , 1) for some unknown θ ∈ R d .
If (cid:96) ( π, θ ) :=
E π,θ [ 1 { (cid:98) i (cid:54) =
i (cid:63) ( θ ) } ] as above then nearly identical results to that of above hold for an analogous function of C MAB ( θ ) (Soare et al., 2014; Karnin, 2016; Fiez et al., 2019).
• Active binary classiﬁcation.
For i = 1 , . . .
, d let φ
i ∈ R p be a feature vector of an unlabeled item (e.g., image) that can be queried for its binary label y i ∈ {− 1 , 1 } where y i ∼ Bernoulli ( θ i ) for some θ ∈ R d .
Let H be an arbitrary set of classiﬁers (e.g., neural nets, random forest, etc.)
such that each h ∈ H assigns a label {− 1 , 1 } to each of the items { φ i } di =1 in the pool.
If items
are chosen sequentially to observe their labels, the objective is to identify the true risk minimizer h (cid:63) ( θ ) =
arg min h ∈H (cid:80) di
=1 E θ
[ 1 { h ( φ i ) (cid:54) =
y i } ] using as few requested labels as possible and (cid:96) ( π, θ ) :
= E π,θ [ 1 { (cid:98) h (cid:54) =
h (cid:63) ( θ ) } ] where (cid:98) h ∈ H is π ’s recommended classiﬁer.
Many candidates for C ( θ ) have been proposed from the agnostic active learning literature (Dasgupta, 2006;
Hanneke, 2007b;a; Dasgupta et al., 2008; Huang et al., 2015; Jain & Jamieson, 2019)
but we believe the most granular candidates come from the combinatorial bandit literature (Chen et al., 2017; Fiez et al., 2019; Cao & Krishnamurthy, 2017; Jain & Jamieson, 2019).
To make the reduction, for each h ∈ H assign a z ( h ) ∈ { 0 , 1 } d such that [ z ( h ) ]
i := 1 { h ( φ i ) = 1 } for all i = 1 , . . .
, d and set Z =
{ z ( h ) : h ∈ H} .
It is easy to check that z (cid:63) ( θ ) :
= arg max z ∈Z (cid:104)
z, θ (cid:105) satisﬁes z (cid:63) ( θ ) =
z ( h (cid:63) ( θ )) .
Thus, requesting the label of example i is equivalent to sampling from Bernoulli ( (cid:104)
e i , θ (cid:105) ) ∈ {− 1 , 1 } , completing the reduction to combinatorial bandits:
X = { e i : i ∈ [ d ] } , Z ⊂ { 0 , 1 } d .
We then apply the exact same C ( θ ) as above for linear bandits.
B G RADIENT E STIMATE D ERIVATION
Here we derive the unbiased gradient estimates (6), (7) and (8) in Algorithm 1.
Since each the gradient estimates in the above averages over M · L identically distributed trajectories, it is therefore sufﬁcient to show that our gradient estimate is unbiased for a single problem (cid:101) θ
i and its rollout trajectory { ( x t , y t ) } Tt =1 .
[Equation]
Observe that if i ∼ SOFTMAX ( w ) and { ( x t , y t ) }
Tt =1 is the result of rolling out a policy π ψ on (cid:101)
θ i then
[Equation]
is an unbiased estimate of ∇
[Equation]
For a feasible set (cid:101) Θ , by deﬁnition of (cid:96) ( π, θ ) ,
[Equation]
θ
[Equation]
( { ( x t , y t ) } Tt =1 ))
(cid:105)(cid:105) where the last equality follows from chain rule and the score-function identity (Aleksandrov et al., 1968).
The quantity inside the expectations, call it g (cid:101) Θ , is then an unbiased estimator of ∇ (cid:101)
Θ E i ∼ SOFTMAX ( w ) (cid:104) (cid:96) ( π ψ , (cid:101) θ i ) (cid:105) given i and { ( x t , y t ) }
Tt =1 are rolled out accordingly.
Note that if L barrier (cid:54) = 0 , ∇ (cid:101) Θ L barrier ( (cid:101) θ i , Ω) is clearly an unbiased gradient estimator of
E i ∼ SOFTMAX ( w ) [ E π, (cid:101) θ
i
[ L barrier (
[Equation] θ i [Equation] Ω)]] given i and rollout are sampled accordingly.
Likewise, for policy,
[Equation]
is an unbiased estimate of ∇ ψ E i ∼ SOFTMAX ( w ) (cid:104) (cid:96) ( π ψ , (cid:101) θ i ) (cid:105) .
C L INEAR B ANDIT P ARAMETERIZATION C. S TATE R EPRESENTATION
We parameterize our state space S as a ﬂattened |X|× 3 matrix where each row represents a distinct x ∈ X .
Speciﬁcally, at time t the row of s t corresponding to some x ∈ X records the number of times that action x has been taken (cid:80) t − 1 s =1 1 { x s = x } , its inverse ( (cid:80) t − 1 s =1 1 { x s = x } ) − 1 , and the sum of the observations (cid:80) t − 1 s =1 1 { x s = x } y s .
C. P OLICY MLP A RCHITECTURE
Our policy π ψ is a multi-layer perceptron with weights ψ .
The policy take a 3 |X| sized state as input and outputs a vector of size |X| which is then pushed through a soft-max to create a probability distribution over X .
At the end of the game, regardless of the policy’s weights, we set (cid:98) z
= argmax z ∈Z (cid:104)
z, (cid:98) θ (cid:105) where (cid:98) θ is the minimum (cid:96) 2 norm solution to argmin θ (cid:80)
Ts =
( y s − (cid:104) x s , θ (cid:105) ) 2 .
Our policy network is a simple 6-layer MLP, with layer sizes { 3 |X| , 256 , 256 , 256 , 256 , |X|} where 3 |X| corresponds to the input layer and |X| is the size of the output layer, which is then pushed through a Softmax function to create a probability over arms.
In addition, all intermediate layers are activated with the leaky ReLU activation units with negative slopes of . 01 .
For the experiments for 1D thresholds and 20 Questions, they share the same network structure as mentioned above with |X| = 25 and |X| = 100 respectively.
D H YPER - PARAMETERS
In this section, we list our hyperparameters.
First we deﬁne λ binary to be a coefﬁcient that gets multiplied to binary loses, so instead of 1 { z (cid:63) ( θ ∗ ) (cid:54) =
(cid:98) z } , we receive loss λ binary · 1 { z (cid:63) ( θ ∗ ) (cid:54) =
(cid:98) z } .
We choose λ binary so that the recieved rewards are approximately at the same scale as S IMPLE R EGRET .
During our experiments, all of the optimizers are Adam.
All budget sizes are T = 20 .
For fairness of evaluation, during each experiment (1D thresholds or 20 Questions), all parameters below are shared for evaluating all of the policies.
To elaborate on training strategy proposed in Algorithmmore, we divide our training into four procedures, as indicated in Table 3:
• Init.
The initialization procedure takes up a rather small portion of iterations primarily for the purpose of optimizing for L barrier so that the particles converge into the constrained difﬁculty sets.
In addition, during the initialization process we initialize and freeze w = (cid:126) 0 , thus putting an uniform distribution over the particles.
This allows us to utilize the entire set of particles without w converge to only a few particles early on.
To initialize (cid:101) Θ , we sample 2 / 3 of the N particles uniformly from [ − 1 , 1] |X| and the rest 1 / 3 of the particles by sampling, for each i ∈ [ |Z| ] , N 3 |Z| particles uniformly from { θ : argmax j (cid:104) θ, z j (cid:105) = i } .
We initialize our policy weights by Xavier initialization with weights sampled from normal distribution and scaled by .
• Regret Training, (cid:101)
π i Training with S IMPLE R EGRET objective usually takes the longest among the Procedures.
The primary purpose for this process is to let the policy converge to a reasonable warm start that already captures some essence of the task.
• Fine-tune π i .
Training with B EST I DENTIFICATION objective run multiple times for each π i with their corresponding complexity set Θ i .
During each run, we start with a warm started policy, and reinitialize the rest of the models by running the initialization procedure followed by optimizing the B EST I DENTIFICATION objective.
• Fine-tune (cid:98) π
This procedure optimizes (2), with baselines min k (cid:96) ( π k , Θ ( r k ) ) evaluated based on each π i learned from the previous procedure.
Similar to ﬁne-tuning each individual π
i , we warm start a policy π (cid:98) K/ 2 (cid:99) and reinitialize w and Θ by running the initialization procedure again.
[Table]
|X|
[Table]
[Table]
|X| =
Jester Joke |X| =
[Table]
N it 20000 (all)
[Table]
[Table] 4 (all) [Table] rate
[Table]
[Table]
[Table]
To provide a general strategy of choosing hyper-parameters, we note that L , ﬁrstly, λ binary , λ Pol-reg are primarily parameters tuned for |X| as the noisiness and scale of the gradients, and entropy over the arms X grows with the size |X| .
Secondly, λ Gen-reg is primarily tuned for |Z| as it penalizes the entropy over the N arms, which is a multiple of |Z| .
Thirdly, learning rate of θ is primarily tuned for the convergence of constraint ρ ∗ into the restricted class, thus L barrier becoming 0 after the speciﬁed number of iterations during initialization is a good indicator.
Finally, we choose N and M by memory constraint of our GPU.
The hyper-parameters for each experiment was tuned with less than 20 hyper-parameter assignments, some metrics to look at while tuning these hyperparameters includes but are not limited to: gradient magnitudes of each component, convergence of each loss and entropy losses for each regularization term (how close it is to the entropy of a uniform probability), etc.
E P OLICY E VALUATION
When evaluating a policy, we are essentially solving the following objective for a ﬁxed policy π :
[Equation]
where Ω is a set of problems.
However, due to non-concavity of this loss function, gradient descent initialized randomly may converge to a local maxima.
To reduce this possibility, we randomly initialize many initial iterates and take gradient steps round-robin, eliminating poorly performing trajectories.
To do this with a ﬁxed amount of computational resource, we apply the successive halving algorithm from Li et al.
Speciﬁcally, we choose hyperparamters:
η = 4 , r = 100 ,
R = 1600 and s = 0 .
This translates to:
• Initialize | (cid:101) Θ | = 1600 , optimize for 100 iterations for each (cid:101) θ i ∈ (cid:101)
Θ • Take the top 400 of them and optimize for another 400 iterations • Take the top 100 of the remaining 400 and optimize for an additional 1600 iterations
We take gradient steps with the Adam optimizer (Kingma & Ba, 2014) with learning rate of 10 −β 1 = .
9 and β 2 = .
G U NCERTAINTY S AMPLING
We deﬁne the symmetric difference of a set of binary vectors, SymDiff ( { z 1 , ..., z n } )
= { i : ∃ j, k ∈ [ n ] s.t., z
( i ) j =
∧ z ( i ) k
= 0 } , as the dimensions where inconsistencies exist.
1 Input: X , Z 2 for t = 1 , ..., T do (cid:98) θ t − 1 = argmin θ (cid:80)
Ts =
( y s − (cid:104) x s , θ (cid:105) )
(cid:98) Z = { z ∈ Z : max z (cid:48) ∈Z (cid:104) z (cid:48) , (cid:98) θ t − 1 (cid:105) =
(cid:104) z, (cid:98) θ t − 1 (cid:105)} if | (cid:98) Z| =
1 then (cid:98) Z t = (cid:98) Z (cid:83) { z ∈ Z : max z (
cid:48) ∈
( Z\ (cid:98) Z ) (cid:104) z (cid:48) , (cid:98) θ t − 1 (cid:105) =
(cid:104) z, (cid:98) θ t − 1 (cid:105)} else (cid:98) Z t
=
(cid:98) Z end
Uniformly sample I t from SymDiff ( (cid:98) Z t )
Pull x I t and observe y t 12 end
H L EARNING TO A CTIVELY L EARN A LGORITHM
To train a policy under the learning to actively learn setting, we aim to solve for the objective
[Equation]
] where our policy and states are parameterized the same way as Appendix C for a fair comparison.
To optimize for the parameter, we take gradient steps like (8) but with the new sampling and rollout where (cid:101) θ i ∼ (cid:98) P .
This gradient step follows from both the classical policy gradient algorithm in reinforcement learning as well as from recent LAL work by Kveton et al.
Moreover, note that the optimal policy for the objective must be deterministic as justiﬁed by deterministic policies being optimal for MDPs.
Therefore, it is clear that, under our experiment setting, the deterministic LAL policy will perform poorly in the adversarial setting (for the same reason why
SGBS performs poorly).
I 20 Q UESTIONS S ETUP
Hu et al.
(2018) collected a dataset of 1000 celebrities and 500 possible questions to ask about each celebrity.
We chose 100 questions out of the 500 by ﬁrst constructing ¯ p (cid:48) , X (cid:48) and Z (cid:48) for thedimensions data, and sampling without replacement 100 of the 500 dimensions from a distribution derived from a static allocation.
We down-sampled the number of questions so our training can run with sufﬁcient M and L to de-noise the gradients while being prototyped with a single GPU.
Speciﬁcally, the dataset from Hu et al.
(2018) consists of probabilities of people answering Yes /
No / Unknown to each celebrity-question pair collected from some population.
To better ﬁt the linear bandit scenario, we re-normalize the probability of getting Yes / No , conditioning on the event that these people did not answer Unknown .
The probability of answering Yes to all 500 questions for each celebrity then constitutes vectors ¯ p (cid:48) (1) , ..., ¯ p (cid:48) (1000) ∈ R 500 , where each dimension of a give ¯ p (cid:48) ( j ) i represents the probability of yes to the i th question about the j th person.
The action set X (cid:48) is then constructed as X (cid:48)
= { e i : i ∈ [500] } , while Z (cid:48) =
{ z ( j ) :
[ z ( j ) i ] = 1 { ¯ p ( j ) i > 1 / 2 }} ⊂ { 0 , 1 } 1000 are binary vectors taking the majority votes.
To sub-sample 100 questions from the 500, we could have uniformly at random selected the questions, but many of these questions are not very discriminative.
Thus, we chose a “good” set of queries based on the design recommended by ρ (cid:63) of Fiez et al.
If questions were being answered noiselessly in response to a particular z ∈ Z (cid:48) , then equivalently we have that for this setting θ = 2 z − 1 .
Since ρ (cid:63) optimizes allocations λ over X (cid:48) that would reduce the number of required queries as much as possible (according to the information theoretic bound of (Fiez et al., 2019)) if we want to ﬁnd a single allocation for all z (cid:
) ∈
Z simultaneously, we can perform the optimization problem
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
We then sample elements from X (cid:48)
according to this optimal λ without replacement and add them to
X until |X| = 100 .
J J ESTER J OKE R ECOMMENDATION S ETUP
We consider the Jester jokes dataset of Goldberg et al.
(2001) that contains jokes ranging from punbased jokes to grossly offensive.
We ﬁlter the dataset to only contain users that rated all 100 jokes, resulting in 14116 users.
A rating of each joke was provided on a [ − 10 , 10] scale which was shrunk to [ − 1 , 1] .
Denote this set of ratings as ˆΘ = { θ i : i ∈ [14116] , θ i ∈ [ − 1 , 1] 100 } , where θ i encodes the ratings of all 100 jokes by user i .
To construct the set of arms Z , we then clustered the ratings of these users to 10 groups to obtain Z =
{ z i : i ∈ [10] , z i ∈ { 0 , 1 } 100 } by minimizing the following metric:
[Equation]
To solve for Z , we adapt the k − means algorithm, with the metric above instead of the L − 2 metric used traditionally.