L EARNING E FFECTIVE E XPLORATION S TRATEGIES FOR C ONTEXTUAL B ANDITS
Anonymous authors
Paper under double-blind review
A BSTRACT
In contextual bandits, an algorithm must choose actions given observed contexts, learning from a reward signal that is observed only for the action chosen.
This leads to an exploration/exploitation trade-off: the algorithm must balance taking actions it already believes are good with taking new actions to potentially discover better choices.
We develop a meta-learning algorithm, Mˆ EL ´ EE , that learns an exploration policy based on simulated, synthetic contextual bandit tasks.
Mˆ EL ´ EE uses imitation learning against these simulations to train an exploration policy that can be applied to true contextual bandit tasks at test time.
We evaluate on both a natural contextual bandit problem derived from a learning to rank dataset as well as hundreds of simulated contextual bandit problems derived from classiﬁcation tasks.
Mˆ EL ´ EE outperforms seven strong baselines on most of these datasets by leveraging a rich feature representation for learning an exploration strategy.
I NTRODUCTION
In a contextual bandit problem, an agent attempts to optimize its behavior over a sequence of rounds based on limited feedback (Kaelbling, 1994; Auer, 2003; Langford & Zhang, 2008).
In each round, the agent chooses an action based on a context (features) for that round, and observes a reward for that action but no others ( § 2).
Contextual bandit problems arise in many real-world settings like online recommendations and personalized medicine.
As in reinforcement learning, the agent must learn to balance exploitation (taking actions that, based on past experience, it believes will lead to high instantaneous reward) and exploration (trying actions that it knows less about).
In this paper, we present a meta-learning approach to automatically learn a good exploration mechanism from data.
To achieve this, we use synthetic supervised learning data sets on which we can simulate contextual bandit tasks in an ofﬂine setting.
Based on these simulations, our algorithm,
Mˆ EL ´ EE (MEta LEarner for Exploration) 1 , learns a good heuristic exploration strategy that should ideally generalize to future contextual bandit problems.
Mˆ EL ´ EE contrasts with more classical approaches to exploration (like (cid:15) -greedy or LinUCB; see § 6), in which exploration strategies are constructed by expert algorithm designers.
These approaches often achieve provably good exploration strategies in the worst case, but are potentially overly pessimistic and are sometimes computationally intractable.
At training time ( § 3.2), Mˆ EL ´ EE simulates many contextual bandit problems from fully labeled synthetic data.
Using this data, in each round, Mˆ EL ´ EE is able to counterfactually simulate what would happen under all possible action choices.
We can then use this information to compute regret estimates for each action, which can be optimized using the AggreVaTe imitation learning algorithm (Ross & Bagnell, 2014).
Our imitation learning strategy mirrors that of the meta-learning approach of Bachman et al.
(2017) in the active learning setting.
We present a simpliﬁed, stylized analysis of the behavior of Mˆ EL ´ EE to ensure that our cost function encourages good behavior ( § 4).
Empirically, we use M ˆ EL ´ EE to train an exploration policy on only synthetic datasets and evaluate this policy on both a contextual bandit task based on a natural learning to rank dataset as well as three hundred simulated contextual bandit tasks ( § 5.2).
We compare the trained policy to a number of alternative exploration algorithms, and show the efﬁcacy of our approach ( § 5.3).
P RELIMINARIES : C ONTEXTUAL B ANDITS AND P OLICY O PTIMIZATION
Contextual bandits is a model of interaction in which an agent chooses actions (based on contexts) and receives immediate rewards for that action alone.
For example, in a simpliﬁed news personalization setting, at each time step t , a user arrives and the system must choose a news article to display to them.
Each possible news article corresponds to an action a , and the user corresponds to a context x t .
After the system chooses an article a t to display, it can observe, for instance, the amount of time that the user spends reading that article, which it can use as a reward r t ( a t ) .
Formally, we largely follow the setup and notation of Agarwal et al.
Let X be an input space of contexts (users) and [ K ] = { 1 , . . .
, K } be a ﬁnite action space (articles).
We consider the statistical setting in which there exists a ﬁxed but unknown distribution D over pairs ( x, r ) ∈
X× [0 , 1] K , where r is a vector of rewards (for convenience, we assume all rewards are bounded in [0 , 1] ).
In this setting, the world operates iteratively over rounds t = 1 , 2 , . . . .
Each round t : 1.
The world draws ( x t , r t ) ∼ D and reveals context x t .
The agent (randomly) chooses action a t ∈
[ K ] based on x t , and observes reward r t ( a t
The goal of an algorithm is to maximize the cumulative sum of rewards over time.
Typically the primary quantity considered is the average regret of a sequence of actions a 1 , . .
. , a T to the behavior of the best possible function in a prespeciﬁed class F :
[Equation] [Equation]
An agent is call no-regret if its average regret is zero in the limit of large T .
To produce a good agent for interacting with the world, we assume access to a function class F and to an oracle policy optimizer for that function class.
For example, F may be a set of single layer neural networks mapping user features x ∈ X to predicted rewards for actions a ∈ [ K ] .
Formally, the observable record of interaction resulting from round t is the tuple ( x t , a t , r t ( a t ) , p t ( a t )) ∈
X× [ K ] ×
[0 , 1] × [0 , 1] , where p t ( a t ) is the probability that the agent chose action a t , and the full history of interaction is h t = (cid:104)
( x i , a i , r i ( a i ) , p i ( a i )) (cid:105) ti =1 .
The oracle policy optimizer, P OL O PT , takes as input a history of user interactions and outputs an f ∈ F with low expected regret.
A standard example of a policy optimizer is to combine inverse propensity scaling (IPS) with a regression algorithm (Dudik et al., 2011).
Here, given a history h , each tuple ( x, a, r, p ) in that history is mapped to a multiple-output regression example.
The input for this regression example is the same x ; the output is a vector of K costs, all of which are zero except the a th component, which takes value r/p .
This mapping is done for all tuples in the history, and a supervised learning algorithm on the function class F is used to produce a low-regret regressor f .
This is the function returned by the policy optimizer.
IPS, and other estimators that have lower-variance than IPS (such as the doubly-robust estimator), have the property of being unbiased. In experiments, we use the direct method (Dudik et al., 2011) largely for its simplicity, however, Mˆ EL ´ EE is agnostic to the type of the estimator used by the policy optimizer.
A PPROACH : L EARNING AND E FFECTIVE E XPLORATION S TRATEGY
In order to have an effective approach to the contextual bandit problem, one must be able to both optimize a policy based on historic data and make decisions about how to explore.
The exploration/exploitation dilemma is fundamentally about long-term payoffs: is it worth trying something potentially suboptimal now in order to learn how to behave better in the future?
A particularly simple and effective form of exploration is (cid:15) -greedy: given a function f output by P OL O PT , act according to f ( x ) with probability (1 − (cid:15) ) and act uniformly at random with probability (cid:15) .
Intuitively, one would hope to improve on such a strategy by taking more (any!) information into account; for instance, basing the probability of exploration on f ’s uncertainty.
In this section, we describe Mˆ EL ´ EE , ﬁrst by describing how it operates at test time when applied to a new contextual bandit problem ( § 3.1), and then by describing how to train it using synthetic simulated contextual bandit problems ( § 3.2).
3. T EST T IME B EHAVIOR OF Mˆ EL ´ EE
Our goal in this paper is to learn how to explore from experience.
The training procedure for
Mˆ EL ´ EE will use ofﬂine supervised learning problems to learn an exploration policy π , which takes two inputs : a function f ∈ F and a context x , and outputs an action.
In our example, f will be the output of the policy optimizer on all historic data, and x will be the current user.
This is used to produce an agent which interacts with the world, maintaining an initially empty history buffer h , as:
The world draws ( x t , r t ) ∼ D and reveals context x t . 2.
The agent computes f t ← P OL O PT ( h ) and a greedy action ˜ a t = π ( f t , x t ) .
The agent plays a t = ˜ a t with probability (1 − µ ) , and a t uniformly at random otherwise.
The agent observes r t ( a t ) and appends ( x t , a t , r t ( a t ) , p t ) to the history h , where p t = µ/K if a t (cid:54) =
˜ a t ; and p t = 1 − µ + µ/K if a t = ˜ a t .
Here, f t is the function optimized on the historical data, and π uses it and x t to choose an action.
Intuitively, π might choose to use the prediction f t ( x t ) most of the time, unless f t is quite uncertain on this example, in which case π might choose to return the second (or third) most likely action according to f t .
The agent then performs a small amount of additional µ -greedy-style exploration: most of the time it acts according to π but occasionally it explores some more.
In practice ( § 5), we ﬁnd that setting µ = 0 is optimal in aggregate, but non-zero µ is necessary for our theory ( § 4).
Importantly, we wish to train π using one set of tasks (for which we have fully supervised data on which to run simulations) and apply it to wholly different tasks (for which we only have bandit feedback).
To achieve this, we allow π to depend representationally on f t in arbitrary ways: for instance, it might use features that capture f t ’s uncertainty on the current example (see § 5.1 for details).
We additionally allow π to depend in a task-independent manner on the history (for instance, which actions have not yet been tried): it can use features of the actions, rewards and probabilities in the history but not depend directly on the contexts x .
This is to ensure that π only learns to explore and not also to solve the underlying task-dependent classiﬁcation problem.
Because π needs to learn to be task independent, we found that if f t ’s predictions were uncalibrated, it was very difﬁcult for π to generalize well to unseen tasks.
Therefore, we additionally allow π to depend on a very small amount of fully labeled data from the task at hand, which we use to allow π to calibrate f t ’s predictions.
In our experiments we use only 30 fully labeled examples, but alternative approaches to calibrating f t that do not require this data would be preferable.
3. T RAINING Mˆ EL ´ EE BY I MITATION L EARNING
The meta-learning challenge is: how do we learn a good exploration policy π ?
We assume we have access to fully labeled data on which we can train π ; this data must include context/reward pairs, but where the reward for all actions is known.
This is a weak assumption: in practice, we use purely synthetic data as this training data; one could alternatively use any fully labeled classiﬁcation dataset (Beygelzimer & Langford, 2009).
Under this assumption about the data, it is natural to think of π ’s behavior as a sequential decision making problem in a simulated setting, for which a natural class of learning algorithms to consider are imitation learning algorithms (Daum´e et al., 2009; Ross et al., 2011; Ross & Bagnell, 2014;
Chang et al., 2015).
Informally, at training time, Mˆ EL ´ EE will treat one of these synthetic datasets as if it were a contextual bandit dataset.
At each time step t , it will compute f t by running P OL O PT on the historical data, and then ask: for each action, what would the long time reward look like if I were to take this action.
Because the training data for Mˆ EL ´ EE is fully labeled, this can be evaluated for each possible action, and a policy π can be learned to maximize these rewards.
More formally, in imitation learning, we assume training-time access to an expert , π (cid:63) , whose behavior we wish to learn to imitate at test-time.
From this, we can deﬁne an optimal reference policy π (cid:63) , which effectively “cheats” at training time by looking at the true labels.
The learning problem is then to estimate π to have as similar behavior to π (cid:63) as possible, but without access to those labels.
The imitation learning algorithm we use is AggreVaTe (Ross & Bagnell, 2014) (closely related to
DAgger (Ross et al., 2011)), and is instantiated for the contextual bandits meta-learning problem in
Alg 1.
AggreVaTe learns to choose actions to minimize the cost-to-go of the expert rather than the
1: for round n = 1 , 2 , . . .
, N do 2: initialize meta-dataset D = {} , choose S at random from { S m } , and set history h
= {} 3: partition and permute S randomly into train Tr and validation Val where | Val |
=
N Val 4: for round t = 1 , 2 , . .
, | Tr | do 5: let ( x t , r t ) =
Tr t 6: for each action a = 1 , . .
, K do 7: optimize f t,a = P OL O PT ( F , h t − 1 ⊕ ( x t , a, r t ( a ) , 1 − K − 1 K µ )) on augmented history 8: roll-out: estimate ˆ c a , the cost-to-go of a , using r t ( a ) and a roll-out policy π out on f t,a 9: end for 10: compute f t =
P OL O PT ( F , h t − 1 ) 11: aggregate D ← D ⊕ (Φ( f t , x t , h t − 1 , Val ) , (cid:104)
ˆ c 1 , . . .
, ˆ c K (cid:105) ) 12: roll-in: a t ∼ µK 1 K + (1 − µ ) π n − 1 ( f t , x t ) with probability p t , where 1 is the ones-vector 13: append history h t ← h t − 1 ⊕ ( x t , a t , r t ( a t ) , p t ) 14: end for 15: update π n =
L EARN ( D ) 16: end for 17: return { π n } Nn = zero-one classiﬁcation loss of mimicking its actions.
On the ﬁrst iteration AggreVaTe collects data by observing the expert perform the task, and in each trajectory, at time t , explores an action a in state s , and observes the cost-to-go Q of the expert after performing this action.
Following the AggreVaTe template, Mˆ EL ´ EE operates in an iterative fashion, starting with an arbitrary π and improving it through interaction with an expert.
Over N rounds, Mˆ EL ´ EE selects random training sets and simulates the test-time behavior on that training set.
The core functionality is to generate a number of states ( f t , x t ) on which to train π , and to use the supervised data to estimate the value of every action from those states.
Mˆ EL ´ EE achieves this by sampling a random supervised training set and setting aside some validation data from it (line 3).
It then simulates a contextual bandit problem on this training data; at each time step t , it tries all actions and “pretends” like they were appended to the current history (line 7) on which it trains a new policy and evaluates it’s roll-out value (line 8).
This yields, for each t , a new training example for π , which is added to π ’s training set (line 11); the features for this example are features of the classiﬁer based on true history (line 10) (and possibly statistics of the history itself), with a label that gives, for each action, the corresponding cost-to-go of that action (the c a s computed in line 8).
Mˆ EL ´ EE then must commit to a roll-in action to actually take; it chooses this according to a roll-in policy (line 12).
Mˆ EL ´ EE has no explicit “exploitation policy”, exploitation happens when π chooses the same action as f t , while exploration happens when it chooses a different action.
In learning to explore, M ˆ EL ´ EE simultaneously learns when to exploit.
Roll-in actions.
The distribution over states visited by Mˆ EL ´ EE depends on the actions taken, and in general it is good to have that distribution match what is seen at test time.
This distribution is determined by a roll-in policy (line 12), controlled in Mˆ EL ´ EE by exploration parameter µ ∈ [0 , 1] .
As µ → 1 , the roll-in policy approaches a uniform random policy; as µ → 0 , the roll-in policy becomes deterministic.
When the roll-in policy does not explore, it acts according to π ( f t , . ) .
Roll-out values.
The ideal value to assign to an action (from the perspective of the imitation learning procedure) is that total reward (or advantage) that would be achieved in the long run if we took this action and then behaved according to our ﬁnal learned policy.
Unfortunately, during training, we do not yet know the ﬁnal learned policy.
Thus, a surrogate roll-out policy π out is used instead.
A convenient, and often computationally efﬁcient alternative, is to evaluate the value assuming all future actions were taken by the expert (Langford & Zadrozny, 2005;
Daum´e et al., 2009; Ross & Bagnell, 2014).
In our setting, at any time step t , the expert has access to the fully supervised reward vector r t for the context x t .
When estimating the roll-out value for an action a , the expert will return the true reward value for this action r t ( a ) and we use this as our estimate for the roll-out value.
T HEORETICAL G UARANTEES
We analyze Mˆ EL ´ EE , showing that the no-regret property of A GGRE V A T E can be leveraged in our meta-learning setting for learning contextual bandit exploration.
In particular, we ﬁrst relate the regret of the learner in line 15 to the overall regret of π .
This will show that, if the underlying classiﬁer improves sufﬁciently quickly, Mˆ EL ´ EE will achieve sublinear regret.
We then show that for a speciﬁc choice of underlying classiﬁer (B ANDITRON ), this is achieved.
Mˆ EL ´ EE is an instantiation of A GGRE V A T E (Ross & Bagnell, 2014); as such, it inherits A GGRE V A T E ’s regret guarantees.
Theorem 1 (Thm 2.2 of Ross & Bagnell (2014), adapted)
After N rounds, if L EARN (line 15) is no-regret algorithm, then as N → ∞ , with probability 1 , it holds
that J (¯ π ) ≤ J ( π (cid:63) ) + 2 T (cid:112) K ˆ (cid:15) class ( T ) , where J ( · ) is the reward of the exploration policy,
¯ π is the average policy returned, and ˆ (cid:15) class ( T ) is the average regression regret for each π n accurately predicting ˆ c , where
[Equation]
is the empirical minimum expected cost-sensitive classiﬁcation regret achieved by policies in the class Π on all the data over the N iterations of training when compared to the Bayes optimal regressor, for U ( T ) the uniform distribution over { 1 , . .
, T } , d
tπ the distribution of states at time t induced by executing policy π , and Q (cid:63) the cost-to-go of the imitation learning expert.
Thus, achieving low regret at the problem of learning π on the training data it observes (“ D ” in
Mˆ EL ´ EE ), i.e. ˆ (cid:15) class ( T ) is small, translates into low regret in the contextual-bandit setting.
At ﬁrst glance this bound looks like it may scale linearly with T .
However, the bound in Theorem 1 is dependent on ˆ (cid:15) class ( T ) .
Note however, that s is a combination of the context vector x t and the classiﬁcation function f t .
As T → ∞ , one would hope that f t improves signiﬁcantly and ˆ (cid:15) class ( T ) decays quickly.
Thus, sublinear regret may still be achievable when f learns sufﬁciently quickly as a function of T .
For instance, if f is optimizing a strongly convex loss function, online gradient descent achieves a regret guarantee of O ( log TT )
(Hazan et al., 2016, Theorem 3.3), potentially leading to a regret for Mˆ EL ´ EE of O ( (cid:112) (log T ) /T ) .
The above statement is informal (it does not take into account the interaction between learning f and π ).
However, we can show a speciﬁc concrete example: we analyze Mˆ EL ´ EE ’s test-time behavior when the underlying learning algorithm is B ANDITRON .
B ANDITRON is a variant of the multiclass
Perceptron that operates under bandit feedback.
Details of this analysis (and proofs, which directly follow the original B ANDITRON analysis) are given in Appendix A; here we state the main result.
Let γ t =
Pr[ r t ( π ( f t , x t ) = 1) | x t ] − Pr[ r t ( f t ( x t ))
=
| x t ] be the edge of π ( f t , . )
over f , and let
Γ = 1 T (cid:80) Tt
=1 E 1 1+ Kγ t be an overall measure of the edge.
For instance if π simply returns f ’s prediction, then all γ t = 0 and Γ = 1 .
We can then show the following:
Theorem 2 Assume that for the sequence of examples, ( x 1 , r 1 ) , ( x 2 , r 2 ) , . . .
, ( x T , r T ) , we have, for all t , || x t || ≤ 1 .
Let W (cid:63) be any matrix, let L be the cumulative hinge-loss of W (cid:63) , let µ be a uniform exploration probability, and let D =
|| W (cid:63) ||
2 F be the complexity of W (cid:63) .
Assume that E γ t ≥ 0 for all t .
Then the number of mistakes M made by Mˆ EL ´ EE with B ANDITRON as P OL
O PT satisﬁes:
[Equation]
where the expectation is taken with respect to the randomness of the algorithm.
Note that under the assumption E γ t ≥ 0 for all t , we have Γ ≤ 1 .
The analysis gives the same mistake bound for B ANDITRON but with the factor of Γ , hence this result improves upon the B ANDITRON analysis only when Γ < 1 (in the realizable setting, the number of mistakes is analogous to the regret).
This result is highly stylized and the assumption that E γ t ≥ 0 is overly strong.
This assumption ensures that π never decreases the probability of a “correct” action.
It does, however, help us understand the behavior of Mˆ EL ´ EE , qualitatively: First, the quantity that matters in Theorem 2, E t γ t is (in the 0/1 loss case) exactly what Mˆ EL ´ EE is optimizing: the expected improvement for choosing an action against f t ’s recommendation.
Second, the beneﬁt of using π within B ANDITRON is a local beneﬁt: because π is trained with expert rollouts, as discussed in § 4, the primary improvement in the analysis is to ensure that π does a better job predicting (in a single step) than f t does.
An obvious open question is whether it is possible to base the analysis on the regret of π (rather than its error) and whether it is possible to extend beyond the B ANDITRON .
E XPERIMENTAL S ETUP AND R ESULTS
Using a collection of synthetically generated classiﬁcation problems, we train an exploration policy π using Mˆ EL ´ EE (Alg 1).
This exploration policy learns to explore on the basis of calibrated probabilistic predictions from f together with a predeﬁned set of exploration features ( § 5.1).
Once π is learned and ﬁxed, we follow the test-time behavior described in § 3.1 to evaluate π on a set of contextual bandit problems.
We evaluate Mˆ EL ´ EE on a natural learning to rank task ( § 5.3.1).
To ensure that the performance of Mˆ EL ´ EE generalizes beyond this single learning to rank task, we additionally perform thorough evaluation on 300 “simulated” contextual bandit problems, derived from standard classiﬁcation tasks ( § 5.3.2).
In all cases, the underlying classiﬁer f is a linear model trained with a policy optimizer that runs stochastic gradient descent (details are in § A.2).
We seek to answer two questions experimentally: (1) How does Mˆ EL ´ EE compare empirically to alternative (expert designed) exploration strategies?
(2) How important are the additional features used by Mˆ EL ´ EE in comparison to using calibrated probability predictions from f as features?
5. T RAINING D ETAILS FOR THE E XPLORATION P OLICY
Exploration Features.
In our experiments, the exploration policy is trained based on features Φ (Alg 1, line 11).
These features are allowed to depend on the current classiﬁer f t , and on any part of the history except the inputs x t in order to maintain task independence.
We additionally ensure that its features are independent of the dimensionality of the inputs, so that π can generalize to datasets of arbitrary dimensions.
The speciﬁc features we use are listed below; these are largely inspired by
Konyushkova et al.
but adapted and augmented to our setting.
The features of f t that we use are: a) predicted probability p ( a t | f t , x t ) , we use a softmax over the predicted rewards from f t to convert them to probabilities; b) entropy of the predicted probability distribution; c) a one-hot encoding for the predicted action f t ( x t ) .
The features of h t − 1 that we use are: a) current time step t ; b) normalized counts for all previous actions predicted so far; c) average observed rewards for each action; d) empirical variance of the observed rewards for each action in the history.
In our experiments, we found that it is essential to calibrate the predicted probabilities of the classiﬁer f t .
We use a very small held-out dataset, of size 30 , to achieve this.
We use Platt’s scaling (Platt, 1999; Lin et al., 2007) method to calibrate the predicted probabilities.
Platt’s scaling works by ﬁtting a logistic regression model to the classiﬁer’s predicted scores.
Training Datasets.
In our experiments, we follow Konyushkova et al.
(2017) (and also Peters et al.
(2014), in a different setting) and train the exploration policy π only on synthetic data .
This is possible because the exploration policy π never makes use of x explicitly and instead only accesses it via f t ’s behavior on it.
We generate datasets with uniformly distributed class conditional distributions.
The datasets are always two-dimensional.
Details are in § A.1.
5. E VALUATION M ETHODOLOGY
For evaluation, we use progressive validation (Blum et al., 1999), which is exactly computing the reward of the algorithm.
Speciﬁcally, to evaluate the performance of an exploration algorithm A on a dataset S of size n , we compute the progressive validation return G ( A )
= 1 n (cid:80) nt =1 r t ( a t ) as the average reward up to n , where a t is the action chosen by the algorithm A and r t is the true reward.
Because our evaluation is over 300 datasets, we report aggregate results in two forms.
The simpler one is Win/Loss Statistics: We compare two exploration methods on a given dataset by counting the number of statistically signiﬁcant wins and losses.
An exploration algorithm
A wins over another algorithm B if the progressive validation return G ( A ) is statistically signiﬁcantly larger than B ’s return G ( B ) at the 0 .
01 level using a paired sample t-test.
We also report cumulative distributions of rewards for each algorithm, following Zhang et al.
In particular, for a given relative reward value ( x ∈ [0 , 1] ), the corresponding CDF value for a given algorithm is the fraction of datasets on which this algorithm achieved reward at least x .
We compute relative reward by Min-
Max normalization: linearly transforming reward y to y (cid:48) =
y − min max − min , where min & max are the minimum & maximum rewards among all exploration algorithms.
In our experiments, we compare to the following baseline exploration methods, keeping the policy optimization method ﬁxed (details in § A.4): (cid:15) -greedy (Sutton, 1996); (cid:15) -decreasing (Sutton & Barto, 1998); EG (cid:15) -greedy
(Li et al., 2010b);
τ -ﬁrst .
We additionally compare to three state-of-the-art exploration methods: LinUCB (Li et al., 2010a);
Cover (Agarwal et al., 2014); and its variant
Cover-NU (Bietti et al., 2018).
We select the best hyperparameters following Bietti et al.
5. E XPERIMENTAL R ESULTS 5.3. L EARNING TO R ANK
We evaluate Mˆ EL ´ EE on a natural learning to rank dataset.
The dataset we consider is the Microsoft
Learning to Rank dataset, variant MSLR-10K from Qin & Liu (2013) 3 .
The dataset consists of feature vectors extracted from query-url pairs along with relevance judgment labels.
The relevance judgments are obtained from a retired labeling set of a commercial web search engine (Microsoft
Bing), which take 5 values from 0 (irrelevant) to 4 (perfectly relevant) and we drop the queries not labelled as any of the two extremes.
In our experiments, we limit the number of labels to the two extremes: 0 and 4 .
A query-url pair is represented by a 136-dimensional feature vector.
The dataset is highly imbalanced as the number of irrelevant queries is much larger than the number of relevant ones.
To address this, we sample the number of irrelevant queries to match that of the relevant ones.
To avoid correlations between the observed query-url pairs, we group the queries by the query ID, and sample a single query from each group.
We convert relevance scores to losses with 0 indicating a perfectly relevant document, and 1 an irrelevant one.
Figure 1 shows the evaluation results on a subset of the MSLR-10K dataset.
Since the performance is closely matched between the different exploration algorithms, we repeat the experiment 16 times with randomly shufﬂed permutations of the MSLR-10K dataset.
Figure 1 (left) shows the learning curve of the trained policy π as well as the baselines.
Here, we see that
Mˆ EL ´ EE quickly achieves high reward, after about 100 examples the two strongest baselines catch up.
By 200 examples all approaches have asymptoted.
We exclude LinUCB from these runs because the required matrix inversions made it too computationally expensive.
4 Figure 1 shows statistically-signiﬁcant win/loss differences for each of the algorithms, across these 16 shufﬂes.
Each row/column entry shows the number of times the row algorithm won against the column, minus the number of losses.
Mˆ EL ´ EE is the only algorithm that always wins more than it loses against other algorithms, and outperforms the nearest competition ( (cid:15) -decreasing) by 3 points.
5.3. S IMULATED C ONTEXTUAL B ANDIT T ASKS
We additionally perform an exhaustive evaluation on simulated contextual bandit tasks to ensure that the performance of Mˆ EL ´ EE generalizes beyond learning to rank.
Following Bietti et al.
(2018), we use a collection of 300 binary classiﬁcation datasets from openml.org for evaluation; the precise list and download instructions is in § A.3.
These datasets cover a variety of different domains including text & image processing, medical, and sensory data.
We convert classiﬁcation datasets into cost-sensitive classiﬁcation problems by using a 0 / 1 encoding.
Given these fully supervised cost-sensitive multi-class datasets, we simulate the contextual bandit setting by only revealing the reward for the selected actions.
In Figure 2 (left), we show a representative learning curve.
Here, we see that as more data becomes available, all the approaches improve (except τ -ﬁrst, which has ceased to learn after 2% of the data).
Mˆ EL ´ EE , in particular, is able to very quickly achieve near-optimal performance (in aroundexamples) in comparison to the best baseline which takes at least 200 .
In Figure 2 (right), we show the CDFs for the different algorithms.
To help read this, at x = 1 . 0 , Mˆ EL ´ EE has a relative reward at least 1 . 0 on more than 40% of datasets, while (cid:15) -decreasing and (cid:15) -greedy achieve this on about 30% of datasets.
We ﬁnd that the two strongest baselines are (cid:15) -decreasing and (cid:15) -greedy (better when reward differences are small, toward the left of the graph).
The two curves for (cid:15) -decreasing and (cid:15) - greedy coincide.
This happens because the exploration probability (cid:15) 0 for (cid:15) -decreasing decays rapidly
Time 0.
P r o g r e ss i v e
R e w a r d
M E L EE - g r ee d y - d e c r e a s i n g
E G - g r ee d y - f i r s t
C o v e r
C o v e r - nu
MELEE -greedy -decreasing
EG -greedy -first
Cover
Cover-nu       -  -     -       - - -     - - - -    - - - - -  - - - - - -
MELEE , : 0.
-decreasing, 0 : 0.
EG -greedy -greedy, : 0.
LinUCB -first, : 0.02 Cover, Bag Size: 16, : 0.
MELEE , : 0.0 -decreasing, 0 : 0.
EG -greedy -greedy, : 0.
LinUCB -first, : 0.
Cover, Bag Size: 16, : 0.
[Figure]
[Figure]
MELEE , : 0.0 -decreasing, 0 : 0.
EG -greedy
[Figure]
approaching zero with a rate of 1 t , where t is the index of the current round.
Mˆ EL ´ EE outperforms the baselines in the “large reward” regimes (right of graph) but under-performs (cid:15) -decreasing and (cid:15) -greedy in low reward regimes (left of graph).
In Figure 3, we show statistically-signiﬁcant win/loss differences for each of the algorithms.
Here, each (row, column) entry shows the number of times the row algorithm won against the column, minus the number of losses.
M ˆ EL ´ EE is the only algorithm that always wins more than it loses against other algorithms, and outperforms the nearest competition ( (cid:15) -decreasing) by 23 points.
To understand more directly how Mˆ EL ´ EE compares to (cid:15) -decreasing, in the left plot of Figure 4, we show a scatter plot of rewards achieved by Mˆ EL ´ EE (x-axis) and (cid:15)
M E L E E
g r ee
d y - d e c r e a s i n g
E G - g r ee d y
L i n U C B - f i r s t
C o v e r
C o v e r - n u
MELEE -greedy -decreasing
EG -greedy
LinUCB -first
Cover
Cover-nu        -        - -       -167 -176 -  - -   -
[Figure]
Mˆ
[Figure]
EE vs (cid:15)
[Figure]
decreasing and points above to (cid:15) -decreasing (124 datasets).
The remaining 29 had no statistically signiﬁcant difference.
Finally, we consider the effect that the additional features have on Mˆ EL ´ EE ’s performance.
In particular, we consider a version of Mˆ EL ´ EE with all features (this is the version used in all other experiments) with an ablated version that only has access to the (calibrated) probabilities of each action from the underlying classiﬁer f .
The comparison is shown as a scatter plot in Figure 4 (right).
Here, we can see that the full feature set does provide lift over just the calibrated probabilities, with a win-minus-loss improvement of 24 by adding additional features from which to learn to explore.
R ELATED W ORK AND D ISCUSSION
The ﬁeld of meta-learning is based on the idea of replacing hand-engineered learning heuristics with heuristics learned from data.
One of the most relevant settings for meta-learning to ours is active learning, in which one aims to learn a decision function to decide which examples, from a pool of unlabeled examples, should be labeled.
Past approaches to meta-learning for active learning include reinforcement learning-based strategies (Woodward & Finn, 2017; Fang et al., 2017), imitation learning-based strategies (Bachman et al., 2017), and batch supervised learning-based strategies (Konyushkova et al., 2017).
Similar approaches have been used to learn heuristics for optimization (Li & Malik, 2016; Andrychowicz et al., 2016), multiarm (non-contextual) bandits Maes et al.
(2012), and neural architecture search (Zoph & Le, 2016), recently mostly based on (deep) reinforcement learning.
While meta-learning for contextual bandits is most similar to meta-learning for active learning, there is a fundamental difference that makes it signiﬁcantly more challenging: in active learning, the goal is to select as few examples as you can to learn, so by deﬁnition the horizon is short; in contextual bandits, learning to explore is fundamentally a long-horizon problem, because what matters is not immediate reward but long term learning.
In reinforcement learning, Gupta et al.
(2018) investigated the task of meta-learning an exploration strategy for a distribution of related tasks by learning a latent exploration space.
Similarly, Xu et al.
(2018) proposed a teacher-student approach for learning to do exploration in off-policy reinforcement learning.
While these approaches are effective if the distribution of tasks is very similar and the state space is shared among different tasks, they fail to generalize when the tasks are different.
Our approach targets an easier problem than exploration in full reinforcement learning environments, and can generalize well across a wide range of different tasks with completely unrelated features spaces.
There has also been a substantial amount of work on constructing “good” exploration policies, in problems of varying complexity: traditional bandit settings (Karnin & Anava, 2016), contextual bandits (Fraud et al., 2016) and reinforcement learning (Osband et al., 2016).
In both bandit settings, most of this work has focused on the learning theory aspect of exploration: what exploration distributions guarantee that learning will succeed (with high probability)?
Mˆ EL ´ EE , lacks such guarantees: in particular, if the data distribution of the observed contexts ( φ ( f t ) ) in some test problem differs substantially from that on which Mˆ EL ´ EE was trained, we can say nothing about the quality of the learned exploration.
Nevertheless, despite fairly substantial distribution mismatch (synthetic → real-world), Mˆ EL ´ EE works well in practice, and our stylized theory ( § 4) suggests that there may be an interesting avenue for developing strong theoretical results for contextual bandit learning with learned exploration policies, and perhaps other meta-learning problems.
In conclusion, we presented Mˆ EL ´ EE , a meta-learning algorithm for learning exploration policies in the contextual bandit setting.
Mˆ EL ´ EE enjoys no-regret guarantees, and empirically it outperforms alternative exploration algorithm in most settings.
One limitation of Mˆ EL ´ EE is the computational resources required during the ofﬂine training phase on the synthetic datasets.
In the future, we will work on improving the computational efﬁciency for Mˆ EL ´ EE in the ofﬂine training phase and scale the experimental analysis to problems with larger number of classes
A S TYLIZED TEST - TIME ANALYSIS FOR B ANDITRON : D ETAILS
The B ANDITRON Mˆ EL ´ EE algorithm is speciﬁed in Alg 2.
The is exactly the same as the typical test time behavior, except it uses a B ANDITRON -type strategy for learning the underlying classiﬁer f in the place of P OL O PT .
P OLICY E LIMINATION M ETA takes as arguments: π (the learned exploration policy) and µ ∈ (0 , 1 / (2 K ))
an added uniform exploration parameter.
The B ANDITRON learns a linear multi-class classiﬁer parameterized by a weight matrix of size K × D , where D is the input dimensionality.
The B ANDITRON assumes a pure multi-class setting in which the reward for one (“correct”) action is 1 and the reward for all other actions is zero.
At each round t , a prediction ˆ a t is made according to f t (summarized by W t ).
We then deﬁne an exploration distribution that “most of the time” acts according to π ( f t , . ) , but smooths each action with µ probability.
The chosen action a t is sampled from this distribution and a binary reward is observed.
The weights of the B ANDITRON are updated according to the B ANDITRON update rule using ˜ U t .
Algorithm 2 B
1: initialize W 1 = 0 ∈
R K × D 2: for rounds t = 1 . . .
T : do 3: observe x
∈ R D 4: compute ˆ a t = f t ( x t ) =
argmax k ∈ K (cid:0) W t x t (cid:1) k 5: deﬁne Q µ ( a ) = µ + (1 − Kµ )
[ a = π ( W t , x t )] 6: sample a t ∼ Q µ 7: observe reward r t ( a t ) ∈ { 0 , 1 } 8: deﬁne ˜
U t ∈ R K × D as: ˜ U ta, · = x t (cid:16) 1 [ r t ( a t )=1] 1 [ a t = a ] Q
[Equation]
( a ) − 1 [ˆ a
[Equation]
=
[Equation]
9: update W t +1 =
W t + ˜ U t 10: end for
The only difference between B ANDITRON Mˆ EL ´ EE and the original B ANDITRON is the introduction of π in the sampling distribution.
The original algorithm achieves the following mistake bound shown below, which depends on the notion of multi-class hinge-loss.
In particular, the hinge-loss of W on ( x, r ) is (cid:96)
[Equation]
W, ( x, r ))
= max a
[Equation]
= a (cid:63) max (cid:8) 0 ,
[Equation]
( Wx ) a (cid:63) + ( Wx ) a (cid:9) , where a (cid:63) is the a for which r ( a ) = 1 .
The overall hinge-loss L is the sum of (cid:96) over the sequence of examples.
Theorem 3 (Thm. 1 and Corr.
2 of Kakade et al.
Assume that for the sequence of examples, ( x 1 , r 1 ) , ( x 2 , r 2 ) , . . .
, ( x T , r T ) , we have, for all t , || x t || ≤ 1 .
Let W (cid:63) be any matrix, let L be the cumulative hinge-loss of W (cid:63) , and let D =
|| W (cid:63) ||
2 F be the complexity of W (cid:63) .
The number of mistakes M made by the B ANDITRON satisﬁes
[Equation]
(4) where the expectation is taken with respect to the randomness of the algorithm.
Furthermore, in a low noise setting (there exists W (cid:63) with ﬁxed complexity d and loss L ≤ O ( √ DKT ) ), then by setting µ = (cid:112) D/ ( TK ) , we obtain E M ≤ O ( √ KDT ) .
We can prove an analogous result for B ANDITRON Mˆ EL ´ EE .
The key quantity that will control how much π improves the execution of B ANDITRON Mˆ EL ´ EE is how much π improves on f t when f t is wrong.
In particular, let γ t =
Pr[ r t ( π ( f t , x t ) = 1) | x t ] − Pr[ r t ( f t ( x t ))
=
| x t ] be the edge of π ( f t , . )
over f , and let
Γ = 1 T (cid:80) Tt
=1 E 1 1+ Kγ t be an overall measure of the edge.
(If π does nothing, then all γ t = 0 and Γ = 1 .)
Given this quantity, we can prove the following Theorem 2.
Proof: [sketch]
The proof is a small modiﬁcation of the original proof of Theorem 3.
The only change is that in the original proof, the following bound is used: E t || ˜
U t
|| 2 / || x t || 2 =
+1 /µ ≤ 2 /µ .
We use, instead: E t || ˜
U t
|| 2 /
|| x t || 2 ≤ 1 + E t 1 µ + γ t ≤ 2 E t 1 1+ γt µ .
The rest of the proof goes through identically.
(cid:3)
A. D ETAILS OF S YNTHETIC D ATASETS
We generate datasets with uniformly distributed class conditional distributions.
We generate 2D datasets by ﬁrst sampling a random variable representing the Bayes classiﬁcation error.
The Bayes error is sampled uniformly from the interval 0 . 0 to 0 .
Next, we generate a balanced dataset where the data for each class lies within a unit rectangle and sampled uniformly.
We overlap the sampling rectangular regions to generate a dataset with the desired Bayes error selected in the ﬁrst step.
A. I MPLEMENTATION D ETAILS
Our implementation is based on scikit-learn (Pedregosa et al., 2011).
We ﬁx the training time exploration parameter µ to 0 .
We train the exploration policy π on 82 synthetic datasets each of size 3000 with uniform class conditional distributions, a total of 246 k samples ( § A.1).
We train π using a linear classiﬁer Breiman (2001) and set the hyper-parameters for the learning rate, and data scaling methods using three-fold cross-validation on the whole meta-training dataset.
For the classiﬁer class F , we use a linear model trained with stochastic gradient descent.
We standardize all features to zero mean and unit variance, or scale the features to lie between zero and one.
To select between the two scaling methods, and tune the classiﬁer’s learning rate, we use three-fold cross-validation on a small fully supervised training set of size 30 samples.
The same set is used to calibrate the predicted probabilities of f t .
A. L IST OF D ATASETS
The datasets we used can be accessed at https://www.openml.org/d/<id> .
The list of ( id , size ) pairs below shows the (id for the datasets we used and the dataset size in number of examples:
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table]
[Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
A. B ASELINE E XPLORATION A LGORITHMS
Our experiments aim to determine how Mˆ EL ´ EE compares to other standard exploration strategies.
In particular, we compare to:
(cid:15) -greedy:
With probability (cid:15) , explore
uniformly
at
random; with probability 1 − (cid:15) act greedily according to f t (Sutton, 1996).
Experimentally, we found (cid:15) =
optimal on average, consistent with the results of Bietti et al.
(cid:15) -decreasing: selects a random action with
probabilities (cid:15) i , where (cid:15) i = (cid:15) 0 /t , (cid:15) 0 ∈ ]0 , 1] and t is the index of the current round.
In our experiments we set (cid:15)
= 0 .
(Sutton & Barto, 1998)
Exponentiated Gradient (cid:15) -greedy: maintains a set of candidate values for (cid:15) -greedy exploration.
At each iteration, it runs a sampling procedure to select a new (cid:15) from a ﬁnite set of candidates.
The probabilities associated with the candidates are initialized uniformly and updated with the
Exponentiated Gradient (EG) algorithm.
Following Li et al.
(2010b), we use the candidate set { (cid:15) i = 0 .
× i + 0 .
01 , i = 1 , · · · , 10 } for (cid:15) .
LinUCB:
Maintains conﬁdence bounds for reward payoffs and selects actions with the highest conﬁdence bound.
It is impractical to run “as is” due to high-dimensional matrix inversions.
We use diagonal approximation to the covariance when the dimensions exceeds 150 .
(Li et al., 2010a)
τ
-ﬁrst: Explore uniformly on the ﬁrst τ fraction of the data; after that, act greedily.
Cover: Maintains a uniform distribution over a ﬁxed number of policies.
The policies are used to approximate a covering distribution over policies that are good for both exploration and exploitation (Agarwal et al., 2014).
Cover Non-Uniform: similar to Cover, but reduces the level of exploration of Cover to be more competitive with the Greedy method.
Cover-Nu doesn’t add extra exploration beyond the actions chose by the covering policies (Bietti et al., 2018).
In all cases, we select the best hyperparameters for each exploration algorithm following Bietti et al.
These hyperparameters are: the choice of (cid:15) in (cid:15) -greedy, τ in τ -ﬁrst, the number of bags, and the tolerance ψ for Cover and Cover-NU.
We set (cid:15) = 0 . 0 , τ = 0 .
02 , bag