D EEP B AYESIAN B ANDITS S
HOWDOWN
A N E MPIRICAL C OMPARISON OF B AYESIAN D EEP N ETWORKS FOR T HOMPSON S
AMPLING
Anonymous authors
Paper under double-blind review
A BSTRACT
Recent interest in decision making with deep neural networks has led to a wide development of practical methods that trade off exploration and exploitation.
Bayesian approaches to deep learning are especially appealing for this purpose as they can provide accurate uncertainty estimates as input for reinforcement learning algorithms.
However, these methods are rarely compared on benchmarks that evaluate the impact of their approximations in terms of decision-making performance, and their empirical effectiveness seems poorly understood.
In this paper, we compare a variety of well-established and recent methods under the lens of
Thompson Sampling over a series of contextual bandit problems.
I NTRODUCTION
Recent advances in areas such as reinforcement learning have sparked renewed interest in sequential decision making with deep neural networks.
These models have proven to be powerful and ﬂexible function approximators, allowing one to learn mappings directly from complex state, e.g. pixels, to estimates of expected reward or value.
While such models can be tremendously accurate on data they have been trained on, quantifying epistemic uncertainty remains challenging.
However, having an understanding of what is not yet known or well understood is critical to some central tasks of machine learning, such as effective exploration for decision making.
Unfortunately, computing the exact Bayesian posterior is generally intractable for such highly parameterized models.
As such, signiﬁcant effort has been dedicated to approximate Bayesian methods for deep neural networks.
These range from variational methods (Graves, 2011; Blundell et al., 2015; Kingma et al., 2015) to stochastic minibatch Markov Chain Monte Carlo (Neal, 1994;
Welling & Teh, 2011; Li et al., 2013; Ahn et al., 2012; Mandt et al., 2016).
Because the exact posterior is intractable, evaluating these approaches is hard.
Furthermore, these methods are rarely compared on benchmarks that measure the quality of their estimates of uncertainty for downstream tasks.
To address this challenge, we develop a benchmark for methods for exploration using deep neural networks.
We compare a variety of well-established and recent methods under the lens of Thompson sampling for contextual bandits, a classical task in sequential decision making.
A fundamental aspect of sequential decision making is the exploration-exploitation dilemma: in order to maximize cumulative reward, agents need to trade-off what is expected to be best at the moment, (i.e., exploitation), with potentially sub-optimal exploratory actions.
Solving this trade-off in an efﬁcient manner to maximize cumulative reward is a signiﬁcant challenge as it requires epistemic uncertainty.
Furthermore, exploratory actions should be coordinated throughout the entire decision making process, known as deep exploration, rather than performed independently at each state.
Exploration in the context of reinforcement learning is a highly active area of research.
Simple strategies such as epsilon-greedy remain extremely competitive (Mnih et al., 2015; Schaul et al., 2016).
However, recently a number of promising techniques have emerged that encourage exploration though carefully adding random noise to the parameters (Plappert et al., 2017; Meire Fortunato, 2017; Gal & Ghahramani, 2016) or bootstrat sampling (Osband et al., 2016) before making decisions.
These methods rely explicitly or implicitly on posterior sampling for exploration.
Thompson sampling is a classic algorithm (Thompson, 1933) which requires only that one can sample from the posterior distribution over plausible problem instances (for example, values or rewards).
At each round, it draws a sample and takes a greedy action under the optimal policy for the sample.
The posterior distribution is then updated after the result of the action is observed.
Thompson sampling has been shown to be extremely effective for bandit problems both in practice (Chapelle & Li, 2011;
Granmo, 2010) and theory (Agrawal & Goyal, 2012).
It is especially appealing for deep neural networks as one rarely has access to the full posterior but can often approximately sample from it.
In this paper, we investigate how different posterior approximations affect the performance of
Thompson Sampling from an empirical standpoint.
For simplicity, we restrict ourselves to one of the most basic sequential decision making scenarios: that of contextual bandits.
As expected, no single algorithm beats all the others in every bandit problem.
We observe, however, some general trends.
We ﬁnd that simple heuristics, including bootstrapping, dropout, and injecting random noise, do not signiﬁcantly and systematically improve performance, while they may still sometimes help.
This suggests that the intrinsic randomness of stochastic gradient descent is enough to explore in many cases.
Other algorithms, like Variational Inference and Monte Carlo approaches, strongly couple their complex representation and uncertainty estimates.
This proves problematic when decisions are made based on partial optimization of both, as online scenarios usually require.
On the other hand, making decisions according to a Bayesian linear regression on the representation provided by the last layer of a deep network offers a robust and easy-to-tune approach.
In Section 2 we discuss Thompson Sampling, and present the contextual bandit problem.
The different algorithmic principles to approximate the posterior distribution fed to Thompson Sampling are introduced in Section 3, while the linear case is described in Section 4.
The main experimental results are presented in Section 5, and discussed in Section 6.
Finally, Section 7 concludes.
D ECISION -M AKING VIA T HOMPSON S AMPLING
The contextual bandit problem works as follows.
At time t = 1 , . . . , n a new context X t ∈ R d arrives and is presented to algorithm A .
The algorithm —based on its internal model and X t — selects one of the k available actions, a t .
Some reward r t =
r t ( X t , a t ) is then generated and returned to the algorithm, that may update its internal model with the new data.
At the end of the process, the reward for the algorithm is given by r = (cid:80) nt =1 r t , and cumulative regret is deﬁned as R A
= E [ r ∗ − r ] , where r ∗ is the cumulative reward of the optimal policy (i.e., the policy that always selects the action with highest expected reward given the context).
The goal is to minimize R A .
The main research question we address in this paper is how approximated model posteriors affect the performance of decision making via Thompson Sampling (Algorithm 1) in contextual bandits.
We study a variety of algorithmic approaches to approximate a posterior distribution, together with different empirical and synthetic data problems that highlight several aspects of decision making.
We consider distributions π over the space of parameters that completely deﬁne a problem instance θ ∈ Θ .
For example, θ could encode the reward distributions of a set of arms in the multi-armed bandit scenario, or –more generally– all the parameters of an MDP in reinforcement learning.
In the following sections we rely on the idea that, if we had access to the actual posterior π t given the observed data at all times t , then choosing actions using Thompson Sampling would lead to near-optimal cumulative regret or, more informally, to good performance.
It is important to remark that in some problems this is not necessarily the case; for example, when actions that have no chance of being optimal still convey useful information about other actions.
Thompson Sampling (or UCB approaches) would never select such actions, even if they are worth their cost (Russo & Van Roy, 2014).
In addition, Thompson Sampling does not take into account the time horizon where the process ends, and if known, exploration efforts could be tuned accordingly (Russo et al., 2017).
Nonetheless, under the assumption that very accurate posterior approximations lead to efﬁcient decisions, the question is: what happens when the approximations are not so accurate?
There are two main possibilities.
In some cases, the mismatch in posteriors may not hurt in terms of decision making, and we will still end up with good decisions.
Unfortunately, in other cases, this mismatch together with its induced feedback loop will degenerate in a signiﬁcant loss of performance.
We would like to understand the main aspects that determine which way it goes.
This is an important practical question as, in large and complex systems, computational sacriﬁces and statistical assumptions are made to favor simplicity and tractability.
But, what is their impact?
A LGORITHMS
In this section, we describe the main algorithmic design principles behind the speciﬁc implementations used to approximately sample from the posterior in our simulations of Section 5.
In Figure 24, we visualize the posteriors of the nonlinear algorithms on a synthetic one dimensional problem.
Linear Methods We can apply well-known closed-form updates for Bayesian linear regression (Bishop, 2006) for exact posterior inference in linear models.
We provide the speciﬁc formulas below, and note that they admit a computationally-efﬁcient online version.
We consider exact linear posteriors as a baseline; i.e., these formulas lead to the right posterior only when the data was generated according to Y
= X T β + (cid:15) where (cid:15) ∼ N (0 , σ 2 ) .
Importantly, we model the joint distribution of β and σ 2 for each action.
Sequentially estimating the noise level σ 2 for each action allows the algorithm to adaptively improve its understanding of the volume of the hyperellipsoid of plausible β ’s, leading, in general, to a more aggressive initial exploration phase (in both β and σ 2 ).
The posterior at time t for action i , after observing X, Y , is π t ( β, σ 2 ) =
π t ( β | σ 2 ) π t ( σ 2 ) , where we assume σ 2 ∼ IG ( a t , b t ) , and β | σ 2 ∼ N ( µ t , σ 2 Σ t ) , an Inverse Gamma and Gaussian distribution, respectively.
Their parameters are given by
[Equation]
We set the prior hyperparameters to µ 0 = 0 , and Λ 0 = λ Id , while a 0 =
b 0 = α > 1 .
We consider two approximations to (1) motivated by function approximators where d is large.
While posterior distributions or conﬁdence ellipsoids should capture dependencies across parameters as observed above (say, a dense Σ t ), in practice, computing the correlations across all pairs of parameters is too expensive, and diagonal covariance approximations are common.
For linear models it may still be feasible to exactly compute (1), whereas in the case of Bayesian neural networks, unfortunately, this may no longer be possible.
Accordingly, we study two linear approximations where Σ t is diagonal.
Our goal is to understand the impact of such approximations in the simplest case, to properly set our expectations for the loss in performance of equivalent approximations in more complex approaches, like mean-ﬁeld variational inference or Stochastic Gradient Langevin Dynamics.
Assume for simplicity the noise standard deviation is known.
In Figure 2a, for d = 2 , we see the posterior distribution β t ∼ N ( µ t , Σ t ) of a linear model based on (1), in red, together with two diagonal approximations.
Each approximation tries to minimize a different objective.
In blue, the
PrecisionDiag posterior approximation ﬁnds the diagonal ˆΣ ∈ R d × d minimizing KL ( N ( µ t , ˆΣ) ||
N ( µ t , Σ t )) , like in mean-ﬁeld variational inference.
In particular, ˆΣ =
Diag (Σ − 1 t ) − 1 .
On the other hand, in green, the Diag posterior approximation ﬁnds the diagonal matrix ¯Σ minimizing
KL ( N ( µ t , Σ t ) || N ( µ t , ¯Σ)) instead, similarly to expectation-propagation algorithms.
In this case, the solution is simply ¯Σ =
Diag (Σ t ) .
Neural Linear
The main problem linear algorithms face is their lack of representational power, which they complement with accurate uncertainty estimates.
A natural attempt at getting the best of both worlds consists in performing a Bayesian linear regression on top of the representation of the last layer of a neural network, similarly to Snoek et al.
The predicted value v i for each action a i is given by v i =
β
Ti z x , where z x is the output of the last hidden layer of the network for context x .
While linear methods directly try to regress values v on x , we can independently train a deep net to learn a representation z , and then use a Bayesian linear regression to regress v on z , obtain uncertainty estimates on the β ’s, and make decisions accordingly via Thompson Sampling.
Note that we do not explicitly consider the weights of the linear output layer of the network to make decisions; further, the network is only used to ﬁnd good representations z .
In addition, we can update the network and the linear regression at different time-scales.
It makes sense to keep an exact linear regression (as in (1) and (2)) at all times, adding each new data point as soon as it arrives.
However, we only update the network after a number of points have been collected.
In our experiments, after updating the network, we perform a forward pass on all the training data to obtain z x , which is then fed to the Bayesian regression.
In practice this may be too expensive, and z could be updated periodically with online updates on the regression.
We call this algorithm Neural Linear .
Neural Greedy We refer to the algorithm that simply trains a neural network and acts greedily (i.e., takes the action whose predicted score for the current context is highest) as RMS , as we train it using the RMSProp optimizer.
This is our non-linear baseline, and we tested several versions of it (based on whether the training step was decayed, reset to its initial value for each re-training or not, and how long the network was trained for).
We also tried the (cid:15) -greedy version of the algorithm, where a random action was selected with probability (cid:15) for some decaying schedule of (cid:15) .
Somewhat surprisingly, this did not help.
Variational Inference Variational approaches approximate the posterior by ﬁnding a distribution within a tractable family that minimizes the KL divergence to the posterior (Hinton & Van Camp, 1993).
Typically (and in our experiments), the posterior is approximated by a mean-ﬁeld or factorized distribution.
Recent advances have scaled these approaches to estimate the posterior of neural networks with millions of parameters (Blundell et al., 2015).
A common criticism of variational inference is that it underestimates uncertainty (e.g., (Bishop, 2006)), which could lead to underexploration.
Dropout (Srivastava et al., 2014) can be seen as optimizing a variational objective (Kingma et al., 2015; Gal & Ghahramani, 2016), which leads to a straight-forward posterior approximation and application to Thompson sampling.
Monte Carlo Monte Carlo sampling remains one of the simplest and reliable tools in the Bayesian toolbox.
Rather than parameterizing the full posterior, Monte Carlo methods estimate the posterior through drawing samples.
This is naturally appealing for highly parameterized deep neural networks for which the posterior is intractable in general and even simple approximations such as multivariate
Gaussian are too expensive (i.e. require computing and inverting a covariance matrix over all parameters).
Among Monte Carlo methods, Hamiltonian Monte Carlo (Neal, 1994) (HMC) is often regarded as a gold standard algorithm for neural networks as it takes advantage of gradient information and momentum to more effectively draw samples.
However, it remains unfeasible for larger datasets as it involves a Metropolis accept-reject step that requires computing the log likelihood over the whole data set.
A variety of methods have been developed to approximate HMC using mini-batch stochastic gradients.
These Stochastic Gradient Langevin Dynamics (SGLD) methods (Neal, 1994;
Welling & Teh, 2011) add Gaussian noise to the model gradients during stochastic gradient updates in such a manner that each update results in an approximate sample from the posterior.
Different strategies have been developed for augmenting the gradients and noise according to a preconditioning matrix.
Li et al.
(2013) show that a preconditioner based on the RMSprop algorithm performs well on deep neural networks.
Patterson & Teh (2013) suggesed using the Fisher information matrix as a preconditioner in SGLD.
Unfortunately theory requires that the approximations of SGLD hold only if the learning rate is asymptotically annealed to zero.
Ahn et al.
(2012) introduced Stochastic
Gradient Fisher Scoring to elegantly remove this requirement by preconditioning according to the
Fisher information (or a diagonal approximation thereof).
Mandt et al.
(2016) develop methods for approximately sampling from the posterior using a constant learning rate in stochastic gradient descent and develop a prescription for a stable version of SGFS.
We evaluate the diagonal- SGFS and constant-SGD algorithms from Mandt et al.
(2016) in this work.
Speciﬁcally for constantSGD we use a constant learning rate for stochastic gradient descent, where the learning rate, (cid:15) is given by (cid:15) = 2 S N BB T where S is the batch size, N the number of data points and BB T is an online average of the diagonal empirical Fisher information matrix.
For Stochastic Gradient Fisher Scoring we use the following stochastic gradient update for the model parameters θ at step t :
[Equation]
where we take the noise covariance EE T to also be BB T and H = 2 N ( (cid:15) BB T + EE T ) − 1 .
Bootstrap A simple and empirical approach to approximate the sampling distribution of any estimator is the Bootstrap (Efron, 1982).
The main idea is to simultaneously train q models, where each model i is based on a different dataset D i .
When all the data D is available in advance, D i is typically created by sampling | D | elements from D at random with replacement.
In our case, however, the data grows one example at a time.
Accordingly, we set a parameter p ∈ (0 , 1] , and append the new datapoint to each D i independently at random with probability p .
In order to emulate Thompson Sampling, we sample a model uniformly at random (i.e., with probability p i = 1 /q .) and take the action predicted to be best by the sampled model.
We tested versions where q = 5 , 10 and p = 0 . 8 , 1 . 0 , with neural network models.
Note that even when p = 1 and the datasets are identical, the random initialization of each network, together with the randomness from SGD, leads to different predictions.
Direct Noise Injection Parameter-Noise (Plappert et al., 2017) is a recently proposed approach for exploration in deep RL that has shown promising results.
The training updates for the network are unchanged, but when selecting actions, the network weights are perturbed with isotropic Gaussian noise.
Crucially, the network uses layer normalization (Ba et al., 2016), which ensures that all weights are on the same scale.
The magnitude of the Gaussian noise is adjusted so that the overall effect of the perturbations is similar in scale to (cid:15) -greedy with a linearly decaying schedule (see (Plappert et al., 2017) for details).
Because the perturbations are done on the model parameters, we might hope that the actions produced by the perturbations are more sensible than (cid:15) -greedy.
Bayesian Non-parametric Gaussian processes (Rasmussen & Williams, 2005) are a gold-standard method for modeling distributions over non-linear continuous functions.
It can be shown that, in the limit of inﬁnite hidden units and under a Gaussian prior, a Bayesian neural network converges to a
Gaussian process (Neal, 1994).
As such, GPs would appear to be a natural baseline.
Unfortunately, standard GPs computationally scale cubically in the number of observations, limiting their applicability to relatively small datasets.
There are a wide variety of methods to approximate Gaussian processes using, for example, pseudo-observations (Snelson & Ghahramani) or variational inference (Titsias, 2009).
However, these are outside the scope of this comparison.
1 Due to the scaling issue, we stop adding inputs to the GP after 1000 observations.
This performed signiﬁcantly better than randomly sampling inputs.
Our implementation is a multi-task Gaussian process (Bonilla et al., 2008) with a linear and Matern 3 / 2 product kernel over the inputs and an exponentiated quadratic kernel over latent vectors for the different tasks.
The hyperparameters of this model and the latent task vectors are optimized over the GP marginal likelihood.
This allows the model to learn correlations between the outputs of the model.
Speciﬁcally, the covariance function K ( · ) of the GP is given by:
[Equation]
6) and the task kernel between tasks t and l are k t ( x k , ˆx l ) =
exp( − r λ m ( v k , ˆv l )) 2 ) where v l indexes the latent vector for task l and r λ ( x , ˆx ) =
| ( x (cid:11) λ ) − ( ˆx (cid:11) λ ) | .
The length-scales, λ m and λ l , and amplitude parameters α , β are optimized via the log marginal likelihood.
There are a variety of other methods, including expectation propagation (Hernández-Lobato & Adams, 2015) and α -divergence minimization of which variational inference is a special case (HernándezLobato et al., 2016).
We leave evaluating these approaches to future work.
F EEDBACK L OOP IN THE L INEAR C ASE
In this section, we illustrate some of the subtleties that arise when uncertainty estimates drive sequential decision-making using simple linear examples.
There is a fundamental difference between static and dynamic scenarios.
In a static scenario, e.g. supervised learning, we are given a model family Θ (like the set of linear models, trees, or neural networks with speciﬁc dimensions), a prior distribution π 0 over Θ , and some observed, and importantly assumed i.i.d., data D .
Our goal is to return an approximate posterior distribution: ˜ π
≈ π = P ( θ | D ) .
Let the distance d
(˜ π, π ) represent the quality of our approximation.
On the other hand, in dynamic settings, our estimate at time t , say ˜ π t , will be used via some mechanism M , in this case Thompson sampling, to collect the next data-point, which is then appended to D t .
In this case, the data-points in D t are no longer independent.
D t will now determine two distributions: the posterior given the data that was actually observed, π t +1 =
P ( θ | D t ) , and our estimate ˜ π t +1 .
When the goal is to make good sequential decisions in terms of cumulative regret, the distance d (˜ π t , π t ) is in general no longer a deﬁnitive proxy for performance.
For instance, a poorly-approximated decision boundary could lead an algorithm, based on ˜ π , to get stuck repeatedly selecting a single sub-optimal action a .
After collecting lots of data for that action, ˜ π t and π t could start to agree (to their capacity) on the models that explain what was observed for a , while both would stick to something close to the prior regarding the other actions.
At that point, d (˜ π t , π t ) may show relatively little disagreement, but the regret would already be terrible.
Let π ∗ t be the posterior distribution P ( θ | D t ) under the Thompson Sampling assumption, that is, data was always collected according to π ∗ j for j < t .
We follow the idea that ˜ π t being close to π ∗ t for all t leads to strong performance.
However, this concept is difﬁcult to formalize: once different decisions are made, data for different actions is collected and it is hard to compare posterior distributions.
We illustrate the previous points with a simple example, see Figure 1.
Data is generated according to a bandit with k = 6 arms.
For a given context X ∼ N ( µ, Σ) , the reward obtained by pulling arm i follows a linear model
r i,X =
X
T β i
+ (cid:15) with (cid:15) ∼ N (0 , σ 2 i ) .
The posterior distribution over β i ∈ R d can be exactly computed using the standard Bayesian linear regression formulas presented in Section 3.
We set the contextual dimension d = 20 , and the prior to be β ∼ N (0 , λ I d ) , for λ > 0 .
In Figure 1, we show the posterior distribution for two dimensions of β i for each arm i after n =pulls.
In particular, in Figure 1a, two independent runs of Thompson Sampling with their posterior distribution are displayed in red and green.
While strongly aligned, the estimates for some arms disagree (especially for arms that are best only for a small fraction of the contexts, like Arm 2 and 3, where fewer data-points are available).
In Figure 1b, we also consider Thompson Sampling with an approximate posterior with diagonal covariance matrix, Diag in red, as deﬁned in Section 3.
Each algorithm collects its own data based on its current posterior (or approximation).
In this case, the posterior disagreement after n = 500 decisions is certainly stronger.
However, as shown in Figure 1c, if we computed the approximate posterior with a diagonal covariance matrix based on the data collected by the actual posterior , the disagreement would be reduced as much as possible within the approximation capacity (i.e., it still cannot capture correlations in this case).
Figure 1b shows then the effect of the feedback loop.
We look next at the impact that this mismatch has on regret.
We illustrate with a similar example how inaccurate posteriors sometimes lead to quite different behaviors in terms of regret.
In Figure 2a, we see the posterior distribution β ∼ N ( µ, Σ) of a linear model in green, together with the two diagonal linear approximations introduced in Section 3: the
Diag (in red) and the PrecisionDiag (in blue) approximations, respectively.
We now assume there are k linear arms, β i ∈ R d for i = 1 , . . .
, k , and decisions are made according to the posteriors in
Figure 2a.
In Figures 2b and 2c we plot the regret of Thompson Sampling when there are k =arms, for both d = 15 and d = 30 .
We see that, while the PrecisionDiag approximation does even outperform the actual posterior, the diagonal covariance approximation truly suffers poor regret when we increase the dimension d , as it is heavily penalized by simultaneously over-exploring in a large number of dimensions and repeateadly acting according to implausible models.
E MPIRICAL E VALUATION
In this section, we present the simulations and outcomes of several synthetic and real-world data bandit problems with each of the algorithms introduced in Section 3.
In particular, we ﬁrst explain how the simulations were set up and run, and the metrics we report.
We then split the experiments according to how data was generated, and the underlying models ﬁt by the algorithms from Section 3.
5. T HE E XPERIMENTAL F RAMEWORK
We run the contextual bandit experiments as described at the beginning of Section 2, and discuss below some implementation details of both experiments and algorithms.
Neural Network Architectures.
All algorithms based on neural networks as function approximators share the same architecture.
In particular, we ﬁt a simple fully-connected feedforward network
[Figure] [Figure]
with two hidden layers with 100 units each and softplus activations.
The input of the network has dimension d (same as the contexts), and there are k outputs, one per action.
Note that for each training point ( X t , a t , r t ) only one action was observed.
Updating Models.
A key question is how often and for how long models are updated.
Ideally, we would like to train after each new observation and for as long as possible.
However, this may limit the applicability of our algorithms in online scenarios where decisions must be made immediately.
We update linear algorithms after each time-step by means of (1) and (2).
For neural networks, the default behavior was to train for t s = 20 mini-batches every t f = 20 timesteps 3 .
The size of each mini-batch was 512.
We experimented with increasing values of t s , and it proved essential for some algorithms like variational inference approaches.
Metrics We report two metrics: cumulative regret and simple regret.
We approximate the latter as the mean cumulative regret in the last 500 time-steps, a proxy for the quality of the ﬁnal policy (see further discussion on pure exploration settings, Bubeck et al.
Hyper-Parameter Tuning Deep learning methods are known to be very sensitive to the selection of a wide variety of hyperparameters and many of the algorithms presented are no exception.
Moreover, that choice is known to be highly dataset dependent.
However, in the bandits scenario, we would not have access to each problem a-priori to perform tuning.
Thus, for each algorithm, the same hyperparameters are used across all tasks; for some algorithms they were chosen through careful tuning over cumulative regret on a small subset of linear bandits (Dropout, SGFS, RMS, BBBN, pNoise) and some (ConstSGD, Neural Linear and all linear methods) required very little tuning.
Buffer After some experimentation, we decided not to use a data buffer as some evidence of catastrophic forgetting was observed.
All observations are sampled with equal probability to be part of a mini-batch.
In addition, as is standard in bandit algorithms, each action was initially selected s
= 3 times using round-robin independently of the context.
5. R EAL -W ORLD D ATA P ROBLEMS WITH N ON -L INEAR M ODELS
We test the algorithms in a broad range of bandit problems created using real-world data.
In particular, we test on the Mushroom, Statlog, Covertype, Financial, Jester, Adult, Census, and Song datasets.
A detailed description of each dataset and its bandit problem is provided in the Appendix, Section A.
They exhibit a broad range of properties: small and large sizes, one dominating action versus more homogeneous optimality, learnable or little signal, stochastic or deterministic rewards, etc.
For space reasons, the outcome of some datasets are directly presented in the Appendix.
The Statlog,
Covertype, Adult, and Census datasets were originally tested in Elmachtoub et al.
The results are displayed in Table 1 (cumulative regret) and Table 5 (simple regret).
5. R EAL -W ORLD D ATA P ROBLEMS WITH L INEAR M ODELS
As the algorithms from Section 3 can be implemented for any model architecture, here we use linear models as a baseline comparison across algorithms.
The datasets are the same as in the previous subsection.
The results are displayed in Table 2 (cumulative regret) and Table 5 (simple regret).
5. T HE W HEEL B ANDIT
Some of the real-data problems presented above do not require signiﬁcant exploration.
We design an artiﬁcial problem where the need for exploration is smoothly parameterized.
The wheel bandit is deﬁned as follows (see Figure 26).
Set d = 2 , and δ ∈ (0 , 1) , the exploration parameter.
Contexts are sampled uniformly at random in the unit circle, X ∼ U
( D ) .
There are k = 5 possible actions.
The ﬁrst action a 1 always offers reward r ∼ N ( µ 1 , σ 2 ) , independently of the context.
For contexts inside the blue circle, i.e. (cid:107) X (cid:107) ≤ δ , the other four actions are equal and sub-optimal, with r ∼ N ( µ 2 , σ 2 ) for µ 2 < µ 1 .
When (cid:107) X (cid:107) >
δ , we are outside the blue circle, and one of the actions a 1 , . . .
, a 4 is optimal depending on the sign of X 1 and X 2 .
If X 1 , X 2 > 0 , action 2 is optimal.
If X 1 > 0 , X 2 < 0 , action 3 is optimal, and so on.
Non-optimal actions still deliver r ∼ N ( µ 2 , σ 2 ) , while the optimal action provides r ∼ N ( µ 3 , σ 2 ) , with µ 3 (cid:29) µ 1 .
We set µ 1 = 1 . 2 , µ 2 = 1 . 0 , µ 3 = 50 . 0 , and σ = 0 . 01 .
Note that the probability of a context randomly falling in the high-reward region is 1 − δ 2 .
The difﬁculty of the problem increases with δ , and we expect algorithms to get stuck repeatedly selecting action a 1 .
The problem can be easily generalized for d > 2 .
Results are shown in Table 6, and Figures 3, 4, 5, 6, and 7.
D ISCUSSION
We discuss next our main ﬁndings for each class of algorithms.
Linear Methods.
Linear methods offer a reasonable baseline, strong in many cases.
While their representation power is certainly a limiting factor, their ability to compute informative uncertainty measures seems to payoff and balance their initial disadvantage.
They do well in several datasets, and are able to react fast to surprising or extreme rewards (maybe as single points can have a heavy impact in ﬁtted models, and their updates are deterministic and exact).
Some datasets clearly need more complex non-linear representations, and linear methods are unable to efﬁciently solve those.
In addition, linear methods obviously offer computational advantages.
NeuralLinear.
The NeuralLinear algorithm sits near a sweet spot that is worth further studying.
In general it seems to consistently beat the RMS neural network it is based on, suggesting its exploration mechanisms add concrete value.
We believe its main strength relies on the fact that it is able to simultaneously learn a data representation that greatly simpliﬁes the task at hand, and to accurately quantify the uncertainty over linear models that explain the observed rewards in terms of the proposed representation.
While the former process may be noisier and heavily dependent on the amount of training steps that were taken and available data, the latter always offers the exact solution to its approximate parent problem.
This, together with the partial success of linear methods with poor representations, may explain its promising results.
In some sense, it knows what it knows.
While conceptually simple, its deployment to large scale systems may involve some technical difﬁculties; mainly, to update the Bayesian estimates when the network is re-trained.
We believe, however, standard solutions to similar problems (like running averages) could greatly mitigate these issues.
Variational Inference.
Overall, BBB performed poorly, even when the model was linear.
Variational methods are known to underestimate uncertainty (Bishop, 2006), so we expected that might lead to poor exploration, however, this does not explain the poor performance entirely.
When the model is linear, LinDiagPrecPost computes the exact solution to the minimization problem BBB optimizes with stochastic variational inference.
The poor performance of BBB relative to LinDiagPrecPost suggests that the problem is more than underexploration.
Figure 25 shows that performance improves as we increase the number of training steps, suggesting that it is not sufﬁcient to partially optimize the variational parameters when the uncertainty estimates directly affect the data being collected.
Optimizing to convergence at every step signiﬁcantly reduces regret, but at major computational cost.
Monte Carlo.
ConstantSGD comes out as the winner on Statlog and Covertype, two problems that both appear to require non-linearity and exploration as evidenced by performance of the linear baseline approaches.
The method is especially appealing as it doesn’t require tuning learning rates or exploration parameters.
SGFS, however, performs signiﬁcantly worse on most benchmarks except the hard exploration variant of the wheel problem where it shines.
The additional injected noise in
SGFS may cause the model to explore more than other approaches and thus incur additional regret.
Bootstrap.
The improvement, if any, provided by bootstrapping neural networks over the base model is surprisingly small.
Moreover, it might not justify the important computational overhead of the method.
The randomness from SGD may be enough for exploration in many bandits problems.
Direct Noise Injection.
Parameter-Noise was based on the RMS2 implementation.
Results suggest that it may only offer modest improvements over its base learner, while we found the algorithm hard to tune, and quite sensitive to the heuristic controlling the injected noise-level.
In addition, developing an intuition for the heuristic is not straightforward as it lacks transparency, and may require previous and repeated access to the decision-making process.
On the other hand, we mainly experimented with two dropout versions: ﬁxed p = 0 .
5 , and p = 0 .
The latter consistently delivered better results, and it is the one we report.
The algorithm required no extra-tuning, and sometimes offered small improvements.
Dropout was used both for training and for decision-making; unfortunately, we did not add a baseline where dropout only applies during training.
Consequently, it is not obvious how to disentangle the contribution of better training from that of better exploration.
This remains as future work.
Bayesian Non-parametrics.
Perhaps unsurprisingly, the Gaussian process performs remarkably well on problems with little data but struggles on larger problems.
This may motivate the use of sparse GP-based approaches in the future.
C ONCLUSIONS AND F UTURE W ORK
In this work, we empirically studied the impact on performance of approximate model posteriors for decision making via Thompson Sampling in contextual bandits.
The methods we found to perform best exactly measure uncertainty (possibly under the wrong model assumptions) on top of complex representations learned in parallel.
More complicated approaches that learn the representation and its uncertainty together seemed to require heavier training, an important drawback in online scenarios, and exhibited stronger hyper-parameter dependence.
Further exploring and developing the promising approaches is an exciting avenue for future work.
A PPENDIX
[Figure] [Figure]
[Table] [Table]
[Table] [Table]
[Table]
[Table]
A R EAL -W ORLD D ATASETS
Mushroom.
The Mushroom Dataset (Schlimmer, 1981) contains 22 attributes per mushroom, and two classes: poisonous and safe.
As in Blundell et al. , we create a bandit problem where the agent must decide whether to eat or not a given mushroom. Eating a safe mushroom provides reward +5. Eating a poisonous mushroom delivers reward +5 with probability 1/2 and reward -35 otherwise.
If the agent does not eat a mushroom, then the reward is 0.
We set n = 50000 . Statlog. The Shuttle Statlog Dataset (asu) provides the value of d = 9 indicators during a space shuttle ﬂight, and the goal is to predict the state of the radiator subsystem of the shuttle. There are k = 7 possible states, and if the agent selects the right state, then reward 1 is generated.
Otherwise, the agent obtains no reward ( r = 0 ).
The most interesting aspect of the dataset is that one action is the optimal one in 80% of the cases, and some algorithms may commit to this action instead of further exploring.
In this case, n = 43500 .
Covertype.
The Covertype Dataset (asu) classiﬁes the cover type of northern Colorado forest areas in k = 7 classes, based on d = 54 features, including elevation, slope, aspect, and soil type.
Again, the agent obtains reward 1 if the correct class is selected, and
0 otherwise. We run the bandit for n = 150000 . Financial. We created the Financial Dataset by pulling the stock prices of d = 21 publicly traded companies in NYSE and Nasdaq, for the last 14 years ( n = 3713) .
For each day, the context was the price difference between the beginning and end of the session for each stock.
We synthetically created the arms, to be a linear combination of the contexts, representing k = 8 different potential portfolios.
By far, this was the smallest dataset, and many algorithms over-explored at the beginning with no time to amortize their investment (Thompson Sampling does not account for the horizon).
Jester.
We create a recommendation system bandit problem as follows.
The Jester Dataset (Goldberg et al., 2001) provides continuous ratings in [ − 10 , 10] for 100 jokes from 73421 users. We ﬁnd a complete subset of n = 19181 users rating all 40 jokes. Following Riquelme et al. (2017), we take d = 32 of the ratings as the context of the user, and k = 8 as the arms.
The agent recommends one joke, and obtains the reward corresponding to the rating of the user for the selected joke.
Adult. The Adult Dataset (Kohavi, 1996; asu) comprises personal information from the US Census Bureau database, and the standard prediction task is to determine if a person makes over $50K a year or not. However, we consider the k = 14 different occupations as feasible actions, based on d =covariates (many of them binarized).
As in previous datasets, the agent obtains reward 1 for making the right prediction, and 0 otherwise.
We set n = 45222 . Census. The US Census (1990) Dataset (asu) contains a number of personal features (age, native language, education...) which we summarize in d = 389 covariates, including binary dummy variables for categorical features. Our goal again is to predict the occupation of the individual among k = 9 classes.
The agent obtains reward 1 for making the right prediction, and 0 otherwise, for each of the n = 250000 randomly selected data points.
Song.
The YearPredictionMSD Dataset is a subset of the Million Song Dataset (Bertin-Mahieux et al., 2011).
The goal is to predict the year a given song was released (1922-2011) based on d =technical audio features.
We divided the years in k = 10 contiguous year buckets containing the same number of songs, and provided decreasing Gaussian rewards as a function of the distance between the interval chosen by the agent and the one containing the year the song was actually released.
We initially selected n = 250000 songs at random from the training set.
The Statlog, Covertype, Adult, and Census datasets were tested in Elmachtoub et al.