H YPERMODELS FOR E XPLORATION
Anonymous authors
Paper under double-blind review
A BSTRACT
We study the use of hypermodels to represent epistemic uncertainty and guide exploration.
This generalizes and extends the use of ensembles to approximate
Thompson sampling.
The computational cost of training an ensemble grows with its size
, and as such
, prior work has typically been limited to ensembles with tens of elements.
We show that alternative hypermodels can enjoy dramatic efﬁciency gains
, enabling behavior that would otherwise require hundreds or thousands of elements
, and even succeed in situations where ensemble methods fail to learn regardless of size.
This allows more accurate approximation of Thompson sampling as well as use of more sophisticated exploration schemes.
In particular, we con sider an approximate form of information directed sampling and demonstrate performance gains relative to Thompson sampling
As alternatives to ensembles, we consider linear and neural network hypermodels
, also known as hypernetworks
We prove that, with neural network base models, a linear hypermodel can represent essentially any distribution over functions, and as such, hypernetworks are no more expressive.
I NTRODUCTION
Consider the sequential decision problem of an agent interacting with an uncertain environment, aiming to maximize cumulative rewards.
Over each time period, the agent must balance between exploiting existing knowledge to accrue immediate reward and investing in exploratory behavior that may increase subsequent rewards.
In order to select informative exploratory actions, the agent must have some understanding of what it is uncertain about.
As such, an ability to represent and resolve epistemic uncertainty is a core capability required of the intelligent agents.
The efﬁcient representation of epistemic uncertainty when estimating complex models like neural networks presents an important research challenge.
Techniques include variational inference (Blundell et al., 2015), dropout 1 (Gal & Ghahramani, 2016) and MCMC (Andrieu et al., 2003).
Another approach has been motivated by the nonparametric bootstrap (Efron & Tibshirani, 1994) and trains an ensemble of neural networks with random permutations applied to each dataset (Lu & Van Roy, 2017).
The spirit is akin to particle ﬁltering, where each element of the ensemble approximates a sample from the posterior and variation between models reﬂects epistemic uncertainty.
Ensembles have proved to be relatively effective and to address some shortcomings of alternative posterior approximation schemes (Osband et al., 2016; 2018).
When training a single large neural network is computationally intensive, training a large ensemble of separate models can be prohibitively expensive.
As such, ensembles in deep learning have typically been limited to tens of models (Riquelme et al., 2018).
In this paper, we show that this parsimony can severely limit the quality of the posterior approximation and ultimately the quality of the learning system.
Further, we consider more general approach based on hypermodels that can realize the beneﬁts of large ensembles without the prohibitive computational requirements.
A hypermodel maps an index drawn from a reference distribution to a base model .
An ensemble is one type of hypermodel; it maps a uniformly sampled base model index to that independently trained base model.
We will consider additional hypermodel classes, including linear hypermodels , which we will use to map a Gaussian-distributed index to base model parameters, and hypernetworks , for which the mapping is a neural network (Ha et al., 2016).
Our motivation is that intelligent hypermodel design might be able to amortize computation across the entire distribution of base models, and in doing so, offer large gains in computational efﬁciency.
We train our hypermodels to estimate a posterior distribution over base models conditioned on observed data, in a spirit similar to that of the Bayesian hypermodel literature (Krueger et al., 2017).
Unlike typical variational approximations to Bayesian deep learning, this approach allows computationally efﬁcient training with complex multimodal distributions.
In this paper, we consider hypermodels trained through stochastic gradient descent on perturbed data (see Section 2.1 for a full description).
Training procedures for hypermodels are an important area of research, and it may be possible to improve on this approach, but that is not the focus of this paper.
Instead, we aim to understand whether more sophisticated hypermodel architectures can substantially improve exploration.
To do this we consider bandit problems of varying degrees of complexity, and investigate the computational requirements to achieve low regret over a long horizon.
To benchmark the quality of posterior approximations, we compare their efﬁcacy when used for
Thompson sampling (Thompson, 1933; Russo et al., 2018).
In its ideal form, Thompson sampling (TS) selects each action by sampling a model from the posterior distribution and optimizing over actions.
For some simple model classes, this approach is computationally tractable.
Hypermodels enable approximate TS in complex systems where exact posterior inference is intractable.
Our results address three questions:
Q: Can alternative hypermodels outperform ensembles?
A:
Yes.
We demonstrate through a simple example that linear hypermodels can offer dramatic improvements over ensembles in the computational efﬁciency of approximate TS.
Further, we demonstrate that linear hypermodels can be effective in contexts where ensembles fail regardless of ensemble size.
Q: Can alternative hypermodels enable more intelligent exploration?
A:
Yes.
We demonstrate that, with neural network hypermodels, a version of information-directed sampling (Russo & Van Roy, 2014; 2018) substantially outperforms TS.
This exploration scheme would
be computationally prohibitive with ensemble hypermodels but becomes viable with a hypernetwork.
Q: Are hypernetworks warranted?
A: Not clear.
We prove a theorem showing that, with neural network base models, linear hypermodels can already represent essentially any distribution over functions.
However, it remains to be seen whether hypernetworks can offer statistical or computational advantages.
Variational methods offer an alternative approach to approximating a posterior distribution and sampling from it.
O’Donoghue et al.
(2018) consider such an approach for approximating Thompson sampling in reinforcement learning.
Approaches to approximating TS and information-directed sampling (IDS) with neural networks base models have been studied in (Lu & Van Roy, 2017;
Riquelme et al., 2018) and Nikolov et al.
(2019), respectively, using ensemble representations of uncertainty.
Hypermodels have been a subject of growing interest over recent years.
Ha et al.
(2016) proposed the notion of hypernetworks as a relaxed form of weight-sharing.
Krueger et al.
(2017) proposed Bayesian hypernetworks for estimation of posterior distributions and a training algorithm based on variational Bayesian deep learning.
A limitation of this approach is in its requirement that the hypernetwork be invertible.
Karaletsos et al.
(2018) studied Bayesian neural networks with correlated priors, speciﬁcally considering prior distributions in which units in the neural network are represented by latent variables and weights between units are drawn conditionally on the values of those latent variables.
Pawlowski et al.
(2017) introduced another variational inference based algorithm that interprets hypernetworks as implicit distributions, i.e. distributions that may have intractable probability density functions but allow for easy sampling.
Hu et al.
(2018) proposes the Stein neural sampler which samples from a given (un-normalized) probability distribution with neural networks trained by minimizing variants of Stein discrepancies.
H YPERMODEL A RCHITECTURES AND T RAINING
We consider base models that are parameterized by an element θ of a parameter space Θ .
Given θ ∈ Θ and an input X t ∈ (cid:60) N x , a base model posits that the conditional expectation of the output
[Figure]
Y t +1 ∈ (cid:60) is given by E
[ Y t +1 | X t , θ ]
= f θ ( X t ) , for some class of functions f indexed by θ .
Figure 1a depicts this class of parameterized base models.
A hypermodel is parameterized by parameters ν , which identify a function g ν : Z (cid:55)→ Θ .
We will refer to each z ∈ Z as an index, as it identiﬁes a speciﬁc instance of the base model.
In particular, given hypermodel parameters ν , base model parameters θ can be generated by selecting z ∈ Z and setting θ =
g ν
( z ) .
This notion of a hypermodel is illustrated in Figure 1b.
Along with a hypermodel, in order to represent a distribution over base models, we must specify a reference distribution p z that can be used to sample an element of Z .
A hypermodel and reference distribution together represent a distribution over base models through offering a mechanism for sampling them by sampling an index and passing it through the mapping.
2. H YPERMODEL T RAINING
Given a set of data pairs { ( X t , Y t +1 ) : t = 0 , . . .
, T − 1 } , a hypermodel training algorithm computes parameters ν
so that the implied distribution over base model parameters approximates its posterior.
It is important that training algorithms be incremental.
This enables scalability and also allows for ongoing modiﬁcations to the data set, as those occurring in the bandit learning context, in which data samples accumulate as time progresses.
One approach to incrementally training a hypermodel involves perturbing data by adding noise to response variables, and then iteratively updating parameters via stochastic gradient descent.
We will assume here that the reference distribution p z is either an N z -dimensional unit Gaussian or a uniform distribution over the N z -dimensional unit hypersphere.
Consider an augmented data set
D = { ( X t , Y t +1 , A t ) : t = 0 , . . .
, T − 1 } , where each A t ∈ (cid:60) N z is a random vector that serves to randomize computations carried out by the algorithm.
Each vector
A t is independently sampled from N (0 , I ) if p z is uniform over the unit hypersphere.
Otherwise, A t is independently sampled from the unit hypersphere.
We consider a stochastic gradient descent algorithm that aims to minimize the loss function
[Equation]
where ν 0 is the initial vector of hypermodel parameters.
Each iteration of the algorithm entails calculating the gradient of terms summed over a minibatch of ( x, y, a ) tuples and random indices z .
Note that σ w a (cid:62) z here represents a random Gaussian perturbation of the response variable y .
In particular, in each iteration, a minibatch ˜ D is constructed by sampling a subset of D uniformly with replacement, and a set ˜ Z of indices is sampled i.i.d.
from p z .
An approximate loss function ˜ L ( ν, [Equation]
[Equation]
  is deﬁned based on these sets.
Hypermodel parameters are updated according to ν ← ν − α ∇ ν ˜ L ( ν, ˜ D , ˜ Z ) / |D| where α , σ 2 w , and σ 2 p are algorithm hyperparameters.
In our experiments, we will take the step size α to be constant over iterations.
It is natural to interpret σ 2 p as a prior variance, as though the prior distribution over base model parameters is N (0 , σ 2 p I ), and σ 2 w as the standard deviation of noise, as though the error distribution is Y t − f θ ( X t ) | θ ∼ N (0 , σ 2 w ) .
Note, though, that a hypermodel can be trained on data generated by any process.
One can think of the hypermodel and base models as inferential tools in the mind of an agent rather than a perfect reﬂection of reality.
E NSEMBLE H YPERMODELS
An ensemble hypermodel is comprised of an ensemble of N ν base models, each identiﬁed by a parameter vector in Θ =
(cid:60) N θ .
Letting indices Z be the set of N ν -dimensional one-hot vectors, we can represent an ensemble in terms of a function g ν : Z (cid:55)→ Θ with parameters ν ∈ Θ N ν .
In particular, given hypermodel parameters ν ∈ Θ N ν , an index z ∈ Z generates base model parameters g ν ( Z ) = νZ .
For an ensemble hypermodel, the reference distribution p z is taken to be uniform over the N ν elements of Z .
2. L INEAR H YPERMODELS
Suppose that Θ = (cid:60) N θ and Z = (cid:60) N z .
Consider a linear hypermodel, deﬁned by g ν ( z )
= a + Bz , where hypermodel parameters are given by ν = ( a ∈ (cid:60) N θ , B ∈ (cid:60) N θ × N z ) and
z ∈ Z is an index with reference distribution p z taken to be the unit Gaussian N (0 , I ) over N z -dimensional vectors.
Such a hypermodel can be used in conjunction with any base model that is parameterized by a vector of real numbers.
The aforementioned linear hypermodel entails a number of parameters that grows with the product of the number N θ of base model parameters and the index dimension N z , since B is a N θ × N z matrix.
This can give rise to onerous computational requirements when dealing with neural network base models.
For example, suppose that we wish to model neural network weights as a Gaussian random vector.
This would require an index of dimension equal to the number of weights, and the number of hypermodel parameters would become quadratic in the number of neural network weights.
For a large neural network, storing and updating that many parameters is impractical.
As such, it is natural to consider linear hypermodels in which the parameters a and B are linearly constrained.
Such linear constraints can, for example, represent independence or conditional independence structure among neural network weights.
2. N EURAL N ETWORK H YPERMODELS
More complex hypermodels are offered by neural networks.
In particular, consider the case in which g ν is a neural network with weights ν , taking N
z
inputs and producing N θ outputs.
Such a representation is alternately refered to as a hypernetwork .
Let the reference distribution p
z be the unit Gaussian N (0 , I ) over N z -dimensional vectors.
As a special case, a neural network hypermodel becomes linear if there are no hidden layers.
2. A DDITIVE P RIOR M ODELS
In order for our stochastic gradient descent algorithm to operate effectively, it is often important to structure the base model so that it is a sum of a prior model , with parameters ﬁxed at initialization, and a differential model , with parameters that evolve while training.
The idea here is for the prior model to represent a sample from a prior distribution and for the differential to learn the difference between prior and posterior as training progresses.
This additive decomposition was ﬁrst introduced in (Osband et al., 2018), which demonstrated its importance in training ensemble hypermodels with neural network base models using stochastic gradient descent.
Without this decomposition, to generate neural networks that represent samples from a sufﬁciently diffuse prior, we would have to initialize with large weights.
Stochastic gradient descent tends to train too slowly and thus becomes impractical if initialized in such a way.
We will consider a decomposition that uses neural network base models (including linear base models as a special case) though the concept is more general.
Consider a neural network model class { ˜ f θ : θ ∈ Θ } with Θ = (cid:60) N θ , where the parameter vector θ includes edge weights and node biases.
Let the index set
Z be (cid:60) N z .
Let D be a diagonal matrix for which each element is the prior standard deviation of corresponding component of θ .
Let B ∈ (cid:60) N θ × N z be a random matrix produced at initialization.
We will take DBz to be parameters of the prior (base) model.
Note that, given an index z ∈ Z , this generates a prior model ˜ f DBz .
When we wish to completely specify a prior model distribution, we will need to deﬁne a distribution for generating the matrix B as well as a reference distribution p z .
Given a prior model of the kind we have described, we consider a base model of the form f θ ( x ) =
˜ f DBz ( x )+ ˆ f θ ( x ) , where { ˆ f θ : θ ∈ Θ } is another neural network model class.
The idea is to compute parameters θ such that f θ approximates a sample from a posterior distribution, conditioned on data.
As such, ˆ f θ represents a difference between prior and posterior.
This decomposition is motivated by the observation that neural network training algorithms are most effective if initialized with small weights and biases.
If we initialize θ with small values, the initial values of ˆ f θ will be small, and in this regime, f θ ≈ ˜ f DBz , which is appropriate since an untrained base model should represent a sample from a prior distribution.
E XPLORATION S CHEMES
Our motivation for studying hypermodels stems from their potential role in improving exploration methods.
As a context for studying exploration, we consider bandit problems.
In particular, we consider the problem faced by an agent making sequential decisions, in each period selecting an action X t ∈ X and observing a response Y t +1 ∈ (cid:60) .
Here, the action set
X is a ﬁnite subset of (cid:60) N x and Y t +1 is interpreted as a reward, which the agent wishes to maximize.
We view the environment as a channel that maps X t to Y t +1 , and conditioned on X t and the environment, Y t +1 is conditionally independent of X 0 , Y 1 , . . .
, X t − 1 , Y t .
In other words, actions do not induce delayed consequences.
However, the agent learns about the environment from applying actions and observing outcomes, and as such, its prediction of an outcome Y t +1 is inﬂuenced by past observations X 0 , Y 1 , . . .
, X t − 1 , Y t .
A base model serves as a possible realization of the environment, while a hypermodel encodes a belief distribution over possible realizations.
We consider an agent that represents beliefs about the environment through a hypermodel, continually updating hypermodel paramerers ν via stochastic gradient descent, as described in Section 2.1, to minimize a loss function based on past actions and observations.
At each time t , the agent selects action X t based on the current hypermodel.
Its selection should balance between exploring to reduce uncertainty indicated by the hypermodel and exploiting knowledge conveyed by the hypermodel to accrue rewards.
3. T HOMPSON S AMPLING
TS is a simple and often very effective exploration scheme that will serve as a baseline in our experiments.
With this scheme, each action X t is selected by sampling an index z ∼ p z from the reference distribution and then optimizing the associated base model to obtain X t ∈ arg max x ∈X f g ν ( z ) ( x ) .
See (Russo et al., 2018) for an understanding when and why TS is effective.
3. I NFORMATION -D IRECTED S AMPLING
IDS (Russo & Van Roy, 2014; 2018) offers an alternative approach to exploration that aims to more directly quantify and optimize the value of information.
There are multiple versions of IDS, and we consider here a sample-based version of variance-IDS (Russo & Van Roy, 2018).
In each time period, this entails sampling a new multiset ˜ Z i.i.d.
from p z .
Then, for each action x ∈ X we compute the sample mean of immediate regret
[Equation]
and a sample variance of reward across possible realizations of the optimal action v
[Equation]
Here, { ˜ Z x ∗ : x ∗ ∈ X} forms a partition of ˜ Z such that, x ∗ is an optimal action for each z ∈ ˜ Z x ∗ ; that is, ˜ Z x ∗ = { z ∈ ˜ Z| x ∗ ∈ arg max x ∈X f g ν ( z ) ( x ) } Then, a probability vector π ∗ ∈ ∆ X is obtained by solving
[Equation]
and action X t sampled from π ∗ .
Note that π ∗ x =
if ˜ Z x is empty.
As established by (Russo &
Van Roy, 2018), the minimum over π ∈ ∆ X is always attained by a probability vector that has at most two nonzero components, and this fact can be used to simplify optimization algorithms.
Producing reasonable estimates of regret and variance calls for many distinct samples, and the number required scales with the number of actions.
An ensemble hypermodel with tens of elements does not sufﬁce, while alternative hypermodels we consider can generate very large numbers of distinct samples.
C AN H YPERMODELS O UTPERFORM E NSEMBLES ?
Because training a large ensemble can be prohibitively expensive, neural network ensembles have typically been limited to tens of models (Riquelme et al., 2018).
In this section, we demonstrate that a linear hypermodel can realize the beneﬁts of a much larger ensemble without the onerous computational requirements.
4. G AUSSIAN B ANDIT WITH I NDEPENDENT A RMS
We consider a Gaussian bandit with K independent arms where the mean reward vector θ ∗ ∈ (cid:60) K is drawn from a Gaussian prior N (0 , σ 2 p I ) .
During each time period t , the agent selects an action X t and observes a noisy reward Y t
+1 =
θ ∗ X t + W t
+1 , where W t +1 is i.i.d.
N (0 , σ 2 w ) .
We let σ 2 p = 2 .and
σ 2 w = 1 , and we ﬁx the time horizon to 10,000 periods.
We compare an ensemble hypermodel and a diagonal linear hypermodel trained via SGD with perturbed data.
Our simulation results show that a diagonal linear hypermodel requires about 50 totimes less computation than an ensemble hypermodel to achieve our target level of performance.
As discussed in Section 2.5, we consider base models of the form f θ ( x ) =
˜ f DBz ( x ) + ˜ f θ ( x ) , where ˜ f DBz ( x ) is an additive prior model, and ˜ f
θ ( x ) is a trainable differential model that aims to learn the difference between prior and posterior.
For an independent Gaussian bandit, ˜ f
˜ θ ( x ) =
˜ θ x .
Although the use of prior models is inessential in this toy example, we include it for consistency and illustration of the approach.
The index z ∈ (cid:60) N z of an ensemble hypermodel is sampled uniformly from the set of N zdimensional one-hot vectors.
Each row of B ∈ (cid:60) K × N z is sampled from N (0 , I ) , and D = σ p I .
The ensemble hypermodel takes the form g ν
( z ) =
νz , where the parameters ν ∈ (cid:60) K × N z are initialized to i.i.d.
N (0 , 0 . 05 2 ) .
Although initializing to small random numbers instead of zeros is unnecessary for a Gaussian bandit, our intention here is to mimic neural network initialization and treat the ensemble hypermodel as a special case of neural network hypermodels.
In a linear hypermodel, to model arms independently, we let z 1 , . . .
, z K ∈ (cid:60) m each be drawn independently from N (0 , I ) , and let the index z ∈ (cid:60) N z be the concatenation of z 1 , . . .
, z K , with
N z =
Km .
Let the prior parameters b 1 , . . .
, b K ∈ (cid:60) m be sampled uniformly from the mdimensional hypershpere, and let B ∈ (cid:60) K
× N z be a block matrix with b (cid:62) 1 , . . .
, b (cid:62) K on the diagonal and zero everywhere else.
Let D = σ p
I .
The diagonal linear hypermodel takes the form g ν
( z ) =
Cz + µ , where µ ∈ (cid:60) K and matrix C ∈ (cid:60) K × N z has a block diagonal structure
C = diag ( c (cid:62) 1 , . . .
, c (cid:62) K ) , with c 1 , . . .
, c K ∈ (cid:60) m .
The hypermodel parameters ν
=
( C, µ ) are initialized to i.i.d.
N (0 , 0 . 05 2 ) .
We train both hypermodels using SGD with perturbed data.
For an ensemble hypermodel, the perturbation of the data point collected at time t is σ w A (cid:62) t z , where A t ∼ N (0 , I ) .
For a diagonal linear hypermodel, the perturbation is σ w A (cid:62) t z X t , where A t is sampled uniformly from the m -dimensional unit hypersphere.
We consider an agent to perform well if its average regret over 10,000 periods is below 0 .
01 √ K .
We compare the computational requirements of ensemble and diagonal linear hypermodels across different numbers of actions.
As a simple machine-independent deﬁnition, we approximate the number of arithmetic operations over each time period: computation = n sgd × n z × n data × n params , where n sgd is the number of SGD steps per time period, n z is the number of index samples per SGD step, n data is the data batch size, and n params is the number of hypermodel parameters involved in each index sample.
We ﬁx the data batch size to 1024 for both agents, and sweep over other hyperparameters separately for each agent.
All results are averaged over 100 runs.
In Figure 2, we plot the computation needed versus number of actions.
Using a diagonal linear hypermodel dramatically reduces the amount of computation needed to perform well relative to using an ensemble hypermodel, with a speed-up of around 50 to 100 times for large numbers of actions.
[Figure]
4. N EURAL N ETWORK B ANDIT
In this section we show that linear hypermodels can also be more effective than ensembles in settings that require generalization between actions.
We consider a bandit problem with rewards generated by a neural network that takes vector-valued actions as inputs.
We consider a ﬁnite action set A ⊂ (cid:60) d with d = 20 , sampled uniformly from the unit hypersphere.
We generate data using a neural network with input dimension 20, 2 hidden layers of size 3, and a scalar output.
The output is perturbed by i.i.d.
N (0 , 1) observation noise.
The weights of each layer are sampled independently from N (0 , 2 . 25) , N (0 , 2 . 25 / 3) , and N (0 , 2 . 25 / 3) , respectively, with biases from N (0 , 1) .
We compare ensemble hypermodels with 10, 30, 100, and 300 particles, and a linear hypermodel with index dimension 30.
Both agents use an additive prior ˜ f DBz ( x ) , where ˜ f is a neural network with the same architecture as the one used to generate rewards.
For the ensemble hypermodel, each row of B is initialized by sampling independently from N (0 , I ) , and D is diagonal with appropriate prior standard deviations.
For the linear hypermodel, we enforce independence of weights across layers by choosing B to be block diagonal with 3 blocks, one for each layer.
Each block has width 10.
Within each block, each row is initialized by sampling uniformly from the 10-dimensional unit hypersphere.
For the trainable differential model, both agents use a neural network architecture with 2 hidden layers of width 10.
The parameters of the ensemble hypermodel are initialized to truncated N (0 , 0 .
The weights of the linear hypermodel are initialized using the Glorot uniform initialization, while the biases are initialized to zero.
In our simulations, we found that training without data perturbation gives lower regret for both agents.
In Figure 3, we plot the cumulative regret of agents trained without data perturbation.
We see that linear hypermodels achieve the least regret in the long run.
The performance of ensemble hypermodels is comparable when the number of actions is 200.
However, there is a large performance gap when the number of actions is greater than 200, which, surprisingly, cannot be compensated by increasing the ensemble size.
We suspect that this may have to do with the reliability of neural network regression, and linear hypermodels are somehow able to circumvent this issue.
We also compare with an (cid:15) -greedy agent with a tuned annealing rate, and an agent that assumes independent actions and applies TS under the Gaussian prior and Gaussian noise assumption.
The gap between the (cid:15) -greedy agent and hypermodel agents grows as the number of actions becomes large, as (cid:15) -greedy explores uniformly and does not write off bad actions.
The performance of the
[Figure]
agent that assumes independent actions degrades quickly as the number of actions increases, as it does not generalize across actions.
Appendix B also compares hypermodel performance with two methods for posterior approximation proposed in the literature, dropout (Gal & Ghahramani, 2016) and variational inference ‘Bayes by backprop’ (Blundell et al., 2015).
In both cases their performance is much worse than our hypermodel approach, even after parameter tuning.
C AN H YPERMODELS E NABLE M ORE I NTELLIGENT E XPLORATION ?
IDS, as we described earlier, requires a large number of independent samples from the (approximate) posterior distribution to generate an action.
One way to obtain these samples is to maintain an ensemble of models, as is done by Nikolov et al.
However, as the number of actions increases, maintaining performance requires a large ensemble, which becomes computationally prohibitive.
More general hypermodels offer an efﬁcient mechanism for generating the required large number of base model samples.
In this section, we present experimental results involving a problem and hypermodel stylized to demonstrate advantages of IDS in a transparent manner.
This context is inspired by the one-sparse linear bandit problem constructed by Russo & Van Roy (2018).
However, the authors of that work do not offer a general computationally practical approach that implements
IDS.
Hypermodels may serve this need.
We generate data according to Y t +
= X (cid:62) t θ ∗ + W t +1 where θ ∗ ∈ (cid:60) N θ is sampled uniformly from one-hot vectors and W t +1 is i.i.d.
N (0 , 1) noise.
We consider a linear base model f θ ( x ) = θ (cid:62) x and hypermodel ( g ν ( z )) m
=
exp( βν m ( z 2 m + α )) / (cid:80) N θ n =1 exp( βν n ( z 2 n + α )) , where α = 0 .
01 , and β = 10 .
As a reference distribution we let p z be N (0 , I ) .
Let the initial hypermodel parameters ν 0 be the vector with each component equal to one.
Note that our hypermodel is designed to allow representation of the prior distribution, as well as uniform distributions over subsets of one-hot vectors.
For simplicity, let N θ be a power of two.
Let I be the set of indicator vectors for all nonsingleton sublists of indices in (1 , . . .
, N θ ) that can be obtained by bisecting the list one or more times.
Note that |I| =
N θ − 2 .
Let the action space X be comprised one hot-vectors and vectors { x/ 2 : x ∈ I} .
As with the one-sparse linear bandit of (Russo & Van Roy, 2018), this problem is designed so that
TS will identify the nonzero component of θ ∗ by applying one-hot actions to rule out one component per period, whereas IDS will essentially carry out a bisection search.
This difference in behavior stems from the fact that TS will only ever apply actions that have some chance of being optimal, which in this context includes only the one-hot vectors, whereas IDS can apply actions known to be suboptimal if they are sufﬁciently informative.
Figure 4 plots regret realized by TS and variance-IDS using the aforementioned hypermodel, trained with perturbed SGD.
As expected, the difference in performance is dramatic.
Each plot is averaged over 500 simulations.
We used SGD hyperparameters σ 2 w = 0 .
01 and σ 2 p = 1 / log
N
θ .
The experiments are with N θ = 200 , 500 samples are used for computing the variance-based information ratio.
[Figure]
A RE H YPERNETWORKS W ARRANTED ?
Results of the previous sections were generated using ensemble and linear hypermodels.
It remains to be seen whether hypernetworks offer substantial beneﬁts.
One might believe that hypernetworks can beneﬁt from the computational advantages enjoyed by linear hypermodels while offering the ability to represent a broader range of probability distributions over base models.
The following result refutes this possibility by establishing that, with neural network base models, linear hypermodels can represent essentially any probability distribution over functions with ﬁnite domain.
We denote by L ∞ ( X , B ) the set of functions f : X (cid:55)→ (cid:60) such that (cid:107) f
(cid:107) ∞ < B , where X is ﬁnite with |X| =
K .
Theorem 1 Let p z be the unit Gaussian distribution in (cid:60) K .
For all (cid:15) > 0 , δ > 0 , B > 0 , and probability measures µ over L ∞ ( X , B ) , there exists a transport map H from p z to µ , a neural network f θ : X (cid:55)→ (cid:60) with a linear output node and ReLU hidden nodes, and a linear hypermodel g ν : Z (cid:55)→ (cid:60) N θ such that
[Equation]
with probability at least 1 − δ , for some ν ∈ (cid:60) N ν , where f ∗ =
H ( z ) .
This result is established in Appendix A. To digest the result, ﬁrst suppose that the inequality is satisﬁed with (cid:15)
= 0 .
Interpret µ as the target probability measure we wish to approximate using the hypermodel.
Note that f g ν ( z ) and f ∗ are determined by z ∼ p z , and f ∗ is distributed according to µ , since H is a transport function that maps p z to µ .
If (cid:107) f g ν ( z ) − f ∗ (cid:107) ∞
=
then f g ν ( z ) is also distributed according to µ , and as such, the hypermodel perfectly represents the distribution.
If (cid:15) > 0 , the representation becomes approximate with tolerance (cid:15) .
Though our result indicates that linear hypermodels sufﬁce to represent essentially all distributions over functions, we do not rule out the possibility of statistical or computational advantages to using hypernetworks.
In particular, there could be situations where hypernetworks generalize more accurately given limited data, or where training algorithms operate more effectively with hypernetworks.
In supervised learning, deep neural networks offer such advantages even though a single hidden layer sufﬁces to represent essentially any function.
Analogous beneﬁts might carry over to hypernetworks, though we leave this question open for future work.
C ONCLUSION
Our results offer initial signs of promise for the role of hypermodels beyond ensembles in improving exploration methods.
We have shown that linear hypermodels can offer large gains in computational efﬁciency, enabling results that would otherwise require ensembles of hundreds or thousands of elements.
Further, these efﬁciency gains enable more sophisticated exploration schemes.
In particular, we experiment with a version of IDS sampling and demonstrate beneﬁts over methods based on
TS.
Finally, we consider the beneﬁts of hypernetworks and establish that, with neural network base models, linear hypermodels are already able to represent essentially any distribution over functions.
Hence, to the extent that hypernetworks offer advantages, this would not be in terms of the class of distributions that can be represented.
A U NIVERSAL A PPROXIMATION VIA L INEAR H YPERMODELS
Assume that X ⊂ (cid:60) is a ﬁnite set with |X| =
K , and µ is a probability measure over the bounded functions f : X → (cid:60) such that (cid:107) f
(cid:107) ∞ ≤ B .
First, we show that we can approximately sample a function from µ using a ReLU model with an linear hypermodel, with the input to the hypermodel drawn from the K -dimensional uniform distribution.
Our main result is summarized below:
Theorem 2 Let p z be the uniform distribution over [0 , 1] K .
For all (cid:15) > 0 , δ ∈ (0 , 1) , B > 0 , and probability measures µ over L ∞ ( X , B ) , there exists a transport map H from p z to µ , a neural network f θ : X (cid:55)→ (cid:60) with a linear output node and ReLU hidden nodes, and a linear hypermodel g ν : Z (cid:55)→ (cid:60) N θ such that
[Equation]
with probability at least 1 − δ , for some ν ∈ (cid:60) N ν , where f ∗ =
H ( z ) .
Proof of Theorem 2: Note that since |X| =
K , the functions f : X → (cid:60) can be represented as vectors in (cid:60) K and hence µ can be viewed as a probability measure over (cid:60) K .
Since p z is absolutely continuous with respect to the Lebesgue measure, from Brenier’s theorem, there exists a measurable transport map H : (cid:60) K
→ (cid:60) K from p z to µ .
Notice that we can always assume H ( z ) = 0 for z / ∈ [0 , 1] K since all the probability mass under p z is in [0 , 1] K , and this assumption does not affect the measurability of H .
To show that each component of H is Lebesgue integrable, let H ( z )[ x ] denote the component of H ( z ) corresponding to x .
Note that
[Equation]
where the inequality follows from the fact that µ is over L ∞ ( X , B ) .
From Theorem 1 in Lu et al.
(2017), for any (cid:15) > 0 and δ ∈ (0 , 1) , there exists a ReLU model ˜
H
[Equation]
→
[Equation]
K s.t.
for any x ∈ X ,
[Equation]
where H ( z )[ x ] and ˜ H ( z )[ x ] are respectively the component of H ( z ) and ˜ H ( z ) corresponding to x .
Note that the above inequality implies:
[Equation]
Hence we have
[Equation]
Note that we can always assume (cid:107) ˜ H ( z ) (cid:107) ∞ ≤ B for z ∈ [0 , 1] K .
If this assumption does not hold, we can add some ReLU layers to cap ˜ H to ensure (cid:107) ˜ H ( z ) (cid:107) ∞ ≤ B .
Since (cid:107) H ( z ) (cid:107) ∞
≤ B almost surely, this cap will not increase E z ∼ p
z (cid:104) (cid:107) H ( z ) − ˜ H ( z ) (cid:107) ∞ (cid:105) .
We now discuss how to implement a ReLU model ˜
h : X × [0 , 1] K → [ − B, B ] s.t. ˜
h ( x, z ) =
˜ H ( z )[ x ] based on the ReLU implementation of the K -dimensional ˜ H ( z ) .
Note that it is straightforward to use ReLU to implement the K -dimensional one-hot encoding for all x ∈ X .
Since (cid:107) ˜ H ( z ) (cid:107) ∞ ≤ B , by deﬁning
[Equation]
we have ˜ h
( x, z ) =
˜ H ( z )[ x ] .
Since both ˜ H ( z ) and the one-hot encoding can be implemented by
ReLU, ˜ h
( x, z ) can also be implemented by ReLU.
Finally, we discuss how to construct the target ReLU model f θ : X → (cid:60) and the linear hypermodel g ν based on ˜ h .
Note that the ReLU corresponding to ˜ h has K + 1 input nodes, one corresponding to x (called InputX ) and K corresponding to z (called InputZ ).
Thus, by treating z as part of the parameter vector θ , f θ : X → (cid:60) is construct as follows: we make InputZ hidden nodes, and the only input node to each node in InputZ is InputX .
Speciﬁcally, the input to the i th node in InputZ is 0 × x
+ z i with scalar weight
and bias z i , where z i is the i th component of z .
Also note that given ˜ h , the components in θ are either constant or components in z , thus g ν is linear.
Note that by deﬁnition, f g ν ( z ) ( x ) =
˜ h
( x, z ) =
˜ H ( z )[ x ] .
By deﬁning f ∗ = H ( z ) , we have E z ∼ p
z (cid:2) (cid:107) f g ν
( z ) − f ∗ (cid:107) ∞ (cid:3) <
(cid:15)δ .
From Markov’s, with probability at least 1 − δ , we have (cid:107) f g ν ( z ) − f ∗ (cid:107) ∞ ≤ (cid:15) .
q.e.d.
Finally, we prove Theorem 1 based on Theorem 2:
Proof of Theorem 1: The proof is similar to that of Theorem 2.
Recall that µ can be viewed as a probability measure over (cid:60) K .
Since p z = N (0 , I K ) is absolutely continuous with respect to the
Lebesgue measure, from Brenier’s theorem, there exists a measurable transport map H : (cid:60) K
→ (cid:60) K from p z to µ , moreover, H ( z ) = ∇ z φ ( z ) for a convex scalar function φ : (cid:60) K
→ (cid:60) .
Notice that for the given δ ∈ (0 , 1) , we can always choose a large enough η > 0 , such that P ( z ∈ B η ) ≥ 1 − δ/ 2 , where B η = { z : (cid:107) z (cid:107) 2 ≤ η } .
We deﬁne an auxiliary function
[Equation]
K →
[Equation]
as
[Equation]
otherwise
Since H is measurable, H (cid:48) is also measurable.
Moreover, since H (cid:48) ( z ) =
H ( z ) = ∇ z φ ( z ) on the compact set B η , thus H (cid:48)
( z ) is bounded in B η .
Thus, for any x ∈ X , we have
[Equation]
Similar to the proof for Theorem 2, for the given (cid:15) and δ , there exists a ReLU model ˜ H : (cid:60) K
→ (cid:60) K s.t.
[Equation]
Notice that
E
[Equation]
where with a little bit abuse of notation p z ( · ) denotes the probability density function of the probability measure p z .
The ﬁrst inequality follows from p z ( z ) ≤ (2 π ) − K/ 2 < 1 .
The subsequent analysis is similar to that in Theorem 2.
Similarly, we can prove that with probability at least 1 − δ/ 2 , we have (cid:107)
f g ν ( z ) − H (cid:48)
( z ) (cid:107) ∞ ≤ (cid:15) , where f θ is a ReLU model and g ν is a linear hypermodel.
Recall that with probability at least 1 − δ/ 2 , we have H (cid:48)
( z ) =
H ( z ) .
Thus, from the union bound, we have with probability at least 1 − δ , we have (cid:107) f g ν ( z ) − H
( z ) (cid:107) ∞ ≤ (cid:15) .
By setting f ∗ = H ( z ) , we have proved the theorem.
q.e.d.
B A LTERNATIVE POSTERIOR APPROXIMATIONS FOR NEURAL NETWORKS
Two alternative popular approaches for approximating posterior distributions for neural networks besides ensembles are dropout (Gal & Ghahramani, 2016) and variational methods such as Bayesby-Backprop (Blundell et al., 2015).
In this section, we show experimental results of these two approaches applied to the neural network bandit problem in Section 4.2.
Our results show that dropout and Bayes-by-Backprop achieve similar regrets on these tasks, which are better than (cid:15) - greedy but worse than ensembles.
The dropout approach applies independent Bernoulli masks to the activations during training, and
Gal & Ghahramani (2016) argue that applying dropout masks once again in a forward pass approximates sampling from the posterior distribution.
In our experiment, we ﬁx the number of hidden layers to 2, same as the other agents, and sweep over the number of hidden units of each layer, as well as the probability of keeping each neuron.
In Figure 6, we see that to achieve low regret, the dropout approach needs to use a neural network that is signiﬁcantly larger than other agents (2 layers
[Figure] [Figure]
with 50 to 100 units each compared with 2 layers with 10 units each).
In Figure
[Figure]
we plot the regret of the best performing dropout agent for each task, which we observe is worse than ensembles and linear hypermodels.
Bayes-by-Backprop is a variational method that approximates the posterior distribution of the weights using independent Gaussian distributions.
The approach in effect uses a diagonal linear hypermodel, but the training algorithm follows the variational inference framework which aims to minimize a KL loss and does not employ additive prior models.
Figure 7 shows its performance for different choices of hyperparameters
[Figure]
and the regrets of the best performing agents are plotted in
Figure 5, which are similar to dropout and worse than ensembles and linear hypermodels
[Figure]
C A DDITIVE PRIORS
Osband et al.
(2018) discuss the beneﬁts of using an additive prior to represent prior uncertainty over models.
In the sequential decision making context, it is particularly crucial to be able to represent prior uncertainties when there is little data available.
In this section
[Figure]
we demonstrate the importance
[Figure] [Figure]
of additive priors by comparing ensembles with and without additive priors on the neural network bandit problem described in Section 4.2.
Both ensembles are initialized randomly using standard neural network initializations.
Since weights are typically initialized to small values (otherwise training could be difﬁcult), the outputs of ensembles that do not use additive priors will be close to zero and will not reﬂect prior uncertainties in general.
We see in Figure 8 that ensembles with additive priors achieve signiﬁcantly lower regrets across different numbers of actions and ensemble sizes.
D A LGORITHM SENSITIVITY A NALYSIS
In order to analyze the sensitivity of the algorithm on different parameters, we present a series of experiments on the neural network bandit, similar to Section 4.2.
The default values of the parameters used in these experiments are same as the values used to generate Figure 3.
The results from the experiments are presented below.
In Figure 9 we present some results from sensitivity analysis on observation noise and perturbation noise.
The plot shows the regret of linear hypermodel at 100 k steps, with different perturbation scales ˜ σ w (standard deviation of the noise added to the response variable in the loss function) and standard deviations of the observation noise σ w .
Both ˜ σ w and σ w take values from { 0 , 1 , 2 } .
Observe that for σ w = 0 and σ w = 1 , perturbation scale of 0 works the best; however, on increasing the observation noise to σ w = 2 , a perturbation scale of 1 performs better than perturbation scale of 0 .
The reason for this could be that SGD step is introducing sufﬁcient amount of noise when σ w is 0 or 1 , but it seems that we need to inject additional noise for larger observation noise.
From this, it is clear that we need to introduce perturbation into the loss function, as the observation noise grows.
Even though there is a discrepancy in the cumulative regret for different perturbation scales, the algorithm seems to be robust to the variation in perturbation scale across different levels of observation noise.
[Figure]
In Figure 10 we present results on how mis-speciﬁcation in prior can affect the performance of the linear hypermodel.
Plot shows the regret of linear hypermodel after 100 k steps, when the prior is mis-speciﬁed.
Prior is mis-speiciﬁed by drawing weights of the prior network from a distribution such that the variance of these weights are a factor m times that of the variance of the weights of the generator, we call this value m as the prior weight multiplier.
We can see that a very small value of m does not induce sufﬁcient exploration and leads to a huge regret.
Similarly, a large value of m can also induce more exploration than desired and leads to some additional regret.
[Figure] In Figure 11, we show how performance of a linear hypermodel is affected by the index dimension.
Recall from Section 4.2 that we use a disjoint segment of the index vector to generate prior weights for each layer. We vary the index dimension per layer as 1 , 3 , 5 and 10 (corresponding to the whole index vector with dimension 3 , 9 , 15 , and 30 ), and observe the cumulative regret attained by the algorithm after 100 k steps.
We observe that an index dimension of 1 per layer performs poorly in comparison to 3 , 5 , and 10 as expected. However, surprisingly, index dimension of 3 , 5 and 10 per layer perform similarly. One reason could be that an index dimension per layer of 3 may be sufﬁcient to generate the randomness needed for the neural network bandit under consideration. Also, note that even with an index dimension per layer of 1 , the linear hypermodel performs similar to other methods like Bayes-by-Backprop and dropout.
[Figure]