F LIPOUT : E FFICIENT P SEUDO -I NDEPENDENT W EIGHT P ERTURBATIONS ON M INI -B ATCHES
A BSTRACT
Stochastic neural net weights are used in a variety of contexts, including regularization, Bayesian neural nets, exploration in reinforcement learning, and evolution strategies.
Unfortunately, due to the large number of weights, all the examples in a mini-batch typically share the same weight perturbation, thereby limiting the variance reduction effect of large mini-batches.
We introduce ﬂipout, an efﬁcient method for decorrelating the gradients within a mini-batch by implicitly sampling pseudo-independent weight perturbations for each example.
Empirically, ﬂipout achieves the ideal linear variance reduction for fully connected networks, convolutional networks, and RNNs.
We ﬁnd signiﬁcant speedups in training neural networks with multiplicative Gaussian perturbations.
Flipout also enables us to vectorize evolution strategies: in our experiments, a single GPU with ﬂipout can handle the same throughput as at least 40 CPU cores using existing methods, equivalent to a factor-of- 4 cost reduction on Amazon Web Services.
I NTRODUCTION
Stochasticity is a key component of many modern neural net architectures and training algorithms.
The most widely used regularization methods are based on randomly perturbing a network’s computations (Srivastava et al., 2014; Ioffe & Szegedy, 2015).
Bayesian neural nets can be trained with variational inference by perturbing the weights (Graves, 2011; Blundell et al., 2015).
Weight noise was found to aid exploration in reinforcement learning (Plappert et al., 2017; Fortunato et al., 2017).
Evolution strategies minimizes a black-box objective by evaluating many weight perturbations in parallel, with impressive performance on robotic control tasks (Salimans et al., 2017).
Some methods perturb a network’s activations (Srivastava et al., 2014; Ioffe & Szegedy, 2015), while others perturb its weights (Graves, 2011; Blundell et al., 2015; Plappert et al., 2017;
Fortunato et al., 2017; Salimans et al., 2017).
Stochastic weights are appealing in the context of regularization or exploration because they can be viewed as a form of posterior uncertainty about the parameters.
However, compared with stochastic activations, they have a serious drawback: because a network typically has many more weights than units, it is very expensive to compute and store separate weight perturbations for every example in a mini-batch.
Therefore, stochastic weight methods are typically done with a single sample per mini-batch.
In contrast, activations are easy to sample independently for different training examples within a mini-batch.
This allows the training algorithm to see orders of magnitude more perturbations in a given amount of time, and the variance of the stochastic gradients decays as 1 /N , where N is the mini-batch size.
We believe this is the main reason stochastic activations are far more prevalent than stochastic weights for neural net regularization.
In other settings such as Bayesian neural nets and evolution strategies (ES), one is forced to use weight perturbations and live with the resulting inefﬁciency.
In order to achieve the ideal 1 /N variance reduction, the gradients within a mini-batch need not be independent, but merely uncorrelated.
In this paper, we present ﬂipout, an efﬁcient method for decorrelating the gradients between different examples without biasing the gradient estimates.
Flipout applies to any perturbation distribution that factorizes by weight and is symmetric around 0—including DropConnect, multiplicative Gaussian perturbations, evolution strategies, and variational Bayesian neural nets—and to many architectures, including fully connected nets, convolutional nets, and RNNs.
In Section 3, we show that ﬂipout gives unbiased stochastic gradients, and discuss its efﬁcient vectorized implementation which incurs only a factor-of-2 computational overhead compared with shared perturbations.
We then analyze the asymptotics of gradient variance with and without ﬂipout, demonstrating strictly reduced variance.
In Section 4, we measure the variance reduction effects on a variety of architectures.
Empirically, ﬂipout gives the ideal 1 /N variance reduction in all architectures we have investigated, just as if the perturbations were done fully independently for each training example.
We demonstrate that ﬂipout achieves competitive performance in regularizing LSTMs, compared to several methods that use dropout.
We also demonstrate speedups in training time in a large batch regime.
Finally, we use ﬂipout to vectorize evolution strategies (Salimans et al., 2017), allowing a single GPU to handle the same throughput as 40 CPU cores using existing approaches; this corresponds to a factor-of-4 cost reduction on Amazon Web Services.
B ACKGROUND 2. W EIGHT PERTURBATIONS
We use the term “weight perturbation” to refer to a class of methods which sample the weights of a neural network stochastically at training time.
More precisely, let f ( x, W ) denote the output of a network with weights W on input x .
The weights are sampled from a distribution q θ parameterized by θ .
We aim to minimize the expected loss E ( x,y ) ∼D ,W ∼ q θ [ L ( f ( x, W ) , y )] , where L is a loss function, and D denotes the data distribution.
The distribution q θ can often be described in terms of perturbations: W = W + ∆ W , where W are the mean weights (typically represented explicitly as part of θ ) and ∆ W is a stochastic perturbation.
We now give some speciﬁc examples of weight perturbations.
Gaussian perturbations.
If the entries ∆ W ij are sampled independently from Gaussian distributions with variance σ 2 ij , this corresponds to the distribution W ij ∼ N
( W ij , σ 2 ij ) .
Using the reparameterization trick (Kingma et al., 2015), this can be rewritten as W ij =
W ij + σ ij (cid:15) ij , where (cid:15) ij ∼ N (0 , 1) ; this representation allows the gradients to be computed using backprop.
A variant of this is multiplicative Gaussian perturbation, where the perturbations are scaled according to the weights: W ij ∼ N
( W ij , σ 2 ij W 2 ij ) , or W ij =
W ij (1 + σ ij (cid:15) ij ) , where again (cid:15) ij ∼ N (0 , 1) .
Multiplicative perturbations can be more effective than additive ones because the information content of the weights is the same regardless of their scale.
DropConnect.
DropConnect (Wan et al., 2013) is a regularization method inspired by dropout (Srivastava et al., 2014) which randomly zeros out a random subset of the weights.
In the case of a 50% drop rate, this can be thought of as a weight perturbation where W = W/ [-
-]2 and each entry ∆ W ij is sampled uniformly from ± W ij .
Variational Bayesian neural nets.
Rather than ﬁtting a point estimate of a neural net’s weights, one can adopt a Bayesian approach of putting a prior distribution p ( W ) over the weights and approximating the posterior distribution p ( W |D )
∝ p ( W ) p ( D| W ) , where D denotes the observed data.
Graves (2011) observed that one could ﬁt an approximation q θ ( W )
≈ p ( W |D )
using variational inference; in particular, one could maximize the evidence lower bound (ELBO) with respect to θ :
[Equation]
The negation of the second term can be viewed as the description length of the data, and the negation of the ﬁrst term can be viewed as the description length of the weights (Hinton & Van Camp, 1993).
Graves (2011) observed that if q is chosen to be a factorial Gaussian, sampling from θ can be thought of as Gaussian weight perturbation where the variance is adapted to maximize F .
Blundell et al.
(2015) later combined this insight with the reparameterization trick (Kingma & Welling, 2014) to derive unbiased stochastic estimates of the gradient of F .
Evolution strategies.
ES (Eigen, 1973) is a family of black box optimization algorithms which use weight perturbations to search for model parameters.
ES was recently proposed as an alternative reinforcement learning algorithm
(Schmidhuber et al., 2007; Salimans et al., 2017).
In each iteration, ES generates a collection of weight perturbations as candidates and evaluates each according to a ﬁtness function
F .
The gradient of the parameters can be estimated given the evaluation result.
ES is highly parallelizable, because perturbations can be generated and evaluated independently by different workers.
Suppose M is the number of workers, W is the model parameter, σ is the standard deviation of the perturbations, α is the learning rate, F is the objective function, and ∆ W m is the Gaussian noise generated at worker m .
The ES algorithm tries to maximize E ∆ W (cid:2) F (cid:0) W + σ ∆ W (cid:1)(cid:3) .
The gradient of the objective function and the update rule can be given as:
[Equation]
, where ∆ W ∼ N (0 , σI ) [Equation]
2. R EPARAMETERIZATION GRADIENT
As mentioned in Section 2.1, the reparameterization trick allows gradients to be computed using backprop (Kingma & Welling, 2014).
Regularizers such as dropout (Srivastava et al., 2014) perturb activations rather than weights.
Because activations can be perturbed independently for every training example in a mini-batch, the variance of the stochastic gradients decreases linearly with the size of the mini-batch.
Unfortunately, it would be extremely expensive to explicitly sample an independent weight perturbation ∆ W for every training example — this would require storing tens to hundreds of copies of the parameters in memory.
Therefore, weight perturbation gradients are typically computed using only a single perturbation shared between all examples in the mini-batch.
It is possible to derive efﬁcient independent weight perturbation methods in particular cases; for instance, independent DropConnect masks could be stored since they require only one bit per weight (Wan et al., 2013).
Kingma et al.
(2015) showed that for fully connected networks with no weight sharing, unbiased stochastic gradients could be computed without explicit weight perturbations using the local reparameterization trick (LRT).
For example, suppose X is the input mini-batch, W is the weight matrix and B = XW is the matrix of activations.
The LRT samples the activations B rather than the weights W .
In the case of a Gaussian posterior, the LRT is given by:
[Equation]
Note that the LRT gives an unbiased gradient estimate when the activations are independent; however, it does not give an unbiased estimate when weight sharing is involved, such as in convolutional neural networks and RNNs.
2. O THER RELATED WORK
Control variates are another general class of strategies for variance reduction, both for black-box optimization (Williams, 1992; Ranganath et al., 2014; Mnih & Gregor, 2014) and for gradient-based optimization (Roeder et al., 2016; Miller et al., 2017;
Louizos et al., 2017).
Control variates are complementary to ﬂipout, so one could potentially combine these techniques to achieve a larger variance reduction.
We also note that the fastfood transform (Le et al., 2013) is based on similar mathematical techniques.
However, whereas fastfood is used to approximately multiply by a large
Gaussian matrix, ﬂipout preserves the random matrix’s distribution and instead decorrelates the gradients between different samples.
M ETHODS
As described above, weight perturbation algorithms suffer from high variance of the gradient estimates because all training examples in a mini-batch share the same perturbation.
More precisely, sharing the perturbation induces correlations between the gradients, implying that the variance can’t be eliminated by averaging.
In this section, we introduce ﬂipout, an efﬁcient way to perturb the weights quasi-independently within a mini-batch.
3. F LIPOUT
We make two assumptions about the weight distribution q θ : (1) the perturbations of different weights are independent; and (2) the perturbation distribution is symmetric around zero.
These are nontrivial constraints, but they encompass important use cases: independent Gaussian perturbations (e.g. as used in variational BNNs and ES) and DropConnect with drop probability 0.5.
We observe that, under these assumptions, the perturbation distribution is invariant to elementwise multiplication by a random sign matrix (i.e. a matrix whose entries are ± 1 ).
In the following, we denote elementwise multiplication by ◦ .
Observation 1.
Let q θ be a perturbation distribution that satisﬁes the above assumptions, and let (cid:100) ∆ W ∼ q θ .
Let E be a random sign matrix that is independent of (cid:100) ∆ W .
Then ∆ W = (cid:100) ∆ W ◦ E is identically distributed to (cid:100) ∆ W .
Furthermore, the loss gradients computed using ∆ W are identically distributed to those computed using (cid:100) ∆ W .
Flipout exploits this fact by using a base perturbation (cid:100) ∆ W shared by all examples in the mini-batch, and multiplies it by a different rank-one sign matrix for each example:
[Equation]
where the subscript denotes the index within the mini-batch, and r n and s n are random vectors whose entries are sampled uniformly from ± 1 .
According to Observation 1, the marginal distribution over gradients computed for individual training examples will be identical to the distribution computed using shared weight perturbations.
Consequently, ﬂipout yields an unbiased estimator for the loss gradients.
However, by decorrelating the gradients between different training examples, we can achieve much lower variance updates when averaging over a mini-batch.
Vectorization.
The advantage of ﬂipout over explicit perturbations is that computations on a minibatch can be written in terms of matrix multiplications.
This enables efﬁcient implementations on
GPUs and modern accelerators such as the Tensor Processing Unit (TPU) (Jouppi et al., 2017).
Let x denote the activations in one layer of a neural net.
The next layer’s activations are given by:
[Equation]
where φ denotes the activation function.
To vectorize these computations, we deﬁne matrices R and S whose rows correspond to the random sign vectors r n and s n for all examples in the mini-batch.
The above equation is vectorized as:
[Equation]
This deﬁnes the forward pass.
Because R and S are sampled independently of W and (cid:100) ∆ W , we can backpropagate through Eqn.
4 to obtain derivatives with respect to W , (cid:100) ∆ W , and X .
Computational cost.
In general, the most expensive operation in the forward pass is matrix multiplication.
Flipout’s forward pass requires two matrix multiplications instead of one, and therefore should be roughly twice as expensive as a forward pass with a single shared perturbation when done in sequence.
However, note that the two matrix multiplications are independent and can be done in parallel; this incurs the same overhead as the local reparameterization trick (Kingma et al., 2015).
A general rule of thumb for neural nets is that the backward pass requires roughly twice as many FLOPs as the forward pass.
This suggests that each update using ﬂipout ought to be about twice as expensive as an update with a single shared perturbation (if the matrix multiplications are done sequentially); this is consistent with our experience.
Evolution strategies.
ES is a highly parallelizable algorithm; however, most ES systems are engineered to run on multi-core CPU machines and are not able to take full advantage of GPU parallelism.
Flipout enables ES to run more efﬁciently on a GPU because it allows each worker to evaluate a batch of quasi-independent perturbations rather than only a single perturbation.
To apply ﬂipout to ES, we can simply replicate the starting state by the number of ﬂipout perturbations N , at each worker.
Instead of Eqn.
1, the update rule using M workers becomes:
[Equation]
where m indexes workers, n indexes the examples in a worker’s batch, and F mn is the reward evaluated with the n th perturbation at worker m .
Hence, each worker is able to evaluate multiple perturbations as a batch, allowing for parallelism on a GPU architecture.
3. V ARIANCE ANALYSIS
In this section, we analyze the variance of stochastic gradients with and without ﬂipout.
We show that ﬂipout is guaranteed to reduce the variance of the gradient estimates compared to the na¨ıve shared perturbation.
Let G x = G ( x, ∆ W ) = ∂∂θ i L ( y, f ( x, W, ∆ W )) denote one entry of the stochastic gradient ∇ θ L ( y, f ( x, W, ∆ W )) under the perturbation ∆ W for a single training example x .
(Note that G x is a random variable which depends on both x and ∆ W .
We analyze a single entry of the gradient so that we can work with scalar-valued variances.)
We denote the gradient averaged over a mini-batch as the random variable G B = 1 N (cid:80) Nn =1 G ( x n , ∆ W n ) , where B = { x n } Nn =1 denotes a mini-batch of size N , and ∆ W n denotes the perturbation for the n th example. (The randomness comes from both the choice of B and the random perturbations.)
For simplicity, we assume that the x n are sampled i.i.d. from the data distribution.
Using the Law of Total Variance, we decompose Var( G B ) into a data term (the variance of the exact mini-batch gradients) and an estimation term (the estimation variance for a ﬁxed mini-batch):
[Equation]
Notice that the data term decays with N while the estimation term may not, due to its dependence on the shared perturbation.
But we can break the estimation term into two parts for which we can analyze the dependence on N . To do this, we reformulate the standard shared perturbation scheme as follows: ∆ W is generated by ﬁrst sampling (cid:100) ∆ W and then multiplying it by a random sign matrix rs (cid:62) as in Eqn. 3 — exactly like ﬂipout, except that the sign matrix is shared by the whole minibatch. According to Observation 1, this yields an identical distribution for ∆ W to the standard shared perturbation scheme.
Based on this, we obtain the following decomposition:
Theorem 2 (Variance Decomposition Theorem) .
Deﬁne α , β , and γ to be
[Equation]
Under the assumptions of Observation
[Equation]
the variance of the gradients under shared perturbations and ﬂipout perturbations can be written in terms of α , β , and γ as follows:
[Equation]
Proof.
Details of the proof are provided in Appendix A.
We can interpret α , β , and γ as follows.
First, α combines the data term from Eqn.
6 with the expected estimation variance for individual data points. This corresponds to the variance of the gradients on individual training examples, so fully independent perturbations yield a total variance of α/N . The other terms, β and γ , reﬂect the covariance between the estimation errors on different training examples as a result of the shared perturbations.
The term β reﬂects the covariance that results from sampling r and s , so it is eliminated by ﬂipout, which samples these vectors independently. Finally, γ reﬂects the covariance that results from sampling (cid:100) ∆ W , which ﬂipout does not eliminate.
Empirically, for all the neural networks we investigated, we found that α (cid:29) β (cid:29) γ . This implies the following behavior for Var( G B ) as a function of N : for small N , the data term α/N dominates, giving a 1 /N variance reduction; with shared perturbations, once N is large enough that α/N < β , the variance Var( G B ) levels off to β .
However, ﬂipout continues to enjoy a 1 /N variance reduction in this regime.
In principle, ﬂipout’s variance should level off at the point where α/N < γ , but in all of our experiments, γ was small enough that this never occurred: ﬂipout’s variance was approximately α/N throughout the entire range of N values we explored, just as if the perturbations were sampled fully independently for every training example.
E XPERIMENTS
We ﬁrst veriﬁed empirically the variance reduction effect of ﬂipout predicted by Theorem 2; we measured the variance of the gradients under different perturbations for a wide variety of neural network architectures and batch sizes.
In Section 4.2, we show that ﬂipout applied to Gaussian perturbations and DropConnect is effective at regularizing LSTM networks.
In Section 4.3, we demonstrate that ﬂipout converges faster than shared perturbations when training with large minibatches.
Finally, in Section 4.4 we present experiments combining Evolution Strategies with ﬂipout in both supervised learning and reinforcement learning tasks. In our experiments, we consider the four architectures shown in Table 1 (details in Appendix B). [Table] LeCun et al.,
[Table]
[Table]
[Table] [Table] [Figure] VARIANCE REDUCTION
Since the main effect of ﬂipout is intended to be variance reduction of the gradients we ﬁrst estimated the gradient variances of several architectures with mini-batch sizes ranging from 1 to(Fig. 1).
We experimented with three perturbation methods: a single shared perturbation per minibatch, the local reparameterization trick (LRT) of Kingma et al.
(2015), and ﬂipout.
For each of FC, ConVGG, and LSTM architectures, we froze a partially trained network to use for all variance estimates and we used multiplicative Gaussian perturbations with σ 2 = 1 .
We computed Monte Carlo estimates of the gradient variance, including both the data and estimation terms in Eqn. 6.
Conﬁdence intervals are based on 50 independent runs of the estimator.
Details are given in Appendix C.
The analysis in Section 3.2 makes strong predictions about the shapes of the curves in Fig.
By Theorem 2, the variance curves for ﬂipout and shared perturbations each have the form a + b/N , where N is the mini-batch size.
On a log-log plot, this functional form appears as a linear regime with slope -1, a constant regime, and a smooth phase transition in between.
Also, because the distribution of individual gradients is identical with and without ﬂipout, the curves must agree for N = 1 .
Our plots are consistent with both of these predictions.
We observe that for shared perturbations, the phase transition consistently occurs for batch sizes somewhere between 100 and 1000.
In contrast, ﬂipout gives the ideal linear variance reduction throughout the range of mini-batch sizes we investigated, i.e., its behavior is indistinguishable from fully independent perturbations.
As analyzed by Kingma et al.
, the LRT gradients are fully independent within a mini-batch, therefore are guaranteed to achieve the ideal 1 /N variance reduction.
Furthermore, they reduce the variance below that of explicit weight perturbations, so we would expect them to achieve smaller variance than ﬂipout, as shown in Fig.
1a.
However, ﬂipout is applicable to a wider variety of architectures, including convolutional nets and RNNs.
4. R EGULARIZATION FOR L ANGUAGE M ODELING
We evaluated the regularization effect of ﬂipout on the character-level and word-level language modeling tasks with the Penn Treebank corpus (PTB) (Marcus et al., 1993).
We compared ﬂipout to several other methods for regularizing RNNs: na¨ıve dropout (Zaremba et al., 2014), variational dropout (Gal & Ghahramani, 2016), recurrent dropout (Semeniuta et al., 2016), zoneout (Krueger et al., 2016), and DropConnect (Merity et al., 2017).
Zaremba et al.
(2014) apply dropout only to the feed-forward connections of an RNN (to the input, output, and connections between layers).
The other methods regularize the recurrent connections as well: Semeniuta et al.
(2016) apply dropout to the cell update vector, with masks sampled either per step or per sequence; Gal & Ghahramani (2016) apply dropout to the forward and recurrent connections, with all dropout masks sampled per sequence.
Merity et al.
(2017) use DropConnect to regularize the hidden-to-hidden weight matrices, with a single DropConnect mask shared between examples in a mini-batch.
We denote their model WD (for weight-dropped LSTM).
Character-Level.
For our character-level experiments, we used a single-layer LSTM withhidden units. We trained each model on non-overlapping sequences of 100 characters in batches of [Table]
[Table] size 32, for 200 epochs, using Adam with learning rate 0.002 and gradient clipping threshold 1.0. Here, we applied ﬂipout to the hidden-to-hidden weight matrix.
More hyperparameter details are given in Appendix D.
The results, measured in bits-per-character (BPC) for the validation and test sequences of PTB, are shown in Table 2.
In the table, shared perturbations and ﬂipout (with Gaussian noise sampling) are denoted by Mult.
Gauss and Mult.
Gauss + Flipout, respectively.
We see that ﬂipout outperforms recurrent dropout (Semeniuta et al., 2016) and zoneout, and is competitive with variational dropout (Gal & Ghahramani, 2016).
Word-Level.
For our word-level experiments, we used a 2-layer LSTM with 650 hidden units per layer and 650-dimensional word embeddings.
We trained on sequences of length 35 in batches of size 40, for 100 epochs.
We used SGD with initial learning rate 30, and decayed the learning rate by a factor of 4 based on the nonmonotonic criterion introduced by Merity et al.
We used ﬂipout to implement DropConnect, as described in Section 2.1, and call this WD+Flipout.
We applied WD+Flipout to the hidden-to-hidden weight matrices for recurrent regularization, and used the same hyperparameters as Merity et al.
We used embedding dropout (setting rows of the embedding matrix to 0) with probability 0.1 for all regularized models except Gal, where we used probability 0.2 as speciﬁed in their paper.
More hyperparameter details are given in Appendix D. We show in Table 3 that WD+Flipout outperforms the other methods with respect to both validation and test perplexity.
In Appendix E.4, we show that WD+Flipout yields signiﬁcant variance reduction for large mini-batches, and that when training with batches of size 8192, it converges faster than WD.
4. LARGE BATCH TRAINING WITH FLIPOUT
Theorem 2 and Fig. 1 suggest that the variance reduction effect of ﬂipout is more pronounced in the large mini-batch regime.
In this section, we train a Bayesian neural network with mini-batches of size 8192 and show that ﬂipout speeds up training in terms of the number of iterations.
We trained the FC and ConvLe networks from Section 4.1 using Bayes by Backprop (Blundell et al., 2015).
Since our primary focus is optimization, we focus on the training loss, shown in Fig.
2a: for FC, we compare ﬂipout with shared perturbations and the LRT; for ConvLe, we compare only to shared perturbations since the LRT does not give an unbiased gradient estimator.
We found that ﬂipout converged in about 3 times fewer iterations than shared perturbations for both models, while achieving comparable performance to the LRT for the FC model.
Because ﬂipout is roughly twice as expensive as shared perturbations (see Section 3.1), this corresponds to a 1.5x speedup overall.
Curves for the training and test error are given in Appendix E.2.
4. EVOLUTION STRATEGIES
ES typically runs on multiple CPU cores. The challenge in making ES GPU-friendly is that each sample requires computing a separate weight perturbation, so each worker can only generate one sample at a time.
In Section 3.1, we showed that ES with ﬂipout allows each worker to evaluate a batch of perturbations, which can be done efﬁciently on a GPU.
However, ﬂipout induces correlations between the samples, so we investigated whether these correlations induce a slowdown in training relative to fully independent perturbations (which we term “IdealES”). In this section, we show empirically that ﬂipout ES is just as sample-efﬁcient as IdealES, and consequently one can obtain signiﬁcantly higher throughput per unit cost using ﬂipout ES on a GPU.
The ES gradient deﬁned in Eqn. 1 has high variance, so a large number of samples are generally needed before applying an update.
We found that 5,000 samples are needed to achieve stable performance in the supervised learning tasks.
Standard ES runs the forward pass 5,000 times with [Figure] independent weight perturbations, which sees little beneﬁt to using a GPU over a CPU.
FlipES allows the same number of samples to be evaluated using a much smaller number of explicit perturbations.
Throughout the experiments, we ran ﬂipout with mini-batches of size 40 (i.e. N = 40 in
Eqn. 5).
We compared IdealES and FlipES with a fully connected network (FC) on the MNIST dataset.
Fig. 2b shows that we incur no loss in performance when using pseudo-independent noise.
Next, we compared FlipES and cpuES (using 40 CPU cores) in terms of the per-update time with respect to the model size.
The result (in Appendix E.3) shows that FlipES scales better because it runs on the GPU.
Finally, we compared FlipES and the backpropagation algorithm with both FC and ConvLe. Fig.
2c and Fig. 2d show that FlipES achieves data efﬁciency comparable with the backpropagation algorithm.
IdealES has a much higher computational cost than backpropagation, due to the large number of forward passes.
FlipES narrows the computational gap between them.
Although ES is more expensive than backpropagation, it can be applied to models which are not fully differentiable, such as models with a discrete loss (e.g., accuracy or BLEU score) or with stochastic units.
C ONCLUSIONS
We have introduced ﬂipout, an efﬁcient method for decorrelating the weight gradients between different examples in a mini-batch.
We showed that ﬂipout is guaranteed to reduce the variance compared with shared perturbations.
Empirically, we demonstrated signiﬁcant variance reduction in the large batch setting for a variety of network architectures, as well as signiﬁcant speedups in training time.
Flipout also makes it practical to apply GPUs to evolution strategies, resulting in substantially increased throughput for a given computational cost.
We believe ﬂipout will make weight perturbations practical in the large batch setting favored by modern accelerators such as Tensor Processing
Units (Jouppi et al., 2017).
A PROOF OF T HEOREM
In this section [Equation] we provide the proof of Theorem 2 (Variance Decomposition Theorem).
Proof.
We use the notations from Section 3.2.
Let x, x (cid:48) denote two training examples from the mini-batch B
[Equation]
and ∆ W, ∆ W (cid:48) denote the weight perturbations they received. We begin with the decomposition into data and estimation terms (Eqn. 6), which we repeat here for convenience: [Equation]
The data term from Eqn. 13 can be simpliﬁed:
[Equation] We break the estimation term from Eqn. 13 into variance and covariance terms: [Equation] We now separately analyze the cases of fully independent perturbations, shared perturbations
[Equation]
and ﬂipout. Fully independent perturbations. If the perturbations are fully independent, the second term in Eqn. 15 disappears.
Hence, combining Eqns. 13, 14, and 15, we are left with [Equation] which is just α/N . Shared perturbations.
Recall that we reformulate the shared perturbations in terms of ﬁrst sampling (cid:100) ∆ W , and then letting ∆ W = (cid:100) ∆ W ◦ rs (cid:62) , where r and s are random sign vectors shared by the whole batch. Using the Law of Total Variance, we break the second term in Eqn. 15 into a part that comes from sampling (cid:100) ∆ W and a part that comes from sampling r and s .
[Equation]
Since the perturbations are shared, ∆ W (cid:48) = ∆ W , so this can be simpliﬁed slightly to:
[Equation]
Plugging these two terms into the second term of Eqn. 15 yields N − 1 N ( β + γ ) , so putting this all together we get Var( G B ) = 1 N α + N − 1 N ( β + γ ) .
Flipout.
Since the perturbations for different examples are independent conditioned on (cid:100) ∆ W , the ﬁrst term of Eqn.
17 vanishes.
However, the second term remains. Therefore, plugging this into Eqn. 15 and combining the result with Eqns. 13 and 14, we are left with Var( G B ) =
1 N α + N − 1 N γ . B N ETWORK CONFIGURATIONS
Here, we provide details of the network conﬁgurations used for our experiments (Section 4).
The FC network is a 3-layer fully-connected network with 512-512-10 hidden units. ConvLe is a LeNet-like network (LeCun et al., 1998) where the ﬁrst two layers are convolutional with 32 and 64 ﬁlters of size [5 , 5] , and use ReLU non-linearities.
A 2 × 2 max pooling layer follows after each convolutional layer.
Dimensionality reduction only takes place in the pooling layer; the stride for pooling is two and padding is used in the convolutional layers to keep the dimension.
Two fully-connected layers with 1024 and 10 hidden units are used to produce the classiﬁcation result.
ConVGG is based on the VGG16 network (Simonyan & Zisserman, 2014).
We modiﬁed the last fully connected layer to have 10 output dimensions for our experiments on CIFAR-10.
We didn’t use batch normalization for the variance reduction experiment since it introduces extra stochasticity.
The architectures used for the LSTM experiments are described in Section 4.2.
The hyperparameters used for the language modelling experiments are provided in Appendix D.
C V ARIANCE REDUCTION EXPERIMENT DETAILS
Given a network architecture, we compute the empirical stochastic gradient update variance as follows.
We start with a moderately pre-trained model, such as a network with 85% training accuracy on MNIST.
Without updating the parameters, we obtain the gradients of all the weights by performing a feed-forward pass, that includes sampling (cid:100) ∆ W , R , and S , followed by backpropagation.
The gradient variance of each weight is computed by repeating this procedure 200 times in the experiments.
Let (cid:93) Var lj denote the estimate of the gradient variance of weight j in layer l . We compute the gradient variance as follows:
[Equation]
where g ilj is the gradient received by weight j in layer l .
We estimate the variance of the gradients in layer l by averaging the variances of the weights in that layer, ˜ V = 1 | J | (cid:80) j (cid:93) Var lj .
In order to compute a conﬁdence interval on the gradient variance estimate, we repeat the above procedure 50 times, yielding a sequence of average variance estimates, (cid:102) V 1 , . . .
, (cid:102) V 50 .
We compute the 90% conﬁdence intervals of the variance estimates with a t-test, for Fig. 1. For ConVGG, multiple GPUs were needed to run the experiment with large mini-batch sizes (such as 4096 and 8192 ).
In such cases, it is computationally efﬁcient to generate independent weight perturbations on different GPUs. However, since our aim was to understand the effects of variance reduction independent of implementation, we shared the base perturbation among all GPUs to produce the plot shown in Fig.
We show in Appendix E that ﬂipout yields lower variance even when we sample independent perturbations on different GPUs. For the LSTM variance reduction experiments, we used the two-layer LSTM described in Section 4.2, trained for 3 epochs on the word-level Penn Treebank dataset. For Fig.
1, we split large
D LSTM R EGULARIZATION E XPERIMENT D ETAILS
Long Short-Term Memory networks (LSTMs) are deﬁned by the following equations: [Equation] [Equation] [Equation]
[Equation]
where i t , f t , and o t are the input, forget, and output gates, respectively, g t is the candidate update, and ◦ denotes elementwise multiplication.
Na¨ıve application of dropout on the hidden state of an
LSTM is not effective, because it leads to signiﬁcant memory loss over long sequences.
Several approaches have been proposed to regularize the recurrent connections, based on applying dropout to speciﬁc terms in the LSTM equations.
Semeniuta et al.
(2016) propose to drop the cell update vector, with a dropout mask d t sampled either per-step or per-sequence: c t =
f t ◦ c t − 1 + i t ◦ ( d t ◦ g t ) .
Gal & Ghahramani (2016) apply dropout to the input and hidden state at each time step, x t ◦ d x and h t − 1 ◦ d h , with dropout masks
d x and d h sampled once per sequence (and repeated in each time step).
Krueger et al.
(2016) propose to zone out units rather than dropping them; the hidden state and cell values are either stochastically updated or maintain their previous value: c t =
d ct ◦ c t − 1 + (1 − d ct ) ◦
( f t ◦ c t − 1 + i t ◦ g t ) and h t =
d ht ◦ h t − 1 + (1 − d ht ) ◦ ( o t ◦ tanh( f t ◦ c t −
+ i t ◦ g t )) , with zoneout masks
d ht and d ct sampled per step.
D. H YPERPARAMETER D ETAILS
For the word-level models (used in Table 3), we used the following hyperparameters:
• For Gal & Ghahramani (2016), we used variational dropout with the parameters given in their paper: 0.35 dropout probability on inputs and outputs, 0.2 hidden state dropout, and 0.2 embedding dropout.
• For Semeniuta et al. , we used 0.1 embedding dropout, 0.5 dropout on inputs and outputs, and 0.3 dropout on cell updates, with per-step mask sampling.
• For Krueger et al. , we used 0.1 embedding dropout, 0.5 dropout on inputs and outputs, and cell and hidden state zoneout probabilities of 0.25 and 0.025, respectively.
• For WD (Merity et al., 2017), we used the parameters given in their paper: 0.1 embedding dropout, 0.4 dropout probability on inputs and outputs, and 0.3 dropout probability on the output between layers (the same masks are used for each step of a sequence).
We use 0. probability for DropConnect applied to the hidden-to-hidden weight matrices.
• For WD+Flipout, we used the same parameters as Merity et al. (2017), given above, but we regularized the hidden-to-hidden weight matrices with the variant of ﬂipout described in Section 2.1, which implements DropConnect with probability 0.5.
For the character-level models (used in Table 2), we used orthogonal initialization for all the LSTM weight matrices, and did not use input or output dropout.
The input characters were represented as one-hot vectors. We used the following hyperparameters for each model:
• For recurrent dropout (Semeniuta et al., 2016), we used 0.25 dropout probability on the cell state, and per-step mask sampling.
• For Zoneout (Krueger et al., 2016), we used 0.5 and 0.05 for the cell and hidden state zoneout probabilities, respectively.
• For the variational LSTM (Gal & Ghahramani, 2016), we used 0.25 recurrent dropout.
• For the ﬂipout and shared perturbation LSTMs, we sampled Gaussian noise with σ = 1 for the hidden-to-hidden weight matrix.
E A DDITIONAL E XPERIMENTS E. V ARIANCE REDUCTION
As discussed in Appendix B, training on multiple GPUs naturally induces independent noise for each sub-batch.
Fig. 3 shows that ﬂipout still achieves lower variance than shared perturbations in such cases.
When estimating the variance with mini-batch size 8192, running on four GPUs naturally induces four independent noise samples, for each sub-batch of size 2048; this yields lower variance than using a single noise sample. Similarly, for mini-batch size 4096, two independent noise samples are generated on separate GPUs.
[Figure]
E. L ARGE B ATCH T RAINING WITH F LIPOUT
Fig.
4 shows the training and test error for the large mini-batch experiments described in Section 4.3.
For both FC and ConvLe networks, we used the Adam optimizer with learning rate 0 .
We downscaled the KL term by a factor of 10 to achieve higher accuracy.
While Fig. 2a shows that ﬂipout converges faster than shared perturbations, Fig. 4 shows that ﬂipout has the same generalization ability as shared perturbations (the faster convergence doesn’t result in overﬁtting).
[Figure]
[Figure]
[Figure]
Fig.
5 shows that the computational cost of cpuES increases as the model size increases, while
FlipES scales better because it runs on the GPU
[Figure] [Figure] [Figure]
FlipES cpuES 0.12s
[Figure]
.51s [Figure] 0.13s [Figure]
[Figure] 18s 5.10s 1.86s 38.0s # Filters FlipES cpuES
[Figure] [Figure] 16s [Figure]
5.48s 46s
[Figure] [Figure] 12s 67s [Figure]
11.77s 132s
The variance reduction offered by ﬂipout allows us to use DropConnect (Wan et al., 2013) efﬁciently in a large mini-batch setting.
Here, we use ﬂipout to implement DropConnect as described in Section 2.1, and use it to regularize an LSTM word-level language model.
We used the LSTM architecture proposed by Merity et al.
(2017), which has 400-dimensional word embedddings and three layers with hidden dimension 1150.
Following Merity et al.
(2017), we tied the weights of the embedding layer and the decoder layer.
Merity et al.
(2017) use DropConnect to regularize the hidden-to-hidden weight matrices, with a single mask shared for all examples in a batch.
We used ﬂipout to achieve a different DropConnect mask per example.
We applied WD+Flipout to both the hidden-to-hidden (h2h) and input-to-hidden (i2h) weight matrices, and compared to the model from Merity et al.
(2017), which we call WD (for weight-dropped LSTM), with DropConnect applied to both h2h and i2h.
Both models use embedding dropout 0.1, output dropout 0.4, and have DropConnect probability 0.5 for the i2h and h2h weights.
Both models were trained using Adam with learning rate 0.001.
Fig.
6 compares the variance of the gradients of the ﬁrst-layer hidden-to-hidden weights between
WD and WD+Flipout, and shows that ﬂipout achieves signiﬁcant variance reduction for mini-batch sizes larger than 256.
Fig.
7 shows the training curves of both models with batch size 8192.
We see that WD+Flipout converges faster than WD, and achieves a lower training perplexity, showcasing the optimization beneﬁts of ﬂipout in large mini-batch settings. [Figure] [Figure]