O N B ATCH A DAPTIVE T RAINING FOR D EEP L EARNING : L OWER L OSS AND L ARGER S TEP S IZE
Anonymous authors
Paper under double-blind review
A BSTRACT
Mini-batch gradient descent and its variants are commonly used in deep learning.
The principle of mini-batch gradient descent is to use noisy gradient calculated on a batch to estimate the real gradient, thus balancing the computation cost per iteration and the uncertainty of noisy gradient.
However, its batch size is a ﬁxed hyper parameter requiring manual setting before training the neural network.
Yin et al.
(2017) proposed a batch adaptive stochastic gradient descent (BA-SGD) that can dynamically choose a proper batch size as learning proceeds.
We extend the BA-SGD to momentum algorithm, and evaluate both the BA-SGD and the batch adaptive momentum (BA-Momentum) on two deep learning tasks from natural language processing to image classiﬁcation.
Experiments conﬁrm that batch adaptive methods can achieve a lower loss compared with mini-batch methods after scanning the same epochs of data.
Furthermore, our BA-Momentum is more robust against larger step sizes, in that it can dynamically enlarge the batch size to reduce the larger uncertainty brought by larger step sizes.
The code implementing batch adaptive framework is now open source, applicable to any gradient-based optimization problems.
I NTRODUCTION
Efﬁciency of training large neural networks becomes increasingly important as deep neural networks tend to have more parameters and require more training data to achieve the state-of-the-art performance on a wide variety of tasks (Goodfellow et al., 2015).
For training deep neural networks, stochastic gradient descent (SGD) (Robbins, 2007) and its variants, including momentum, which utilizes past updates with an exponential decay (Qian, 1999), and other methods that can adapt different learning rates for each dimension, such as ADAGRAD (Duchi et al., 2010), ADADELTA (Zeiler, 2012) and ADAM (Kingma & Ba, 2014), are commonly used.
SGD approximates the gradient by only using a single data instance in each iteration, which may lead to uncertainty of approximation.
This uncertainty can be reduced by adopting a batch of instances to do the approximation.
In mini-batch SGD, the batch size is a ﬁxed hyper parameter requiring manual setting before training the neural network.
Setting the batch size typically involves a tuning procedure in which the best batch size is chosen by a series of attempts.
Yin et al.
(2017) has developed a batch adaptive stochastic gradient descent (BA-SGD) that can dynamically choose a proper batch size as learning proceeds.
BA-SGD models the decrease of objective value as a Gaussian random walk game with rebound on the basis of Taylor extension and central limit theorem.
Its core idea is to only update the parameters when the ratio between the expected decrease of objective value and the current batch size is large enough, otherwise enlarge the batch size to better approximate the gradient.
It claimed that by smartly choosing the batch size, the BA-SGD not only conserves the fast convergence of SGD algorithm but also avoids too frequent model updates, and compared with mini-batch SGD, its objective value decreases more, after scanning the same amount of data.
However, the experiment in Yin et al.
(2017) was only conducted on some simple classiﬁcation tasks using fully connected neural network with one input layer, one output layer and two hidden layers.
What about the evaluation on some complex neural networks, such as convolutional neural network (CNN) and recurrent neural network (RNN)?
How well would the batch adaptive algorithm perform on other complicated tasks related to natural language processing and computer vision?
Furthermore, empirical studies reveal that SGD usually performs not so well in some deep and complex neural networks (Sutskever et al., 2013).
Can this batch adaptive framework be extended to other gradient based optimization algorithms except SGD?
Therefore, in this paper we extend the batch adaptive framework to momentum algorithm, and evaluate both the batch adaptive SGD (BA-SGD) and the batch adaptive momentum (BA-Momentum) on two deep learning tasks from natural language processing to image classiﬁcation.
These two tasks use RNN and CNN respectively, which cover most of the deep learning models.
In our experiments, we have the following observations.
First, for batch adaptive methods, their loss functions converge to lower values after scanning same epoches of data, compared with ﬁxedbatch-size methods.
Second, BA-Momentum is more robust against large step sizes by dynamically enlarging the batch size to counteract with the larger noise brought by larger step sizes.
Third, we observed a batch size boom , a concentrated period where the batch size frequently increases to larger values, in the training of BA-Momentum.
The batch size boom is of signiﬁcance in that it always appears at the point where mini-batch method starts to reach its lowest possible loss and it helps BA-Momentum keep dropping to even lower loss.
More details on these observations and their analysis can be found in Section 4.
The code implementing the batch adaptive framework using Theano (AlR) is now open source 1 , which is applicable to any gradient-based optimization problems.
This paper is organized as follows.
In Section 2, we brieﬂy introduce the batch adaptive framework proposed by Yin et al.
In Section 3, we extend the batch adaptive framework to momentum algorithm.
In Section 4, we demonstrate the performance of BA-M and BA-SGD on
Fashion-MNIST (Xiao et al., 2017) and relation extraction task, and then reveal the robustness of
BA-Momentum against large step sizes.
In Section 5, we discuss some efﬁciency issue concerned with implementing this batch adaptive framework, and also propose several promising applications based on this framework.
P REREQUISITES
In this section we brieﬂy summarize the batch adaptive stochastic gradient descent proposed by Yin et al.
2. N OTATIONS
We use X and Y to respectively denote the training data set and its random subset.
The vector of model parameters is denoted by (cid:126)θ and is subscripted by t to denote an iteration.
F is used to denote objective function, while f (cid:126)θ is the partial derivative of function F with model parameters (cid:126)θ (i.e. the gradient, computed over the whole data set).
2. B ATCH A DAPTIVE F RAMEWORK
Let (cid:126)ξ i denote the difference between the approximate gradient computed on an individual instance g i(cid:126)θ and the real gradient f (cid:126)θ , thus, ˆ f (cid:126)θ , the approximate gradient computed on a batch Y can be written as:
[Equation]
The term
[Equation]
| can be viewed as a random variable determined by the randomly sampled batch
Y .
On the basis of Central Limit Theorem (CLT), it should satisfy a multi-dimension normal distribution N ( 0 , Σ |Y| ) , where Σ is the covariance matrix of (cid:126)ξ j . f
(cid:126)θ is the real gradient, computed on the whole data set and treated as a constant given the parameters (cid:126) θ .
Thus we have ˆ f (cid:126)θ ∼ N ( f (cid:126)θ , Σ |Y| ) .
After modelling the estimation of gradient on a batch as a normally distributed random variable,
Yin et al.
(2017) uses ﬁrst-order Taylor extension to approximate the objective function F ( (cid:126)θ ) at any _Adaptive_Framework parameter conﬁguration.
The equation is shown below.
[Equation]
where the function h (cid:126)θ 0 ( (cid:126)θ ) is the remainder term satisfying lim
[Equation]
=
[Equation]
If SGD optimization algorithm is adopted to update the parameters, with Equation (1) and Equation (2), the decrease of objective value can be expressed as:
[Equation]
where η is the learning rate, and the noise term satisﬁes (cid:126)ε Y ∼ N [Equation] 0 , Σ |Y| ) , then (cid:126)ε T Y · f (cid:126)θ 0 can be viewed as a weighted sum of each dimension of vector (cid:126)ε Y , thus satisfying a one-dimension Gaussian distribution, i.e. (cid:126)ε T Y · f (cid:126)θ 0 ∼ N (0 ,
[Equation]
T(cid:126)θ
[Equation]
0 |Y| ) .
Then ∆ F ( (cid:126)θ 0 ) also satisﬁes a Gaussian distribution, i.e.
[Equation]
0 f (cid:126)θ 0 , [Equation] [Equation] In practice, they use the approximate gradient ˆ f (cid:126)θ 0 to calculate the mean and variance of ∆ F ( (cid:126) θ 0 ) .
For the covariance matrix of (cid:126)ε Y , Σ , its unbiased estimation ˆ Σ is:
[Equation]
After modelling the decrease of objective value as a normally distributed random variable, Yin et al.
(2017) abstract the process of objective value change as a random walk game with a Gaussian dice.
In the game, the objective value is regarded as a state, and the decrease of objective value, namely the transfer from the current state to the next state is determined by a Gaussian dice, of which the mean solely depend on the current state and the variance is controlled by the state and the batch to be chosen.
They deﬁne the domain of game state as a half closed set of real numbers [ S ∗ , + ∞ ) , where S ∗ is the minimum objective value that the learning can possibly achieve.
The game starts with a random state and the goal is to move as close as possible to S ∗ .
There are two ways of state transfer: one directly decreasing from S i to S j , another ﬁrst reaching minimum point S ∗ and then rebounding to S j .
Formally, for state s t , the moving step (cid:52) s t is generated by a Gaussian dice N ( µ t , σ 2 t m ) , where µ t = ˆ f T(cid:126)θ t ˆ f (cid:126)θ t , σ 2 t =
ˆ f T(cid:126)θ t ˆ Σ ˆ f (cid:126)θ t , m = |Y| , denoting the batch size.
The state transition equation is presented below.
[Equation]
Let p s t m ( (cid:52) s t ) denote the probability density function for a random moving step (cid:52) s t ∼ N (cid:16) µ t , σ 2 t m (cid:17) .
The expected value of next state can be expressed as follows.
[Equation]
To decide the best batch size, one should consider both the variance of estimated gradients and computation cost.
Large batch can reduce the variance and therefore make more accurate updates, whereas it also requires more computations.
Yin et al.
(2017) then deﬁne a utility function to ﬁnd a balance.
Maximizing the utility means achieving the largest expected decrease of loss per data instance.
[Equation]
where m ∗ is the best batch size for the ( t + 1) -th iteration.
For more speciﬁcations on the BA-SGD algorithm, see Yin et al.
B ATCH A DAPTIVE M OMENTUM
The main appeal of momentum is its ability to reduce oscillations and accelerate convergence (Goh, 2017).
The idea behind momentum is that it accelerates learning along dimensions where gradient continues pointing to the same direction, and slows down those where sign of gradient constantly changes (Zeiler, 2012).
Recent work on momentum, called YellowFin, an automatic tuner for both momentum and learning rate, helps momentum optimizer converge in even fewer iterations than
ADAM on large ResNet and LSTM models (Zhang et al., 2017).
Since it is both powerful and popular, we would like to also apply the batch adaptive framework to momentum optimizer.
3. M OMENTUM
Momentum utilizes past parameter updates with an exponential decay.
Its way of update is given by
[Equation]
where m t denotes momentum at the t -th iteration and ρ is the rate controlling the decay of the previous parameter updates.
Equation 10 can also be written in the following form.
[Equation]
3. D ERIVATION
Referring to Equation 3, we can analogically estimate the decrease of objective value in the following expression.
[Equation]
where (cid:126)ε τ =
[Equation]
j |Y τ | is the noise term which represents the difference between the real gradient and the estimated gradient calculated on a batch
Y τ chosen at the τ -th iteration.
Though the estimated gradient from the previous iterations (i.e. τ = 0 ...
t − 1 ) has noise, their batches which respectively determine their noise have already been selected, thus their noise is no longer a random variable but a constant.
However, for the t -th iteration, we have not decided which batch to be sampled, therefore (cid:126)ε t is indeed a random variable, and on the basis of CLT we know it is normally distributed, i.e. (cid:126)ε t ∼ N ( 0 , Σ t |Y t | ) .
Based on this and the fact that real gradients, f (cid:126)θ τ ( τ = 0 ...t ), are all constants for the t -th iteration, we then have the decrease of objective value, ∆ F t , satisfying a one-dimensional Gaussian distribution, which is also experimentally veriﬁed in Appendix B.
We need to calculate the mean and variance of ∆ F t , but we prefer not to record all the ˆ f (cid:126)θ τ from previous iterations.
Therefore, we construct a recurrence formula to avoid the trouble.
Let P t = (cid:80) tτ
=
ρ t − τ ( f (cid:126)θ τ + (cid:126)ε τ ) , then we have the following recurrence formula.
[Equation]
Thus the mean and variance of P t can be calculated in the following forms.
[Equation]
[Equation]
Now we have ∆ F t ∼ N ( − η ˆ f T(cid:126)θ t µ P t , η 2 ˆ f T(cid:126)θ t Σ P t ˆ f (cid:126)θ t ) .
In the Gaussian walk game with rebound illustrated in Section 2, the Gaussian dice here satisﬁes (cid:52) s t ∼ N (ˆ f T(cid:126)θ t µ P t , ˆ f T(cid:126)θ t Σ P t ˆ f (cid:126)θ t ) .
For simplicity, we let σ 2 t =
ˆ f T(cid:126)θ t ˆΣ t ˆ f (cid:126)θ t , µ t = ˆ f T(cid:126)θ t µ P t , m = |Y t | , thus (cid:52) s t ∼ N ( µ t , σ 2 t m ) .
Then the expected value of the next state shares the same expression with Equation 6, though the mean of (cid:52) s t is different.
For batch adaptive momentum algorithm, we also adopt the utility function in Equation 7, and the best batch size is the one that maximizes the utility function, calculated through Equation 8.
3. A LGORITHM
At the end of this section, we would like to summarize how the batch adaptive momentum algorithm is implemented by presenting the pseudo code below.
In the pseudo code, the M stands for the total budget, i.e. the total number of instances used for training, the m 0 means sampling step, the smallest batch increment.
The rest symbols have been introduced before.
When implementing the algorithm, one may note that it is time-consuming to calculate the covariance matrix, ˆΣ t .
We discuss a tradeoff in Appendix C.
1: procedure BA-M OMENTUM ( X , (cid:126)θ, η, ρ, M, m 0 ) 2: while M > 0 do 3:
Y t ← ∅ 4: repeat 5: random sample Z from X − Y t with |Z| = m6:
Y t ← Y t (cid:83) Z 7: calculate ˆ f (cid:126)θ t , ˆΣ t with Y t (cid:46) Equation (1) and (4) 8: calculate µ P t with µ P t − 1 and ˆ f (cid:126)θ (cid:46) Equation (14) 9: s t ← F ( (cid:126)θ |Y t ) , µ t ← ˆ f T(cid:126) θ t µ P t , σ t ← (cid:113) ˆ f T (cid:126)θ ˆΣ t ˆ f (cid:126)θ 10: m ∗ ← arg max m u ( m, s t ) where S ∗ =11: until |Y t | ≥ min { m ∗ , |X|} 12: m t = ρ m t − 1 + η ˆ f (cid:126)θ t 13: (cid:126)θ ← (cid:126)θ − m t 14:
M ← M − |Y t | 15: t = t +16: end while 17: return (cid:126)θ 18: end procedure
E XPERIMENT
In this section, we present the learning efﬁciency of different learning algorithms on two deep learning models.
One is the CNN for image classiﬁcation on the dataset of Fashion MNIST (Xiao et al., 2017), and the other is a more complex RNN model for relation extraction on ﬁnancial documents (Miwa & Bansal, 2016).
To evaluate our learning efﬁciency regardless of the size of the data set, we use epoch as the unit of scanned data and one epoch means all instances in the whole training set.
We calculate the training loss on the whole training set each time when the model has scanned one more epoch of data.
4. M ODEL AND D ATASET
Fashion MNIST is a data set of Zalando’s article images, which consists of 60000 training samples and 10000 test samples.
Each sample is a 28 × 28 grayscale image, associated with a label fromclasses.
We design a CNN model with 3 convolutional layers for this experiment.
Another model we use for relation extraction task is a bi-directional LSTM RNN (Gers, 2001).
To train this model, we use 3855 training instances.
A detailed description on the speciﬁc task and architecture of networks for the two experiments can be found in Appendix A.
4. L OWER L OSS WITH B UDGET L IMIT
In both experiments, 10 different optimization algorithms are used.
They are BA-Momentum, BASGD, mini-batch momentum with a ﬁxed size of 32, 64, 128, 256 respectively, and mini-batch SGD with a ﬁxed size of 32, 64, 128, 256 respectively.
For simplicity, we denote mini-batch momentum with a ﬁxed size of 32 as ”Mini-32-Momentum”.
The same rule applies to ”Mini-256-SGD” etc.
We choose mini-batch methods with different sizes ranging from 32 to 256 as baselines.
This is because with a budget limit, i.e. ﬁxed number of epochs used for training, the small batch method can have more but less accurate updates, while the large batch method can make fewer but more accurate updates.
More updates and accurate updates both can to some extend help the model achieve a lower loss, thus we use both small and large batch methods as baselines to make the comparative test with batch adaptive methods more convincing.
For BA-Momentum and BA-SGD, the smallest batch increment, m 0 , is 32, which means the batch size for each iteration in BA-Momentum and
BA-SGD is always a multiple 32.
The result of image classiﬁcation task on Fashion MNIST is plotted in Figure 1 and Figure 3a.
The observation shows that for ﬁve different momentum-based optimization algorithms, BA-Momentum achieves a much lower loss than the rest four methods after 100 epochs of training.
Furthermore,
BA-Momentum has fewer ﬂuctuations in later training stage.
For the rest four mini-batch methods, the smaller the batch size is, the lower their loss gets and the more they ﬂuctuate.
The same trend can be seen in SGD-based methods, where BA-SGD also achieves the lowest loss after a limited budget of 100 epochs.
Now we take a look at the batch size change per iteration, plotted in Figure 3a.
For both BAMomentum and BA-SGD, the batch size keeps almost constant at 32 in an early stage, while it more frequently increases to larger size in later training stage.
The tendency is especially evident for
BA-SGD.
The lines in Figure 3a for BA-SGD gets denser as iterations increase, indicating the batch size rises more frequently from 32 to larger sizes.
Also, the largest possible batch size for BA-SGD enlarges from 64 to almost 1000 as learning proceeds.
We display the result of relation extraction task in Figure 2 and Figure 3b.
BA-Momentum still achieves the lowest cost compared with other four mini-batch momentum methods.
However, BASGD almost shares the same curve with Mini-32-SGD, though ﬂuctuates a bit less in later stage.
The batch size change for BA-Momentum and BA-SGD on relation extraction task have the same trend observed in the image classiﬁcation result, though the batch sizes in this task change less frequently, as the sparser lines in Figure 3b suggest.
In terms of the two batch adaptive methods, BA-Momentum and BA-SGD, BA-Momentum shows a faster and lower training loss convergence than BA-SGD in both experiments.
Here we analyze why the BA-Momentum and BA-SGD can achieve the lowest loss on both tasks.
The batch adaptive methods adopt a smaller batch in an early stage, which allows them to have more iterations per epoch and thus might accelerate the decrease of loss, then they dynamically enlarge their batch size in later stage to reduce the noise of estimated gradients, because the noise has a larger impact on the accuracy of parameter update in later training stage.
Therefore the batch adaptive methods can conserve the fast decrease of Mini-32 methods in early stage and meanwhile keep decreasing rather than severely ﬂuctuate like Mini-32 methods do in later stage
[Figure]
[Figure]
4. R OBUSTNESS TO L ARGE S TEP S IZE
In the ﬁne tuning process of BA-Momentum
[Figure]
we ﬁnd that BA-Momentum is robust against large step sizes.
Speciﬁcally, under a certain range of learning rates, when we tune the learning rate to a higher value
[Figure]
BA-Momentum can still achieve the same or even lower loss while the performance of other mini-batch methods will be degraded, converging to higher loss and ﬂuctuating more.
For simplicity, we only choose Mini-32-Momentum as baseline, since it performs the best in the four momentum-based mini-batch methods.
We choose four different learning rates to test the robustness of BA-Momentum.
They are 0.005, 0.01, 0.02, and 0.05.
The result is displayed in Figure 4.
As shown in Figure 4a, when learning rate rises from 0.005 to 0.01 and then 0.02, BA-Momentum ends up with lower loss from 1.13e-4 to 1.69e-5 and then 5.77e6.
In contrast, Mini-32-Momentum ﬂuctuates more when the learning rate rises from 0.005 to 0.01, and it ends up with much higher loss when the learning rate increases from 0.01 to 0.02.
However, when the learning rate is tuned to 0.05, both BA-Momentum and Mini-32-Momentum suffer from a very high loss.
These observations conﬁrm that BA-Momentum is more robust against larger step sizes within a certain reasonable range.
[Figure]
We offer an explanation on the robustness of BA-Momentum.
As the density of lines in Figure 4b suggests, the higher the learning rate is given, the larger our batch size tends to become.
BAMomentum enlarges the batch size to counteract with the larger uncertainty of parameter updates brought by higher learning rate.
In this way, it successfully avoids the ﬂuctuations and meanwhile beneﬁts from the larger step size to converge faster
[Figure]
4. B ATCH S IZE B OOM
We ﬁnd an intriguing phenomenon when conducting experiments testing the robustness of BAMomentum.
In the process of learning, the batch size of BA-Momentum will experience a batch size boom , a concentrated period where the batch size frequently increases to larger values.
For example
[Figure]
the batch size boom in Figure 5a is from 80000 iterations to 110000 iterations, while the batch size boom in Figure 5b is from 60000 iterations to 80000 iterations.
In the process of learning, when a batch size boom appears, the curve of BA-Momentum starts to diverge with the curve of
Mini-32-Momentum.
This makes sense because using a larger batch can help BA-Momentum make more accurate updates and thus decrease to a lower loss.
Interestingly, the batch size boom always appears at the point in which Mini-32-Momentum reaches its lowest possible loss and after which it ﬂuctuates around that loss.
This can be observed in all three plots in Figure 5.
As learning rate increases, Mini-32-Momentum reaches its lowest loss earlier, and the batch size boom also appears earlier, helping BA-Momentum keep decreasing to lower loss.
[Figure]
C ONCLUSION AND D ISCUSSION
In this work we developed BA-Momentum algorithm, an extension of the BA-SGD proposed by Yin et al.
We also evaluate the two algorithms on natural language processing and image classiﬁcation tasks using RNN and CNN respectively.
The experiments show that both batch adaptive methods can achieve lower loss than mini-batch methods after scanning same epochs of data.
Furthermore, we also conﬁrm that within a certain range of step sizes, BA-Momentum is more robust against large step size compared with mini-batch methods.
In the experiments, we did not evaluate the decrease of training loss with respect to training time.
This is because, in the BA-SGD and BA-Momentum algorithm, we have to calculate the derivatives of the loss of each instance from a batch with respect to parameters, and then derive a covariance matrix through Equation 4 from the derivatives.
Computing derivatives by backpropagation is time consuming, and now we have to compute all the derivatives of every instance in a batch.
However, in mini-batch gradient descent, it is a common practice to calculate an average loss from a batch and then the derivative of this average loss, which requires less time.
A feasible approach to reduce the computation cost might be to modify the way Theano do the derivation for a batch of instances and return the square sum of the derivatives, which we plan to study in future work.
The batch adaptive framework can have many important applications.
It can be adapted to accelerate distributed deep learning.
For distributed deep learning, communication cost for synchronizing gradients and parameters among workers and parameter server is its well-known bottleneck (Li et al., 2014a;b; Wen et al., 2017).
A larger batch may help make more accurate updates, thus reducing the total number of iterations needed to converge, lowering the communication cost.
However, a larger batch also causes a higher computation cost per iteration.
In this update-costly environment, the batch adaptive framework may be modiﬁed to take both the computation and communication cost into consideration when deciding a proper batch size, which is worth further exploring.
Another application is that the batch adaptive framework may help remedy the generalization degradation of using large batch studied by Keskar et al.
They provided solid numeric evidence suggesting that using a larger batch will degrade the quality of the model, as measured by its ability to generalize.
They also studied the cause for this generalization drop and presented evidence supporting the view that large-batch methods tend to converge to sharp minimizers of the training and testing functions, which causes this generalization drop.
Several strategies to help large-batch methods eliminate this generalization drop was proposed in their work.
The most promising one is to warm-start with certain epochs of small-batch regime, and then use large batch for the rest of the training.
However, the number of epochs needed to warm start with small batch varies for different data sets, thus a batch adaptive method that can dynamically change the batch size against the characteristics of data is the key to solving this problem.
The batch adaptive framework sheds light on this issue.
Difﬁculty lies in how to identify a sharp minima accurately and efﬁciently in the process of learning and limit the batch size when encountering a sharp minima, which we plan to study in future work.
A A RCHITECTURE OF N ETWORKS
For the classiﬁcation task on Fashion MNIST, we use 3 convolutional layers, where each layer has 32, 64, 128 ﬁlters respectively, with MaxPool after each convolutional layer.
For all layers, we use
ReLU activations and Dropout (Srivastava et al., 2014).
The relation extraction task aims to extract quadruples in which the elements are correctly matched with each other.
The input training instance consists of a sequence representing a sentence from ﬁnancial documents, and all possible quadruples in this sequence.
For the model, a bi-directional
LSTM RNN is used.
There are two layers of bi-directional LSTM, and a softmax outputs the probability of each quadruple being correct.
B M ODEL V ERIFICATION
In our BA-Momentum method, we model the decrease of objective value as a random variable satisfying a one-dimensional Gaussian distribution in Equation 12.
This derivation is on the basis of ﬁrst order Taylor extension and central limit theorem, involving some approximation.
After modelling the decrease of objective value as normally distributed, we then estimated its mean and variance in Equation 14 and Equation 15.
Here we would like to use real data from Fashion-MNIST to verify that the decrease of objective value truly satisﬁes a Gaussian distribution and our estimation is close to its real mean and variance.
We compute the decrease of objective value on the whole data set after 10 iterations for 500 times with batches randomly sampled and size ﬁxed at 100.
The result is plotted in Figure 6.
The observation conﬁrms that the decrease of objective value distributes normally.
Its mean and standard deviation are -7.94253e-10 and 3.48414e-11 respectively, while its estimated mean and standard deviation are -8.58165e-10 and 4.58062e-12.
Therefore, the decrease of objective value in our BA-M algorithm satisﬁes a Gaussian distribution and our estimation for its mean and variance is close to the real ones.
−9. −9.
−8. −8.
[Figure]
C I MPLEMENTATION I SSUES
When calculating the covariance matrix in Equation 4, one should note that it takes a space of O ( d 2 ) with d denoting the dimension of the model parameter vector to store the covariance matrix.
The cost is quite high for complex models with a large amount of parameters.
Yin et al.
(2017) has proposed a practical tradeoff assuming that model parameters are independent of each other, then the covariance matrix becomes a diagonal one, greatly reducing the space and time cost.
Here we use a small data set from UCI machine learning repository (Lichman, 2013) to test whether the tradeoff will affect the performance of BA-Momentum.
We use two different BA-Momentum.
One calculates the exact covariance matrix, while the other calculates its estimated diagonal matrix.
The task is a classiﬁcation task with 3 classes andfeatures.
The training set is composed of around 60000 data instances.
After training for 14 epochs, we yield the following result.
The Figure 7a show the batch size change as learning proceeds for matrix method and diagonal method respectively.
The Figure 7b demonstrates different training loss per epoch for two different methods.
From the ﬁgures we can see that the trend of batch size change for two methods are almost the same, therefore they perform equally well on the training loss descent.
This indicates estimating the covariance matrix by its diagonal matrix will not cause degradation to the performance of BA-Momentum.
[Figure]