Original: A R ESIZABLE M INI - BATCH G RADIENT D ESCENT BASED ON A M ULTI -A RMED B ANDIT
Revised: A R ESIZABLE M INI - BATCH G RADIENT D ESCENT BASED ON A M ULTI -A RMED B ANDIT

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Determining the appropriate batch size for mini-batch gradient descent is always time consuming as it often relies on grid search.
Revised: Determining the appropriate batch size for mini-batch gradient descent is always time consuming as it often relies on grid search.

Original: This paper considers a resizable mini-batch gradient descent (RMGD) algorithm based on a multi-armed bandit that achievesequivalent performance  to that of best ﬁxed batch-size.
Revised: This paper considers a resizable mini-batch gradient descent (RMGD) algorithm based on a multi-armed bandit that achieves performance equivalent to that of best ﬁxed batch-size.

Original: At each epoch, the RMGD samples a batch size according to  certain probability distribution of a batch being successful in reducing the loss function.
Revised: At each epoch, the RMGD samples a batch size according to a certain probability distribution proportional to a batch being successful in reducing the loss function.

Original: Sampling from this probability provides a mechanism for exploring different batch size and exploiting batch sizes with history of success.
Revised: Sampling from this probability provides a mechanism for exploring different batch size and exploiting batch sizes with history of success.

Original: After obtaining the validation loss at each epoch with the sampled batch size, the probability distribution is updated to incorporate the effectiveness of the sampled batch size.
Revised: After obtaining the validation loss at each epoch with the sampled batch size, the probability distribution is updated to incorporate the effectiveness of the sampled batch size.

Original: Experimental results show that the RMGD achieves performance better than the best performing single batch size.
Revised: Experimental results show that the RMGD achieves performance better than the best performing single batch size.

Original:  Furthermore, it, obviously, attains this performance in a shorter amount of time than grid search.It is surprising that the RMGD achieves better performance than grid search.
Revised: It is surprising that the RMGD achieves better performance than grid search. Furthermore, it attains this performance in a shorter amount of time than grid search.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Gradient descent (GD) is a common optimization algorithm for ﬁnding the minimum of the expected loss.
Revised: Gradient descent (GD) is a common optimization algorithm for ﬁnding the minimum of the expected loss.

Original: It takes iterative steps proportional to the negative gradient of the loss function at each iteration.
Revised: It takes iterative steps proportional to the negative gradient of the loss function at each iteration.

Original: It is based on the observation that if the multi-variable loss functions f ( w ) is differentiable at point w , then f ( w ) decreases fastest in the direction of the negative gradient of f at w , i.e., −∇ f ( w ) .
Revised: It is based on the observation that if the multi-variable loss functions f ( w ) is differentiable at point w , then f ( w ) decreases fastest in the direction of the negative gradient of f at w , i.e., −∇ f ( w ) .

Original: The model parameters are updated iteratively in GD as follows:
Revised: The model parameters are updated iteratively in GD as follows:

Original: [Equation]
Revised: [Equation]

Original: where w t , g t , and η t are the model parameters, gradients of f with respect to w , and learning rate at time t respectively.
Revised: where w t , g t , and η t are the model parameters, gradients of f with respect to w , and learning rate at time t respectively.

Original: For small enough η t , f ( w t ) ≥ f ( w t +1 ) and ultimately the sequence of w t will move down toward a local minimum.
Revised: For small enough η t , f ( w t ) ≥ f ( w t +1 ) and ultimately the sequence of w t will move down toward a local minimum.

Original: For a convex loss function, GD is guaranteed to converge to a global minimum with an appropriate learning rate.
Revised: For a convex loss function, GD is guaranteed to converge to a global minimum with an appropriate learning rate.

Original: There are various issues to consider in gradient-based optimization.
Revised: There are various issues to consider in gradient-based optimization.

Original: First, GD can be extremely slow and impractical for large dataset: gradients of all the data have to be evaluated for each iteration.
Revised: First, GD can be extremely slow and impractical for large dataset: gradients of all the data have to be evaluated for each iteration.

Original: With larger data size, the convergence rate, the computational cost and memory become critical, and special care is required to minimize these factors.
Revised: With larger data size, the convergence rate, the computational cost and memory become critical, and special care is required to minimize these factors.

Original: Second, for non-convex function which is often encountered in deep learning, GD can get stuck in a local minimum without the hope of escaping.
Revised: Second, for non-convex function which is often encountered in deep learning, GD can get stuck in a local minimum without the hope of escaping.

Original: Third, stochastic gradient descent (SGD), which is based on the gradient of a single training sample, has large gradient variance, and it requires a large number of iterations.
Revised: Third, stochastic gradient descent (SGD), which is based on the gradient of a single training sample, has large gradient variance, and it requires a large number of iterations.

Original: This ultimately translates to slow convergence.
Revised: This ultimately translates to slow convergence.

Original: Mini-batch gradient descent (MGD), which is based on the gradient over a small batch of training data, trades off between the robustness of SGD and the stability of GD.
Revised: Mini-batch gradient descent (MGD), which is based on the gradient over a small batch of training data, trades off between the robustness of SGD and the stability of GD.

Original: There are three advantages for using MGD over GD and SGD: 1) The batching allows both the efﬁciency of memory usage and implementations; 2)
Revised: There are three advantages for using MGD over GD and SGD: 1) The batching allows both the efﬁciency of memory usage and implementations; 2)

Original: The model update frequency is higher than GD which allows for a more robust convergence avoiding local minimum; 3) MGD requires less iteration per epoch and provides a more stable update than SGD.
Revised: The model update frequency is higher than GD which allows for a more robust convergence avoiding local minimum; 3) MGD requires less iteration per epoch and provides a more stable update than SGD.

Original: For these reasons, MGD has been a popular algorithm for machine learning.
Revised: For these reasons, MGD has been a popular algorithm for machine learning.

Original: However, selecting an appropriate batch size is difﬁcult.
Revised: However, selecting an appropriate batch size is difﬁcult.

Original: Various studies suggest that there is a close link between performance and batch size used in MGD Breuel (2015);
Revised: Various studies suggest that there is a close link between performance and batch size used in MGD Breuel (2015);

Original: Keskar et al.
Revised: Keskar et al.

Original: ; Wilson & Martinez (2003).
Revised: ; Wilson & Martinez (2003).

Original: There are various guidelines for selecting a batch size but have not been completely practical Bengio (2012).
Revised: There are various guidelines for selecting a batch size but have not been completely practical Bengio (2012).

Original: Grid search is a popular method but it comes at the expense of search time.
Revised: Grid search is a popular method but it comes at the expense of search time.

Original: There are a small number of adaptive MGD algorithms to replace grid search Byrd et al.
Revised: There are a small number of adaptive MGD algorithms to replace grid search Byrd et al.

Original: ; De et al. (2016);
Revised: ; De et al. (2016);

Original: Friedlander & Schmidt (2012).
Revised: Friedlander & Schmidt (2012).

Original: These algorithms increase the batch size gradually according to their own criterion.
Revised: These algorithms increase the batch size gradually according to their own criterion.

Original: However, these algorithms are based on convex loss function and hard to be applied to deep learning.
Revised: However, these algorithms are based on convex loss function and hard to be applied to deep learning.

Original: For non-convex optimization, it is difﬁcult to determine the optimal batch size for best performance.
Revised: For non-convex optimization, it is difﬁcult to determine the optimal batch size for best performance.

Original: This paper considers a resizable mini-batch gradient descent (RMGD) algorithm based on a multi-armed bandit for achieving best performance in grid search by selecting an appropriate batch size at each epoch with a probability deﬁned as a function of its previous success/failure.
Revised: This paper considers a resizable mini-batch gradient descent (RMGD) algorithm based on a multi-armed bandit for achieving best performance in grid search by selecting an appropriate batch size at each epoch with a probability deﬁned as a function of its previous success/failure.

Original: At each epoch, RMGD samples a batch size from its probability distribution, then uses the selected batch size for mini-batch gradient descent.
Revised: At each epoch, RMGD samples a batch size from its probability distribution, then uses the selected batch size for mini-batch gradient descent.

Original: After obtaining the validation loss at each epoch, the probability distribution is updated to incorporate the effectiveness of the sampled batch size.
Revised: After obtaining the validation loss at each epoch, the probability distribution is updated to incorporate the effectiveness of the sampled batch size.

Original: The beneﬁt of RMGD is that it avoids the need for cumbersome grid search to achieve best performance and that it is simple enough to apply to any optimization algorithm using MGD.
Revised: The beneﬁt of RMGD is that it avoids the need for cumbersome grid search to achieve best performance and that it is simple enough to apply to any optimization algorithm using MGD.

Original: The detailed algorithm of RMGD are described in Section 4, and experimental results are presented in Section 5.
Revised: The detailed algorithm of RMGD are described in Section 4, and experimental results are presented in Section 5.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: There are only a few published results on the topic of batch size.
Revised: There are only a few published results on the topic of batch size.

Original: It was empirically shown that
Revised: It was empirically shown that

Original: SGD converged faster than GD on a large speech recognition database Wilson & Martinez (2003).
Revised: SGD converged faster than GD on a large speech recognition database Wilson & Martinez (2003).

Original: It was determined that the range of learning rate resulting in low test errors was considerably getting smaller as the batch size increased on convolutional neural networks and that small batch size yielded the best test error, while large batch size could not yield comparable low error rate Breuel (2015).
Revised: It was determined that the range of learning rate resulting in low test errors was considerably getting smaller as the batch size increased on convolutional neural networks and that small batch size yielded the best test error, while large batch size could not yield comparable low error rate Breuel (2015).

Original: It was observed that larger batch size are more liable to converge to a sharp local minimum thus leading to poor generalization Keskar et al.
Revised: It was observed that larger batch size are more liable to converge to a sharp local minimum thus leading to poor generalization Keskar et al.

Original: It was found that the learning rate and the batch size controlled the trade-off between the depth and width of the minima in MGD Jastrzkebski et al.
Revised: It was found that the learning rate and the batch size controlled the trade-off between the depth and width of the minima in MGD Jastrzkebski et al.

Original: A small number of adaptive MGD algorithms have been proposed.
Revised: A small number of adaptive MGD algorithms have been proposed.

Original: Byrd et al.
Revised: Byrd et al.

Original: (2012) introduced a methodology for using varying sample size in MGD.
Revised: (2012) introduced a methodology for using varying sample size in MGD.

Original: A relatively small batch size is chosen at the start, then the algorithm chooses a larger batch size when the optimization step does not produce improvement in the target objective function.
Revised: A relatively small batch size is chosen at the start, then the algorithm chooses a larger batch size when the optimization step does not produce improvement in the target objective function.

Original: They assumed that using a small batch size allowed rapid progress in the early stages, while a larger batch size yielded high accuracy.
Revised: They assumed that using a small batch size allowed rapid progress in the early stages, while a larger batch size yielded high accuracy.

Original: However, this assumption did not corresponded with later researches that reported the degradation of performance with large batch size Breuel (2015); Keskar et al. (2016); Mishkin et al.
Revised: However, this assumption did not corresponded with later researches that reported the degradation of performance with large batch size Breuel (2015); Keskar et al. (2016); Mishkin et al.

Original: Another similar adaptive algorithm, which increases the batch size gradually as the iteration proceeded, was done by
Revised: Another similar adaptive algorithm, which increases the batch size gradually as the iteration proceeded, was done by

Original: Friedlander & Schmidt (2012).
Revised: Friedlander & Schmidt (2012).

Original: The algorithm uses relatively few samples to approximate the gradient, and gradually increase the number of samples with a constant learning rate.
Revised: The algorithm uses relatively few samples to approximate the gradient, and gradually increase the number of samples with a constant learning rate.

Original: It was observed that increasing the batch size is more effective than decaying the learning rate for reducing the number of iterations Smith et al.
Revised: It was observed that increasing the batch size is more effective than decaying the learning rate for reducing the number of iterations Smith et al.

Original: However, these increasing batch size algorithms lack ﬂexibility since it is unidirectional.
Revised: However, these increasing batch size algorithms lack ﬂexibility since it is unidirectional.

Original: Balles et al.
Revised: Balles et al.

Original: (2017) proposed a dynamic batch size adaptation algorithm.
Revised: (2017) proposed a dynamic batch size adaptation algorithm.

Original: It estimates the variance of the stochastic gradients and adapts the batch size to decrease the variance.
Revised: It estimates the variance of the stochastic gradients and adapts the batch size to decrease the variance.

Original: However, this algorithm needs to ﬁnd the gradient variance and its computation depends on the number of model parameters.
Revised: However, this algorithm needs to ﬁnd the gradient variance and its computation depends on the number of model parameters.

Original: Batch size can also be considered as a hyperparameter, and there have been some proposals based on bandit-based hyperparameter (but not batch size) optimization which maybe applicable for determining the best ﬁxed batch size.
Revised: Batch size can also be considered as a hyperparameter, and there have been some proposals based on bandit-based hyperparameter (but not batch size) optimization which maybe applicable for determining the best ﬁxed batch size.

Original: Jamieson & Talwalkar (2016) introduced a successive halving algorithm.
Revised: Jamieson & Talwalkar (2016) introduced a successive halving algorithm.

Original: This algorithm uniformly allocates a budget to a set of hyperparameter conﬁgurations, evaluates the performance of all conﬁgurations, and throws out the worst half until one conﬁguration remains.
Revised: This algorithm uniformly allocates a budget to a set of hyperparameter conﬁgurations, evaluates the performance of all conﬁgurations, and throws out the worst half until one conﬁguration remains.

Original: Li et al.
Revised: Li et al.

Original: (2017) introduced a novel bandit-based hyperparameter optimization algorithm referred as HYPERBAND.
Revised: (2017) introduced a novel bandit-based hyperparameter optimization algorithm referred as HYPERBAND.

Original: This algorithm considers the optimization problem as a resource allocation problem.
Revised: This algorithm considers the optimization problem as a resource allocation problem.

Original: The two algorithms mentioned above are not adaptive, and for searching a small hyperparameter space, the two algorithms will not be very effective.
Revised: The two algorithms mentioned above are not adaptive, and for searching a small hyperparameter space, the two algorithms will not be very effective.

Original: The experimental results in this paper show that adaptive MGD tends to perform better than ﬁxed MGD.
Revised: The experimental results in this paper show that adaptive MGD tends to perform better than ﬁxed MGD.

Original:  S ETUP
Revised:  S ETUP

Original: Let B = { b k } Kk =1 be the set of possible batch size and π = { π k } Kk =1 be the probability distribution of batch size where b k , π k , and K are the k th batch size, the probability of b k to be selected, and number of batch sizes respectively.
Revised: Let B = { b k } Kk =1 be the set of possible batch size and π = { π k } Kk =1 be the probability distribution of batch size where b k , π k , and K are the k th batch size, the probability of b k to be selected, and number of batch sizes respectively.

Original: This paper considers algorithm for multi-armed bandit over B according to Algorithm 1.
Revised: This paper considers algorithm for multi-armed bandit over B according to Algorithm 1.

Original: Let w τ ∈ W be the model parameters at epoch τ , and ˜ w t be the temporal parameters at sub iteration t .
Revised: Let w τ ∈ W be the model parameters at epoch τ , and ˜ w t be the temporal parameters at sub iteration t .

Original: Let J : W → R be the training loss function and let g = ∇ J ( w ) be the gradients of training loss function with respect to the model parameters.
Revised: Let J : W → R be the training loss function and let g = ∇ J ( w ) be the gradients of training loss function with respect to the model parameters.

Original: η τ is the learning rate at epoch τ .
Revised: η τ is the learning rate at epoch τ .

Original: Let (cid:96) :
Revised: Let (cid:96) :

Original: W → R be the validation loss function, and y k ∈ { 0 , 1 } be the cost of choosing the batch size b k .
Revised: W → R be the validation loss function, and y k ∈ { 0 , 1 } be the cost of choosing the batch size b k .

Original: In here, y k = 

Revised: In here, y k =

Original: if the validation loss decreases by the selected batch size b k (well-updating) and y k
Revised: if the validation loss decreases by the selected batch size b k (well-updating) and y k

Original: = 1 otherwise (misupdating).
Revised: = 1 otherwise (misupdating).

Original: The aim of the algorithm is to have low misupdating.
Revised: The aim of the algorithm is to have low misupdating.

Original: For the cost function y k , graduated losses such as hinge loss and percentage of nonnegative changes in validation loss can be variations of 0-1 loss.
Revised: For the cost function y k , graduated losses such as hinge loss and percentage of nonnegative changes in validation loss can be variations of 0-1 loss.

Original: However, there are no differences in regret bound among them in this setting and it is experimentally conﬁrmed that there are little performance gaps among them.
Revised: However, there are no differences in regret bound among them in this setting and it is experimentally conﬁrmed that there are little performance gaps among them.

Original: Therefore, this paper introduces the 0-1 loss, which is simple and basic.
Revised: Therefore, this paper introduces the 0-1 loss, which is simple and basic.

Original:  R ESIZABLE M INI - BATCH G RADIENT D ESCENT
Revised:  R ESIZABLE M INI - BATCH G RADIENT D ESCENT

Original: The resizable mini-batch gradient descent (RMGD) sets the batch sizes as multi arms, and at each epoch it samples one of the batch sizes from probability distribution.
Revised: The resizable mini-batch gradient descent (RMGD) sets the batch sizes as multi arms, and at each epoch it samples one of the batch sizes from probability distribution.

Original: Then, it suffers a cost of selecting this batch size.
Revised: Then, it suffers a cost of selecting this batch size.

Original: Using the cost, probability distribution is updated.
Revised: Using the cost, probability distribution is updated.

Original: 4. A LGORITHMS
Revised: 4. A LGORITHMS

Original: The overall framework of the RMGD algorithm is shown in Figure 1.
Revised: The overall framework of the RMGD algorithm is shown in Figure 1.

Original: The RMGD consists of two components: batch size selector and parameter optimizer.
Revised: The RMGD consists of two components: batch size selector and parameter optimizer.

Original: The selector samples a batch size from probability distribution and updates the distribution.
Revised: The selector samples a batch size from probability distribution and updates the distribution.

Original: The optimizer is usual mini-batch gradient.
Revised: The optimizer is usual mini-batch gradient.

Original: Selector samples a batch size b k τ ∈ B from the probability distribution π τ at each epoch τ where k τ is selected index.
Revised: Selector samples a batch size b k τ ∈ B from the probability distribution π τ at each epoch τ where k τ is selected index.

Original: Here b k is associated with probability π k .
Revised: Here b k is associated with probability π k .

Original: The selected batch size b k τ is applied to optimizer for MGD at each epoch, and the selector gets cost y k τ from optimizer.
Revised: The selected batch size b k τ is applied to optimizer for MGD at each epoch, and the selector gets cost y k τ from optimizer.

Original: Then, the selector
Revised: Then, the selector

Original: Input :
Revised: Input :

Original: B = { b k } Kk =1 : Set of batch sizes π 0 = { 1 /K, . . .
Revised: B = { b k } Kk =1 : Set of batch sizes π 0 = { 1 /K, . . .

Original: , 1 /K } : Prior probability distribution
Revised: , 1 /K } : Prior probability distribution

Original: Procedure: 1: Initialize model parameters w2: for epoch τ = 0 , 1 , 2 , . . .
Revised: Procedure: 1: Initialize model parameters w2: for epoch τ = 0 , 1 , 2 , . . .

Original: Select batch size b k τ ∈ B from π τ 4:
Revised: Select batch size b k τ ∈ B from π τ 4:

Original: Set temporal parameters ˜ w 0 =
Revised: Set temporal parameters ˜ w 0 =

Original: w τ 5: for t = 0 , 1 , . .
Revised: w τ 5: for t = 0 , 1 , . .

Original: , T − 1 where T = (cid:100) m/b k τ (cid:101) 6:
Revised: , T − 1 where T = (cid:100) m/b k τ (cid:101) 6:

Original: Compute gradient g t = ∇ J ( ˜ w t ) 7:
Revised: Compute gradient g t = ∇ J ( ˜ w t ) 7:

Original: Update ˜ w t +1 =
Revised: Update ˜ w t +1 =

Original: ˜ w t − η τ g t 8: end for 9:
Revised: ˜ w t − η τ g t 8: end for 9:

Original: Update w τ
Revised: Update w τ

Original: +

Revised: +

Original: =
Revised: =

Original: ˜ w T 10:
Revised: ˜ w T 10:

Original: Observe validation loss (cid:96) ( w τ +1 ) 11: if (cid:96) ( w τ +1 ) <
Revised: Observe validation loss (cid:96) ( w τ +1 ) 11: if (cid:96) ( w τ +1 ) <

Original: (cid:96) ( w τ ) 12:
Revised: (cid:96) ( w τ ) 12:

Original: Get cost y k τ =13: else 14:
Revised: Get cost y k τ =13: else 14:

Original: Get cost y k τ =15: end if 16: for i = 1 , 2 , . .
Revised: Get cost y k τ =15: end if 16: for i = 1 , 2 , . .

Original: , K 17: if i = k τ 18:
Revised: , K 17: if i = k τ 18:

Original: Set temporal probability ˜ π i =
Revised: Set temporal probability ˜ π i =

Original: π iτ e − βy
Revised: π iτ e − βy

Original: kτ /π iτ 19: else 20:
Revised: kτ /π iτ 19: else 20:

Original: Set temporal probability ˜ π i = π iτ 21: end if 22: end for 23:
Revised: Set temporal probability ˜ π i = π iτ 21: end if 22: end for 23:

Original: Update ∀ i ∈
Revised: Update ∀ i ∈

Original: [ K ] , π iτ +

Revised: [ K ] , π iτ +

Original: =
Revised: =

Original: ˜
Revised: ˜

Original: π
Revised: π

Original: i / (cid:80) j ˜ π j 24: end for updates probabilities by randomized weighted majority, for i = k τ , ˜ π i = π iτ e − βy
Revised: i / (cid:80) j ˜ π j 24: end for updates probabilities by randomized weighted majority, for i = k τ , ˜ π i = π iτ e − βy

Original: kτ /π iτ for i (cid:54) =
Revised: kτ /π iτ for i (cid:54) =

Original: k τ , ˜ π i = π iτ ∀
Revised: k τ , ˜ π i = π iτ ∀

Original: i, π iτ +

Revised: i, π iτ +

Original: =
Revised: =

Original: ˜
Revised: ˜

Original: π
Revised: π

Original: i / (cid:88)
Revised: i / (cid:88)

Original: j ˜ π j where β ∈ (0 , 1) is positive hyperparameter.
Revised: j ˜ π j where β ∈ (0 , 1) is positive hyperparameter.

Original: When τ = 0 , π τ = { 1 /K, . . .
Revised: When τ = 0 , π τ = { 1 /K, . . .

Original: , 1 /K } .
Revised: , 1 /K } .

Original: Optimizer updates the model parameters w .
Revised: Optimizer updates the model parameters w .

Original: For each epoch, temporal parameters ˜ w 0 is set to w τ , and MGD iterates T = (cid:100) m/b k τ (cid:101) 1 times using the selected batch size b k τ where m is the total number of training samples:
Revised: For each epoch, temporal parameters ˜ w 0 is set to w τ , and MGD iterates T = (cid:100) m/b k τ (cid:101) 1 times using the selected batch size b k τ where m is the total number of training samples:

Original: [Equation]
Revised: [Equation]

Original: After T iterations at epoch τ , the model parameters is updated as w τ +1 =
Revised: After T iterations at epoch τ , the model parameters is updated as w τ +1 =

Original: ˜ w T .
Revised: ˜ w T .

Original: Then, the optimizer obtains validation loss (cid:96) , and outputs cost as follows:
Revised: Then, the optimizer obtains validation loss (cid:96) , and outputs cost as follows:

Original: [Equation]
Revised: [Equation]

Original: The RMGD samples an appropriate batch size from a probability distribution at each epoch.
Revised: The RMGD samples an appropriate batch size from a probability distribution at each epoch.

Original: This probability distribution encourages exploration of different batch size and then later exploits batch size with history of success, which means decreasing validation loss.
Revised: This probability distribution encourages exploration of different batch size and then later exploits batch size with history of success, which means decreasing validation loss.

Original: Figure 2 shows an example of training progress of RMGD.
Revised: Figure 2 shows an example of training progress of RMGD.

Original: The ﬁgure represents the probability distribution with respect to epoch.
Revised: The ﬁgure represents the probability distribution with respect to epoch.

Original: The white dot represents the selected batch size at each epoch.
Revised: The white dot represents the selected batch size at each epoch.

Original: In the early stage of training, commonly, all batch sizes tend to decrease validation loss: π is uniform.
Revised: In the early stage of training, commonly, all batch sizes tend to decrease validation loss: π is uniform.

Original: Thus, all batch size have equal probability of being sampled ( exploration ).
Revised: Thus, all batch size have equal probability of being sampled ( exploration ).

Original: In the later stages of training, the probability distribution varies based on success and failure.
Revised: In the later stages of training, the probability distribution varies based on success and failure.

Original: Thus, better performing batch size gets higher probability to be sampled ( exploitation ).
Revised: Thus, better performing batch size gets higher probability to be sampled ( exploitation ).

Original: In this case, 256 is the best performing batch size.
Revised: In this case, 256 is the best performing batch size.

Original: 4. R EGRET B OUND
Revised: 4. R EGRET B OUND

Original: The regret bound of the RMGD follows the regret bound derived in Shalev-Shwartz et al.
Revised: The regret bound of the RMGD follows the regret bound derived in Shalev-Shwartz et al.

Original: The goal of this algorithm is to have low regret for not selecting the best performing batch size such that
Revised: The goal of this algorithm is to have low regret for not selecting the best performing batch size such that

Original: [Equation]
Revised: [Equation]

Original: where the expectation is over the algorithm’s randomness of batch size selection and the second term on the right-hand side is the cumulative sum of the cost by the best ﬁxed batch size which minimizes the cumulative sum of the cost.
Revised: where the expectation is over the algorithm’s randomness of batch size selection and the second term on the right-hand side is the cumulative sum of the cost by the best ﬁxed batch size which minimizes the cumulative sum of the cost.

Original: The regret of the RMGD is bounded,
Revised: The regret of the RMGD is bounded,

Original: [Equation]
Revised: [Equation]

Original: In particular, setting β = (cid:112) log( K ) / ( K T ) , the regret is bounded by 2 (cid:112) K log( K ) T , which is sublinear with T .
Revised: In particular, setting β = (cid:112) log( K ) / ( K T ) , the regret is bounded by 2 (cid:112) K log( K ) T , which is sublinear with T .

Original: The detailed derivation of regret bound is described in the appendix A.
Revised: The detailed derivation of regret bound is described in the appendix A.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: This section describes various experimental results on MNIST, CIFAR10, and CIFAR100 dataset.
Revised: This section describes various experimental results on MNIST, CIFAR10, and CIFAR100 dataset.

Original: In the experiments, simple convolutional neural networks (CNN) is used for MNIST and ‘All-CNNC’
Revised: In the experiments, simple convolutional neural networks (CNN) is used for MNIST and ‘All-CNNC’

Original: Springenberg et al.
Revised: Springenberg et al.

Original: (2014) is used for CIFAR10 and CIFAR100.
Revised: (2014) is used for CIFAR10 and CIFAR100.

Original: The details of the dataset and experimental settings are presented in the appendix B.
Revised: The details of the dataset and experimental settings are presented in the appendix B.

Original: 5. MNIST DATASET
Revised: 5. MNIST DATASET

Original: The validity of the RMGD was assessed by performing image classiﬁcation on the MNIST dataset using AdamOptimizer and AdagradOptimizer as optimizer.
Revised: The validity of the RMGD was assessed by performing image classiﬁcation on the MNIST dataset using AdamOptimizer and AdagradOptimizer as optimizer.

Original: The experiments were repeatedtimes for each algorithm and each optimizer, then the results were analyzed for signiﬁcance.
Revised: The experiments were repeatedtimes for each algorithm and each optimizer, then the results were analyzed for signiﬁcance.

Original: Figure 3 shows the probability distribution and the selected batch size with respect to epoch during training for the RMGD.
Revised: Figure 3 shows the probability distribution and the selected batch size with respect to epoch during training for the RMGD.

Original: The white dot represents the batch size selected at each epoch.
Revised: The white dot represents the batch size selected at each epoch.

Original: The top ﬁgure is the case that small batch size (32) performs better.
Revised: The top ﬁgure is the case that small batch size (32) performs better.

Original: After epoch 50, batch size 32 gets high probability and is selected more than others.
Revised: After epoch 50, batch size 32 gets high probability and is selected more than others.

Original: It means that batch size 32 has less misupdating in this case.
Revised: It means that batch size 32 has less misupdating in this case.

Original: The gradually increasing batch size algorithm may not perform well in this case.
Revised: The gradually increasing batch size algorithm may not perform well in this case.

Original: The middle ﬁgure is the case that large batch size (512) performs better.
Revised: The middle ﬁgure is the case that large batch size (512) performs better.

Original: After epoch 60, batch size 512 gets high probability and selected more than others.
Revised: After epoch 60, batch size 512 gets high probability and selected more than others.

Original: The bottom ﬁgure shows that the best performing batch size varies with epoch.
Revised: The bottom ﬁgure shows that the best performing batch size varies with epoch.

Original: During epoch from 40 to 55, batch size of 256 performs best, and best performing batch size switches to 128 during epoch from 60 to 70, then better performing batch size backs toafter epoch 80.
Revised: During epoch from 40 to 55, batch size of 256 performs best, and best performing batch size switches to 128 during epoch from 60 to 70, then better performing batch size backs toafter epoch 80.

Original: In the results, any batch size can be a successful batch size in the later stages without any particular order.
Revised: In the results, any batch size can be a successful batch size in the later stages without any particular order.

Original: The RMGD is more ﬂexible for such situation than the MGD or directional adaptive MGD such as gradually increasing batch size algorithm.
Revised: The RMGD is more ﬂexible for such situation than the MGD or directional adaptive MGD such as gradually increasing batch size algorithm.

Original: Figure 4 shows the test accuracy of each algorithm.
Revised: Figure 4 shows the test accuracy of each algorithm.

Original: The error bar is standard error.
Revised: The error bar is standard error.

Original: The number in parenthesis next to
Revised: The number in parenthesis next to

Original: Among the MGD algorithms, relatively small batch sizes (16 - 64) lead to higher performance than large batch sizes (128 - 512) and batch size 64 achieves the best performance in grid search.
Revised: Among the MGD algorithms, relatively small batch sizes (16 - 64) lead to higher performance than large batch sizes (128 - 512) and batch size 64 achieves the best performance in grid search.

Original: These
Revised: These

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: 283 ±
Revised: 283 ±

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: results correspond with other studies Breuel (2015); Keskar et al.
Revised: results correspond with other studies Breuel (2015); Keskar et al.

Original: ; Mishkin et al.
Revised: ; Mishkin et al.

Original: Most RMGD settings outperform all ﬁxed MGD algorithms in both case.
Revised: Most RMGD settings outperform all ﬁxed MGD algorithms in both case.

Original: Although the performance of RMGD is not signiﬁcantly increased compared to the best MGD, the purpose of this algorithm is not to improve performance, but to ensure that the best performance is achieved without performing a grid search on the batch size.
Revised: Although the performance of RMGD is not signiﬁcantly increased compared to the best MGD, the purpose of this algorithm is not to improve performance, but to ensure that the best performance is achieved without performing a grid search on the batch size.

Original: Rather, the improved performance of the RMGD is a surprising result.
Revised: Rather, the improved performance of the RMGD is a surprising result.

Original: Therefore, the RMGD is said to be valid.
Revised: Therefore, the RMGD is said to be valid.

Original: There are little performance gap among RMGD settings.
Revised: There are little performance gap among RMGD settings.

Original: The ’sub’ setting outperforms the ’basic’ setting in left ﬁgure, but the opposite result is shown in right ﬁgure.
Revised: The ’sub’ setting outperforms the ’basic’ setting in left ﬁgure, but the opposite result is shown in right ﬁgure.

Original: Therefore, there is no clear tendency of performance change depending on the size of the batch size set.
Revised: Therefore, there is no clear tendency of performance change depending on the size of the batch size set.

Original: Table 1 and 2 present iterations and real time for training, mean, maximum, and minimum of test accuracies for each algorithm with AdamOptimizer and AdagradOptimizer respectively.
Revised: Table 1 and 2 present iterations and real time for training, mean, maximum, and minimum of test accuracies for each algorithm with AdamOptimizer and AdagradOptimizer respectively.

Original: The MGD (total) is the summation of the iterations and real time of whole MGDs for grid search.
Revised: The MGD (total) is the summation of the iterations and real time of whole MGDs for grid search.

Original: The RMGD (basic) outperforms best performing MGD and is, also, faster than best performing MGD.
Revised: The RMGD (basic) outperforms best performing MGD and is, also, faster than best performing MGD.

Original: Furthermore, it is 8 times faster than grid search in both cases.
Revised: Furthermore, it is 8 times faster than grid search in both cases.

Original: In the results, the RMGD is effective regardless of the optimizer.
Revised: In the results, the RMGD is effective regardless of the optimizer.

Original: 5. CIFAR10 AND CIFAR100 DATASET
Revised: 5. CIFAR10 AND CIFAR100 DATASET

Original: The CIFAR10 and CIFAR100 dataset were, also, used to assess effectiveness of the RMGD.
Revised: The CIFAR10 and CIFAR100 dataset were, also, used to assess effectiveness of the RMGD.

Original: The experiments were repeated 25 times and 10 times, respectively.
Revised: The experiments were repeated 25 times and 10 times, respectively.

Original: In these experiments, all images
Revised: In these experiments, all images

Original: [Table] [Table]
Revised: [Table] [Table]

Original: are whitened and contrast normalized before being input to the network.
Revised: are whitened and contrast normalized before being input to the network.

Original: Figure 5 shows the test accuracy for each algorithm.
Revised: Figure 5 shows the test accuracy for each algorithm.

Original: The left ﬁgure represents the test accuracy on CIFAR10.
Revised: The left ﬁgure represents the test accuracy on CIFAR10.

Original: In contrast to the MNIST results, relatively large batch sizes (128 - 256) lead to higher performance than small batch sizes (16 - 64) and batch size 256 achieves the best performance in grid search.
Revised: In contrast to the MNIST results, relatively large batch sizes (128 - 256) lead to higher performance than small batch sizes (16 - 64) and batch size 256 achieves the best performance in grid search.

Original: The right ﬁgure represents the test accuracy on CIFAR100 and batch size 128 achieves the best performance in grid search.
Revised: The right ﬁgure represents the test accuracy on CIFAR100 and batch size 128 achieves the best performance in grid search.

Original: The results on MNIST, CIFAR10 and CIFAR100 indicate that it is difﬁcult to know which batch size is optimal before performing a grid search.
Revised: The results on MNIST, CIFAR10 and CIFAR100 indicate that it is difﬁcult to know which batch size is optimal before performing a grid search.

Original: Meanwhile, all RMGD settings have again exceeded the best performance of ﬁxed MGD.
Revised: Meanwhile, all RMGD settings have again exceeded the best performance of ﬁxed MGD.

Original: There are no signiﬁcant performance gaps among
Revised: There are no signiﬁcant performance gaps among

Original: RMGD settings, so there is no need to worry about choosing appropriate batch size set or selecting cost function.
Revised: RMGD settings, so there is no need to worry about choosing appropriate batch size set or selecting cost function.

Original: Table 3 and 4 present the detailed results on CIFAR10 and CIFAR100 dataset.
Revised: Table 3 and 4 present the detailed results on CIFAR10 and CIFAR100 dataset.

Original: The RMGD (basic) is a little slower than single best performing MGD (256 for CIFAR10 and 128 for CIFAR100), however, it was much faster than grid search -about 4.6 times on CIFAR10 and 5.0 times on CIFAR faster.
Revised: The RMGD (basic) is a little slower than single best performing MGD (256 for CIFAR10 and 128 for CIFAR100), however, it was much faster than grid search -about 4.6 times on CIFAR10 and 5.0 times on CIFAR faster.

Original: Therefore, this results, also, show the effectiveness of the RMGD.
Revised: Therefore, this results, also, show the effectiveness of the RMGD.

Original: It is difﬁcult to compare the RMGD with other adaptive batch size algorithm, e.g. coupling adaptive batch sizes (CABS)
Revised: It is difﬁcult to compare the RMGD with other adaptive batch size algorithm, e.g. coupling adaptive batch sizes (CABS)

Original: Balles et al.
Revised: Balles et al.

Original: (2017), directly since the underlying goals are different.
Revised: (2017), directly since the underlying goals are different.

Original: While the goal of the RMGD is to reduce the validation loss in terms of generalization performance, the CABS determines the batch size to balance between the gradient variance and computation.
Revised: While the goal of the RMGD is to reduce the validation loss in terms of generalization performance, the CABS determines the batch size to balance between the gradient variance and computation.

Original: However, it is obvious that the RMGD is simpler and easier to implement than any other adaptive algorithm cited in this paper, and comparing the test accuracy between the RMGD and the CABS on the CIFAR and CIFAR100 using the same experimental settings with ’All-CNN-C’ shows that the performance of the RMGD is higher than that of the CABS (CIFAR10: 87.862 ± 0.142, CIFAR100: 60.
Revised: However, it is obvious that the RMGD is simpler and easier to implement than any other adaptive algorithm cited in this paper, and comparing the test accuracy between the RMGD and the CABS on the CIFAR and CIFAR100 using the same experimental settings with ’All-CNN-C’ shows that the performance of the RMGD is higher than that of the CABS (CIFAR10: 87.862 ± 0.142, CIFAR100: 60.

Original: ± 0.421).
Revised: ± 0.421).

Original: And again, the purpose of this algorithm is not to outperform other algorithms, but to guarantee that the best performance is reached without grid search.
Revised: And again, the purpose of this algorithm is not to outperform other algorithms, but to guarantee that the best performance is reached without grid search.

Original: C ONCLUSION
Revised: C ONCLUSION

Original: Selecting batch size affects the model quality and training efﬁciency, and determining the appropriate batch size is time consuming and requires considerable resources as it often relies on grid search.
Revised: Selecting batch size affects the model quality and training efﬁciency, and determining the appropriate batch size is time consuming and requires considerable resources as it often relies on grid search.

Original: The focus of this paper is to design a simple robust algorithm that is theoretically sound and applicable in many situations.
Revised: The focus of this paper is to design a simple robust algorithm that is theoretically sound and applicable in many situations.

Original: This paper considers a resizable mini-batch gradient descent (RMGD) algorithm based on a multiarmed bandit that achieves equivalent performance to that of best ﬁxed batch-size.
Revised: This paper considers a resizable mini-batch gradient descent (RMGD) algorithm based on a multiarmed bandit that achieves equivalent performance to that of best ﬁxed batch-size.

Original: At each epoch, the
Revised: At each epoch, the

Original: RMGD samples a batch size according to certain probability distribution of a batch being successful in reducing the loss function.
Revised: RMGD samples a batch size according to certain probability distribution of a batch being successful in reducing the loss function.

Original: Sampling from this probability provides a mechanism for exploring different batch size and exploiting batch sizes with history of success.
Revised: Sampling from this probability provides a mechanism for exploring different batch size and exploiting batch sizes with history of success.

Original: After obtaining the validation loss at each epoch with the sampled batch size, the probability distribution is updated to incorporate the effectiveness of the sampled batch size.
Revised: After obtaining the validation loss at each epoch with the sampled batch size, the probability distribution is updated to incorporate the effectiveness of the sampled batch size.

Original: The goal of this algorithm is not to achieve state-of-the-art accuracy but rather to select appropriate batch size which leads low misupdating and performs better.
Revised: The goal of this algorithm is not to achieve state-of-the-art accuracy but rather to select appropriate batch size which leads low misupdating and performs better.

Original: The RMGD essentially assists the learning process to explore the possible domain of the batch size and exploit successful batch size.
Revised: The RMGD essentially assists the learning process to explore the possible domain of the batch size and exploit successful batch size.

Original: The beneﬁt of RMGD is that it avoids the need for cumbersome grid search to achieve best performance and that it is simple enough to apply to various ﬁeld of machine learning including deep learning using MGD.
Revised: The beneﬁt of RMGD is that it avoids the need for cumbersome grid search to achieve best performance and that it is simple enough to apply to various ﬁeld of machine learning including deep learning using MGD.

Original: Experimental results show that the RMGD achieves the best grid search performance on various dataset, networks, and optimizers.
Revised: Experimental results show that the RMGD achieves the best grid search performance on various dataset, networks, and optimizers.

Original: Furthermore, it, obviously, attains this performance in a shorter amount of time than the grid search.
Revised: Furthermore, it, obviously, attains this performance in a shorter amount of time than the grid search.

Original: Also, there is no need to worry about which batch size set or cost function to choose when setting RMGD.
Revised: Also, there is no need to worry about which batch size set or cost function to choose when setting RMGD.

Original: In conclusion, the RMGD is effective and ﬂexible
Revised: In conclusion, the RMGD is effective and ﬂexible

Original: mini-batch gradient descent algorithm.
Revised: mini-batch gradient descent algorithm.

Original: A PPENDIX A R EGRET B OUND
Revised: A PPENDIX A R EGRET B OUND

Original: In the RMGD algorithm, there are K batch sizes as multi arms with the probability distribution π ∈ S , and at each epoch the algorithm should select one of the batch sizes b k τ .
Revised: In the RMGD algorithm, there are K batch sizes as multi arms with the probability distribution π ∈ S , and at each epoch the algorithm should select one of the batch sizes b k τ .

Original: Then it receives a cost of selecting this arm, y k τ τ ∈ { 0 , 1 } by testing the validation loss (cid:96) .
Revised: Then it receives a cost of selecting this arm, y k τ τ ∈ { 0 , 1 } by testing the validation loss (cid:96) .

Original: The vector y τ ∈ { 0 , 1 } K represents the selecting cost for each batch size.
Revised: The vector y τ ∈ { 0 , 1 } K represents the selecting cost for each batch size.

Original: The goal of this algorithm is to have low regret for not selecting the best performing batch size.
Revised: The goal of this algorithm is to have low regret for not selecting the best performing batch size.

Original: [Equation]
Revised: [Equation]

Original: where the expectation is over the algorithm’s randomness of batch size selection.
Revised: where the expectation is over the algorithm’s randomness of batch size selection.

Original: Let S be the probability simplex, the selecting loss functions be f τ ( π ) =
Revised: Let S be the probability simplex, the selecting loss functions be f τ ( π ) =

Original: (cid:104) π , y τ (cid:105) 2 and R :
Revised: (cid:104) π , y τ (cid:105) 2 and R :

Original: S → R be a regularization function that is often chosen to be strongly convex with respect to some norm || · || .
Revised: S → R be a regularization function that is often chosen to be strongly convex with respect to some norm || · || .

Original: The algorithm select a batch size with probability P [ b k τ ] =
Revised: The algorithm select a batch size with probability P [ b k τ ] =

Original: π k τ τ and therefore f τ ( π τ ) is the expected cost of the selected batch size at epoch τ .
Revised: π k τ τ and therefore f τ ( π τ ) is the expected cost of the selected batch size at epoch τ .

Original: The gradient of the selecting loss function is y τ .
Revised: The gradient of the selecting loss function is y τ .

Original: However, only one element y k τ τ is known at each epoch.
Revised: However, only one element y k τ τ is known at each epoch.

Original: To estimate gradient, random vector z τ is deﬁned as follows:
Revised: To estimate gradient, random vector z τ is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: otherwise and expectation of z τ satisﬁes,
Revised: otherwise and expectation of z τ satisﬁes,

Original: [Equation]
Revised: [Equation]

Original: The most natural learning rule is to set the probability distribution which has minimal cost on all past epochs.
Revised: The most natural learning rule is to set the probability distribution which has minimal cost on all past epochs.

Original: It is referred to as Follow-the-Regularized-Leader (FTRL) in online learning:
Revised: It is referred to as Follow-the-Regularized-Leader (FTRL) in online learning:

Original: [Equation]
Revised: [Equation]

Original: where β is positive hyperparameter.
Revised: where β is positive hyperparameter.

Original: The FTRL has a problem that it requires solving an optimization problem at each epoch.
Revised: The FTRL has a problem that it requires solving an optimization problem at each epoch.

Original: To solve this problem, Online Mirror Descent (OMD) is applied.
Revised: To solve this problem, Online Mirror Descent (OMD) is applied.

Original: The OMD computes the current probability distribution iteratively based on a gradient update rule and the previous probability distribution and lies in the update being carried out in a ’dual’ space, deﬁned by regularizer.
Revised: The OMD computes the current probability distribution iteratively based on a gradient update rule and the previous probability distribution and lies in the update being carried out in a ’dual’ space, deﬁned by regularizer.

Original: This follows from considering ∇ R as a mapping from R K onto itself.
Revised: This follows from considering ∇ R as a mapping from R K onto itself.

Original: The OMD relies on Bregman divergence.
Revised: The OMD relies on Bregman divergence.

Original: The Bregman divergence between π and ˜ π with respect to the regularizer
Revised: The Bregman divergence between π and ˜ π with respect to the regularizer

Original: R is given as:
Revised: R is given as:

Original: [Equation]
Revised: [Equation]

Original: and a Bregman projection of ˜ π onto simplex S :
Revised: and a Bregman projection of ˜ π onto simplex S :

Original: [Equation]
Revised: [Equation]

Original: Then the probability distribution is updated by the OMD as follows:
Revised: Then the probability distribution is updated by the OMD as follows:

Original: [Equation]
Revised: [Equation]

Original: In general, if R is strongly convex, then ∇ R becomes a bijective mapping, thus ˜ π τ +1 can be recovered by the inverse gradient mapping ( ∇ R ) − 1 .
Revised: In general, if R is strongly convex, then ∇ R becomes a bijective mapping, thus ˜ π τ +1 can be recovered by the inverse gradient mapping ( ∇ R ) − 1 .

Original: Given that R is strongly convex, the OMD and
Revised: Given that R is strongly convex, the OMD and

Original: FTRL produce equivalent predictions:
Revised: FTRL produce equivalent predictions:

Original: [Equation]
Revised: [Equation]

Original: by the Lemma 1 in Hazan & Kale (2010).
Revised: by the Lemma 1 in Hazan & Kale (2010).

Original: It makes sense to use the negative entropic regularization for R in RMGD setting:
Revised: It makes sense to use the negative entropic regularization for R in RMGD setting:

Original: [Equation]
Revised: [Equation]

Original: Then, ∇ R ( π ) i = log( π i ) + 1 .
Revised: Then, ∇ R ( π ) i = log( π i ) + 1 .

Original: From the OMD, ˜ π τ +1 is updated as follows:
Revised: From the OMD, ˜ π τ +1 is updated as follows:

Original: [Equation]
Revised: [Equation]

Original: The Bregman projection with respect to the negative entropy function becomes scaling by the (cid:96) 1norm.
Revised: The Bregman projection with respect to the negative entropy function becomes scaling by the (cid:96) 1norm.

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: The probability distribution π τ is updated by the rule of the normalized exponentiated gradient (normalized-EG) algorithm described in Algorithm 1.
Revised: The probability distribution π τ is updated by the rule of the normalized exponentiated gradient (normalized-EG) algorithm described in Algorithm 1.

Original: Also, the selecting loss function is linear and it is satisﬁed that ∀ τ, i we have βz iτ ≥ 0 .
Revised: Also, the selecting loss function is linear and it is satisﬁed that ∀ τ, i we have βz iτ ≥ 0 .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: by the Theorem 2.22 in Shalev-Shwartz et al.
Revised: by the Theorem 2.22 in Shalev-Shwartz et al.

Original: (2012), where π ∗ ∈ S is a ﬁxed vector which minimizes the cumulative selecting loss,
Revised: (2012), where π ∗ ∈ S is a ﬁxed vector which minimizes the cumulative selecting loss,

Original: [Equation]
Revised: [Equation]

Original: Since f τ is convex and z τ is estimated gradients for all τ ,
Revised: Since f τ is convex and z τ is estimated gradients for all τ ,

Original: [Equation]
Revised: [Equation]

Original: by the Theorem 4.1 in Shalev-Shwartz et al.
Revised: by the Theorem 4.1 in Shalev-Shwartz et al.

Original: The last term is bounded as follows:
Revised: The last term is bounded as follows:

Original: [Equation]
Revised: [Equation]

Original: Therefore, the regret of the RMGD is bounded,
Revised: Therefore, the regret of the RMGD is bounded,

Original: [Equation]
Revised: [Equation]

Original: In particular, setting β = (cid:112) log( K ) / ( K T ) , the regret is bounded by 2 (cid:112) K log( K ) T , which is sublinear with T .
Revised: In particular, setting β = (cid:112) log( K ) / ( K T ) , the regret is bounded by 2 (cid:112) K log( K ) T , which is sublinear with T .

Original: B E XPERIMENTAL S ETTINGS D ATASET
Revised: B E XPERIMENTAL S ETTINGS D ATASET

Original: MNIST is a dataset of handwritten digits that is commonly used for image classiﬁcation.
Revised: MNIST is a dataset of handwritten digits that is commonly used for image classiﬁcation.

Original: Each sample is a black and white image and 28 × 28 in size.
Revised: Each sample is a black and white image and 28 × 28 in size.

Original: The MNIST is split into three parts: 55, samples for training, 5,000 samples for validation, and 10,000 samples for test.
Revised: The MNIST is split into three parts: 55, samples for training, 5,000 samples for validation, and 10,000 samples for test.

Original: CIFAR10 consists of 60,000 32 × 32 color images in 10 classes (airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck), with 6,000 images per class.
Revised: CIFAR10 consists of 60,000 32 × 32 color images in 10 classes (airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck), with 6,000 images per class.

Original: The CIFAR10 is split into three parts: 45,000 samples for training, 5,000 samples for validation, and 10,000 samples for test.
Revised: The CIFAR10 is split into three parts: 45,000 samples for training, 5,000 samples for validation, and 10,000 samples for test.

Original: CIFAR100 consists of 60,000 32 × 32 color images in 100 classes.
Revised: CIFAR100 consists of 60,000 32 × 32 color images in 100 classes.

Original: The CIFAR100 is split into three parts: 45,000 samples for training, 5,000 samples for validation, and 10,000 samples for test.
Revised: The CIFAR100 is split into three parts: 45,000 samples for training, 5,000 samples for validation, and 10,000 samples for test.

Original: S ETTINGS
Revised: S ETTINGS

Original: The simple CNN consists of two convolution layers with 5 × 5 ﬁlter and 1 × 1 stride, two max pooling layers with 2 × 2 kernel and 2 × 2 stride, single fully-connected layer, and softmax classiﬁer.
Revised: The simple CNN consists of two convolution layers with 5 × 5 ﬁlter and 1 × 1 stride, two max pooling layers with 2 × 2 kernel and 2 × 2 stride, single fully-connected layer, and softmax classiﬁer.

Original: Description of the ’All-CNN-C’ is provided in Table 5.
Revised: Description of the ’All-CNN-C’ is provided in Table 5.

Original: For MNIST, AdamOptimizer with η = 10 − 4 and AdagradOptimizer with η = 0 .
Revised: For MNIST, AdamOptimizer with η = 10 − 4 and AdagradOptimizer with η = 0 .

Original: 1 are used as optimizer.
Revised: 1 are used as optimizer.

Original: The basic batch size set B = { 16 , 32 , 64 , 128 , 256 , 512 } , subset of basic B − = { 16 , 64 , 256 } , and superset of basic
Revised: The basic batch size set B = { 16 , 32 , 64 , 128 , 256 , 512 } , subset of basic B − = { 16 , 64 , 256 } , and superset of basic

Original: B + = { 16 , 24 , 32 , 48 , 64 , 96 , 128 , 192 , 256 , 384 , 512 } .
Revised: B + = { 16 , 24 , 32 , 48 , 64 , 96 , 128 , 192 , 256 , 384 , 512 } .

Original: The model is trained for a total ofepochs.
Revised: The model is trained for a total ofepochs.

Original: For CIFAR10 and CIFAR100, MomentumOptimizer with ﬁxed momentum of 0.9 is used as optimizer.
Revised: For CIFAR10 and CIFAR100, MomentumOptimizer with ﬁxed momentum of 0.9 is used as optimizer.

Original: The learning rate η k is scaled up proportionately to the batch size ( η k = 0 .
Revised: The learning rate η k is scaled up proportionately to the batch size ( η k = 0 .

Original: 05 ∗ b k / 256 )
Revised: 05 ∗ b k / 256 )

Original: and decayed by a schedule S =
Revised: and decayed by a schedule S =

Original: [200 , 250 , 300] in which η k is multiplied by a ﬁxed multiplier of 0.1 after 200, 250, and 300 epochs respectively.
Revised: [200 , 250 , 300] in which η k is multiplied by a ﬁxed multiplier of 0.1 after 200, 250, and 300 epochs respectively.

Original: The model is trained for a total of 350 epochs.
Revised: The model is trained for a total of 350 epochs.

Original: Dropout is applied to the input image as well as after each convolution layer with stride 2.
Revised: Dropout is applied to the input image as well as after each convolution layer with stride 2.

Original: The dropout probabilities are 20% for dropping out inputs and 50% otherwise.
Revised: The dropout probabilities are 20% for dropping out inputs and 50% otherwise.

Original: The model is regularized with weight decay λ = 0 .
Revised: The model is regularized with weight decay λ = 0 .

Original: The basic batch size set B = { 16 , 32 , 62 , 128 , 256 } , subset of basic B − = { 16 , 64 , 256 } , and superset of basic B + = { 16 , 24 , 32 , 48 , 64 , 96 , 128 , 192 , 256 } .
Revised: The basic batch size set B = { 16 , 32 , 62 , 128 , 256 } , subset of basic B − = { 16 , 64 , 256 } , and superset of basic B + = { 16 , 24 , 32 , 48 , 64 , 96 , 128 , 192 , 256 } .

Original: For all experiments, rectiﬁed linear unit (ReLU) is used as activation function.
Revised: For all experiments, rectiﬁed linear unit (ReLU) is used as activation function.

Original: For RMGD, β is set to (cid:112) log(6) / (6 ∗ 100) ≈ 0 .
Revised: For RMGD, β is set to (cid:112) log(6) / (6 ∗ 100) ≈ 0 .

Original: 055 for MNIST and (cid:112) log(5) / (5 ∗ 350)
Revised: 055 for MNIST and (cid:112) log(5) / (5 ∗ 350)

Original: ≈ 0 .
Revised: ≈ 0 .

Original: for CIFAR10 and
Revised: for CIFAR10 and

Original: CIFAR100.
Revised: CIFAR100.

Original: The basic batch size selecting cost is 0-1 loss, hinge loss is max { 0 , (cid:96) τ − (cid:96) τ − 1 } , and ratio loss is max { 0 , ( (cid:96) τ − (cid:96) τ − 1 ) /(cid:96)
Revised: The basic batch size selecting cost is 0-1 loss, hinge loss is max { 0 , (cid:96) τ − (cid:96) τ − 1 } , and ratio loss is max { 0 , ( (cid:96) τ − (cid:96) τ − 1 ) /(cid:96)

Original: τ − 1 } .
Revised: τ − 1 } .

Original: [Table]
Revised: [Table]