Original: C RITICAL B ATCH S IZE M INIMIZES S TOCHASTIC F
Revised: C RITICAL B ATCH S IZE M INIMIZES S TOCHASTIC F

Original: IRST
Revised: IRST

Original: O RDER
Revised: O RDER

Original: O RACLE C OMPLEXITY OF D EEP L EARNING O PTIMIZER USING H YPERPARAMETERS C LOSE TO O NE
Revised: O RACLE C OMPLEXITY OF D EEP L EARNING O PTIMIZER USING H YPERPARAMETERS C LOSE TO O NE

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Practical results have shown that deep learning optimizers using small constant learning rates, hyperparameters close to one, and large batch sizes can ﬁnd the model parameters of deep neural networks that minimize the loss functions.
Revised: Practical results have shown that deep learning optimizers using small constant learning rates, hyperparameters close to one, and large batch sizes can ﬁnd the model parameters of deep neural networks that minimize the loss functions.

Original: We ﬁrst show theoretical evidence that the momentum method (Momentum) and adaptive moment estimation (Adam) perform well in the sense that the upper bound of the theoretical performance measure is small with a small constant learning rate, hyperparameters close to one, and a large batch size.
Revised: We ﬁrst show theoretical evidence that the momentum method (Momentum) and adaptive moment estimation (Adam) perform well in the sense that the upper bound of the theoretical performance measure is small with a small constant learning rate, hyperparameters close to one, and a large batch size.

Original: Next, we show that there exists a batch size called the critical batch size minimizing the stochastic ﬁrst-order oracle (SFO) complexity, which is the stochastic gradient computation cost, and that SFO complexity increases once the batch size exceeds the critical batch size.
Revised: Next, we show that there exists a batch size called the critical batch size minimizing the stochastic ﬁrst-order oracle (SFO) complexity, which is the stochastic gradient computation cost, and that SFO complexity increases once the batch size exceeds the critical batch size.

Original: Finally, we provide numerical results that support our theoretical results.
Revised: Finally, we provide numerical results that support our theoretical results.

Original: That is, the numerical results indicate that Adam using a small constant learning rate, hyperparameters close to one, and the critical batch size minimizing
Revised: That is, the numerical results indicate that Adam using a small constant learning rate, hyperparameters close to one, and the critical batch size minimizing

Original: SFO complexity has faster convergence than Momentum and stochastic gradient descent (SGD).
Revised: SFO complexity has faster convergence than Momentum and stochastic gradient descent (SGD).

Original:  I NTRODUCTION 1. B ACKGROUND
Revised:  I NTRODUCTION 1. B ACKGROUND

Original: Useful deep learning optimizers have been proposed to ﬁnd the model parameters of the deep neural networks that minimize loss functions called the expected risk and empirical risk, such as stochastic gradient descent (SGD) (Robbins & Monro, 1951; Zinkevich, 2003; Nemirovski et al., 2009; Ghadimi & Lan, 2012; 2013), momentum methods (Polyak, 1964; Nesterov, 1983), and adaptive methods.
Revised: Useful deep learning optimizers have been proposed to ﬁnd the model parameters of the deep neural networks that minimize loss functions called the expected risk and empirical risk, such as stochastic gradient descent (SGD) (Robbins & Monro, 1951; Zinkevich, 2003; Nemirovski et al., 2009; Ghadimi & Lan, 2012; 2013), momentum methods (Polyak, 1964; Nesterov, 1983), and adaptive methods.

Original: The various adaptive methods include Adaptive Gradient (AdaGrad) (Duchi et al., 2011), Root Mean Square Propagation (RMSProp) (Tieleman & Hinton, 2012), Adaptive Moment Estimation (Adam) (Kingma & Ba, 2015), Adaptive Mean Square Gradient (AMSGrad) (Reddi et al., 2018), Yogi (Zaheer et al., 2018), Adam with decoupled weight decay (AdamW) (Loshchilov & Hutter, 2019), and AdaBelief (named for adapting stepsizes by the belief in observed gradients)
Revised: The various adaptive methods include Adaptive Gradient (AdaGrad) (Duchi et al., 2011), Root Mean Square Propagation (RMSProp) (Tieleman & Hinton, 2012), Adaptive Moment Estimation (Adam) (Kingma & Ba, 2015), Adaptive Mean Square Gradient (AMSGrad) (Reddi et al., 2018), Yogi (Zaheer et al., 2018), Adam with decoupled weight decay (AdamW) (Loshchilov & Hutter, 2019), and AdaBelief (named for adapting stepsizes by the belief in observed gradients)

Original: (Zhuang et al., 2020).
Revised: (Zhuang et al., 2020).

Original: Theoretical analyses of adaptive methods for nonconvex optimization were presented in (Zaheer et al., 2018;
Revised: Theoretical analyses of adaptive methods for nonconvex optimization were presented in (Zaheer et al., 2018;

Original: Zou et al., 2019; Chen et al., 2019;
Revised: Zou et al., 2019; Chen et al., 2019;

Original: Zhou et al., 2020; Zhuang et al., 2020; Chen et al., 2021) (see (Jain et al., 2018; Fehrman et al., 2020; Chen et al., 2020;
Revised: Zhou et al., 2020; Zhuang et al., 2020; Chen et al., 2021) (see (Jain et al., 2018; Fehrman et al., 2020; Chen et al., 2020;

Original: Scaman & Malherbe, 2020; Loizou et al., 2021) for convergence analyses of SGD).
Revised: Scaman & Malherbe, 2020; Loizou et al., 2021) for convergence analyses of SGD).

Original: A particularly interesting feature of adaptive methods is the use of hyperparameters, denoted by β 1 and β 2 , that can be set to inﬂuence the method performance P( K ) :
Revised: A particularly interesting feature of adaptive methods is the use of hyperparameters, denoted by β 1 and β 2 , that can be set to inﬂuence the method performance P( K ) :

Original: = 1 K P K k =1 E [ ∥∇ f ( θ k ) ∥ 2 ] , where ∇ f is the gradient of a loss function f : R d → R , ( θ k )
Revised: = 1 K P K k =1 E [ ∥∇ f ( θ k ) ∥ 2 ] , where ∇ f is the gradient of a loss function f : R d → R , ( θ k )

Original: Kk =1 is the sequence generated by an optimizer, and K is the number of steps.
Revised: Kk =1 is the sequence generated by an optimizer, and K is the number of steps.

Original: The previous results are summarized in Table 1 indicating that using β 1 and/or β 2 close to 0 makes the upper bound of P( K ) small.
Revised: The previous results are summarized in Table 1 indicating that using β 1 and/or β 2 close to 0 makes the upper bound of P( K ) small.

Original: Meanwhile, practical results for adaptive methods were presented in (Kingma & Ba, 2015;
Revised: Meanwhile, practical results for adaptive methods were presented in (Kingma & Ba, 2015;

Original: Reddi et al., 2018;
Revised: Reddi et al., 2018;

Original: Zaheer et al., 2018; Zou et al., 2019; Chen et al., 2019; Zhuang et al., 2020;
Revised: Zaheer et al., 2018; Zou et al., 2019; Chen et al., 2019; Zhuang et al., 2020;

Original: Chen et al., 2021).
Revised: Chen et al., 2021).

Original: These studies have shown that using, for example, β 1 ∈ { 0 . 9 , 0 . 99 } and β 2 ∈ { 0 . 99 , 0 . 999 } provides superior performance for training deep neural networks.
Revised: These studies have shown that using, for example, β 1 ∈ { 0 . 9 , 0 . 99 } and β 2 ∈ { 0 . 99 , 0 . 999 } provides superior performance for training deep neural networks.

Original: The practically useful β 1 and β 2 are each close to 1 , whereas in contrast, the theoretical results (Table 1) show that using β 1 and/or β 2 close to 0 makes the upper bounds of the performance measures small.
Revised: The practically useful β 1 and β 2 are each close to 1 , whereas in contrast, the theoretical results (Table 1) show that using β 1 and/or β 2 close to 0 makes the upper bounds of the performance measures small.

Original: [Table]
Revised: [Table]

Original: (Chen et al., 2019)
Revised: (Chen et al., 2019)

Original: AMSGrad
Revised: AMSGrad

Original: [Equation]
Revised: [Equation]

Original: (Zhou et al., 2020)
Revised: (Zhou et al., 2020)

Original: AdaBelief
Revised: AdaBelief

Original: [Equation]
Revised: [Equation]

Original: (Zhuang et al., 2020)
Revised: (Zhuang et al., 2020)

Original: Padam
Revised: Padam

Original: [Equation]
Revised: [Equation]

Original: (Chen et al., 2021)
Revised: (Chen et al., 2021)

Original: Adam,
Revised: Adam,

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: (this paper)
Revised: (this paper)

Original: The practical performance of a deep learning optimizer strongly depends on the batch size.
Revised: The practical performance of a deep learning optimizer strongly depends on the batch size.

Original: In (Smith et al., 2018), it was numerically shown that using an enormous batch size leads to a reduction in the number of parameter updates and model training time.
Revised: In (Smith et al., 2018), it was numerically shown that using an enormous batch size leads to a reduction in the number of parameter updates and model training time.

Original: The theoretical results in (Zaheer et al., 2018) showed that using large batch sizes makes the upper bound of P( K ) of an adaptive method small (Table 1).
Revised: The theoretical results in (Zaheer et al., 2018) showed that using large batch sizes makes the upper bound of P( K ) of an adaptive method small (Table 1).

Original: Convergence analyses of SGD in (Cotter et al., 2011; Chen et al., 2020;
Revised: Convergence analyses of SGD in (Cotter et al., 2011; Chen et al., 2020;

Original: Arjevani et al., 2022) indicated that running SGD with a decaying learning rate and large batch size for sufﬁciently many steps leads to convergence to a local minimizer of a loss function.
Revised: Arjevani et al., 2022) indicated that running SGD with a decaying learning rate and large batch size for sufﬁciently many steps leads to convergence to a local minimizer of a loss function.

Original: Accordingly, the practical results for large batch sizes match the theoretical ones.
Revised: Accordingly, the practical results for large batch sizes match the theoretical ones.

Original: In (Shallue et al., 2019; Zhang et al., 2019), it was studied how increasing the batch size affects the performances of deep learning optimizers.
Revised: In (Shallue et al., 2019; Zhang et al., 2019), it was studied how increasing the batch size affects the performances of deep learning optimizers.

Original: In both studies, it was numerically shown that increasing batch size tends to decrease the number of steps K needed for training deep neural networks, but with diminishing returns.
Revised: In both studies, it was numerically shown that increasing batch size tends to decrease the number of steps K needed for training deep neural networks, but with diminishing returns.

Original: Moreover, it was shown that momentum methods can exploit larger batches than SGD (Shallue et al., 2019), and that K-FAC and Adam can exploit larger batches than momentum methods (Zhang et al., 2019).
Revised: Moreover, it was shown that momentum methods can exploit larger batches than SGD (Shallue et al., 2019), and that K-FAC and Adam can exploit larger batches than momentum methods (Zhang et al., 2019).

Original: 1. M OTIVATION 1.2. H YPERPARAMETERS CLOSE TO ONE AND CONSTANT LEARNING RATE
Revised: 1. M OTIVATION 1.2. H YPERPARAMETERS CLOSE TO ONE AND CONSTANT LEARNING RATE

Original: As described in Section 1.1, the practically useful β 1 and β 2 are each close to 1 , whereas in contrast, the theoretical results show that using β 1 and/or β 2 close to 0 makes the upper bounds of the performance measures small.
Revised: As described in Section 1.1, the practically useful β 1 and β 2 are each close to 1 , whereas in contrast, the theoretical results show that using β 1 and/or β 2 close to 0 makes the upper bounds of the performance measures small.

Original: Hence, there is a gap between theory ( β 1 , β 2 ≈ 0 ) and practice ( β 1 , β 2 ≈ 1 ) for adaptive methods.
Revised: Hence, there is a gap between theory ( β 1 , β 2 ≈ 0 ) and practice ( β 1 , β 2 ≈ 1 ) for adaptive methods.

Original: As a consequence, the ﬁrst motivation of this paper is to bridge this gap.
Revised: As a consequence, the ﬁrst motivation of this paper is to bridge this gap.

Original: Since using small constant learning rates is robust for training deep neural networks (Kingma & Ba, 2015; Reddi et al., 2018;
Revised: Since using small constant learning rates is robust for training deep neural networks (Kingma & Ba, 2015; Reddi et al., 2018;

Original: Zaheer et al., 2018; Zou et al., 2019; Chen et al., 2019; Zhuang et al., 2020;
Revised: Zaheer et al., 2018; Zou et al., 2019; Chen et al., 2019; Zhuang et al., 2020;

Original: Chen et al., 2021), we focus on using a small constant learning rate α .
Revised: Chen et al., 2021), we focus on using a small constant learning rate α .

Original: Here, we note that using a learning rate depending on the Lipschitz constant L of the gradient of a loss function f would be unrealistic.
Revised: Here, we note that using a learning rate depending on the Lipschitz constant L of the gradient of a loss function f would be unrealistic.

Original: This is because computing the Lipschitz constant L is NP-hard (Virmaux & Scaman, 2018, Theorem 2).
Revised: This is because computing the Lipschitz constant L is NP-hard (Virmaux & Scaman, 2018, Theorem 2).

Revised: The results using varying learning rates are given in Appendix A.

Original: 1.2. C RITICAL BATCH SIZE
Revised: 1.2. C RITICAL BATCH SIZE

Original: The second motivation of this paper is to clarify theoretically the relationship between the diminishing returns reported in (Shallue et al., 2019; Zhang et al., 2019) and batch size.
Revised: The second motivation of this paper is to clarify theoretically the relationship between the diminishing returns reported in (Shallue et al., 2019; Zhang et al., 2019) and batch size.

Original: Numerical evaluations in (Shallue et al., 2019; Zhang et al., 2019) have deﬁnitively shown that, for deep learning optimizers, the number of steps K needed to train a deep neural network halves for each doubling of the batch size b and that there is a region of diminishing returns beyond the critical batch size b ⋆ .
Revised: Numerical evaluations in (Shallue et al., 2019; Zhang et al., 2019) have deﬁnitively shown that, for deep learning optimizers, the number of steps K needed to train a deep neural network halves for each doubling of the batch size b and that there is a region of diminishing returns beyond the critical batch size b ⋆ .

Original: This implies that there is a positive number C such that
Revised: This implies that there is a positive number C such that

Original: [Equation]
Revised: [Equation]

Original: (1) where K and b are deﬁned for i, j ∈ N by K = 2 i and b = 2 j (For example, Figure 1 in Section 4 shows C
Revised: where K and b are deﬁned for i, j ∈ N by K = 2 i and b = 2 j (For example, Figure 1 in Section 4 shows C

Original: ≈ 20 and b ⋆ ≈ 2 11 for Adam used to train ResNet-20 on the CIFAR-10 dataset).
Revised: ≈ 20 and b ⋆ ≈ 2 11 for Adam used to train ResNet-20 on the CIFAR-10 dataset).

Original: We deﬁne the stochastic ﬁrst-order oracle (SFO) complexity of a deep learning optimizer as N :=
Revised: We deﬁne the stochastic ﬁrst-order oracle (SFO) complexity of a deep learning optimizer as N :=

Original: Kb on the basis of the number of steps K needed for training the deep neural network and the batch size b used in the optimizer.
Revised: Kb on the basis of the number of steps K needed for training the deep neural network and the batch size b used in the optimizer.

Original: Let b ⋆ be a critical batch size such that there are diminishing returns for all batch sizes beyond b ⋆ , as asserted in (Shallue et al., 2019; Zhang et al., 2019).
Revised: Let b ⋆ be a critical batch size such that there are diminishing returns for all batch sizes beyond b ⋆ , as asserted in (Shallue et al., 2019; Zhang et al., 2019).

Original: This fact, expressed in (1), implies that, while SFO complexity N initially almost does not change (i.e., K halves for each doubling of b ), N is minimized at critical batch size b ⋆ , and there are diminishing returns once the batch size exceeds b ⋆ .
Revised: This fact, expressed in (1), implies that, while SFO complexity N initially almost does not change (i.e., K halves for each doubling of b ), N is minimized at critical batch size b ⋆ , and there are diminishing returns once the batch size exceeds b ⋆ .

Original: 1. C ONTRIBUTION
Revised: 1. C ONTRIBUTION

Original: Our results are summarized in Table 2 (see also the “(this paper)” column of Table 1).
Revised: Our results are summarized in Table 2 (see also the “(this paper)” column of Table 1).

Original: Our goal is to ﬁnd a local minimizer of a loss function f over R d , i.e., a stationary point θ ⋆ ∈ R d satisfying ∇ f ( θ ⋆ ) = 0 , which is equivalent to the variational inequality (VI) deﬁned for all θ ∈ R d by ∇
Revised: Our goal is to ﬁnd a local minimizer of a loss function f over R d , i.e., a stationary point θ ⋆ ∈ R d satisfying ∇ f ( θ ⋆ ) = 0 , which is equivalent to the variational inequality (VI) deﬁned for all θ ∈ R d by ∇

Original: f ( θ ⋆ ) ⊤ ( θ ⋆ − θ ) ≤ 0 .
Revised: f ( θ ⋆ ) ⊤ ( θ ⋆ − θ ) ≤ 0 .

Original: Here, we show the relationship between (i) E [ ∇
Revised: Here, we show the relationship between (i) E [ ∇

Original: f ( θ k ) ⊤ ( θ k − θ )] ≤ ϵ ( θ ∈ R d , k ∈ N ) and (ii) E [ ∥∇ f ( θ k ) ∥ 2 ] ≤ ϵ ( k ∈ N ) , where ϵ > 0 is the precision.
Revised: f ( θ k ) ⊤ ( θ k − θ )] ≤ ϵ ( θ ∈ R d , k ∈ N ) and (ii) E [ ∥∇ f ( θ k ) ∥ 2 ] ≤ ϵ ( k ∈ N ) , where ϵ > 0 is the precision.

Original: Let us assume that ( θ k ) k ∈ N is bounded.
Revised: Let us assume that ( θ k ) k ∈ N is bounded.

Original: Suppose that (i) holds.
Revised: Suppose that (i) holds.

Original: Then, there exists a subsequence ( θ k i ) i ∈ N of ( θ k ) k ∈ N such that ( θ k i ) i ∈ N converges to θ ∗ .
Revised: Then, there exists a subsequence ( θ k i ) i ∈ N of ( θ k ) k ∈ N such that ( θ k i ) i ∈ N converges to θ ∗ .

Original: The continuity of ∇ f thus implies that, for all θ ∈ R d , E [ ∇ f ( θ ∗ ) ⊤ ( θ ∗ − θ )] ≤ ϵ .
Revised: The continuity of ∇ f thus implies that, for all θ ∈ R d , E [ ∇ f ( θ ∗ ) ⊤ ( θ ∗ − θ )] ≤ ϵ .

Original: Putting θ := θ ∗ −∇ f ( θ ∗ ) ensures that E [ ∥∇ f ( θ ∗ ) ∥ 2 ] ≤ ϵ .
Revised: Putting θ := θ ∗ −∇ f ( θ ∗ ) ensures that E [ ∥∇ f ( θ ∗ ) ∥ 2 ] ≤ ϵ .

Original: Suppose that (ii) holds.
Revised: Suppose that (ii) holds.

Original: Then, the deﬁnition of the inner product and Jensen’s inequality imply that, for all θ ∈ R d , E [ ∇ f ( θ k ) ⊤ ( θ k − θ )] ≤ Dist( θ ) √ ϵ , where Dist( θ ) :
Revised: Then, the deﬁnition of the inner product and Jensen’s inequality imply that, for all θ ∈ R d , E [ ∇ f ( θ k ) ⊤ ( θ k − θ )] ≤ Dist( θ ) √ ϵ , where Dist( θ ) :

Original: = sup {∥ θ k − θ ∥ : k ∈ N } < + ∞ .
Revised: = sup {∥ θ k − θ ∥ : k ∈ N } < + ∞ .

Original: Therefore, it is adequate to use (i) and ϵ -approximation VI( K, θ ) :
Revised: Therefore, it is adequate to use (i) and ϵ -approximation VI( K, θ ) :

Original: = 1 K P Kk =1 E [ ∇ f ( θ k ) ⊤ ( θ k − θ )] ≤ ϵ (Table 2) as the performance measure of an optimizer.
Revised: = 1 K P Kk =1 E [ ∇ f ( θ k ) ⊤ ( θ k − θ )] ≤ ϵ (Table 2) as the performance measure of an optimizer.

Original: 1.3. A DVANTAGE OF SETTING A SMALL CONSTANT LEARNING RATE AND HYPERPARAMETERS CLOSE TO ONE
Revised: 1.3. A DVANTAGE OF SETTING A SMALL CONSTANT LEARNING RATE AND HYPERPARAMETERS CLOSE TO ONE

Original: We can show that the upper bound C 1 K + C 2 b
Revised: We can show that the upper bound C 1 K + C 2 b

Original: + C 3 of VI( K, θ ) becomes small when α is small, βand β 2 are close to 1 , and K is large.
Revised: + C 3 of VI( K, θ ) becomes small when α is small, βand β 2 are close to 1 , and K is large.

Original: This implies that Momentum and Adam perform well when α is small and β 1 and β 2 are each set close to 1 .
Revised: This implies that Momentum and Adam perform well when α is small and β 1 and β 2 are each set close to 1 .

Original: Section 3.1 shows this result in detail.
Revised: Section 3.1 shows this result in detail.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: C RITICAL BATCH SIZE
Revised: C RITICAL BATCH SIZE

Original: As described in Section 1.2.
Revised: As described in Section 1.2.

Original: [Table]
Revised: [Table]

Original: the practical performance of a deep learning optimizer strongly depends on the batch size (Shallue et al., 2019; Zhang et al., 2019).
Revised: the practical performance of a deep learning optimizer strongly depends on the batch size (Shallue et al., 2019; Zhang et al., 2019).

Original: The advantage of this paper is to clarify theoretically the relationship between batch size and the performance of deep learning optimizers and develop a theory demonstrating the existence of critical batch sizes, which was shown numerically by (Shallue et al., 2019; Zhang et al., 2019).
Revised: The advantage of this paper is to clarify theoretically the relationship between batch size and the performance of deep learning optimizers and develop a theory demonstrating the existence of critical batch sizes, which was shown numerically by (Shallue et al., 2019; Zhang et al., 2019).

Original: Motivated by the results in (Shallue et al., 2019; Zhang et al., 2019) and Section 1.2.2, we use SFO complexity N :=
Revised: Motivated by the results in (Shallue et al., 2019; Zhang et al., 2019) and Section 1.2.2, we use SFO complexity N :=

Original: Kb as the performance measure of a deep learning optimizer.
Revised: Kb as the performance measure of a deep learning optimizer.

Original: We ﬁrst show that the number of steps K to satisfy VI( K, θ ) ≤ ϵ can be deﬁned as in Table 2.
Revised: We ﬁrst show that the number of steps K to satisfy VI( K, θ ) ≤ ϵ can be deﬁned as in Table 2.

Original: As a function, K is convex and monotone decreasing with respect to batch size b .
Revised: As a function, K is convex and monotone decreasing with respect to batch size b .

Original: Next, we show that SFO complexity N deﬁned as in Table 2 is convex with respect to batch size b .
Revised: Next, we show that SFO complexity N deﬁned as in Table 2 is convex with respect to batch size b .

Original: This result agrees with the fact of (1).
Revised: This result agrees with the fact of (1).

Original: Moreover, SFO complexity N is minimized at b ⋆ deﬁned as in Table 2.
Revised: Moreover, SFO complexity N is minimized at b ⋆ deﬁned as in Table 2.

Original: This result guarantees the existence of the critical batch size b ⋆ .
Revised: This result guarantees the existence of the critical batch size b ⋆ .

Original: Section 3.2 shows the above results in detail.
Revised: Section 3.2 shows the above results in detail.

Original: However, the accurate setting of the critical batch size b
Revised: However, the accurate setting of the critical batch size b

Original: ⋆ deﬁned as in Table 2 would be difﬁcult since b ⋆ involves unknown parameters, such as
Revised: ⋆ deﬁned as in Table 2 would be difﬁcult since b ⋆ involves unknown parameters, such as

Original: G and D ( θ ) (see Section 2.2.3).
Revised: G and D ( θ ) (see Section 2.2.3).

Original: The advantage of our analysis is that we can estimate appropriate batch sizes using the formula for b ⋆ before implementing deep learning optimizers.
Revised: The advantage of our analysis is that we can estimate appropriate batch sizes using the formula for b ⋆ before implementing deep learning optimizers.

Original: Section 4 will discuss estimation of appropriate batch sizes in detail.
Revised: Section 4 will discuss estimation of appropriate batch sizes in detail.

Original:  N ONCONVEX O PTIMIZATION AND D EEP L EARNING O PTIMIZERS
Revised:  N ONCONVEX O PTIMIZATION AND D EEP L EARNING O PTIMIZERS

Original: This section gives a nonconvex optimization problem in deep neural networks and optimizers for solving the problem under standard assumptions.
Revised: This section gives a nonconvex optimization problem in deep neural networks and optimizers for solving the problem under standard assumptions.

Original: 2. N ONCONVEX OPTIMIZATION IN DEEP LEARNING
Revised: 2. N ONCONVEX OPTIMIZATION IN DEEP LEARNING

Original: Let R d be a d -dimensional Euclidean space with inner product ⟨ x , y ⟩ := x ⊤ y inducing the norm ∥ x ∥ and N be the set of nonnegative integers.
Revised: Let R d be a d -dimensional Euclidean space with inner product ⟨ x , y ⟩ := x ⊤ y inducing the norm ∥ x ∥ and N be the set of nonnegative integers.

Original: Deﬁne [ n ] := { 1 , 2 , . . . , n } for n ≥ 1 .
Revised: Deﬁne [ n ] := { 1 , 2 , . . . , n } for n ≥ 1 .

Original: Given a parameter θ ∈ R d and a data point z in a data domain Z , a machine learning model provides a prediction whose quality is measured by a differentiable nonconvex loss function ℓ ( θ ; z ) .
Revised: Given a parameter θ ∈ R d and a data point z in a data domain Z , a machine learning model provides a prediction whose quality is measured by a differentiable nonconvex loss function ℓ ( θ ; z ) .

Original: We aim to minimize the expected loss deﬁned for all θ ∈ R d by f ( θ ) =
Revised: We aim to minimize the expected loss deﬁned for all θ ∈ R d by f ( θ ) =

Original: E z
Revised: E z

Original: ∼D
Revised: ∼D

Original: [ ℓ ( θ ; z )]
Revised: [ ℓ ( θ ; z )]

Original: =
Revised: =

Original: E [ ℓ ξ ( θ )] , where D is a probability distribution over Z , ξ denotes a random variable with distribution function P , and E [ · ] denotes the expectation taken with respect to ξ .
Revised: E [ ℓ ξ ( θ )] , where D is a probability distribution over Z , ξ denotes a random variable with distribution function P , and E [ · ] denotes the expectation taken with respect to ξ .

Original: A particularly interesting example of f ( θ ) is the empirical average loss deﬁned for all θ ∈ R d by f ( θ ; S )
Revised: A particularly interesting example of f ( θ ) is the empirical average loss deﬁned for all θ ∈ R d by f ( θ ; S )

Original: = 1 n P i ∈ [ n ] ℓ ( θ ; z i ) =
Revised: = 1 n P i ∈ [ n ] ℓ ( θ ; z i ) =

Original: 1 n P i ∈ [ n ] ℓ i ( θ ) , where S
Revised: 1 n P i ∈ [ n ] ℓ i ( θ ) , where S

Original: =
Revised: =

Original: ( z 1 , z 2 , . . .
Revised: ( z 1 , z 2 , . . .

Original: , z n ) denotes the training set and ℓ i ( · ) :=
Revised: , z n ) denotes the training set and ℓ i ( · ) :=

Original: ℓ ( · ; z i ) denotes the loss function corresponding to the i -th training data z i .
Revised: ℓ ( · ; z i ) denotes the loss function corresponding to the i -th training data z i .

Original: 2. D EEP LEARNING OPTIMIZERS 2.2. C ONDITIONS
Revised: 2. D EEP LEARNING OPTIMIZERS 2.2. C ONDITIONS

Original: We assume that a stochastic ﬁrst-order oracle (SFO) exists such that, for a given θ ∈ R d , it returns a stochastic gradient G ξ ( θ ) of the function f , where a random variable ξ is supported on Ξ independently of θ .
Revised: We assume that a stochastic ﬁrst-order oracle (SFO) exists such that, for a given θ ∈ R d , it returns a stochastic gradient G ξ ( θ ) of the function f , where a random variable ξ is supported on Ξ independently of θ .

Original: The following are standard conditions when considering a deep learning optimizer.
Revised: The following are standard conditions when considering a deep learning optimizer.

Original: (C1)
Revised: (C1)

Original: f : R d → R is continuously differentiable.
Revised: f : R d → R is continuously differentiable.

Original: (C2)
Revised: (C2)

Original: Let ( θ k ) k ∈ N ⊂ R
Revised: Let ( θ k ) k ∈ N ⊂ R

Original: d be the sequence generated by a deep learning optimizer.
Revised: d be the sequence generated by a deep learning optimizer.

Original: For each iteration k , E ξ k
Revised: For each iteration k , E ξ k

Original: [ G ξ k ( θ k )] = ∇ f ( θ k ) , where ξ 0 , ξ 1 , . .
Revised: [ G ξ k ( θ k )] = ∇ f ( θ k ) , where ξ 0 , ξ 1 , . .

Original: . are independent samples and the random variable ξ k is independent of ( θ l ) kl =0 .
Revised: . are independent samples and the random variable ξ k is independent of ( θ l ) kl =0 .

Original: There exists a nonnegative constant σ 2 such that E
Revised: There exists a nonnegative constant σ 2 such that E

Original: [Equation]
Revised: [Equation]

Original: (C3)
Revised: (C3)

Original: For each iteration k , the optimizer samples a batch B k of size b independently of k and estimates the full gradient ∇ f as ∇ f B k ( θ k ) :
Revised: For each iteration k , the optimizer samples a batch B k of size b independently of k and estimates the full gradient ∇ f as ∇ f B k ( θ k ) :

Original: = 1 b P i ∈ [ b ] G ξ k,i ( θ k ) , where ξ k,i is a random variable generated by the i
Revised: = 1 b P i ∈ [ b ] G ξ k,i ( θ k ) , where ξ k,i is a random variable generated by the i

Original: -th sampling in the k -th iteration.
Revised: -th sampling in the k -th iteration.

Original: A DAM
Revised: A DAM

Original: Algorithm 1 is the Adam optimizer (Kingma & Ba, 2015) under (C1)–(C3).
Revised: Algorithm 1 is the Adam optimizer (Kingma & Ba, 2015) under (C1)–(C3).

Original: The symbol ⊙ in step 6 is deﬁned for all x = ( x i ) di
Revised: The symbol ⊙ in step 6 is deﬁned for all x = ( x i ) di

Original: =1 ∈ R d , x ⊙ x := ( x 2 i ) di
Revised: =1 ∈ R d , x ⊙ x := ( x 2 i ) di

Original: =1 ∈ R d , and diag ( x i ) in step 8 is a diagonal matrix with diagonal components x 1 , x 2 , . . .
Revised: =1 ∈ R d , and diag ( x i ) in step 8 is a diagonal matrix with diagonal components x 1 , x 2 , . . .

Original: , x d .
Revised: , x d .

Original: Require: α ∈ (0 , + ∞ ) , b ∈ (0 , + ∞ ) , β 1 ∈ (0 , 1) , β 2 ∈ [0 , 1) 1: k ← 0 , θ 0 ∈ R d , m − 1 := 0 , v − 1 :=2: loop 3: ∇ f B k ( θ k ) :
Revised: Require: α ∈ (0 , + ∞ ) , b ∈ (0 , + ∞ ) , β 1 ∈ (0 , 1) , β 2 ∈ [0 , 1) 1: k ← 0 , θ 0 ∈ R d , m − 1 := 0 , v − 1 :=2: loop 3: ∇ f B k ( θ k ) :

Original: = 1 b
Revised: = 1 b

Original: P i ∈ [ b ] G ξ k,i ( θ k ) 4: m k :=
Revised: P i ∈ [ b ] G ξ k,i ( θ k ) 4: m k :=

Original: β 1 m k − 1 + (1 − β 1 ) ∇ f B k ( θ k ) 5:
Revised: β 1 m k − 1 + (1 − β 1 ) ∇ f B k ( θ k ) 5:

Original: ˆ m k :=
Revised: ˆ m k :=

Original: (1 − β k + ) − 1 m k 6: v k :=
Revised: (1 − β k + ) − 1 m k 6: v k :=

Original: β 2 v k − 1 + (1 − β 2 ) ∇ f B k ( θ k ) ⊙ ∇ f B k ( θ k ) 7: ˆ v k :=
Revised: β 2 v k − 1 + (1 − β 2 ) ∇ f B k ( θ k ) ⊙ ∇ f B k ( θ k ) 7: ˆ v k :=

Original: (1 − β k + ) − 1 v k 8:
Revised: (1 − β k + ) − 1 v k 8:

Original: H k := diag ( p ˆ v k,i ) 9: θ k +1 :
Revised: H k := diag ( p ˆ v k,i ) 9: θ k +1 :

Original: = θ k − α H − 1 k ˆ m k 10: k ← k +11: end loop
Revised: = θ k − α H − 1 k ˆ m k 10: k ← k +11: end loop

Original: The SGD optimizer under (C1)–(C3) is Algorithm 1 when β 1 = 

Revised: The SGD optimizer under (C1)–(C3) is Algorithm 1 when β 1 =

Original: and H k is the identity matrix.
Revised: and H k is the identity matrix.

Original: The Momentum optimizer under (C1)–(C3) is deﬁned for all k ∈ N by θ k
Revised: The Momentum optimizer under (C1)–(C3) is deﬁned for all k ∈ N by θ k

Original: +1 :
Revised: +1 :

Original: = θ k − α m k .
Revised: = θ k − α m k .

Original: 2.2. A SSUMPTIONS
Revised: 2.2. A SSUMPTIONS

Original: We assume the following conditions that were used in (Kingma & Ba, 2015, Theorem 4.1):
Revised: We assume the following conditions that were used in (Kingma & Ba, 2015, Theorem 4.1):

Original: (A1)
Revised: (A1)

Original: There exist positive numbers G and B such that, for all k ∈ N , ∥∇ f ( θ k ) ∥ ≤ G and ∥∇ f B k ( θ k ) ∥ ≤ B . (A2)
Revised: There exist positive numbers G and B such that, for all k ∈ N , ∥∇ f ( θ k ) ∥ ≤ G and ∥∇ f B k ( θ k ) ∥ ≤ B . (A2)

Original: For all θ ∈ R d , there exists a positive number Dist( θ ) such that, for all k ∈ N , ∥ θ k − θ ∥ ≤ Dist( θ ) .
Revised: For all θ ∈ R d , there exists a positive number Dist( θ ) such that, for all k ∈ N , ∥ θ k − θ ∥ ≤ Dist( θ ) .

Original: Let ( g 2 k,i ) di =1 :=
Revised: Let ( g 2 k,i ) di =1 :=

Original: ∇ f B k ( θ k ) ⊙ ∇ f B k ( θ k ) ( k ∈ N ) .
Revised: ∇ f B k ( θ k ) ⊙ ∇ f B k ( θ k ) ( k ∈ N ) .

Original: Assumption (A1) implies that M := sup { max i ∈ [ d ] g 2 k,i : k ∈ N } < + ∞ .
Revised: Assumption (A1) implies that M := sup { max i ∈ [ d ] g 2 k,i : k ∈ N } < + ∞ .

Original: Assumption (A2) implies that D ( θ ) :
Revised: Assumption (A2) implies that D ( θ ) :

Original: = sup { max i ∈
Revised: = sup { max i ∈

Original: [ d ] ( θ k,i − θ i ) 2 : k ∈ N } < + ∞ .
Revised: [ d ] ( θ k,i − θ i ) 2 : k ∈ N } < + ∞ .

Original: We deﬁne v ∗ := inf { min i ∈
Revised: We deﬁne v ∗ := inf { min i ∈

Original: [ d ] v k,i : k ∈ N } .
Revised: [ d ] v k,i : k ∈ N } .

Original: Theorem 3 in (Reddi et al., 2018) shows that there exists a stochastic convex optimization problem such that Adam using β 1 < √ β 2 (e.g., β 1 = 0 .
Revised: Theorem 3 in (Reddi et al., 2018) shows that there exists a stochastic convex optimization problem such that Adam using β 1 < √ β 2 (e.g., β 1 = 0 .

Original: 9 and β 2 = 0 .
Revised: 9 and β 2 = 0 .

Original: 999 ) does not converge to the optimal solution.
Revised: 999 ) does not converge to the optimal solution.

Original: If for all k ∈ N and all i ∈
Revised: If for all k ∈ N and all i ∈

Original: [ d ] , ˆ v k,i in Adam satisﬁes
Revised: [ d ] , ˆ v k,i in Adam satisﬁes

Original: [Equation]
Revised: [Equation]

Original: then Adam with a decaying learning rate α k
Revised: then Adam with a decaying learning rate α k

Original: = O (1 / √ k ) and β 1 and β 2 satisfying β 1 < √ βcan solve the stochastic convex optimization problem (Reddi et al., 2018, (2), Theorem 4).
Revised: = O ( 1 √ k ) and β 1 and β 2 satisfying β 1 < √ β 2 can solve the stochastic convex optimization problem (Reddi et al., 2018, (2), Theorem 4).

Original: We thus assume condition (2) for Adam to guarantee the convergence of Adam.
Revised: We thus assume condition (2) for Adam to guarantee the convergence of Adam.

Original:  O UR R ESULTS
Revised:  O UR R ESULTS

Original: This section states our theoretical results (Theorem 3.1) in Table 2 and our contribution (Sections 3.1 and 3.2) in detail.
Revised: This section states our theoretical results (Theorem 3.1) in Table 2 and our contribution (Sections 3.1 and 3.2) in detail.

Original: The proof of Theorem 3.1 is given in Appendix A.
Revised: The proof of Theorem 3.1 is given in Appendix A.

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: The sequence ( θ k ) k ∈ N generated by each of SGD, Momentum, and Adam with (2) under (C1)–(C3) and (A1) and (A2) satisﬁes the following: (i
Revised: The sequence ( θ k ) k ∈ N generated by each of SGD, Momentum, and Adam with (2) under (C1)–(C3) and (A1) and (A2) satisﬁes the following: (i)

Original: [Upper bound of VI( K, θ ) ]
Revised: [Upper bound of VI( K, θ ) ]

Original: For all K ≥ 1 and all θ ∈ R d 
Revised: For all K ≥ 1 and all θ ∈ R d ,

Original: [Equation]
Revised: [Equation]

Original: where C i ( i = 1 , 2 , 3) for SGD are
Revised: where C i ( i = 1 , 2 , 3) for SGD are

Original: [Equation]
Revised: [Equation]

Original: C i ( i = 1 , 2 , 3) for Momentum are
Revised: C i ( i = 1 , 2 , 3) for Momentum are

Original: [Equation]
Revised: [Equation]

Original: C i ( i = 1 , 2 , 3) for Adam with (2) are
Revised: C i ( i = 1 , 2 , 3) for Adam with (2) are

Original: [Equation]
Revised: [Equation]

Original: and the parameters are deﬁned as in Section 2.2.
Revised: and the parameters are deﬁned as in Section 2.2.

Original: (ii) [Steps to satisfy VI( K, θ ) ≤ ϵ 
Revised: (ii) [Steps to satisfy VI( K, θ ) ≤ ϵ ]

Original: ] The number of steps K deﬁned by
Revised: The number of steps K deﬁned by

Original: [Equation]
Revised: [Equation]

Original: satisﬁes VI( K θ ) ≤ ϵ and the function K ( b ) deﬁned by  is convex and monotone decreasing with 
Revised: satisﬁes VI( K, θ ) ≤ ϵ and the function K ( b ) deﬁned by (3) is convex and monotone decreasing with respect to batch size b ( > C 2 ϵ − C 3 > 0)

Original: respect

Original: to batch size b ( > C 2 ϵ − C

Original: > 0) .

Original: (iii)
Revised: (iii)

Original: [Minimization of SFO complexity
Revised: [Minimization of SFO complexity]

Original: The SFO complexity deﬁned by
Revised: The SFO complexity deﬁned by

Original: [Equation]
Revised: [Equation]

Revised: is convex

Original: is convex with respect to batch size b ( > C2 ϵ − C3 > 0) .
Revised: with respect to batch size b ( > C ϵ − C

Revised: >

Original: The batch size [Equation]
Revised: The batch size [Equation]

Original: attains the minimum value N ⋆ = K ( b ⋆ ) b ⋆ = 2 C 1 C 2 ( ϵ − C 3 ) 2 of N .
Revised: attains the minimum value N ⋆ = K ( b ⋆ ) b ⋆ = 2 C 1 C 2 ( ϵ − C 3 ) 2 of N .

Original: The proof of Theorem 3.1(i) ensures that C i ( i = 1 , 2 , 3) of AMSGrad (Algorithm 1 with ˆ m k
Revised: The proof of Theorem 3.1(i) ensures that C i ( i = 1 , 2 , 3) of AMSGrad (Algorithm 1 with ˆ m k

Original: = m k , ˆ v k = v k , ˜ v k,i := max { ˜ v k − 1 ,i , v k,i } (i.e., ˜ v k +1 ,i ≥ ˜ v k,i ; see (2)), and
Revised: = m k , ˆ v k = v k , ˜ v k,i := max { ˜ v k − 1 ,i , v k,i } (i.e., ˜ v k +1 ,i ≥ ˜ v k,i ; see (2)), and

Original: H k :=
Revised: H k :=

Original: diag ( p ˜ v k,i ) ) are
Revised: diag ( p ˜ v k,i ) ) are

Original: C

Original: [Equation]
Revised: [Equation]

Original: ( θ ) [Equation] We give a brief outline of the proof strategy of Theorem 3.1, with an emphasis on the main difﬁculty that has to be overcome in order not to assume Lipschitz smoothness of f (i.e., ∇ f is Lipschitz continuous with the Lipschitz constant L ).
Revised: We give a brief outline of the proof strategy of Theorem 3.1, with an emphasis on the main difﬁculty that has to be overcome in order not to assume Lipschitz smoothness of f (i.e., ∇ f is Lipschitz continuous with the Lipschitz constant L ).

Original: First, we show that (C2) and (A1) imply that ( E [ ∥ m k ∥ ]) k ∈ N and ( E [ ∥ d k ∥ ]) k ∈ N are bounded, where d k := − H − 1 k ˆ m k .
Revised: First, we show that (C2) and (A1) imply that ( E [ ∥ m k ∥ ]) k ∈ N and ( E [ ∥ d k ∥ ]) k ∈ N are bounded, where d k := − H − 1 k ˆ m k .

Original: Since we do not assume Lipschitz smoothness of f , we cannot use the descent lemma, i.e., f ( y ) ≤ f ( x ) + ∇ f ( x ) ⊤ ( y − x ) + L 2 ∥ y − x ∥
Revised: Since we do not assume Lipschitz smoothness of f , we cannot use the descent lemma, i.e., f ( y ) ≤ f ( x ) + ∇ f ( x ) ⊤ ( y − x ) + L 2 ∥ y − x ∥

Original: ( x , y ∈ R d ) .
Revised: ( x , y ∈ R d ) .

Original: This is the main difﬁculty to prove Theorem 3.1.
Revised: This is the main difﬁculty to prove Theorem 3.1.

Original: Almost all of the previous analyses of adaptive methods are based on the descent lemma, and hence, they can use the expectation of the squared norm of the full gradient E [ ∥∇ f ( θ k ) ∥ 2 ] as the performance measure.
Revised: Almost all of the previous analyses of adaptive methods are based on the descent lemma, and hence, they can use the expectation of the squared norm of the full gradient E [ ∥∇ f ( θ k ) ∥ 2 ] as the performance measure.

Original: Accordingly, we must use other performance measures that are different from E [ ∥∇ f ( θ k ) ∥ 2 ] .
Revised: Accordingly, we must use other performance measures that are different from E [ ∥∇ f ( θ k ) ∥ 2 ] .

Original: This paper uses the performance measure VI( K, θ ) .
Revised: This paper uses the performance measure VI( K, θ ) .

Original: We next show that P Kk =1 E [ m ⊤ k − 1 ( θ k − θ )] ≤ a K + b K + c K ≤
Revised: We next show that P Kk =1 E [ m ⊤ k − 1 ( θ k − θ )] ≤ a K + b K + c K ≤

Original: C 1 + C 2 Kb + ˜ C 3 K , where C 1 and C 2 are deﬁned as in Theorem 3.1 and ˜ C 3 > 0 .
Revised: C 1 + C 2 Kb + ˜ C 3 K , where C 1 and C 2 are deﬁned as in Theorem 3.1 and ˜ C 3 > 0 .

Original: In particular, (2), (A1), and (A2) imply a K ≤ C 1 , the boundedness condition of ( E [ ∥ d k ∥ ]) k ∈ N implies b K ≤ C 2 Kb , and (A2) and the Cauchy–Schwarz inequality imply c K ≤ ˜ C 3 K .
Revised: In particular, (2), (A1), and (A2) imply a K ≤ C 1 , the boundedness condition of ( E [ ∥ d k ∥ ]) k ∈ N implies b K ≤ C 2 Kb , and (A2) and the Cauchy–Schwarz inequality imply c K ≤ ˜ C 3 K .

Original: The deﬁnition of m k , the Cauchy–
Revised: The deﬁnition of m k , the Cauchy–

Original: Schwarz inequality, the triangle inequality, and Jensen’s inequality imply Theorem 3.1(i).
Revised: Schwarz inequality, the triangle inequality, and Jensen’s inequality imply Theorem 3.1(i).

Original: Theorem 3.1(i) and C 1 K + C 2 b + C 3 =
Revised: Theorem 3.1(i) and C 1 K + C 2 b + C 3 =

Original: ϵ lead to Theorem 3.1(ii) and (iii).
Revised: ϵ lead to Theorem 3.1(ii) and (iii).

Original: 3. A DVANTAGE OF SETTING A SMALL CONSTANT LEARNING RATE AND HYPERPARAMETERS CLOSE TO ONE
Revised: 3. A DVANTAGE OF SETTING A SMALL CONSTANT LEARNING RATE AND HYPERPARAMETERS CLOSE TO ONE

Original: We ﬁrst show theoretical evidence that Adam using a small constant learning rate α , β 1 and β 2 close to 1 , and a large number of steps K performs well.
Revised: We ﬁrst show theoretical evidence that Adam using a small constant learning rate α , β 1 and β 2 close to 1 , and a large number of steps K performs well.

Original: Theorem 3.1(i) indicates that the upper bound of
Revised: Theorem 3.1(i) indicates that the upper bound of

Original: VI( K, θ ) for Adam is
Revised: VI( K, θ ) for Adam is

Original: [Equation]
Revised: [Equation]

Original: where β 1 ∈ (0 , 1) , β 2 ∈ [0 , 1) , and
Revised: where β 1 ∈ (0 , 1) , β 2 ∈ [0 , 1) , and

Original: [Equation]
Revised: [Equation]

Original: (the strict evaluation (6) of Theorem 3.1(i) comes from (28) in Appendix A).
Revised: (the strict evaluation (6) of Theorem 3.1(i) comes from (28) in Appendix A).

Original: Since the function h ( β 1 ) deﬁned by (7) is monotone decreasing, using β 1 close to 1 makes h ( β 1 ) small.
Revised: Since the function h ( β 1 ) deﬁned by (7) is monotone decreasing, using β 1 close to 1 makes h ( β 1 ) small.

Original: Since 1 β 1 (1 − β 1 ) is monotone increasing for β 1 ≥ 1 / 2 , using β 1 close to 1 makes 1 β 1 (1 − β 1 ) large.
Revised: Since 1 β 1 (1 − β 1 ) is monotone increasing for β 1 ≥ 1 / 2 , using β 1 close to 1 makes 1 β 1 (1 − β 1 ) large.

Original: Hence, we need to set a small α to make α ( σ 2 b − 1 + G 2 ) 2 √ v ∗ β 1 (1 − β 1 ) small.
Revised: Hence, we need to set a small α to make α ( σ 2 b − 1 + G 2 ) 2 √ v ∗ β 1 (1 − β 1 ) small.

Original: The function q
Revised: The function q

Original: [Equation]
Revised: [Equation]

Original: is monotone decreasing with respect to β 2 , while using β 2 close to 1 makes 1 √ 1 − β 2 large.
Revised: is monotone decreasing with respect to β 2 , while using β 2 close to 1 makes 1 √ 1 − β 2 large.

Original: When β 2 close to 1 and a small learning rate α are used, we need to use a large number of steps K to make
Revised: When β 2 close to 1 and a small learning rate α are used, we need to use a large number of steps K to make

Original: [Equation]
Revised: [Equation]

Original: 3. C RITICAL BATCH SIZE
Revised: 3. C RITICAL BATCH SIZE

Original: Theorem 3.1(ii) indicates that the number of steps K to satisfy VI( K, θ ) ≤ ϵ can be expressed as (3).
Revised: Theorem 3.1(ii) indicates that the number of steps K to satisfy VI( K, θ ) ≤ ϵ can be expressed as (3).

Original: The function K ( b ) deﬁned by (3) is convex and monotone decreasing.
Revised: The function K ( b ) deﬁned by (3) is convex and monotone decreasing.

Original: Hence, the form of
Revised: Hence, the form of

Original: K deﬁned by (3) supports theoretically the relationship between K and b shown in (Shallue et al., 2019; Zhang et al., 2019) (see also Figures 1 and 3 in this paper).
Revised: K deﬁned by (3) supports theoretically the relationship between K and b shown in (Shallue et al., 2019; Zhang et al., 2019) (see also Figures 1 and 3 in this paper).

Original: Theorem 3.1(iii) indicates that
Revised: Theorem 3.1(iii) indicates that

Original: SFO complexity deﬁned by (4) is convex with respect to batch size b .
Revised: SFO complexity deﬁned by (4) is convex with respect to batch size b .

Original: This result agrees with the fact of (1) (see also Figures 2 and 4 in this paper).
Revised: This result agrees with the fact of (1) (see also Figures 2 and 4 in this paper).

Original: Moreover, SFO complexity N :=
Revised: Moreover, SFO complexity N :=

Original: Kb is minimized at b ⋆ deﬁned by (5); e.g., b ⋆ S for SGD, b ⋆ M for Momentum, and b ⋆ A for Adam are respectively
Revised: Kb is minimized at b ⋆ deﬁned by (5); e.g., b ⋆ S for SGD, b ⋆ M for Momentum, and b ⋆ A for Adam are respectively

Original: [Equation]
Revised: [Equation]

Original: where σ S , σ M , and σ A are positive constants depending on SGD, Momentum, and Adam and C 3 =
Revised: where σ S , σ M , and σ A are positive constants depending on SGD, Momentum, and Adam and C 3 =

Original: C 3 , S , C 3 , M , C 3 , A are positive constants deﬁned as in Theorem 3.1.
Revised: C 3 , S , C 3 , M , C 3 , A are positive constants deﬁned as in Theorem 3.1.

Original: From (8), the lower bounds b ∗ of b ⋆ S , b ⋆ M , and b ⋆ A are respectively
Revised: From (8), the lower bounds b ∗ of b ⋆ S , b ⋆ M , and b ⋆ A are respectively

Original: [Equation]
Revised: [Equation]

Original:  N UMERICAL R ESULTS
Revised:  N UMERICAL R ESULTS

Original: We evaluated the performances of SGD, Momentum, and Adam with different batch sizes.
Revised: We evaluated the performances of SGD, Momentum, and Adam with different batch sizes.

Original: The metrics were the number of steps K and the SFO complexity Kb satisfying f ( θ K ) ≤ 10 − 1 , where θ K is generated for each of SGD, Momentum, and Adam using batch size b .
Revised: The metrics were the number of steps K and the SFO complexity Kb satisfying f ( θ K ) ≤ 10 − 1 , where θ K is generated for each of SGD, Momentum, and Adam using batch size b .

Original: The stopping condition was 200 epochs.
Revised: The stopping condition was 200 epochs.

Original: The experimental environment consisted of two Intel(R)
Revised: The experimental environment consisted of two Intel(R)

Original: Xeon(R) Gold 6148 2.4GHz CPUs with 20 cores each, a 16-GB NVIDIA Tesla V100 900-Gbps GPU, and the Red Hat
Revised: Xeon(R) Gold 6148 2.4GHz CPUs with 20 cores each, a 16-GB NVIDIA Tesla V100 900-Gbps GPU, and the Red Hat

Original: Enterprise Linux 7.6 OS.
Revised: Enterprise Linux 7.6 OS.

Original: The code was written in Python 3.8.2 using the NumPy 1.17.3 and PyTorch 1.3.0 packages.
Revised: The code was written in Python 3.8.2 using the NumPy 1.17.3 and PyTorch 1.3.0 packages.

Original: A constant learning rate α = 10 − 3 was commonly used.
Revised: A constant learning rate α = 10 − 3 was commonly used.

Original: Momentum used β 1 = 0 .
Revised: Momentum used β 1 = 0 .

Original: Adam used β 1 = 0 .
Revised: Adam used β 1 = 0 .

Original: 9 and β 2 = 0 .
Revised: 9 and β 2 = 0 .

Original: 999 (Kingma & Ba, 2015).
Revised: 999 (Kingma & Ba, 2015).

Original: Here, we use (9) and estimate appropriate batch sizes in the sense that SFO complexity is minimized.
Revised: Here, we use (9) and estimate appropriate batch sizes in the sense that SFO complexity is minimized.

Original: The deﬁnitions of v ∗ and v k,i (see also (23) in Appendix A) imply that, for k ∈
Revised: The deﬁnitions of v ∗ and v k,i (see also (23) in Appendix A) imply that, for k ∈

Original: [ K ] and all i ∈ [ d ] ,
Revised: [ K ] and all i ∈ [ d ] ,

Original: [Equation]
Revised: [Equation]

Original: implies that E [ ∥∇ f B k ( θ k ) ∥ 2 ] ≤ σ 2 b + E [ ∥∇ f ( θ k ) ∥ 2 ] (see also (12) in Appendix A).
Revised: implies that E [ ∥∇ f B k ( θ k ) ∥ 2 ] ≤ σ 2 b + E [ ∥∇ f ( θ k ) ∥ 2 ] (see also (12) in Appendix A).

Original: Conditions (C2) and (C3) imply that, if b is large, then σ is small.
Revised: Conditions (C2) and (C3) imply that, if b is large, then σ is small.

Original: Hence, assuming that σ 2 b ≈ 0 implies that v ∗ ≤ ∥∇ f
Revised: Hence, assuming that σ 2 b ≈ 0 implies that v ∗ ≤ ∥∇ f

Original: ( θ k ∗ ) ∥ 2 =
Revised: ( θ k ∗ ) ∥ 2 =

Original: [Equation]
Revised: [Equation]

Original: P
Revised: P

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: Since deep learning optimizers can approximate stationary points of f , we assume that, for example, G k ∗ ,i ≈ ϵ .
Revised: Since deep learning optimizers can approximate stationary points of f , we assume that, for example, G k ∗ ,i ≈ ϵ .

Original: Then, (9) implies that
Revised: Then, (9) implies that

Original: [Equation]
Revised: [Equation]

Original: Conditions (C2) and (C3) imply that, if b is large, then σ is small.
Revised: Conditions (C2) and (C3) imply that, if b is large, then σ is small.

Original: If SGD and Momentum can exploit large batch sizes, then σ S and σ M are small.
Revised: If SGD and Momentum can exploit large batch sizes, then σ S and σ M are small.

Original: However, (10) implies that b ∗ S and b ∗ M must be small when σ S and σ M are small.
Revised: However, (10) implies that b ∗ S and b ∗ M must be small when σ S and σ M are small.

Original: Accordingly, SGD and Momentum would not be able to use large batch sizes.
Revised: Accordingly, SGD and Momentum would not be able to use large batch sizes.

Original: Meanwhile, from (10), we expect that Adam can exploit a large batch size b ⋆ A > b ∗ A > b ∗∗ A , for example, when b ∗∗ A < 2 11 (CIFAR-10; n = 50000 , ResNet-20; d ≈ 1 . 1 × 10 7 , b ∗∗ A ≈ 1600 ) and 2 10 <
Revised: Meanwhile, from (10), we expect that Adam can exploit a large batch size b ⋆ A > b ∗ A > b ∗∗ A , for example, when b ∗∗ A < 2 11 (CIFAR-10; n = 50000 , ResNet-20; d ≈ 1 . 1 × 10 7 , b ∗∗ A ≈ 1600 ) and 2 10 <

Original: b
Revised: b

Original: ∗∗
Revised: ∗∗

Original: A < 2 11 (MNIST; n = 60000 , ResNet-18; d ≈ 1 . 

Revised: A < 2 11 (MNIST; n = 60000 , ResNet-18; d ≈ 1 .

Original: × 10 7 , b ∗∗ A ≈ 2000 )
Revised: × 10 7 , b ∗∗ A ≈ 2000 )

Original: (He et al., 2016;
Revised: (He et al., 2016;

Original: Leong et al., 2020), where σ 2A ≈ 10 − 2 and ϵ ≈ 10 − 3 are used.
Revised: Leong et al., 2020), where σ 2A ≈ 10 − 2 and ϵ ≈ 10 − 3 are used.

Original: 4. R ES N ET -20 ON THE CIFAR-10 DATASET
Revised: 4. R ES N ET -20 ON THE CIFAR-10 DATASET

Original: [Figure]
Revised: [Figure]

Original: There such that the number of steps K for Adam is inversely proportional to batch size b .
Revised: There such that the number of steps K for Adam is inversely proportional to batch size b .

Original: Adam has critical batch size b ⋆ = 2 11 .
Revised: Adam has critical batch size b ⋆ = 2 11 .

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: S F O ×

Revised: S F O ×

Original: SGDMomentumAdam
Revised: SGDMomentumAdam

Original: Let us consider training ResNet-20 on the CIFAR-10 dataset with n = 50000 .
Revised: Let us consider training ResNet-20 on the CIFAR-10 dataset with n = 50000 .

Original: Figure 1 shows the number of steps for SGD, Momentum, and Adam versus batch size.
Revised: Figure 1 shows the number of steps for SGD, Momentum, and Adam versus batch size.

Original: For SGD and Momentum, the number of steps K needed for f ( θ K ) ≤ 10 − 1 initially decreased.
Revised: For SGD and Momentum, the number of steps K needed for f ( θ K ) ≤ 10 − 1 initially decreased.

Original: However, SGD with b ≥ 2 6 and
Revised: However, SGD with b ≥ 2 6 and

Original: Momentum with b ≥ 2 10 did not satisfy f ( θ K ) ≤ 10 − 1 before the stopping condition was reached.
Revised: Momentum with b ≥ 2 10 did not satisfy f ( θ K ) ≤ 10 − 1 before the stopping condition was reached.

Original: Adam had an initial period of perfect scaling (indicated by dashed line) such that the number of steps K needed for f ( θ K ) ≤ 10 − 1 was inversely proportional to batch size b , and critical batch size b ⋆ = 2 11 such that K was not inversely proportional to the batch size beyond b ⋆ ; i.e., there were diminishing returns.
Revised: Adam had an initial period of perfect scaling (indicated by dashed line) such that the number of steps K needed for f ( θ K ) ≤ 10 − 1 was inversely proportional to batch size b , and critical batch size b ⋆ = 2 11 such that K was not inversely proportional to the batch size beyond b ⋆ ; i.e., there were diminishing returns.

Original: Figure 2 plots the SFO complexities for SGD, Momentum, and Adam versus batch size.
Revised: Figure 2 plots the SFO complexities for SGD, Momentum, and Adam versus batch size.

Original: For SGD,
Revised: For SGD,

Original: SFO complexity was minimum at b ⋆ = 2 2 ; for Momentum, it was minimum at b ⋆ = 2 3 .
Revised: SFO complexity was minimum at b ⋆ = 2 2 ; for Momentum, it was minimum at b ⋆ = 2 3 .

Original: This implies that SGD and Momentum could not use large batch sizes, as shown in (10).
Revised: This implies that SGD and Momentum could not use large batch sizes, as shown in (10).

Original: For Adam,
Revised: For Adam,

Original: SFO complexity was minimum at the critical batch size b ⋆ = 2 11 that was close to estimation of critical batch
Revised: SFO complexity was minimum at the critical batch size b ⋆ = 2 11 that was close to estimation of critical batch

Original: b ⋆ A > b ∗∗ A with 1600 ≈ b ∗∗ A < b ⋆ A = 2 11 = 2048 , as shown in (10).
Revised: b ⋆ A > b ∗∗ A with 1600 ≈ b ∗∗ A < b ⋆ A = 2 11 = 2048 , as shown in (10).

Original: We also checked that the elapsed time for Adam monotonically decreased for b ≤ 2 11 and that the elapsed time for critical batch size b ⋆ = 2 11 was the shortest.
Revised: We also checked that the elapsed time for Adam monotonically decreased for b ≤ 2 11 and that the elapsed time for critical batch size b ⋆ = 2 11 was the shortest.

Original: The elapsed time for b ≥ 2 12 increased with the SFO complexity, as shown in Figure 2 (see Tables 3, 4, 5, and 6 in Appendix A).
Revised: The elapsed time for b ≥ 2 12 increased with the SFO complexity, as shown in Figure 2 (see Tables 3, 4, 5, and 6 in Appendix A).

Original: 4. R ES N ET -18 ON THE MNIST DATASET
Revised: 4. R ES N ET -18 ON THE MNIST DATASET

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Let us consider training ResNet-18 on the MNIST dataset with n = 60000 .
Revised: Let us consider training ResNet-18 on the MNIST dataset with n = 60000 .

Original: Figures 3 and 4 indicate that Adam could exploit larger batch sizes than SGD and Momentum.
Revised: Figures 3 and 4 indicate that Adam could exploit larger batch sizes than SGD and Momentum.

Original: Moreover, these ﬁgures indicate that Momentum minimized SFO complexity at the critical batch size b ⋆ = 2 3 and Adam minimized SFO complexity at the critical batch size b ⋆ = 2 10 that was close to estimation of critical batch
Revised: Moreover, these ﬁgures indicate that Momentum minimized SFO complexity at the critical batch size b ⋆ = 2 3 and Adam minimized SFO complexity at the critical batch size b ⋆ = 2 10 that was close to estimation of critical batch

Original: b ⋆ A > b ∗∗ A with 2 10 < b ∗∗ A ≈ 2000 < 2 11 , as shown in (10).
Revised: b ⋆ A > b ∗∗ A with 2 10 < b ∗∗ A ≈ 2000 < 2 11 , as shown in (10).

Original: We can also check that the elapsed time for critical batch size b ⋆ = 2 10 was the shortest (see Tables 7, 8, 9, and 10 in Appendix A).
Revised: We can also check that the elapsed time for critical batch size b ⋆ = 2 10 was the shortest (see Tables 7, 8, 9, and 10 in Appendix A).

Original: We also estimated appropriate batch sizes of Adam for (i) CNN on MNIST ( n = 60000 , d ≈ 7 .
Revised: We also estimated appropriate batch sizes of Adam for (i) CNN on MNIST ( n = 60000 , d ≈ 7 .

Original: 7 × 10 6 ) and (ii) ResNet-32 on CIFAR-10 ( n = 50000 , d ≈ 2 . 

Revised: 7 × 10 6 ) and (ii) ResNet-32 on CIFAR-10 ( n = 50000 , d ≈ 2 .

Original: × 10 7 ) from b ∗∗ A in (10) and checked that actual critical batch sizes in (Zhang et al., 2019, Figure 5(a), (e)) are close to estimated batch sizes ((i) b ⋆ A = 2 11 = 2048 >
Revised: × 10 7 ) from b ∗∗ A in (10) and checked that actual critical batch sizes in (Zhang et al., 2019, Figure 5(a), (e)) are close to estimated batch sizes ((i) b ⋆ A = 2 11 = 2048 >

Original: b ∗∗ A ≈ 2000 , (ii) b ⋆ A = 2 10 = 1024 ≈ b ∗∗ A ≈ 1200 ).
Revised: b ∗∗ A ≈ 2000 , (ii) b ⋆ A = 2 10 = 1024 ≈ b ∗∗ A ≈ 1200 ).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper showed the relationship between batch size b and the number of steps K to achieve an ϵ -approximation of deep learning optimizers using a small constant learning rate α and hyperparameters β 1 and β 2 close to 1 .
Revised: This paper showed the relationship between batch size b and the number of steps K to achieve an ϵ -approximation of deep learning optimizers using a small constant learning rate α and hyperparameters β 1 and β 2 close to 1 .

Original: From the convexity of SFO complexity N , there exists a global minimizer b ⋆ of N that is the critical batch size.
Revised: From the convexity of SFO complexity N , there exists a global minimizer b ⋆ of N that is the critical batch size.

Original: We also gave numerical results indicating that Adam using a small constant learning rate, hyperparameters close to one, and the critical batch size has faster convergence than Momentum and SGD.
Revised: We also gave numerical results indicating that Adam using a small constant learning rate, hyperparameters close to one, and the critical batch size has faster convergence than Momentum and SGD.

Original: Moreover, we estimated appropriate batch sizes from our formula for b ⋆ and showed that actual critical batch sizes are close to estimated batch sizes.
Revised: Moreover, we estimated appropriate batch sizes from our formula for b ⋆ and showed that actual critical batch sizes are close to estimated batch sizes.

Original: A A PPENDIX
Revised: A A PPENDIX

Original: Unless stated otherwise all relationships between random variables are supposed to hold almost surely.
Revised: Unless stated otherwise all relationships between random variables are supposed to hold almost surely.

Original: Let S be a positive deﬁnite matrix which is denoted by S ∈ S d ++ .
Revised: Let S be a positive deﬁnite matrix which is denoted by S ∈ S d ++ .

Original: The S -inner product of
Revised: The S -inner product of

Original: R d is deﬁned for all x , y ∈ R d by ⟨ x , y ⟩ S :
Revised: R d is deﬁned for all x , y ∈ R d by ⟨ x , y ⟩ S :

Original: = ⟨ x , S y ⟩ = x ⊤ ( S y ) and the S -norm is deﬁned by ∥ x ∥ S :
Revised: = ⟨ x , S y ⟩ = x ⊤ ( S y ) and the S -norm is deﬁned by ∥ x ∥ S :

Original: = p ⟨ x , S x ⟩ .
Revised: = p ⟨ x , S x ⟩ .

Original: A. L EMMAS
Revised: A. L EMMAS

Original: Lemma A.1.
Revised: Lemma A.1.

Original: Suppose that (C1), (C2), and (C3) hold.
Revised: Suppose that (C1), (C2), and (C3) hold.

Original: Then, Adam satisﬁes the following: for all k ∈ N and all θ ∈ R d ,
Revised: Then, Adam satisﬁes the following: for all k ∈ N and all θ ∈ R d ,

Original: [Equation]
Revised: [Equation]

Original: where d k := − H − 1 k ˆ m k , ˆ β 1 := 1 − β 1 , and ˜ β 1 k := 1 − β k + .
Revised: where d k := − H − 1 k ˆ m k , ˆ β 1 := 1 − β 1 , and ˜ β 1 k := 1 − β k + .

Original: Proof.
Revised: Proof.

Original: Let θ ∈ R d and k ∈ N .
Revised: Let θ ∈ R d and k ∈ N .

Original: The deﬁnition θ k +1 :
Revised: The deﬁnition θ k +1 :

Original: = θ k + α d k implies that
Revised: = θ k + α d k implies that

Original: [Equation]
Revised: [Equation]

Original: Moreover
Revised: Moreover

Original: [Equation]
Revised: [Equation]

Original: the deﬁnitions of d k , m k
Revised: the deﬁnitions of d k , m k

Original: [Equation]
Revised: [Equation]

Original: and ˆ m k ensure that
Revised: and ˆ m k ensure that

Original: [Equation]
Revised: [Equation]

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: Conditions (C2) and (C3) guarantee that
Revised: Conditions (C2) and (C3) guarantee that

Original: [Equation]
Revised: [Equation]

Original: Therefore, the lemma follows from taking the expectation on both sides of (11).
Revised: Therefore, the lemma follows from taking the expectation on both sides of (11).

Original: This completes the proof.
Revised: This completes the proof.

Original: The discussion in the proof of Lemma A.1 also gives the following lemma.
Revised: The discussion in the proof of Lemma A.1 also gives the following lemma.

Original: Lemma A.2.
Revised: Lemma A.2.

Original: Suppose that (C1), (C2), and (C3) hold.
Revised: Suppose that (C1), (C2), and (C3) hold.

Original: Then, SGD satisﬁes the following: for all k ∈ N and all θ ∈ R d ,
Revised: Then, SGD satisﬁes the following: for all k ∈ N and all θ ∈ R d ,

Original: [Equation]
Revised: [Equation]

Original: Moreover, Momentum satisﬁes the following: for all k ∈ N and all θ ∈ R d ,
Revised: Moreover, Momentum satisﬁes the following: for all k ∈ N and all θ ∈ R d ,

Original: [Equation]
Revised: [Equation]

Original: where ˆ β 1 := 1 − β 1 .
Revised: where ˆ β 1 := 1 − β 1 .

Original: The following lemma indicates the bounds on ( E [ ∥ m k ∥ 2 ]) k ∈ N and ( E [ ∥ d k ∥ 2 H k ]) k ∈ N .
Revised: The following lemma indicates the bounds on ( E [ ∥ m k ∥ 2 ]) k ∈ N and ( E [ ∥ d k ∥ 2 H k ]) k ∈ N .

Original: Lemma A.3.
Revised: Lemma A.3.

Original: Adam under (C2) and (A1), for all k ∈ N satisﬁes
Revised: Adam under (C2) and (A1), for all k ∈ N satisﬁes

Original: [Equation]
Revised: [Equation]

Original: where v ∗ := inf { min i ∈
Revised: where v ∗ := inf { min i ∈

Original: [ d ] v k,i : k ∈ N } , ˜ β 1 k := 1 − β k +1 1 k , and ˜ β 2 k := 1 − β k +1 2 k .
Revised: [ d ] v k,i : k ∈ N } , ˜ β 1 k := 1 − β k +1 1 k , and ˜ β 2 k := 1 − β k +1 2 k .

Original: Proof.
Revised: Proof.

Original: Condition (C2) implies that
Revised: Condition (C2) implies that

Original: [Equation]
Revised: [Equation]

Original: which, together with (C2) and (A1), in turn implies that
Revised: which, together with (C2) and (A1), in turn implies that

Original: [Equation]
Revised: [Equation]

Original: The convexity of ∥ · ∥ 2 , together with the deﬁnition of m k and (13), guarantees that, for all k ∈ N ,
Revised: The convexity of ∥ · ∥ 2 , together with the deﬁnition of m k and (13), guarantees that, for all k ∈ N ,

Original: [Equation]
Revised: [Equation]

Original: Induction thus ensures that, for all k ∈ N ,
Revised: Induction thus ensures that, for all k ∈ N ,

Original: [Equation]
Revised: [Equation]

Original: where m − 1 = 0 .
Revised: where m − 1 = 0 .

Original: For k ∈ N , H k ∈ S d
Revised: For k ∈ N , H k ∈ S d

Original: +
Revised: +

Original: + guarantees the existence of a unique matrix
Revised: + guarantees the existence of a unique matrix

Original: H k ∈ S d ++ such that H k
Revised: H k ∈ S d ++ such that H k

Original: = H 2 k (Horn & Johnson, 1985, Theorem 7.2.6).
Revised: = H 2 k (Horn & Johnson, 1985, Theorem 7.2.6).

Original: We have that, for all x ∈ R d ,
Revised: We have that, for all x ∈ R d ,

Original: ∥ x ∥ 2 H k = ∥
Revised: ∥ x ∥ 2 H k = ∥

Original: H k x ∥ 2 .
Revised: H k x ∥ 2 .

Original: Accordingly, the deﬁnitions of d k and ˆ m k imply that, for all k ∈ N ,
Revised: Accordingly, the deﬁnitions of d k and ˆ m k imply that, for all k ∈ N ,

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Moreover, the deﬁnition of
Revised: Moreover, the deﬁnition of

Original: [Equation]
Revised: [Equation]

Original: inf {
Revised: inf {

Original: [Equation]
Revised: [Equation]

Original: k,i
Revised: k,i

Original: [Equation]
Revised: [Equation]

Original: : k ∈
Revised: : k ∈

Original: [Equation]
Revised: [Equation]

Original: } and (14) imply that, for all k ∈ N ,
Revised: } and (14) imply that, for all k ∈ N ,

Original: [Equation]
Revised: [Equation]

Original: completing the proof.
Revised: completing the proof.

Original: A. SGD
Revised: A. SGD

Original: We show Theorem 3.1 for SGD.
Revised: We show Theorem 3.1 for SGD.

Original: Proof.
Revised: Proof.

Original: (i) Lemma A.2 and (13) imply that, for all k ∈ N and all θ ∈ R d ,
Revised: (i) Lemma A.2 and (13) imply that, for all k ∈ N and all θ ∈ R d ,

Original: [Equation]
Revised: [Equation]

Original: Summing the above inequality from k = 1 to k = K leads to the ﬁnding that, for all K ≥ 1 , 2 α
Revised: Summing the above inequality from k = 1 to k = K leads to the ﬁnding that, for all K ≥ 1 , 2 α

Original: K X
Revised: K X

Original: [Equation]
Revised: [Equation]

Original: , which implies that, for all K ≥ 1 and all θ ∈ R d ,
Revised: K, which implies that, for all K ≥ 1 and all θ ∈ R d ,

Original: [Equation]
Revised: [Equation]

Original: (ii) Let θ ∈ R d and ϵ > 0 .
Revised: (ii) Let θ ∈ R d and ϵ > 0 .

Original: Condition C 1 /K + C 2 /b + C 3 =
Revised: Condition C 1 K + C 2 b + C 3 =

Original: ϵ is equivalent to
Revised: ϵ is equivalent to

Original: [Equation]
Revised: [Equation]

Original: Since ϵ = C 1 /K + C 2 /b + C 3 > C 3 , we consider the case b > C 2/ ( ϵ − C 3) > 0 to guarantee that K > 0 .
Revised: Since ϵ = C 1 K + C 2 b + C 3 > C 3 , we consider the case b > C 2 ϵ − C 3 > 0 to guarantee that K > 0 .

Original: From (15), the function K deﬁned by (16) satisﬁes VI( K, θ ) ≤ C 1 /K
Revised: From (15), the function K deﬁned by (16) satisﬁes VI( K, θ ) ≤ C 1 K + C 2 b

Original: + C [-2 /b + C 
-] = ϵ .
Revised: + C [-2 /b + C 
-]3 = ϵ .

Original: Moreover, from (16), 
Revised: Moreover, from (16), d K

Revised: [Equation]

Revised: b

Original: [Equation]
Revised: [Equation]

Original: which implies that K is convex and monotone decreasing with respect to b .
Revised: which implies that K is convex and monotone decreasing with respect to b .

Original: (iii)
Revised: (iii)

Original: We have that
Revised: We have that

Original: [Equation]
Revised: [Equation]

Original: Accordingly,
Revised: Accordingly,

Original: [Equation]
Revised: [Equation]

Original: which implies that K ( b ) b is convex with respect to b and
Revised: which implies that K ( b ) b is convex with respect to b and

Original: [Equation]
Revised: [Equation]

Original: The point b ⋆ attains the minimum value K ( b ⋆ ) b ⋆ = 2 C 1 C 2/ ( ϵ − C 3 ) 2 of K ( b ) b .
Revised: The point b ⋆ attains the minimum value K ( b ⋆ ) b ⋆ = 2 C 1 C 2 ( ϵ − C 3 ) 2 of K ( b ) b .

Original: This completes the proof.
Revised: This completes the proof.

Original: A. M OMENTUM
Revised: A. M OMENTUM

Original: We show Theorem 3.1 for Momentum.
Revised: We show Theorem 3.1 for Momentum.

Original: Proof.
Revised: Proof.

Original: (i) Lemma A.2 ensures that, for all k ∈ N and all θ ∈ R d ,E (cid:2)
Revised: (i) Lemma A.2 ensures that, for all k ∈ N and all θ ∈ R d ,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: which, together with Lemma A.3, the Cauchy–Schwarz inequality, and (A1) and (A2), implies that
Revised: which, together with Lemma A.3, the Cauchy–Schwarz inequality, and (A1) and (A2), implies that

Original: E (cid:2) ( θ k − θ )
Revised: E (cid:2) ( θ k − θ )

Original: ⊤ m k − 1 (cid:3) ≤ 2 αβn
Revised: ⊤ m k − 1 (cid:3) ≤ 2 αβn

Original: E h ∥
Revised: E h ∥

Original: θ k

Original: [Equation]

Original: θ  ∥i − Eh ∥ θ k
Revised: θ k − θ ∥i − E

Original: [Equation]

Original: θ ∥

Original: [Equation]

Original: io

Original: [Equation]
Revised: [Equation]

Original: α

Original: [Equation]

Original: + ˆ

Original: [Equation]

Original: β 1 Dist( θ ) G.

Original: Summing the above inequality from k = 1 to k = K gives a relation that implies that
Revised: Summing the above inequality from k = 1 to k = K gives a relation that implies that

Original: K X
Revised: K X

Original: [Equation]
Revised: [Equation]

Original: and hence,
Revised: and hence,

Original: [Equation]
Revised: [Equation]

Original: Moreover, we have that, for all k ∈ N and all θ ∈ R d ,
Revised: Moreover, we have that, for all k ∈ N and all θ ∈ R d ,

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst equality comes from the deﬁnition of m k and the ﬁrst inequality comes from the
Revised: where the ﬁrst equality comes from the deﬁnition of m k and the ﬁrst inequality comes from the

Original: Cauchy–Schwarz inequality, the triangle inequality
Revised: Cauchy–Schwarz inequality, the triangle inequality

Original: [Equation]
Revised: [Equation]

Original: and (A2).
Revised: and (A2).

Original: Hence, from Lemma A.3, (13),
Revised: Hence, from Lemma A.3, (13),

Original: Jensen’s inequality
Revised: Jensen’s inequality

Original: [Equation]
Revised: [Equation]

Original: and b ≥ 1 ,
Revised: and b ≥ 1 ,

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: for all K ≥ 1 and all θ ∈ R d ,
Revised: for all K ≥ 1 and all θ ∈ R d ,

Original: [Equation]
Revised: [Equation]

Original: The deﬁnition of m k ensures that
Revised: The deﬁnition of m k ensures that

Original: [Equation]
Revised: [Equation]

Original: which, together with the Cauchy–Schwarz inequality, the triangle inequality
Revised: which, together with the Cauchy–Schwarz inequality, the triangle inequality

Original: [Equation]
Revised: [Equation]

Original: and (A2)
Revised: and (A2)

Original: [Equation]
Revised: [Equation]

Original: implies that
Revised: implies that

Original: [Equation]
Revised: [Equation]

Original: Lemma A.3, (13), Jensen’s inequality
Revised: Lemma A.3, (13), Jensen’s inequality

Original: [Equation]
Revised: [Equation]

Original: and b ≥ 1 guarantee that
Revised: and b ≥ 1 guarantee that

Original: [Equation]
Revised: [Equation]

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original: + G
Revised: + G

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: (18) ensures that, for all K ≥ 1 and all θ ∈ R d ,
Revised: (18) ensures that, for all K ≥ 1 and all θ ∈ R d ,

Original: [Equation]
Revised: [Equation]

Original: (ii) A discussion similar to the one showing (ii) in Theorem 3.1 for SGD would show (ii) in Theorem 3.1 for Momentum.
Revised: (ii) A discussion similar to the one showing (ii) in Theorem 3.1 for SGD would show (ii) in Theorem 3.1 for Momentum.

Original: (iii)
Revised: (iii)

Original: An argument similar to that which obtained (iii) in Theorem 3.1 for SGD would prove (iii) in Theorem 3.1 for Momentum.
Revised: An argument similar to that which obtained (iii) in Theorem 3.1 for SGD would prove (iii) in Theorem 3.1 for Momentum.

Original: A. A DAM
Revised: A. A DAM

Original: We show Theorem 3.1 for Adam.
Revised: We show Theorem 3.1 for Adam.

Original: Proof.
Revised: Proof.

Original: (i) Let θ ∈ R d .
Revised: (i) Let θ ∈ R d .

Original: Lemma A.1 guarantees that for all k ∈ N ,
Revised: Lemma A.1 guarantees that for all k ∈ N ,

Original: [Equation]
Revised: [Equation]

Original: We deﬁne γ k := ˜ 
Revised: We deﬁne γ k := ˜ β

Original: β 1 k / (2 β 1 α) ( k ∈ N ) .
Revised: 1 k 2 β 1 α ( k ∈ N ) .

Original: Then, for all K ≥ 1 ,
Revised: Then, for all K ≥ 1 ,

Original: K X k =
Revised: K X k =

Original: [Equation]
Revised: [Equation]

Original: Since H k ∈ S d ++ exists such that H k
Revised: Since H k ∈ S d ++ exists such that H k

Original: = H 2 k , we have ∥ x
Revised: = H 2 k , we have ∥ x

Original: ∥ 2 H k = ∥
Revised: ∥ 2 H k = ∥

Original: H k x ∥ 2 for all x ∈ R d .
Revised: H k x ∥ 2 for all x ∈ R d .

Original: Accordingly, we also have
Revised: Accordingly, we also have

Original: [Equation]
Revised: [Equation]

Original: H k ( θ k − θ ) (cid:13)(cid:13) 2 − γ k − 1 (cid:13)(cid:13)
Revised: H k ( θ k − θ ) (cid:13)(cid:13) 2 − γ k − 1 (cid:13)(cid:13)

Original: H k − 1 ( θ k − θ ) (cid:13)(cid:13)o# .
Revised: H k − 1 ( θ k − θ ) (cid:13)(cid:13)o# .

Original: From H k = diag (ˆ v 1 / 4 k,i ) , we have that, for all x = ( x i ) di =1 ∈ R d , ∥ H k x ∥ 2 = P di = p ˆ v k,i x 2 i .
Revised: From H k = diag (ˆ v 1 / 4 k,i ) , we have that, for all x = ( x i ) di =1 ∈ R d , ∥ H k x ∥ 2 = P di = p ˆ v k,i x 2 i .

Original: Hence, for all K ≥ 2 , Γ K = E " K X
Revised: Hence, for all K ≥ 2 , Γ K = E " K X

Original: k
Revised: k

Original: =
Revised: =

Original: d X i = (cid:16) γ k p ˆ v k,i − γ k −p ˆ v k − 1 ,i (cid:17) ( θ k,i − θ i )# .
Revised: d X i = (cid:16) γ k p ˆ v k,i − γ k −p ˆ v k − 1 ,i (cid:17) ( θ k,i − θ i )# .

Original: Condition (2) and γ k ≥ γ k − 1 ( k ≥ 1 ) imply that, for all k ≥ 1 and all i ∈ [ d ] , γ k p ˆ v k,i − γ k −p ˆ v k − 1 ,i ≥ 0 .
Revised: Condition (2) and γ k ≥ γ k − 1 ( k ≥ 1 ) imply that, for all k ≥ 1 and all i ∈ [ d ] , γ [Equation] p [Equation]

Original:  Moreover, (A2) ensures that D ( θ ) :
Revised: − [Equation] ≥ 0 . Moreover, (A2) ensures that D ( θ ) :

Original: = sup { max i ∈
Revised: = sup { max i ∈

Original: [ d ] ( θ k,i − θ i ) 2 : k ∈ N } < + ∞ .
Revised: [ d ] ( θ k,i − θ i ) 2 : k ∈ N } < + ∞ .

Original: Accordingly, for all K ≥ 2 , Γ K ≤ D ( θ ) E " K
Revised: Accordingly, for all K ≥ 2 , Γ K ≤ D ( θ ) E " K

Original: X
Revised: X

Original: k
Revised: k

Original: =
Revised: =

Original: d X i = (cid:16) γ k p ˆ v k,i − γ k −p ˆ v k − 1 ,i (cid:17)
Revised: d X i = (cid:16) γ k p ˆ v k,i − γ k −p ˆ v k − 1 ,i (cid:17)

Original: # = D ( θ ) E " d
Revised: # = D ( θ ) E " d

Original: X i = (cid:16) γ K p ˆ v K,i − γp ˆ v 1 ,i (cid:17)# .
Revised: X i = (cid:16) γ K p ˆ v K,i − γp ˆ v 1 ,i (cid:17)# .

Original: Let ∇ f B k
Revised: Let ∇ f B k

Original: ( θ k ) ⊙ ∇ f B k ( θ k ) :
Revised: ( θ k ) ⊙ ∇ f B k ( θ k ) :

Original: = ( g 2 k,i
Revised: = ( g 2 k,i

Original: ) ∈
Revised: ) ∈

Original: R d + .
Revised: R d + .

Original: Assumption (A1) ensures that there exists M ∈ R such that, for all k ∈ N , max i ∈
Revised: Assumption (A1) ensures that there exists M ∈ R such that, for all k ∈ N , max i ∈

Original: [ d ] g 2 k,i ≤ M .
Revised: [ d ] g 2 k,i ≤ M .

Original: The deﬁnition of v k guarantees that, for all i ∈
Revised: The deﬁnition of v k guarantees that, for all i ∈

Original: [ d ] and all k ∈ N ,
Revised: [ d ] and all k ∈ N ,

Original: [Equation]
Revised: [Equation]

Original: Induction thus ensures that, for all i ∈
Revised: Induction thus ensures that, for all i ∈

Original: [ d ] and all k ∈ N ,
Revised: [ d ] and all k ∈ N ,

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where v 0 = ( v 0 ,i ) = 0 .
Revised: where v 0 = ( v 0 ,i ) = 0 .

Original: From the deﬁnition of ˆ v k , we have that, for all i ∈ [ d ] and all k ∈ N ,
Revised: From the deﬁnition of ˆ v k , we have that, for all i ∈ [ d ] and all k ∈ N ,

Original: [Equation]
Revised: [Equation]

Original: Therefore, (21), E [ ∥ θ 1 − θ ∥ 2 H 1 ] ≤ D ( θ ) E
Revised: Therefore, (21), E [ ∥ θ 1 − θ ∥ 2 H 1 ] ≤ D ( θ ) E

Original: [ P di =1 p ˆ v 1 ,i ] , and (24) imply, for all K ≥ 1 ,
Revised: [ P di =1 p ˆ v 1 ,i ] , and (24) imply, for all K ≥ 1 ,

Original: K X k =
Revised: K X k =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: Inequality (25) with ˜ β 1 K := 

Revised: Inequality (25) with ˜ β 1 K :=

Original: − β K +1≤ 1 and ˜ β 2 K := 1 − β K +1≥ 1 − β 2 =: ˆ β 2 implies that
Revised: − β K +1≤ 1 and ˜ β 2 K := 1 − β K +1≥ 1 − β 2 =: ˆ β 2 implies that

Original: K X k =
Revised: K X k =

Original: [Equation]
Revised: [Equation]

Original: Lemma A.3 guarantees that, for all k ∈ N ,
Revised: Lemma A.3 guarantees that, for all k ∈ N ,

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Inequality (27) with ˜ β 1 k := 1 − β k + ≥ 1 − β 

Revised: Inequality (27) with ˜ β 1 k := 1 − β k + ≥ 1 − β

Original: =: ˆ β 1 and ˜ β 2 k := 1 − β k + ≤ 1 implies that
Revised: =: ˆ β 1 and ˜ β 2 k := 1 − β k + ≤ 1 implies that

Original: [Equation]
Revised: [Equation]

Original: The Cauchy–Schwarz inequality and (A2) imply that, for all k ∈ N ,
Revised: The Cauchy–Schwarz inequality and (A2) imply that, for all k ∈ N ,

Original: [Equation]
Revised: [Equation]

Original: Hence, (20), (26), (28), and (29) ensure that, for all K ≥ 1 ,
Revised: Hence, (20), (26), (28), and (29) ensure that, for all K ≥ 1 ,

Original: [Equation]
Revised: [Equation]

Original: Therefore, from (17), for all K ≥ 1 ,
Revised: Therefore, from (17), for all K ≥ 1 ,

Original: [Equation]
Revised: [Equation]

Original: From (19) and (30), for all K ≥ 1 ,
Revised: From (19) and (30), for all K ≥ 1 ,

Original: [Equation]
Revised: [Equation]

Original: (ii) A discussion similar to the one showing (ii) in Theorem 3.1 for SGD would show (ii) in Theorem 3.1 for Adam.
Revised: (ii) A discussion similar to the one showing (ii) in Theorem 3.1 for SGD would show (ii) in Theorem 3.1 for Adam.

Original: (iii)
Revised: (iii)

Original: An argument similar to that which obtained (iii) in Theorem 3.1 for SGD would prove (iii) in 
Revised: An argument similar to that which obtained (iii) in Theorem 3.1 for SGD would prove (iii) in Theorem 3.1 for Adam.

Revised: A. R ESULTS FOR VARYING LEARNING RATES

Revised: We can establish the upper bound of VI( K, θ ) for Adam with varying learning rates from a discussion similar to the one for proving Theorem 3.1(i).

Revised: If ( α k ) k ∈ N is monotone decreasing, then we have

Original: Theorem 3.1 for Adam.
Revised: [Equation] Let us consider the case where α k = 1 √ k ( k ≥ 1) . From (31) [Equation]

Revised: we have that

Revised: [Equation]

Revised: , where 1 − β k + ≤ 1 and P Kk =1 1 √ k ≤√

Revised: K .

Revised: Hence,

Revised: [Equation]

Revised: Theorem 3.1(i) indicates that Adam with a constant learning rate α satisﬁes

Revised: [Equation]

Revised: Hence, using α k = 1 √ k mitigates the variance (the term ˜ C 2 b √ K ) in contrast to the term C 2 b using α k = α .

Revised: However, the bias term C 1 K using α k

Revised: = α would be better than the bias term ˜ C 1 √ K using α k = 1 √ k in the sense of minimizing the upper bound of VI( K, θ ) .

Revised: Let T ≥ 1 , γ ∈ (0 , 1) , α > 0 , and P := KT .

Revised: Next, let us consider the following varying learning rate

Revised: [Equation]

Revised: where α jT = ( γ j − 1 α, γ j − 1 α, . .

Revised: , γ j − 1 α ) ( j = 1 , 2 , . . .

Revised: , P ) .

Revised: That is, ( α k ) is (( α, α, . . .

Original: A. A DDITIONAL NUMERICAL RESULTS
Revised: , α ) , ( γα, γα, . . . , γα ) , . . , ( γ P − 1 α, γ P − 1 α, . . , γ P − 1 α )) .

Revised: Let α > 0 be the lower bound of α k .

Revised: From

Revised: [Equation]

Revised: and (31), we have that

Revised: [Equation]

Revised: Here, we recall that Adam with α k = α satisﬁes

Revised: [Equation]

Revised: and Adam with α k = 1 √ k satisﬁes

Original: [Table]
Revised: [Equation]

Original: [Table]
Revised: Therefore, using [Equation] is more desirable to minimize the upper bound of VI( K, θ ) for Adam than using α k = α, 1 √ k .

Revised: A. A DDITIONAL NUMERICAL RESULTS

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]