Original: C RITICAL B ATCH S IZE M INIMIZES S TOCHASTIC F IRST -O RDER O RACLE C OMPLEXITY OF D EEP L EARNING O PTIMIZER USING H YPERPARAMETERS C LOSE TO O NE
Revised: C RITICAL B ATCH S IZE M INIMIZES S TOCHASTIC F IRST O RDER O RACLE C OMPLEXITY OF D EEP L EARNING O PTIMIZER USING H YPERPARAMETERS C LOSE TO O NE

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Practical results have shown that deep learning optimizers using small constant learning rates, hyperparameters close to one, and large batch sizes can ﬁnd the model parameters of deep neural networks that minimize the loss functions.
Revised: Practical results have shown that deep learning optimizers using small constant learning rates, hyperparameters close to one, and large batch sizes can ﬁnd the model parameters of deep neural networks that minimize the loss functions.

Original: We ﬁrst show theoretical evidence that the momentum method (Momentum) and adaptive moment estimation (Adam) perform well in the sense that the upper bound of the theoretical performance measure is small with a small constant learning rate, hyperparameters close to one, and a large batch size.
Revised: We ﬁrst show theoretical evidence that the momentum method (Momentum) and adaptive moment estimation (Adam) perform well in the sense that the upper bound of the theoretical performance measure is small with a small constant learning rate, hyperparameters close to one, and a large batch size.

Original: Next, we show that there exists a batch size called the critical batch size minimizing the stochastic ﬁrst-order oracle (SFO) complexity, which is the stochastic gradient computation cost, and that SFO complexity increases once the batch size exceeds the critical batch size.
Revised: Next, we show that there exists a batch size called the critical batch size minimizing the stochastic ﬁrst-order oracle (SFO) complexity, which is the stochastic gradient computation cost, and that SFO complexity increases once the batch size exceeds the critical batch size.

Original: Finally, we provide numerical results that support our theoretical results.
Revised: Finally, we provide numerical results that support our theoretical results.

Original: That is, the numerical results indicate that Adam using a small constant learning rate, hyperparameters close to one, and the critical batch size minimizing
Revised: That is, the numerical results indicate that Adam using a small constant learning rate, hyperparameters close to one, and the critical batch size minimizing

Original: SFO complexity has faster convergence than Momentum and stochastic gradient descent (SGD).
Revised: SFO complexity has faster convergence than Momentum and stochastic gradient descent (SGD).

Original:  I NTRODUCTION 1. B ACKGROUND
Revised:  I NTRODUCTION 1. B ACKGROUND

Original: Useful deep learning optimizers have been proposed to ﬁnd the model parameters of the deep neural networks that minimize loss functions called the expected risk and empirical risk, such as stochastic gradient descent (SGD) (Robbins & Monro, 1951; Zinkevich, 2003; Nemirovski et al., 2009; Ghadimi & Lan, 2012; 2013), momentum methods (Polyak, 1964; Nesterov, 1983), and adaptive methods.
Revised: Useful deep learning optimizers have been proposed to ﬁnd the model parameters of the deep neural networks that minimize loss functions called the expected risk and empirical risk, such as stochastic gradient descent (SGD) (Robbins & Monro, 1951; Zinkevich, 2003; Nemirovski et al., 2009; Ghadimi & Lan, 2012; 2013), momentum methods (Polyak, 1964; Nesterov, 1983), and adaptive methods.

Original: The various adaptive methods include Adaptive Gradient (AdaGrad) (Duchi et al., 2011), Root Mean Square Propagation (RMSProp) (Tieleman & Hinton, 2012), Adaptive Moment Estimation (Adam) (Kingma & Ba, 2015), Adaptive Mean Square Gradient (AMSGrad) (Reddi et al., 2018), Yogi (Zaheer et al., 2018), Adam with decoupled weight decay (AdamW) (Loshchilov & Hutter, 2019), and AdaBelief (named for adapting stepsizes by the belief in observed gradients)
Revised: The various adaptive methods include Adaptive Gradient (AdaGrad) (Duchi et al., 2011), Root Mean Square Propagation (RMSProp) (Tieleman & Hinton, 2012), Adaptive Moment Estimation (Adam) (Kingma & Ba, 2015), Adaptive Mean Square Gradient (AMSGrad) (Reddi et al., 2018), Yogi (Zaheer et al., 2018), Adam with decoupled weight decay (AdamW) (Loshchilov & Hutter, 2019), and AdaBelief (named for adapting stepsizes by the belief in observed gradients)

Original: (Zhuang et al., 2020).
Revised: (Zhuang et al., 2020).

Original: Theoretical analyses of adaptive methods for nonconvex optimization were presented in (Zaheer et al., 2018;
Revised: Theoretical analyses of adaptive methods for nonconvex optimization were presented in (Zaheer et al., 2018;

Original: Zou et al., 2019; Chen et al., 2019;
Revised: Zou et al., 2019; Chen et al., 2019;

Original: Zhou et al., 2020; Zhuang et al., 2020; Chen et al.,2021; Iiduka, 2021) (see (Fehrman et al., 2020; Chen et al., 2020; Scaman & Malherbe, 2020; Loizou et al., 2021) for convergence analyses of SGD for nonconvex optimization).
Revised: Zhou et al., 2020; Zhuang et al., 2020; Chen et al., 2021) (see (Jain et al., 2018; Fehrman et al., 2020; Chen et al., 2020; Scaman & Malherbe, 2020; Loizou et al., 2021) for convergence analyses of SGD).

Original: A particularly interesting feature of adaptive methods is the use of hyperparameters, denoted by β 1 and β 2 , that can be set to inﬂuence the method performance.
Revised: A particularly interesting feature of adaptive methods is the use of hyperparameters, denoted by β 1 and β 2 , that can be set to inﬂuence the method performance P( K ) :

Original: Almost all of the previous results for convergence analyses of adaptive methods showed that using β 1 and β 2 close to 0 makes the upper bound of the performance measure (1 /K )

Original:  P Kk =1 E [ ∥∇ f ( θ k ) ∥ 2 ] small, where ∇ f is the gradient of a loss function f : R d → R , ( θ k )
Revised: = 1 K P K k =1 E [ ∥∇ f ( θ k ) ∥ 2 ] , where ∇ f is the gradient of a loss function f : R d → R , ( θ k )

Original: Kk =1 is the sequence generated by an optimizer, and K is the number of steps.
Revised: Kk =1 is the sequence generated by an optimizer, and K is the number of steps.

Revised: The previous results are summarized in Table 1 indicating that using β 1 and/or β 2 close to 0 makes the upper bound of P( K ) small.

Original: Meanwhile, practical results for adaptive methods were presented in (Kingma & Ba, 2015;
Revised: Meanwhile, practical results for adaptive methods were presented in (Kingma & Ba, 2015;

Original: Reddi et al., 2018;
Revised: Reddi et al., 2018;

Original: Zaheer et al., 2018; Zou et al., 2019; Chen et al., 2019; Zhuang et al., 2020;
Revised: Zaheer et al., 2018; Zou et al., 2019; Chen et al., 2019; Zhuang et al., 2020;

Original: Chen et al., 2021).
Revised: Chen et al., 2021).

Original: These studies have shown that using, for example, β 1 ∈ { 0 . 9 , 0 . 99 } and β 2 ∈ { 0 . 99 , 0 . 999 } provides superior performance for training deep neural networks.
Revised: These studies have shown that using, for example, β 1 ∈ { 0 . 9 , 0 . 99 } and β 2 ∈ { 0 . 99 , 0 . 999 } provides superior performance for training deep neural networks.

Original: The practically useful β 1 and β 2 are each close to 1 , whereas in contrast, the theoretical results (the above paragraph) show that using β 1 and β 2 close to 0 makes the upper bounds of the performance measures small. 
Revised: The practically useful β 1 and β 2 are each close to 1 , whereas in contrast, the theoretical results (Table 1) show that using β 1 and/or β 2 close to 0 makes the upper bounds of the performance measures small. [Table] (Chen et al., 2019) AMSGrad

Revised: [Equation]

Revised: (Zhou et al., 2020)

Original: The practical performance of a deep learning optimizer strongly depends on the batch size.
Revised: AdaBelief [Equation] (Zhuang et al., 2020) Padam

Revised: [Equation]

Revised: (Chen et al., 2021)

Revised: Adam,

Revised: [Table]

Revised: [Equation]

Revised: (this paper)

Revised: The practical performance of a deep learning optimizer strongly depends on the batch size.

Original: In (Smith et al., 2018), it was numerically shown that using an enormous batch size leads to a reduction in the number of parameter updates and model training time.
Revised: In (Smith et al., 2018), it was numerically shown that using an enormous batch size leads to a reduction in the number of parameter updates and model training time.

Original: The theoretical results in (Zaheer et al., 2018) showed that using large batch sizes makes the upper bound of E [ ∥∇ f ( θ k )∥ 2 ] of an adaptive method small.
Revised: The theoretical results in (Zaheer et al., 2018) showed that using large batch sizes makes the upper bound of P( K ) of an adaptive method small (Table 1).

Original: Convergence analyses of SGD in (Chen et al., 2020) indicated that running SGD with a decaying learning rate and large batch size for sufﬁciently many steps leads to convergence to a local minimizer of a loss function.
Revised: Convergence analyses of SGD in (Cotter et al., 2011; Chen et al., 2020; Arjevani et al., 2022) indicated that running SGD with a decaying learning rate and large batch size for sufﬁciently many steps leads to convergence to a local minimizer of a loss function.

Original: Accordingly, the practical results for large batch sizes match the theoretical ones.
Revised: Accordingly, the practical results for large batch sizes match the theoretical ones.

Original: In (Shallue et al., 2019; Zhang et al., 2019), it was studied how increasing the batch size affects the performances of deep learning optimizers.
Revised: In (Shallue et al., 2019; Zhang et al., 2019), it was studied how increasing the batch size affects the performances of deep learning optimizers.

Original: In both studies, it was numerically shown that increasing batch size tends to decrease the number of steps K needed for training deep neural networks, but with diminishing returns.
Revised: In both studies, it was numerically shown that increasing batch size tends to decrease the number of steps K needed for training deep neural networks, but with diminishing returns.

Original: Moreover, it was shown that momentum methods can exploit larger batches than SGD (Shallue et al., 2019), and that K-FAC and Adam can exploit larger batches than momentum methods (Zhang et al., 2019).
Revised: Moreover, it was shown that momentum methods can exploit larger batches than SGD (Shallue et al., 2019), and that K-FAC and Adam can exploit larger batches than momentum methods (Zhang et al., 2019).

Original: Thus, it was shown that momentum and adaptive methods can signiﬁcantly reduce the number of steps K needed for training deep neural networks (Shallue et al., 2019, Figure 4), (Zhang et al., 2019, Figure 5).

Original: 1. M OTIVATION 1.2. H YPERPARAMETERS CLOSE TO ONE AND CONSTANT LEARNING RATE
Revised: 1. M OTIVATION 1.2. H YPERPARAMETERS CLOSE TO ONE AND CONSTANT LEARNING RATE

Original: As described in Section 1.1, the practically useful β 1 and β 2 are each close to 1 , whereas in contrast, the theoretical results show that using β 1 and β 2 close to 0 makes the upper bounds of the performance measures small.
Revised: As described in Section 1.1, the practically useful β 1 and β 2 are each close to 1 , whereas in contrast, the theoretical results show that using β 1 and/or β 2 close to 0 makes the upper bounds of the performance measures small.

Original: Hence, there is a gap between theory ( β 1 , β 2 ≈ 0 ) and practice ( β 1 , β 2 ≈ 1 ) for adaptive methods.
Revised: Hence, there is a gap between theory ( β 1 , β 2 ≈ 0 ) and practice ( β 1 , β 2 ≈ 1 ) for adaptive methods.

Original: As a consequence, the ﬁrst motivation of this paper is to bridge this gap.
Revised: As a consequence, the ﬁrst motivation of this paper is to bridge this gap.

Original: Since using small constant learning rates(e.g., 10 − 2 , 10 − 3 , and 10 − 4 ) is robust for training deep neural networks (Kingma & Ba, 2015; Reddi et al., 2018;
Revised: Since using small constant learning rates is robust for training deep neural networks (Kingma & Ba, 2015; Reddi et al., 2018;

Original: Zaheer et al., 2018; Zou et al., 2019; 
Revised: Zaheer et al., 2018; Zou et al., 2019; Chen et al., 2019; Zhuang et al., 2020;

Original: Chen et al.,2019; Zhuang et al., 2020; Chen et al., 2021), we focus on using a small constant learning rate α .
Revised: Chen et al., 2021), we focus on using a small constant learning rate α .

Original: Here, we note that using a learning rate depending on the Lipschitz constant L of the gradient of a loss function f would be unrealistic.
Revised: Here, we note that using a learning rate depending on the Lipschitz constant L of the gradient of a loss function f would be unrealistic.

Original: This is because computing the Lipschitz constant L is NP-hard (Virmaux & Scaman, 2018, Theorem 2).
Revised: This is because computing the Lipschitz constant L is NP-hard (Virmaux & Scaman, 2018, Theorem 2).

Original: 1.2. C RITICAL BATCH SIZE
Revised: 1.2. C RITICAL BATCH SIZE

Original: The second motivation of this paper is to clarify theoretically the relationship between the diminishing returns reported in (Shallue et al., 2019; Zhang et al., 2019) and batch size.
Revised: The second motivation of this paper is to clarify theoretically the relationship between the diminishing returns reported in (Shallue et al., 2019; Zhang et al., 2019) and batch size.

Original: Numerical evaluations in (Shallue et al., 2019; Zhang et al., 2019) have deﬁnitively shown that, for deep learning optimizers, the number of steps K needed to train a deep neural network halves for each doubling of the batch size b and that there is a region of diminishing returns beyond the critical batch size b ⋆ .
Revised: Numerical evaluations in (Shallue et al., 2019; Zhang et al., 2019) have deﬁnitively shown that, for deep learning optimizers, the number of steps K needed to train a deep neural network halves for each doubling of the batch size b and that there is a region of diminishing returns beyond the critical batch size b ⋆ .

Original: This implies that there is a positive number C such that
Revised: This implies that there is a positive number C such that

Original: [Equation]
Revised: [Equation]

Original:  where K and b are deﬁned for i, j ∈ N by K = 2 i and b = 2 j (For example, Figure 1 in Section 4 shows C
Revised: (1) where K and b are deﬁned for i, j ∈ N by K = 2 i and b = 2 j (For example, Figure 1 in Section 4 shows C

Original: ≈ 20 and b ⋆ ≈ 2 11 for Adam used to train ResNet-20 on the CIFAR-10 dataset).
Revised: ≈ 20 and b ⋆ ≈ 2 11 for Adam used to train ResNet-20 on the CIFAR-10 dataset).

Original: We deﬁne the stochastic ﬁrst-order oracle (SFO) complexity of a deep learning optimizer as  Kb on the basis of the number of steps K needed for training the deep neural network and the batch size b used in the optimizer.
Revised: We deﬁne the stochastic ﬁrst-order oracle (SFO) complexity of a deep learning optimizer as N := Kb on the basis of the number of steps K needed for training the deep neural network and the batch size b used in the optimizer.

Original: Let b ⋆ be a critical batch size such that there are diminishing returns for all batch sizes beyond b ⋆ , as asserted in (Shallue et al., 2019; Zhang et al., 2019).
Revised: Let b ⋆ be a critical batch size such that there are diminishing returns for all batch sizes beyond b ⋆ , as asserted in (Shallue et al., 2019; Zhang et al., 2019).

Original: This fact, expressed in (1), implies that, while SFO complexity Kb initially almost does not change (i.e., K halves for each doubling of b ), Kb is minimized at critical batch size b ⋆ , and there are diminishing returns once the batch size exceeds b ⋆ .
Revised: This fact, expressed in (1), implies that, while SFO complexity N initially almost does not change (i.e., K halves for each doubling of b ), N is minimized at critical batch size b ⋆ , and there are diminishing returns once the batch size exceeds b ⋆ .

Original: 1. C ONTRIBUTION
Revised: 1. C ONTRIBUTION

Original: Our results are summarized in Table 1.
Revised: Our results are summarized in Table 2 (see also the “(this paper)” column of Table 1).

Original: Our goal is to ﬁnd a local minimizer of a loss function f over R d , i.e., a stationary point θ ⋆ ∈ R d satisfying ∇ f ( θ ⋆ ) = 0 , which is equivalent to the variational inequality (VI) deﬁned for all θ ∈ R d by ∇
Revised: Our goal is to ﬁnd a local minimizer of a loss function f over R d , i.e., a stationary point θ ⋆ ∈ R d satisfying ∇ f ( θ ⋆ ) = 0 , which is equivalent to the variational inequality (VI) deﬁned for all θ ∈ R d by ∇

Original: f ( θ ⋆ ) ⊤ ( θ ⋆ − θ ) ≤ 0 .
Revised: f ( θ ⋆ ) ⊤ ( θ ⋆ − θ ) ≤ 0 .

Original: Hence, we use an ϵ -approximation
Revised: Here, we show the relationship between (i) E [ ∇ f ( θ k ) ⊤ ( θ k − θ )] ≤ ϵ ( θ ∈ R d , k ∈ N ) and (ii) E [ ∥∇ f ( θ k ) ∥ 2 ] ≤ ϵ ( k ∈ N ) , where ϵ > 0 is the precision. Let us assume that ( θ k ) k ∈ N is bounded. Suppose that (i) holds.

Original: VI( K, θ  ) ≤ ϵ (Table 1) as the performance measure ofthe sequence ( θ k ) k ∈ N generated by a deep learning optimizer, where ϵ > 0 is the precision (see (5) for other performance measure of ( θ k  ) k ∈ N  ) 
Revised: Then, there exists a subsequence ( θ k i ) i ∈ N of ( θ k ) k ∈ N such that ( θ k i ) i ∈ N converges to θ ∗ . The continuity of ∇ f thus implies that, for all θ ∈ R d , E [ ∇ f ( θ ∗ ) ⊤ ( θ ∗ − θ )] ≤ ϵ .

Original: [Table] [Table] [Table] [Equation]
Revised: Putting θ := θ ∗ −∇ f ( θ ∗ ) ensures that E [ ∥∇ f ( θ ∗ ) ∥ 2 ] ≤ ϵ .

Original: Critical Batch
Revised: Suppose that (ii) holds. Then, the deﬁnition of the inner product and Jensen’s inequality imply that, for all θ ∈ R d , E [ ∇ f ( θ k ) ⊤ ( θ k − θ )] ≤ Dist( θ ) √ ϵ , where Dist( θ ) : = sup {∥ θ k − θ ∥ : k ∈ N } < + ∞ . Therefore, it is adequate to use (i) and ϵ -approximation VI( K, θ ) :

Original: [Equation]
Revised: = 1 K P Kk =1 E [ ∇ f ( θ k ) ⊤ ( θ k − θ )] ≤ ϵ (Table 2) as the performance measure of an optimizer.

Original: 1.3. A DVANTAGE OF SETTING A SMALL CONSTANT LEARNING RATE AND HYPERPARAMETERS CLOSE TO ONE
Revised: 1.3. A DVANTAGE OF SETTING A SMALL CONSTANT LEARNING RATE AND HYPERPARAMETERS CLOSE TO ONE

Original: We can show that the upper bound C 1 /K
Revised: We can show that the upper bound C 1 K + C 2 b

Original: + C2 /b + C 3 of VI( K, θ ) becomes small when α is small, β 1 and β 2 are close to 1 , and K is large.
Revised: + C 3 of VI( K, θ ) becomes small when α is small, βand β 2 are close to 1 , and K is large.

Original: This implies that Momentum and Adam perform well when α is small and β 1 and β 2 are each set close to 1 .
Revised: This implies that Momentum and Adam perform well when α is small and β 1 and β 2 are each set close to 1 .

Original: Section 3.1 shows this result in detail.
Revised: Section 3.1 shows this result in detail.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: C RITICAL BATCH SIZE
Revised: C RITICAL BATCH SIZE

Original: We can check that the upper bound C 1 /K
Revised: As described in Section 1.2. [Table]

Original: + C 2 /b + C 3 ofVI( K, θ ) becomes small when a  batch size b is large.
Revised: the practical performance of a deep learning optimizer strongly depends on the batch size (Shallue et al., 2019; Zhang et al., 2019). The advantage of this paper is to clarify theoretically the relationship between batch size and the performance of deep learning optimizers and develop a theory demonstrating the existence of critical batch sizes, which was shown numerically by (Shallue et al., 2019; Zhang et al., 2019).

Original: Motivated by the results in (Shallue et al., 2019; Zhang et al., 2019) and Section 1.2.2, we use SFO complexity  Kb as the performance measure of a deep learning optimizer.
Revised: Motivated by the results in (Shallue et al., 2019; Zhang et al., 2019) and Section 1.2.2, we use SFO complexity N := Kb as the performance measure of a deep learning optimizer.

Original: We ﬁrst show that the number of steps K to satisfy VI( K, θ ) ≤ ϵ can be deﬁned as in Table 1.
Revised: We ﬁrst show that the number of steps K to satisfy VI( K, θ ) ≤ ϵ can be deﬁned as in Table 2.

Original: As a function, K is convex and monotone decreasing with respect to batch size b .
Revised: As a function, K is convex and monotone decreasing with respect to batch size b .

Original: Next, we show that SFO complexity Kb deﬁned as in Table 1 is convex with respect to batch size b .
Revised: Next, we show that SFO complexity N deﬁned as in Table 2 is convex with respect to batch size b .

Original: This result agrees with the fact of (1).
Revised: This result agrees with the fact of (1).

Original: Moreover, SFO complexity Kb is minimized at 
Revised: Moreover, SFO complexity N is minimized at b ⋆ deﬁned as in Table 2.

Original: [Equation]
Revised: This result guarantees the existence of the critical batch size b ⋆ .

Original: Section 3.2 shows the above results in detail.
Revised: Section 3.2 shows the above results in detail.

Original: From the above discussion, we conclude that a deep learning optimizer should use the following parameters: • a small constant learning rate α (e.g., α = 10 − 3 ) and hyperparameters β 1 and β 2 close to 1 (e.g., β 1 , β 2 ∈ { 0 . 9 , 0 . 99 , 0 . 999 } ) • the critical batch size b⋆ deﬁned by (2)
Revised: However, the accurate setting of the critical batch size b

Original: The accurate setting of the critical batch size b ⋆ deﬁned by (2) would be difﬁcult since b ⋆in (2) involves unknown parameters, such as G and D ( θ ) (see Section 2.2.3).
Revised: ⋆ deﬁned as in Table 2 would be difﬁcult since b ⋆ involves unknown parameters, such as G and D ( θ ) (see Section 2.2.3).

Original: Hence, we would like to estimate appropriate batch sizes using the formula(2) for b ⋆ before implementing deep learning optimizers.
Revised: The advantage of our analysis is that we can estimate appropriate batch sizes using the formula for b ⋆ before implementing deep learning optimizers.

Original: Section 4 will discuss estimation of appropriate batch sizes in detail.
Revised: Section 4 will discuss estimation of appropriate batch sizes in detail.

Original:  N ONCONVEX O PTIMIZATION AND D EEP L EARNING O PTIMIZERS
Revised:  N ONCONVEX O PTIMIZATION AND D EEP L EARNING O PTIMIZERS

Original: This section gives a nonconvex optimization problem in deep neural networks and optimizers for solving the problem under standard assumptions.
Revised: This section gives a nonconvex optimization problem in deep neural networks and optimizers for solving the problem under standard assumptions.

Original: 2. N ONCONVEX OPTIMIZATION IN DEEP LEARNING
Revised: 2. N ONCONVEX OPTIMIZATION IN DEEP LEARNING

Original: Let R d be a d -dimensional Euclidean space with inner product ⟨ x , y ⟩ := x ⊤ y inducing the norm ∥ x ∥ and N be the set of nonnegative integers.
Revised: Let R d be a d -dimensional Euclidean space with inner product ⟨ x , y ⟩ := x ⊤ y inducing the norm ∥ x ∥ and N be the set of nonnegative integers.

Original: Deﬁne [ n ] := { 1 , 2 , . . . , n } for n ≥ 1 .
Revised: Deﬁne [ n ] := { 1 , 2 , . . . , n } for n ≥ 1 .

Original: Given a parameter θ ∈ R d and a data point z in a data domain Z , a machine learning model provides a prediction whose quality is measured by a differentiable nonconvex loss function ℓ ( θ ; z ) .
Revised: Given a parameter θ ∈ R d and a data point z in a data domain Z , a machine learning model provides a prediction whose quality is measured by a differentiable nonconvex loss function ℓ ( θ ; z ) .

Original: We aim to minimize the expected loss deﬁned for all θ ∈ R d by 
Revised: We aim to minimize the expected loss deﬁned for all θ ∈ R d by f ( θ ) =

Original: [Equation]
Revised: E z ∼D [ ℓ ( θ ; z )] =

Original:  where D is a probability distribution over Z , ξ denotes a random variable with distribution function P , and E [ · ] denotes the expectation taken with respect to ξ .
Revised: E [ ℓ ξ ( θ )] , where D is a probability distribution over Z , ξ denotes a random variable with distribution function P , and E [ · ] denotes the expectation taken with respect to ξ .

Original: A particularly interesting example of (3) is the empirical average loss deﬁned for all θ ∈ R d by 
Revised: A particularly interesting example of f ( θ ) is the empirical average loss deﬁned for all θ ∈ R d by f ( θ ; S )

Original: [Equation]
Revised: = 1 n P i ∈ [ n ] ℓ ( θ ; z i ) =

Original:  where S =
Revised: 1 n P i ∈ [ n ] ℓ i ( θ ) , where S =

Original: ( z 1 , z 2 , . . .
Revised: ( z 1 , z 2 , . . .

Original: , z n ) denotes the training set and ℓ i ( · ) :=
Revised: , z n ) denotes the training set and ℓ i ( · ) :=

Original: ℓ ( · ; z i ) denotes the loss function corresponding to the i -th training data z i .
Revised: ℓ ( · ; z i ) denotes the loss function corresponding to the i -th training data z i .

Original: We would like to ﬁnd a stationary point θ ⋆ ∈ R d of f deﬁned by (3).

Original: A point θ ⋆ ∈ R d is a stationary point of f if and only if θ ⋆ ∈ R d satisﬁes the following variational inequality (VI): for all θ ∈ R d , ∇ f ( θ ⋆ ) ⊤ ( θ ⋆ − θ ) ≤ 0 .

Original: Hence, we use VI( K, θ ) :

Original: = (1 /K

Original: [Equation]

Original: θ k ) ⊤ ( θ k − θ )] ≤ ϵ as the performance measure of the sequence ( θ k ) k ∈ N generated by a deep learning optimizer, where ϵ >is the precision and K denotes the number of steps of the optimizer (see also Table 1).

Original: We also use the following performance measure:

Original: [Equation]

Original: The performance measure (5) is an ϵ –approximation of the sequence ( θ k ) k ∈ N in the sense that the inner product of the vector θ k − θ ⋆ and the inverse direction m k of the search direction − m k is less than or equal to ϵ .

Original: Thanks to the previous results in (Zaheer et al., 2018; Zou et al., 2019; Chen et al., 2019; Zhou et al., 2020; Zhuang et al., 2020; Chen et al., 2021; Iiduka, 2021), it is guaranteed that

Original: Adam and its variants can ﬁnd θ ⋆ , which implies that 0 ≤ M( K ) .

Original: If M( K ) ≤ ϵ , then the deﬁnition of the inner product implies that θ K approximates θ ⋆ .

Original: Hence, it is sufﬁcient to check whether or not a deep learning optimizer satisﬁes (5).

Original: 2. D EEP LEARNING OPTIMIZERS 2.2. C ONDITIONS
Revised: 2. D EEP LEARNING OPTIMIZERS 2.2. C ONDITIONS

Original: We assume that a stochastic ﬁrst-order oracle (SFO) exists such that, for a given θ ∈ R d , it returns a stochastic gradient G ξ ( θ ) of the function f deﬁned by (3), where a random variable ξ is supported on Ξ independently of θ .
Revised: We assume that a stochastic ﬁrst-order oracle (SFO) exists such that, for a given θ ∈ R d , it returns a stochastic gradient G ξ ( θ ) of the function f , where a random variable ξ is supported on Ξ independently of θ .

Original: The following are standard conditions when considering a deep learning optimizer.
Revised: The following are standard conditions when considering a deep learning optimizer.

Original: (C1)
Revised: (C1)

Original: f : R d → Rdeﬁned by (3) is continuously differentiable.
Revised: f : R d → R is continuously differentiable.

Original: (C2)
Revised: (C2)

Original: Let ( θ k ) k ∈ N ⊂ R
Revised: Let ( θ k ) k ∈ N ⊂ R

Original: d be the sequence generated by a deep learning optimizer.
Revised: d be the sequence generated by a deep learning optimizer.

Original: For each iteration k , 
Revised: For each iteration k , E ξ k

Original: [Equation] where ξ 0 , ξ 1 , . .
Revised: [ G ξ k ( θ k )] = ∇ f ( θ k ) , where ξ 0 , ξ 1 , . .

Original: . are independent samples and the random variable ξ k is independent of ( θ l ) kl =0 .
Revised: . are independent samples and the random variable ξ k is independent of ( θ l ) kl =0 .

Original: There exists a nonnegative constant σ 2 such that 
Revised: There exists a nonnegative constant σ 2 such that E

Original: [Equation]
Revised: [Equation]

Original: (C3)
Revised: (C3)

Original: For each iteration k , the optimizer samples a batch B k of size b independently of k and estimates the full gradient ∇ f as [Equation]
Revised: For each iteration k , the optimizer samples a batch B k of size b independently of k and estimates the full gradient ∇ f as ∇ f B k ( θ k ) :

Original:  where ξ k,i is a random variable generated by the i -th sampling in the k -th iteration.In the case that f is deﬁned by (4), we have that, for each k , B k ⊂ [ n ] and ∇ f B k ( θ k ) = (1 /b ) P i ∈ [ b ] ∇ ℓ ξ k,i ( θ k ) .
Revised: = 1 b P i ∈ [ b ] G ξ k,i ( θ k ) , where ξ k,i is a random variable generated by the i -th sampling in the k -th iteration.

Original: 2.2. A DAM
Revised: A DAM

Original: Algorithm 1 is the Adam optimizer (Kingma & Ba, 2015) under (C1)–(C3).
Revised: Algorithm 1 is the Adam optimizer (Kingma & Ba, 2015) under (C1)–(C3).

Original: The symbol ⊙ in step 6 is deﬁned for all x = ( x i ) di
Revised: The symbol ⊙ in step 6 is deﬁned for all x = ( x i ) di

Original: =1 ∈ R d , x ⊙ x := ( x 2 i ) di
Revised: =1 ∈ R d , x ⊙ x := ( x 2 i ) di

Original: =1 ∈ R d , and diag ( x i ) in step 8 is a diagonal matrix with diagonal components x 1 , x 2 , . . .
Revised: =1 ∈ R d , and diag ( x i ) in step 8 is a diagonal matrix with diagonal components x 1 , x 2 , . . .

Original: , x d .
Revised: , x d .

Original: Algorithm 1 Adam (Kingma & Ba, 2015)

Original: Require: α ∈ (0 , + ∞ ) , b ∈ (0 , + ∞ ) , β 1 ∈ (0 , 1) , β 2 ∈ [0 , 1) 1: k ← 0 , θ 0 ∈ R d , m − 1 := 0 , v − 1 :=2: loop 3: [Equation] [Equation]
Revised: Require: α ∈ (0 , + ∞ ) , b ∈ (0 , + ∞ ) , β 1 ∈ (0 , 1) , β 2 ∈ [0 , 1) 1: k ← 0 , θ 0 ∈ R d , m − 1 := 0 , v − 1 :=2: loop 3: ∇ f B k ( θ k ) :

Original: [Equation]
Revised: = 1 b

Original: [Equation]
Revised: P i ∈ [ b ] G ξ k,i ( θ k ) 4: m k := β 1 m k − 1 + (1 − β 1 ) ∇ f B k ( θ k ) 5:

Original: [Equation]
Revised: ˆ m k :=

Original: H
Revised: (1 − β k + ) − 1 m k 6: v k := β 2 v k − 1 + (1 − β 2 ) ∇ f B k ( θ k ) ⊙ ∇ f B k ( θ k ) 7: ˆ v k :=

Original: [Equation]
Revised: (1 − β k + ) − 1 v k 8:

Original:  := diag ( p ˆ v k,i ) 9: θ k +1 :
Revised: H k := diag ( p ˆ v k,i ) 9: θ k +1 :

Original: = θ k − α H − 1 k ˆ m k 10: k ← k +11: end loop
Revised: = θ k − α H − 1 k ˆ m k 10: k ← k +11: end loop

Original: The SGD optimizer under (C1)–(C3) is Algorithm 1 when β 1 = 

Revised: The SGD optimizer under (C1)–(C3) is Algorithm 1 when β 1 =

Original: and H k is the identity matrix.
Revised: and H k is the identity matrix.

Original: The Momentum optimizer under (C1)–(C3) is deﬁned for all k ∈ N by θ k
Revised: The Momentum optimizer under (C1)–(C3) is deﬁned for all k ∈ N by θ k

Original: +1 :
Revised: +1 :

Original: = θ k − α m k .
Revised: = θ k − α m k .

Original:  A SSUMPTIONS
Revised: 2.2. A SSUMPTIONS

Original: We assume the following conditions that were used in (Kingma & Ba, 2015, Theorem 4.1):
Revised: We assume the following conditions that were used in (Kingma & Ba, 2015, Theorem 4.1):

Original: (A1)
Revised: (A1)

Original: There exist positive numbers G and B such that, for all k ∈ N , ∥∇ f ( θ k ) ∥ ≤ G and ∥∇ f B k ( θ k ) ∥ ≤ B . (A2)
Revised: There exist positive numbers G and B such that, for all k ∈ N , ∥∇ f ( θ k ) ∥ ≤ G and ∥∇ f B k ( θ k ) ∥ ≤ B . (A2)

Original: For all θ ∈ R d , there exists a positive number Dist( θ ) such that, for all k ∈ N , ∥ θ k − θ ∥ ≤ Dist( θ ) .
Revised: For all θ ∈ R d , there exists a positive number Dist( θ ) such that, for all k ∈ N , ∥ θ k − θ ∥ ≤ Dist( θ ) .

Original: Let ( g 2 k,i ) di =1 :=
Revised: Let ( g 2 k,i ) di =1 :=

Original: ∇ f B k ( θ k ) ⊙ ∇ f B k ( θ k ) ( k ∈ N ) .
Revised: ∇ f B k ( θ k ) ⊙ ∇ f B k ( θ k ) ( k ∈ N ) .

Original: Assumption (A1) implies that M := sup { max i ∈ [ d ] g 2 k,i : k ∈ N } < + ∞ .
Revised: Assumption (A1) implies that M := sup { max i ∈ [ d ] g 2 k,i : k ∈ N } < + ∞ .

Original: Assumption (A2) implies that D ( θ ) :
Revised: Assumption (A2) implies that D ( θ ) :

Original: = sup { max i ∈
Revised: = sup { max i ∈

Original: [ d ] ( θ k,i − θ i ) 2 : k ∈ N } < + ∞ .
Revised: [ d ] ( θ k,i − θ i ) 2 : k ∈ N } < + ∞ .

Original: We deﬁne v ∗ := inf { min i ∈
Revised: We deﬁne v ∗ := inf { min i ∈

Original: [ d ] v k,i : k ∈ N } .
Revised: [ d ] v k,i : k ∈ N } .

Original: Theorem 3 in (Reddi et al., 2018) shows that there exists a stochastic convex optimization problem such that Adam using β 1 < √ β 2 (e.g., β 1 = 0 .
Revised: Theorem 3 in (Reddi et al., 2018) shows that there exists a stochastic convex optimization problem such that Adam using β 1 < √ β 2 (e.g., β 1 = 0 .

Original: 9 and β 2 = 0 .
Revised: 9 and β 2 = 0 .

Original: 999 ) does not converge to the optimal solution.
Revised: 999 ) does not converge to the optimal solution.

Original: If for all k ∈ N and all i ∈
Revised: If for all k ∈ N and all i ∈

Original: [ d ] , ˆ v k,i in Adam satisﬁes
Revised: [ d ] , ˆ v k,i in Adam satisﬁes

Original: [Equation]
Revised: [Equation]

Original: (8) then Adam(AMSGrad) with a decaying learning rate α k
Revised: then Adam with a decaying learning rate α k

Original: = O (1 / √ k ) and β 1 and β 2 satisfying β 1 < √ β 2 can solve the stochastic convex optimization problem (Reddi et al., 2018, (2), Theorem 4).
Revised: = O (1 / √ k ) and β 1 and β 2 satisfying β 1 < √ βcan solve the stochastic convex optimization problem (Reddi et al., 2018, (2), Theorem 4).

Original: We thus assume condition (8) for Adam to guarantee the convergence of Adam.
Revised: We thus assume condition (2) for Adam to guarantee the convergence of Adam.

Original:  O UR R ESULTS
Revised:  O UR R ESULTS

Original: This section states our theoretical results (Theorem 3.1) in Table 1 and our contribution (Sections 3.1 and 3.2) in detail.
Revised: This section states our theoretical results (Theorem 3.1) in Table 2 and our contribution (Sections 3.1 and 3.2) in detail.

Original: The proof of Theorem 3.1 is given in Appendix A.
Revised: The proof of Theorem 3.1 is given in Appendix A.

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: The sequence ( θ k ) k ∈ N generated by each of SGD, Momentum, and Adam with (8) under (C1)–(C3) and (A1) and (A2) satisﬁes the following: (i)
Revised: The sequence ( θ k ) k ∈ N generated by each of SGD, Momentum, and Adam with (2) under (C1)–(C3) and (A1) and (A2) satisﬁes the following: (i

Original: [Upper bound of VI( K, θ ) ]
Revised: [Upper bound of VI( K, θ ) ]

Original: For all K ≥ 1 and all θ ∈ R d,
Revised: For all K ≥ 1 and all θ ∈ R d

Original: [Equation]
Revised: [Equation]

Original: where C i ( i = 1 , 2 , 3) for SGD are
Revised: where C i ( i = 1 , 2 , 3) for SGD are

Original: [Equation]
Revised: [Equation]

Original: C i ( i = 1 , 2 , 3) for Momentum are
Revised: C i ( i = 1 , 2 , 3) for Momentum are

Original: [Equation]
Revised: [Equation]

Original: C i ( i = 1 , 2 , 3) for Adam with (8) are
Revised: C i ( i = 1 , 2 , 3) for Adam with (2) are

Original: [Equation]
Revised: [Equation]

Original: , and the parameters are deﬁned as in Section 2.2.
Revised: and the parameters are deﬁned as in Section 2.2.

Original: (ii) [Steps to satisfy VI( K, θ ) ≤ ϵ]
Revised: (ii) [Steps to satisfy VI( K, θ ) ≤ ϵ

Original:  The number of steps K deﬁned by
Revised: ] The number of steps K deﬁned by

Original: [Equation]
Revised: [Equation]

Original: satisﬁes VI( K, θ ) ≤ ϵ and the function K ( b ) deﬁned by(9) is convex and monotone decreasing withrespect to batch size b ( > C 2 / ( ϵ − C 3 ) > 0) .
Revised: satisﬁes VI( K θ ) ≤ ϵ and the function K ( b ) deﬁned by is convex and monotone decreasing with

Revised: respect

Revised: to batch size b ( > C 2 ϵ − C

Original:  (iii)
Revised: > 0) . (iii)

Original: [Minimization of SFO complexity]
Revised: [Minimization of SFO complexity

Original: The SFO complexity deﬁned by
Revised: The SFO complexity deﬁned by

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: is convex with respect to batch size b ( > C 2/ ( ϵ − C 3) > 0) .
Revised: is convex with respect to batch size b ( > C 2 ϵ − C 3 > 0) .

Original: The batch size
Revised: The batch size

Original: [Equation]
Revised: [Equation]

Original: attains the minimum value  K ( b ⋆ ) b 
Revised: attains the minimum value N ⋆ = K ( b ⋆ ) b ⋆ = 2 C 1 C 2 ( ϵ − C 3 ) 2 of N . The proof of Theorem 3.1(i) ensures that C i ( i = 1 , 2 , 3) of AMSGrad (Algorithm 1 with ˆ m k = m k , ˆ v k = v k , ˜ v k,i := max { ˜ v k − 1 ,i , v k,i } (i.e., ˜ v k +1 ,i ≥ ˜ v k,i ; see (2)), and H k :=

Original: ⋆ = 2 C 1 C 2 / ( ϵ − C 3 ) 2 of K ( b )b .
Revised: diag ( p ˜ v k,i ) ) are C [Equation] ( θ )

Revised: [Equation]

Revised: We give a brief outline of the proof strategy of Theorem 3.1, with an emphasis on the main difﬁculty that has to be overcome in order not to assume Lipschitz smoothness of f (i.e., ∇ f is Lipschitz continuous with the Lipschitz constant L ).

Revised: First, we show that (C2) and (A1) imply that ( E [ ∥ m k ∥ ]) k ∈ N and ( E [ ∥ d k ∥ ]) k ∈ N are bounded, where d k := − H − 1 k ˆ m k .

Revised: Since we do not assume Lipschitz smoothness of f , we cannot use the descent lemma, i.e., f ( y ) ≤ f ( x ) + ∇ f ( x ) ⊤ ( y − x ) + L 2 ∥ y − x ∥

Revised: ( x , y ∈ R d ) .

Revised: This is the main difﬁculty to prove Theorem 3.1.

Revised: Almost all of the previous analyses of adaptive methods are based on the descent lemma, and hence, they can use the expectation of the squared norm of the full gradient E [ ∥∇ f ( θ k ) ∥ 2 ] as the performance measure.

Revised: Accordingly, we must use other performance measures that are different from E [ ∥∇ f ( θ k ) ∥ 2 ] .

Revised: This paper uses the performance measure VI( K, θ ) .

Revised: We next show that P Kk =1 E [ m ⊤ k − 1 ( θ k − θ )] ≤ a K + b K + c K ≤

Revised: C 1 + C 2 Kb + ˜ C 3 K , where C 1 and C 2 are deﬁned as in Theorem 3.1 and ˜ C 3 > 0 .

Revised: In particular, (2), (A1), and (A2) imply a K ≤ C 1 , the boundedness condition of ( E [ ∥ d k ∥ ]) k ∈ N implies b K ≤ C 2 Kb , and (A2) and the Cauchy–Schwarz inequality imply c K ≤ ˜ C 3 K .

Revised: The deﬁnition of m k , the Cauchy–

Revised: Schwarz inequality, the triangle inequality, and Jensen’s inequality imply Theorem 3.1(i).

Revised: Theorem 3.1(i) and C 1 K + C 2 b + C 3 =

Revised: ϵ lead to Theorem 3.1(ii) and (iii).

Original: 3. A DVANTAGE OF SETTING A SMALL CONSTANT LEARNING RATE AND HYPERPARAMETERS CLOSE TO ONE
Revised: 3. A DVANTAGE OF SETTING A SMALL CONSTANT LEARNING RATE AND HYPERPARAMETERS CLOSE TO ONE

Original: We ﬁrst show theoretical evidence that Adam using a small constant learning rate α , β 1 and β 2 close to 1 , and a large number of steps K performs well.
Revised: We ﬁrst show theoretical evidence that Adam using a small constant learning rate α , β 1 and β 2 close to 1 , and a large number of steps K performs well.

Original: Theorem 3.1(i) indicates that the upper bound of
Revised: Theorem 3.1(i) indicates that the upper bound of

Original: VI( K, θ ) for Adam is
Revised: VI( K, θ ) for Adam is

Original: [Equation]
Revised: [Equation]

Original: where β 1 ∈ (0 , 1) , β 2 ∈ [0 , 1) , and
Revised: where β 1 ∈ (0 , 1) , β 2 ∈ [0 , 1) , and

Original: [Equation]
Revised: [Equation]

Original: (while Theorem 3.1(i) shows (13), the strict evaluation (12) comes from (37) in Appendix A).
Revised: (the strict evaluation (6) of Theorem 3.1(i) comes from (28) in Appendix A).

Original: Since the function h ( β 1 ) deﬁned by (14) is monotone decreasing, using β 1 close to 1 makes h ( β 1 ) small.
Revised: Since the function h ( β 1 ) deﬁned by (7) is monotone decreasing, using β 1 close to 1 makes h ( β 1 ) small.

Original: Since 1/ ( β 1 (1 − β 1 )) is monotone increasing for β 1 ≥ 1 / 2 , using β 1 close to 1 makes 1/ ( β 1 (1 − β 1 )) large.
Revised: Since 1 β 1 (1 − β 1 ) is monotone increasing for β 1 ≥ 1 / 2 , using β 1 close to 1 makes 1 β 1 (1 − β 1 ) large.

Original: Hence, we need to set a small α to make α ( σ 2 /b + G 2 ) / (2 √ v ∗ β 1 (1 − β 1 )) small.
Revised: Hence, we need to set a small α to make α ( σ 2 b − 1 + G 2 ) 2 √ v ∗ β 1 (1 − β 1 ) small.

Original: The function q1 − β k + is monotone decreasing with respect to β 2 , while using β 2 close to 1 makes 1 / √ 1 − βlarge. When β 2 close to 1 and a small learning rate α are used, we need to use a large number of steps K to make dD ( θ )
Revised: The function q

Original: [Equation]
Revised: [Equation]

Original:  − β [Equation]
Revised: is monotone decreasing with respect to β 2 , while using β 2 close to 1 makes 1 √ 1 − β 2 large.

Original: K ) small. Let θ ⋆ be a stationary point of f . We can also check that the upper bound of M( K ) deﬁned by (5) for Adam is
Revised: When β 2 close to 1 and a small learning rate α are used, we need to use a large number of steps K to make

Original: [Equation]
Revised: [Equation]

Original: where A ( K, θ ⋆ ) is deﬁned as in (13) (see (39)).

Original: Accordingly, Adam performs well in the sense that the upper bound of M( K ) is small with a small constant learning rate α , β 1 and β 2 close to 1 , and a large number of steps K .

Original: See (27) in Appendix A for the upper bound of M( K ) for Momentum.

Original: 3. C RITICAL BATCH SIZE
Revised: 3. C RITICAL BATCH SIZE

Original: Here, we can see that the upper bound of VI( K, θ ⋆ ) for SGD is [Equation] Compared with (13) and (15), it seems that, for a ﬁxed small constant learning rate α (e.g., α = 10 − 3 ), SGD performs better than Momentum and Adam in terms of the upper bounds of VI( K, θ ) and M( K ) . Meanwhile, we can check that the upper bounds of VI( K, θ⋆ ) and M( K ) in (13), (15), and (16) are small when batch size b is large.
Revised: Theorem 3.1(ii) indicates that the number of steps K to satisfy VI( K, θ ) ≤ ϵ can be expressed as (3).

Original: Accordingly, we need to check carefully how to set batch sizes to improve theoretical and practical performances of deep learning optimizers.

Original: Motivated by the results in (Shallue et al., 2019; Zhang et al., 2019) and Section 1.2.2, we use SFO complexity

Original: Kb as the performance measure of a deep learning optimizer.

Original: Theorem 3.1(ii) indicates that the number of steps K to satisfy VI( K, θ ) ≤ ϵ can be expressed as (9) ( K to satisfy M( K ) ≤ ϵ can be obtained by replacing C 3 with ˜ C 3 in (15)).

Original: The function K ( b ) deﬁned by (9) is convex and monotone decreasing.
Revised: The function K ( b ) deﬁned by (3) is convex and monotone decreasing.

Original: Hence, the form of K deﬁned by (9) supports theoretically the relationship between K and b shown in (Shallue et al., 2019; Zhang et al., 2019) (see also Figures 1 and 3 in this paper).
Revised: Hence, the form of K deﬁned by (3) supports theoretically the relationship between K and b shown in (Shallue et al., 2019; Zhang et al., 2019) (see also Figures 1 and 3 in this paper).

Original: Theorem 3.1(iii) indicates that SFO complexity deﬁned by (10) is convex with respect to batch size b .
Revised: Theorem 3.1(iii) indicates that SFO complexity deﬁned by (4) is convex with respect to batch size b .

Original: This result agrees with the fact of (1) (see also Figuresand 4 in this paper).
Revised: This result agrees with the fact of (1) (see also Figures 2 and 4 in this paper).

Original: Moreover, SFO complexity  Kb is minimized at b ⋆ deﬁned by (11); e.g., b ⋆ S for SGD, b ⋆ M for Momentum, and b ⋆ A for Adam are respectively
Revised: Moreover, SFO complexity N := Kb is minimized at b ⋆ deﬁned by (5); e.g., b ⋆ S for SGD, b ⋆ M for Momentum, and b ⋆ A for Adam are respectively

Original: [Equation]
Revised: [Equation]

Original: where σ S , σ M , and σ A are positive constants depending on SGD, Momentum, and Adam and C 3 =
Revised: where σ S , σ M , and σ A are positive constants depending on SGD, Momentum, and Adam and C 3 =

Original: C 3 , S , C 3 , M , C 3 , A are positive constants deﬁned as in Theorem 3.1 (see also ˜ C 3 in (15)).
Revised: C 3 , S , C 3 , M , C 3 , A are positive constants deﬁned as in Theorem 3.1.

Original: From (17), the lower bounds b ∗ of b ⋆ S , b ⋆ M , and b ⋆ A are respectively
Revised: From (8), the lower bounds b ∗ of b ⋆ S , b ⋆ M , and b ⋆ A are respectively

Original: [Equation]
Revised: [Equation]

Original: [-If σ S = σ M = σ A , α and β 1 are ﬁxed (e.g., α = 10 − 3 and β 1 = 0 . 9 ), and √ v ∗ (1 − β 1 ) < 
-]
Revised: [-If σ S = σ M = σ A , α and β 1 are ﬁxed (e.g., α = 10 − 3 and β 1 = 0 . 9 ), and √ v ∗ (1 − β 1 ) < 
-]

Original: (e.g., 0 . 1 √ v ∗ < 1 when β 1 = 0 . 9 ), then

Original: [Equation]

Original: [Equation]

Original: Hence, b ∗ A is larger than b ∗ S and b ∗ M .

Original: Section 4 discusses estimation of appropriate batch sizes in detail.

Original:  N UMERICAL R ESULTS
Revised:  N UMERICAL R ESULTS

Original: We evaluated the performances of SGD, Momentum, and Adam with different batch sizes.
Revised: We evaluated the performances of SGD, Momentum, and Adam with different batch sizes.

Original: The metrics were the number of steps K and the SFO complexity Kb satisfying f ( θ K ) ≤ 10 − 1 , where θ K is generated for each of SGD, Momentum, and Adam using batch size b .
Revised: The metrics were the number of steps K and the SFO complexity Kb satisfying f ( θ K ) ≤ 10 − 1 , where θ K is generated for each of SGD, Momentum, and Adam using batch size b .

Original: The stopping condition was 200 epochs.
Revised: The stopping condition was 200 epochs.

Original: The experimental environment consisted of two Intel(R)
Revised: The experimental environment consisted of two Intel(R)

Original: Xeon(R) Gold 6148 2.4GHz CPUs with 20 cores each, a 16-GB NVIDIA Tesla V100 900-Gbps GPU, and the Red Hat
Revised: Xeon(R) Gold 6148 2.4GHz CPUs with 20 cores each, a 16-GB NVIDIA Tesla V100 900-Gbps GPU, and the Red Hat

Original: Enterprise Linux 7.6 OS.
Revised: Enterprise Linux 7.6 OS.

Original: The code was written in Python 3.8.2 using the NumPy 1.17.3 and PyTorch 1.3.0 packages.
Revised: The code was written in Python 3.8.2 using the NumPy 1.17.3 and PyTorch 1.3.0 packages.

Original: A constant learning rate α = 10 − 3 was commonly used.
Revised: A constant learning rate α = 10 − 3 was commonly used.

Original: Momentum used β 1 = 0 .
Revised: Momentum used β 1 = 0 .

Original: Adam used β 1 = 0 .
Revised: Adam used β 1 = 0 .

Original: 9 and β 2 = 0 .
Revised: 9 and β 2 = 0 .

Original: 999 (Kingma & Ba, 2015).
Revised: 999 (Kingma & Ba, 2015).

Original: Here, we use (18) and estimate appropriate batch sizes in the sense that SFO complexity is minimized.
Revised: Here, we use (9) and estimate appropriate batch sizes in the sense that SFO complexity is minimized.

Original: The deﬁnitions of v ∗ and v k,i (see also (32) in Appendix A) imply that, for k ∈
Revised: The deﬁnitions of v ∗ and v k,i (see also (23) in Appendix A) imply that, for k ∈

Original: [ K ] and all i 
Revised: [ K ] and all i ∈ [ d ] ,

Original: [Equation]
Revised: [Equation]

Original: - tion (C2)(6) implies that E [ ∥∇ f B k ( θ k ) ∥ 2 ] ≤ σ 2 /b + E [ ∥∇ f ( θ k ) ∥ 2 ] (see also (21) in Appendix A).
Revised: implies that E [ ∥∇ f B k ( θ k ) ∥ 2 ] ≤ σ 2 b + E [ ∥∇ f ( θ k ) ∥ 2 ] (see also (12) in Appendix A).

Original: Conditions (C2)(7) and (C3) imply that, if b is large, then σ is small.
Revised: Conditions (C2) and (C3) imply that, if b is large, then σ is small.

Original: Hence, assuming that σ 2 /b ≈implies that v ∗ ≤ ∥∇ f
Revised: Hence, assuming that σ 2 b ≈ 0 implies that v ∗ ≤ ∥∇ f

Original: ( θ k ∗ ) ∥ 2 =∥ (1 /n )
Revised: ( θ k ∗ ) ∥ 2 =

Original: P i ∈ [ n ] ∇ ℓ i ( θ k ∗ ) ∥ 2 = (1 /n 2 ) ∥
Revised: [Equation]

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: ∈ [ n ]

Original: [Equation]
Revised: [Equation]

Original: i

Original: [Equation]
Revised: [Equation]

Original: i . Since deep learning optimizers can approximate stationary points of f , we assume that, for example, G k ∗ ,i ≈ ϵ .
Revised: Since deep learning optimizers can approximate stationary points of f , we assume that, for example, G k ∗ ,i ≈ ϵ .

Original: Then, (18) implies that
Revised: Then, (9) implies that

Original: [Equation]
Revised: [Equation]

Original: Conditions (C2)(7) and (C3) imply that, if b is large, then σ is small.
Revised: Conditions (C2) and (C3) imply that, if b is large, then σ is small.

Original: If SGD and Momentum can exploit large batch sizes, then σ S and σ M are small.
Revised: If SGD and Momentum can exploit large batch sizes, then σ S and σ M are small.

Original: However, (19) implies that b ∗ S and b ∗ M must be small when σ S and σ M are small.
Revised: However, (10) implies that b ∗ S and b ∗ M must be small when σ S and σ M are small.

Original: Accordingly, SGD and Momentum would not be able to use large batch sizes.
Revised: Accordingly, SGD and Momentum would not be able to use large batch sizes.

Original: Meanwhile, from (19), we expect that Adam can exploit a large batch size b ⋆ A > b ∗ A > b ∗∗ A , for example, when b ∗∗ A < 2 11 (CIFAR-10; n = 50000 , ResNet-20; d ≈ 1 . 1 × 10 7 , b ∗∗ A ≈ 1600 ) and 2 10 <
Revised: Meanwhile, from (10), we expect that Adam can exploit a large batch size b ⋆ A > b ∗ A > b ∗∗ A , for example, when b ∗∗ A < 2 11 (CIFAR-10; n = 50000 , ResNet-20; d ≈ 1 . 1 × 10 7 , b ∗∗ A ≈ 1600 ) and 2 10 <

Original: b
Revised: b

Original: ∗∗
Revised: ∗∗

Original: A < 2 11 (MNIST; n = 60000 , ResNet-18; d ≈ 1 . 

Revised: A < 2 11 (MNIST; n = 60000 , ResNet-18; d ≈ 1 .

Original: × 10 7 , b ∗∗ A ≈ 2000 )
Revised: × 10 7 , b ∗∗ A ≈ 2000 )

Original: (He et al., 2016; Leong et al., 2020), where σ 2A ≈ 10 − 2 and ϵ ≈ 10 − 3 are used.
Revised: (He et al., 2016; Leong et al., 2020), where σ 2A ≈ 10 − 2 and ϵ ≈ 10 − 3 are used.

Original: 4. R ES N ET -20 ON THE CIFAR-10 DATASET
Revised: 4. R ES N ET -20 ON THE CIFAR-10 DATASET

Original: [Figure]
Revised: [Figure]

Original: size needed to train ResNet-20 on CIFAR-10. Thereis an initial period of perfect scaling (indicated by dashed line) such that the number of steps K for Adam is inversely proportional to batch size b .
Revised: There such that the number of steps K for Adam is inversely proportional to batch size b .

Original: Adam has critical batch size b ⋆ = 2 11 .
Revised: Adam has critical batch size b ⋆ = 2 11 .

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Original: S F O ×

Revised: S F O ×

Original: SGDMomentumAdam
Revised: SGDMomentumAdam

Original: Let us consider training ResNet-20 on the CIFAR-10 dataset with n = 50000 .
Revised: Let us consider training ResNet-20 on the CIFAR-10 dataset with n = 50000 .

Original: Figure 1 shows the number of steps for SGD, Momentum, and Adam versus batch size.
Revised: Figure 1 shows the number of steps for SGD, Momentum, and Adam versus batch size.

Original: For SGD and Momentum, the number of steps K needed for f ( θ K ) ≤ 10 − 1 initially decreased.
Revised: For SGD and Momentum, the number of steps K needed for f ( θ K ) ≤ 10 − 1 initially decreased.

Original: However, SGD with b ≥ 2 6 and
Revised: However, SGD with b ≥ 2 6 and

Original: Momentum with b ≥ 2 10 did not satisfy f ( θ K ) ≤ 10 − 1 before the stopping condition was reached.
Revised: Momentum with b ≥ 2 10 did not satisfy f ( θ K ) ≤ 10 − 1 before the stopping condition was reached.

Original: Adam had an initial period of perfect scaling (indicated by dashed line) such that the number of steps K needed for f ( θ K ) ≤ 10 − 1 was inversely proportional to batch size b , and critical batch size b ⋆ = 2 11 such that K was not inversely proportional to the batch size beyond b ⋆ ; i.e., there were diminishing returns.
Revised: Adam had an initial period of perfect scaling (indicated by dashed line) such that the number of steps K needed for f ( θ K ) ≤ 10 − 1 was inversely proportional to batch size b , and critical batch size b ⋆ = 2 11 such that K was not inversely proportional to the batch size beyond b ⋆ ; i.e., there were diminishing returns.

Original: Figure 2 plots the SFO complexities for SGD, Momentum, and Adam versus batch size.
Revised: Figure 2 plots the SFO complexities for SGD, Momentum, and Adam versus batch size.

Original: For SGD,
Revised: For SGD,

Original: SFO complexity was minimum at b ⋆ = 2 2 ; for Momentum, it was minimum at b ⋆ = 2 3 .
Revised: SFO complexity was minimum at b ⋆ = 2 2 ; for Momentum, it was minimum at b ⋆ = 2 3 .

Original: This implies that SGD and Momentum could not use large batch sizes, as shown in (19).
Revised: This implies that SGD and Momentum could not use large batch sizes, as shown in (10).

Original: For Adam,
Revised: For Adam,

Original: SFO complexity was minimum at the critical batch size b ⋆ = 2 11 that was close to estimation of critical batch
Revised: SFO complexity was minimum at the critical batch size b ⋆ = 2 11 that was close to estimation of critical batch

Original: b ⋆ A > b ∗∗ A with 1600 ≈ b ∗∗ A < b ⋆ A = 2 11 = 2048 , as shown in (19).
Revised: b ⋆ A > b ∗∗ A with 1600 ≈ b ∗∗ A < b ⋆ A = 2 11 = 2048 , as shown in (10).

Original: We also checked that the elapsed time for Adam monotonically decreased for b ≤ 2 11 and that the elapsed time for critical batch size b ⋆ = 2 11 was the shortest.
Revised: We also checked that the elapsed time for Adam monotonically decreased for b ≤ 2 11 and that the elapsed time for critical batch size b ⋆ = 2 11 was the shortest.

Original: The elapsed time for b ≥ 2 12 increased with the SFO complexity, as shown in Figure 2 (see Tables2, 3, 4,  and 5 in Appendix A).
Revised: The elapsed time for b ≥ 2 12 increased with the SFO complexity, as shown in Figure 2 (see Tables 3, 4, 5, and 6 in Appendix A).

Original: 4. R ES N ET -18 ON THE MNIST DATASET
Revised: 4. R ES N ET -18 ON THE MNIST DATASET

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Let us consider training ResNet-18 on the MNIST dataset with n = 60000 .
Revised: Let us consider training ResNet-18 on the MNIST dataset with n = 60000 .

Original: Figures 3 and 4 indicate that Adam could exploit larger batch sizes than SGD and Momentum.
Revised: Figures 3 and 4 indicate that Adam could exploit larger batch sizes than SGD and Momentum.

Original: Moreover, these ﬁgures indicate that Momentum minimized SFO complexity at the critical batch size b ⋆ = 2 3 and Adam minimized SFO complexity at the critical batch size b ⋆ = 2 10 that was close to estimation of critical batch
Revised: Moreover, these ﬁgures indicate that Momentum minimized SFO complexity at the critical batch size b ⋆ = 2 3 and Adam minimized SFO complexity at the critical batch size b ⋆ = 2 10 that was close to estimation of critical batch

Original: b ⋆ A > b ∗∗ A with 2 10 < b ∗∗ A ≈ 2000 < 2 11 , as shown in (19).
Revised: b ⋆ A > b ∗∗ A with 2 10 < b ∗∗ A ≈ 2000 < 2 11 , as shown in (10).

Original: We can also check that the elapsed time for critical batch size b ⋆ = 2 10 was the shortest (see Tables6, 7, 8,  and 9 in Appendix A).
Revised: We can also check that the elapsed time for critical batch size b ⋆ = 2 10 was the shortest (see Tables 7, 8, 9, and 10 in Appendix A).

Original: We also estimated appropriate batch sizes of Adam for (i) CNN on MNIST ( n = 60000 , d ≈ 7 .
Revised: We also estimated appropriate batch sizes of Adam for (i) CNN on MNIST ( n = 60000 , d ≈ 7 .

Original: 7 × 10 6 ) and (ii) ResNet-32 on CIFAR-10 ( n = 50000 , d ≈ 2 . 

Revised: 7 × 10 6 ) and (ii) ResNet-32 on CIFAR-10 ( n = 50000 , d ≈ 2 .

Original: × 10 7 ) from b ∗∗ A in (19) and checked that actual critical batch sizes in (Zhang et al., 2019, Figure 5(a), (e)) are close to estimated batch sizes ((i) b ⋆ A = 2 11 = 2048 >
Revised: × 10 7 ) from b ∗∗ A in (10) and checked that actual critical batch sizes in (Zhang et al., 2019, Figure 5(a), (e)) are close to estimated batch sizes ((i) b ⋆ A = 2 11 = 2048 >

Original: b ∗∗ A ≈ 2000 , (ii) b ⋆ A = 2 10 = 1024 ≈ b ∗∗ A ≈ 1200 ).
Revised: b ∗∗ A ≈ 2000 , (ii) b ⋆ A = 2 10 = 1024 ≈ b ∗∗ A ≈ 1200 ).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper showed the relationship between batch size b and the number of steps K to achieve an ϵ -approximation of deep learning optimizers using a small constant learning rate α and hyperparameters β 1 and β 2 close to 1 .
Revised: This paper showed the relationship between batch size b and the number of steps K to achieve an ϵ -approximation of deep learning optimizers using a small constant learning rate α and hyperparameters β 1 and β 2 close to 1 .

Original: From the convexity of SFO complexity Kb , there exists a global minimizer b ⋆ of Kb that is the critical batch size.
Revised: From the convexity of SFO complexity N , there exists a global minimizer b ⋆ of N that is the critical batch size.

Original: We also gave numerical results indicating that Adam using a small constant learning rate, hyperparameters close to one, and the critical batch size has faster convergence than Momentum and SGD.
Revised: We also gave numerical results indicating that Adam using a small constant learning rate, hyperparameters close to one, and the critical batch size has faster convergence than Momentum and SGD.

Original: Moreover, we estimated appropriate batch sizes from our formula for b ⋆ and showed that actual critical batch sizes are close to estimated batch sizes.
Revised: Moreover, we estimated appropriate batch sizes from our formula for b ⋆ and showed that actual critical batch sizes are close to estimated batch sizes.

Original: A A PPENDIX
Revised: A A PPENDIX

Original: Unless stated otherwise, all relationships between random variables are supposed to hold almost surely.
Revised: Unless stated otherwise all relationships between random variables are supposed to hold almost surely.

Original: Let S be a positive deﬁnite matrix, which is denoted by S ∈ S d ++ .
Revised: Let S be a positive deﬁnite matrix which is denoted by S ∈ S d ++ .

Original: The S -inner product of
Revised: The S -inner product of

Original: R d is deﬁned for all x , y ∈ R d by ⟨ x , y ⟩ S :
Revised: R d is deﬁned for all x , y ∈ R d by ⟨ x , y ⟩ S :

Original: = ⟨ x , S y ⟩ = x ⊤ ( S y ), and the S -norm is deﬁned by ∥ x ∥ S :
Revised: = ⟨ x , S y ⟩ = x ⊤ ( S y ) and the S -norm is deﬁned by ∥ x ∥ S :

Original: = p ⟨ x , S x ⟩ .
Revised: = p ⟨ x , S x ⟩ .

Original: A. L EMMAS
Revised: A. L EMMAS

Original: Lemma A.1.
Revised: Lemma A.1.

Original: Suppose that (C1), (C2)(6), and (C3) hold.
Revised: Suppose that (C1), (C2), and (C3) hold.

Original: Then, Adam satisﬁes the following: for all k ∈ N and all θ ∈ R d ,
Revised: Then, Adam satisﬁes the following: for all k ∈ N and all θ ∈ R d ,

Original: [Equation]
Revised: [Equation]

Original: where d k := − H − 1 k ˆ m k , ˆ β 1 := 1 − β 1 , and ˜ β 1 k := 1 − β k +1.
Revised: where d k := − H − 1 k ˆ m k , ˆ β 1 := 1 − β 1 , and ˜ β 1 k := 1 − β k + .

Original: Proof.
Revised: Proof.

Original: Let θ ∈ R d and k ∈ N .
Revised: Let θ ∈ R d and k ∈ N .

Original: The deﬁnition θ k +1 :
Revised: The deﬁnition θ k +1 :

Original: = θ k + α d k implies that
Revised: = θ k + α d k implies that

Original: [Equation]
Revised: [Equation]

Original: Moreover
Revised: Moreover

Original: [Equation]
Revised: [Equation]

Original: the deﬁnitions of d k [Equation] m k
Revised: the deﬁnitions of d k , m k

Original: [Equation]
Revised: [Equation]

Original: and ˆ m k ensure that
Revised: and ˆ m k ensure that

Original: [Equation]
Revised: [Equation]

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: Conditions (6) and (C3) guarantee that
Revised: Conditions (C2) and (C3) guarantee that

Original: [Equation]
Revised: [Equation]

Original: Therefore, the lemma follows from taking the expectation on both sides of (20).
Revised: Therefore, the lemma follows from taking the expectation on both sides of (11).

Original: This completes the proof.
Revised: This completes the proof.

Original: The discussion in the proof of Lemma A.1 also gives the following lemma.
Revised: The discussion in the proof of Lemma A.1 also gives the following lemma.

Original: Lemma A.2.
Revised: Lemma A.2.

Original: Suppose that (C1), (C2)(6), and (C3) hold.
Revised: Suppose that (C1), (C2), and (C3) hold.

Original: Then, SGD satisﬁes the following: for all k ∈ N and all θ ∈ R d ,
Revised: Then, SGD satisﬁes the following: for all k ∈ N and all θ ∈ R d ,

Original: [Equation]
Revised: [Equation]

Original: Moreover, Momentum satisﬁes the following: for all k ∈ N and all θ ∈ R d ,
Revised: Moreover, Momentum satisﬁes the following: for all k ∈ N and all θ ∈ R d ,

Original: [Equation]
Revised: [Equation]

Original: where ˆ β 1 := 1 − β 1 .
Revised: where ˆ β 1 := 1 − β 1 .

Original: The following lemma indicates the bounds on ( E [ ∥ m k ∥ 2 ]) k ∈ N and ( E [ ∥ d k ∥ 2 H k ]) k ∈ N .
Revised: The following lemma indicates the bounds on ( E [ ∥ m k ∥ 2 ]) k ∈ N and ( E [ ∥ d k ∥ 2 H k ]) k ∈ N .

Original: Lemma A.3.
Revised: Lemma A.3.

Original: Adam under (C2)(6), (7), and (A1), for all k ∈ N satisﬁes
Revised: Adam under (C2) and (A1), for all k ∈ N satisﬁes

Original: [Equation]
Revised: [Equation]

Original: where v ∗ := inf { min i ∈
Revised: where v ∗ := inf { min i ∈

Original: [ d ] v k,i : k ∈ N } , ˜ β 1 k := 1 − β k +11 k , and ˜ β 2 k := 1 − β k +12 k .
Revised: [ d ] v k,i : k ∈ N } , ˜ β 1 k := 1 − β k +1 1 k , and ˜ β 2 k := 1 − β k +1 2 k .

Original: Proof.
Revised: Proof.

Original: Condition (C2)(6) implies that
Revised: Condition (C2) implies that

Original: [Equation]
Revised: [Equation]

Original: which, together with (C2)(7) and (A1) [Equation] in turn implies that
Revised: which, together with (C2) and (A1), in turn implies that

Original: [Equation]
Revised: [Equation]

Original: The convexity of ∥ · ∥ 2 , together with the deﬁnition of m k and (22), guarantees that, for all k ∈ N ,
Revised: The convexity of ∥ · ∥ 2 , together with the deﬁnition of m k and (13), guarantees that, for all k ∈ N ,

Original: [Equation]
Revised: [Equation]

Original: Induction thus ensures that, for all k ∈ N ,
Revised: Induction thus ensures that, for all k ∈ N ,

Original: [Equation]
Revised: [Equation]

Original: where m − 1 = 0 .
Revised: where m − 1 = 0 .

Original: For k ∈ N , H k ∈ S d
Revised: For k ∈ N , H k ∈ S d

Original: +
Revised: +

Original: + guarantees the existence of a unique matrix
Revised: + guarantees the existence of a unique matrix

Original: H k ∈ S d ++ such that H k
Revised: H k ∈ S d ++ such that H k

Original: = H 2 k (Horn & Johnson, 1985, Theorem 7.2.6).
Revised: = H 2 k (Horn & Johnson, 1985, Theorem 7.2.6).

Original: We have that, for all x ∈ R d ,
Revised: We have that, for all x ∈ R d ,

Original: ∥ x ∥ 2 H k = ∥
Revised: ∥ x ∥ 2 H k = ∥

Original: H k x ∥ 2 .
Revised: H k x ∥ 2 .

Original: Accordingly, the deﬁnitions of d k and ˆ m k imply that, for all k ∈ N ,
Revised: Accordingly, the deﬁnitions of d k and ˆ m k imply that, for all k ∈ N ,

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Moreover, the deﬁnition of v ∗ := inf {v k
Revised: Moreover, the deﬁnition of [Equation] inf {

Original: [Equation]
Revised: [Equation]

Original: i
Revised: k,i

Original: [Equation]
Revised: [Equation]

Original: : k ∈ N } and (23) imply that, for all k ∈ N ,
Revised: : k ∈ [Equation] } and (14) imply that, for all k ∈ N ,

Original: [Equation]
Revised: [Equation]

Original: completing the proof.
Revised: completing the proof.

Original: A. SGD
Revised: A. SGD

Original: We show Theorem 3.1 for SGD.
Revised: We show Theorem 3.1 for SGD.

Original: Proof.
Revised: Proof.

Original: (i) Lemma A.2 and (22) imply that, for all k ∈ N and all θ ∈ R d ,
Revised: (i) Lemma A.2 and (13) imply that, for all k ∈ N and all θ ∈ R d ,

Original: [Equation]
Revised: [Equation]

Original: Summing the above inequality from k = 1 to k = K leads to the ﬁnding that, for all K ≥ 1 , 2 αK X k =
Revised: Summing the above inequality from k = 1 to k = K leads to the ﬁnding that, for all K ≥ 1 , 2 α

Original: [Equation] (cid:18) [Equation] b
Revised: K X

Original: [Equation]
Revised: [Equation]

Original: (cid:19)

Original: K, which implies that, for all K ≥ 1 and all θ ∈ R d ,
Revised: , which implies that, for all K ≥ 1 and all θ ∈ R d ,

Original: [Equation]
Revised: [Equation]

Original: (ii) Let θ ∈ R d and ϵ > 0 .
Revised: (ii) Let θ ∈ R d and ϵ > 0 .

Original: Condition C 1 /K
Revised: Condition C 1 /K

Original: + C 2 /b + C 3 =
Revised: + C 2 /b + C 3 =

Original: ϵ is equivalent to
Revised: ϵ is equivalent to

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Since ϵ = C 1 /K + C 2 /b + C 3 > C 3 , we consider the case b > C 2 / ( ϵ − C 3 ) > 0 to guarantee that
Revised: Since ϵ = C 1 /K + C 2 /b + C 3 > C 3 , we consider the case b > C 2 / ( ϵ − C 3 ) > 0 to guarantee that

Original: K > 0 .
Revised: K > 0 .

Original: From (24), the function K deﬁned by (25) satisﬁes VI( K, θ ) ≤ C 1 /K
Revised: From (15), the function K deﬁned by (16) satisﬁes VI( K, θ ) ≤ C 1 /K

Original: + C 2 /b + C 

Revised: + C 2 /b + C

Original: = ϵ .
Revised: = ϵ .

Original: Moreover, from (25),
Revised: Moreover, from (16),

Original: [Equation]
Revised: [Equation]

Original: which implies that K is convex and monotone decreasing with respect to b .
Revised: which implies that K is convex and monotone decreasing with respect to b .

Original: (iii)
Revised: (iii)

Original: We have that
Revised: We have that

Original: [Equation]
Revised: [Equation]

Original: Accordingly,
Revised: Accordingly,

Original: [Equation]
Revised: [Equation]

Original: which implies that K ( b ) b is convex with respect to b and
Revised: which implies that K ( b ) b is convex with respect to b and

Original: [Equation]
Revised: [Equation]

Original: The point b ⋆ attains the minimum value K ( b ⋆ ) b ⋆ = 2 C 1 C 2 / ( ϵ − C 3 ) 2 of K ( b ) b .
Revised: The point b ⋆ attains the minimum value K ( b ⋆ ) b ⋆ = 2 C 1 C 2 / ( ϵ − C 3 ) 2 of K ( b ) b .

Original: This completes the proof.
Revised: This completes the proof.

Original: A. M OMENTUM
Revised: A. M OMENTUM

Original: We show Theorem 3.1 for Momentum.
Revised: We show Theorem 3.1 for Momentum.

Original: Proof.
Revised: Proof.

Original: (i) Lemma A.2 ensures that, for all k ∈ N and all θ ∈ R d , 
Revised: (i) Lemma A.2 ensures that, for all k ∈ N and all θ ∈ R d , E (cid:2)

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: which, together with Lemma A.
Revised: which, together with Lemma A.3, the Cauchy–Schwarz inequality, and (A1) and (A2), implies that E (cid:2) ( θ k − θ ) ⊤ m k − 1 (cid:3) ≤ 2 αβn E h ∥

Revised: θ k

Original: [Equation]
Revised: [Equation]

Original: the Cauchy–Schwarz inequality
Revised: θ ∥i − E h ∥ θ k [Equation] θ ∥

Original: [Equation]
Revised: [Equation]

Original: and (A1) and (A2), implies that
Revised: io [Equation] α [Equation]

Revised: + ˆ

Original: [Equation]
Revised: [Equation]

Revised: β 1 Dist( θ ) G.

Original: Summing the above inequality from k = 1 to k = K gives a relation that implies that
Revised: Summing the above inequality from k = 1 to k = K gives a relation that implies that

Original: K X
Revised: K X

Original: [Equation]
Revised: [Equation]

Original: and hence,
Revised: and hence,

Original: K

Original: [Equation]
Revised: [Equation]

Original: Moreover, we have that, for all k ∈ N and all θ ∈ R d ,
Revised: Moreover, we have that, for all k ∈ N and all θ ∈ R d ,

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst equality comes from the deﬁnition of m k and the ﬁrst inequality comes from the
Revised: where the ﬁrst equality comes from the deﬁnition of m k and the ﬁrst inequality comes from the

Original: Cauchy–Schwarz inequality, the triangle inequality
Revised: Cauchy–Schwarz inequality, the triangle inequality

Original: [Equation]
Revised: [Equation]

Original: and (A2).
Revised: and (A2).

Original: Hence, from Lemma A.3, (22),
Revised: Hence, from Lemma A.3, (13),

Original: Jensen’s inequality
Revised: Jensen’s inequality

Original: [Equation]
Revised: [Equation]

Original: and b ≥ 1 ,
Revised: and b ≥ 1 ,

Original: [Equation]
Revised: [Equation]

Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: Therefore, for all K ≥ 1 and all θ ∈ R d ,
Revised: for all K ≥ 1 and all θ ∈ R d ,

Original: [Equation]
Revised: [Equation]

Original: The deﬁnition of m k ensures that
Revised: The deﬁnition of m k ensures that

Original: [Equation]
Revised: [Equation]

Original: , which, together with the Cauchy–Schwarz inequality, the triangle inequality
Revised: which, together with the Cauchy–Schwarz inequality, the triangle inequality

Original: [Equation]
Revised: [Equation]

Original: and (A2), implies that
Revised: and (A2)

Original: [Equation]
Revised: [Equation]

Original: θ k [Equation] k
Revised: implies that

Original: [Equation]
Revised: [Equation]

Original: Lemma A.3, (22), Jensen’s inequality
Revised: Lemma A.3, (13), Jensen’s inequality

Original: [Equation]
Revised: [Equation]

Original: and b ≥ 1 guarantee that
Revised: and b ≥ 1 guarantee that

Original: [Equation]
Revised: [Equation]

Revised: p

Revised: [Equation]

Revised: + G

Original: Therefore, (27) ensures that, for all K ≥ 1 and all θ ∈ R d ,
Revised: [Equation] Therefore [Equation] (18) ensures that, for all K ≥ 1 and all θ ∈ R d ,

Original: [Equation]
Revised: [Equation]

Original: C. (ii) A discussion similar to the one showing (ii) in Theorem 3.1 for SGD would show (ii) in Theorem 3.1 for Momentum.
Revised: (ii) A discussion similar to the one showing (ii) in Theorem 3.1 for SGD would show (ii) in Theorem 3.1 for Momentum.

Original: (iii)
Revised: (iii)

Original: An argument similar to that which obtained (iii) in Theorem 3.1 for SGD would prove (iii) in Theorem 3.1 for Momentum.
Revised: An argument similar to that which obtained (iii) in Theorem 3.1 for SGD would prove (iii) in Theorem 3.1 for Momentum.

Original: A. A DAM
Revised: A. A DAM

Original: We show Theorem 3.1 for Adam.
Revised: We show Theorem 3.1 for Adam.

Original: Proof.
Revised: Proof.

Original: (i) Let θ ∈ R d .
Revised: (i) Let θ ∈ R d .

Original: Lemma A.1 guarantees that for all k ∈ N ,
Revised: Lemma A.1 guarantees that for all k ∈ N ,

Original: [Equation]
Revised: [Equation]

Original: We deﬁne γ k := ˜
Revised: We deﬁne γ k := ˜

Original: β 1 k / (2 β 1 α ) ( k ∈ N ) .
Revised: β 1 k / (2 β 1 α ) ( k ∈ N ) .

Original: Then, for all K ≥ 1 ,
Revised: Then, for all K ≥ 1 ,

Original: K X 
Revised: K X k =

Original: [Equation]
Revised: [Equation]

Original: Since H k ∈ S d ++ exists such that H k
Revised: Since H k ∈ S d ++ exists such that H k

Original: = H 2 k [Equation] we have ∥ x
Revised: = H 2 k , we have ∥ x

Original: ∥ 2 H k = ∥
Revised: ∥ 2 H k = ∥

Original: H k x ∥ 2 for all x ∈ R d .
Revised: H k x ∥ 2 for all x ∈ R d .

Original: Accordingly, we also have
Revised: Accordingly, we also have

Original: [Equation]
Revised: [Equation]

Original: From H k = diag (ˆ v 1 / 4 k,i ) [Equation] we have that, for all x =
Revised: H k ( θ k − θ ) (cid:13)(cid:13) 2 − γ k − 1 (cid:13)(cid:13)

Original: [Equation]
Revised: H k − 1 ( θ k − θ ) (cid:13)(cid:13)o# .

Original:  x i [Equation] [Equation] p ˆ v k,i x 2 i .
Revised: From H k = diag (ˆ v 1 / 4 k,i ) , we have that, for all x = ( x i ) di =1 ∈ R d , ∥ H k x ∥ 2 = P di = p ˆ v k,i x 2 i .

Original: Hence, for all K ≥ 2 , [Equation]
Revised: Hence, for all K ≥ 2 , Γ K = E " K X

Original: E
Revised: k

Original: [Equation] (cid:17) 
Revised: = d X i = (cid:16) γ k p ˆ v k,i − γ k −p ˆ v k − 1 ,i (cid:17) ( θ k,i − θ i )# .

Original: [Equation] Condition (8) and γ k ≥ γ k − 1 ( k ≥ 1 ) imply that, for all k ≥ 1 and all i ∈ [ d ] , [Equation]
Revised: Condition (2) and γ k ≥ γ k − 1 ( k ≥ 1 ) imply that, for all k ≥ 1 and all i ∈ [ d ] , γ k p ˆ v k,i − γ k −p ˆ v k − 1 ,i ≥ 0 .

Original: Moreover, (A2) ensures that D ( θ ) :
Revised: Moreover, (A2) ensures that D ( θ ) :

Original: = sup { max i ∈
Revised: = sup { max i ∈

Original: [ d ] ( θ k,i − θ i ) 2 : k ∈ N } < + ∞ .
Revised: [ d ] ( θ k,i − θ i ) 2 : k ∈ N } < + ∞ .

Original: Accordingly, for all K ≥ 2 , Γ K ≤ D ( θ ) E " K
Revised: Accordingly, for all K ≥ 2 , Γ K ≤ D ( θ ) E " K

Original: X
Revised: X

Original: k
Revised: k

Original: =
Revised: =

Original: d X i = (cid:16) [Equation] p ˆ v  k [Equation]
Revised: d X i = (cid:16) γ k p ˆ v k,i − γ k −p ˆ v k − 1 ,i (cid:17)

Original: γ k − [Equation] p ˆ v k − [Equation]
Revised: # = D ( θ ) E " d

Original: (cid:17)# = D ( θ ) E " d X i = (cid:16) γ K p ˆ v K,i − γp ˆ v 1 ,i (cid:17)# .
Revised: X i = (cid:16) γ K p ˆ v K,i − γp ˆ v 1 ,i (cid:17)# .

Original: Let ∇ f B k
Revised: Let ∇ f B k

Original: ( θ k ) ⊙ ∇ f B k ( θ k ) :
Revised: ( θ k ) ⊙ ∇ f B k ( θ k ) :

Original: = ( g 2 k,i
Revised: = ( g 2 k,i

Original: ) ∈
Revised: ) ∈

Original: R d + .
Revised: R d + .

Original: Assumption (A1) ensures that there exists M ∈ R such that, for all k ∈ N , max i ∈
Revised: Assumption (A1) ensures that there exists M ∈ R such that, for all k ∈ N , max i ∈

Original: [ d ] g 2 k,i ≤ M .
Revised: [ d ] g 2 k,i ≤ M .

Original: The deﬁnition of v k guarantees that, for all i ∈
Revised: The deﬁnition of v k guarantees that, for all i ∈

Original: [ d ] and all k ∈ N ,
Revised: [ d ] and all k ∈ N ,

Original: [Equation]
Revised: [Equation]

Original: Induction thus ensures that, for all i ∈
Revised: Induction thus ensures that, for all i ∈

Original: [ d ] and all k ∈ N ,
Revised: [ d ] and all k ∈ N ,

Original: [Equation]
Revised: [Equation]

Original: (32) where v 0 = ( v 0 ,i ) = 0 .
Revised: where v 0 = ( v 0 ,i ) = 0 .

Original: From the deﬁnition of ˆ v k , we have that, for all i ∈ [ d ] and all k ∈ N ,
Revised: From the deﬁnition of ˆ v k , we have that, for all i ∈ [ d ] and all k ∈ N ,

Original: [Equation]
Revised: [Equation]

Original: Therefore, (30), E [ ∥ θ 1 − θ ∥ 2 H 1 ] ≤ D ( θ ) E
Revised: Therefore, (21), E [ ∥ θ 1 − θ ∥ 2 H 1 ] ≤ D ( θ ) E

Original: [ P di =1 p ˆ v 1 ,i ] , and (33) imply, for all K ≥ 1 ,
Revised: [ P di =1 p ˆ v 1 ,i ] , and (24) imply, for all K ≥ 1 ,

Original: K X 
Revised: K X k =

Revised: [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Inequality (34) with ˜ β 1 K := 

Revised: Inequality (25) with ˜ β 1 K :=

Original: − β K +1≤ 1 and ˜ β 2 K := 1 − β K +1≥ 1 − β 2 =: ˆ β 2 implies that
Revised: − β K +1≤ 1 and ˜ β 2 K := 1 − β K +1≥ 1 − β 2 =: ˆ β 2 implies that

Original: K 
Revised: K X k =

Original: [Equation]
Revised: [Equation]

Original: Lemma A.3 guarantees that, for all k ∈ N ,
Revised: Lemma A.3 guarantees that, for all k ∈ N ,

Original: [Equation]
Revised: [Equation]

Original: Inequality (36) with ˜ β 1 k := 1 − β k + ≥ 1 − β 1 =: ˆ β 1 and ˜ β 2 k :

Original: [Equation]
Revised: [Equation]

Original: {+Inequality (27) with ˜ β 1 k := 1 − β k + ≥ 1 − β 
 =: ˆ β 1 and ˜ β 2 k := 1 − β k + ≤+} 1 implies that[Equation]
Revised: {+Inequality (27) with ˜ β 1 k := 1 − β k + ≥ 1 − β 
 =: ˆ β 1 and ˜ β 2 k := 1 − β k + ≤+} 1 implies that

Original: [Equation]
Revised: [Equation]

Original: The Cauchy–Schwarz inequality and (A2) imply that, for all k ∈ N ,
Revised: The Cauchy–Schwarz inequality and (A2) imply that, for all k ∈ N ,

Original: [Equation]
Revised: [Equation]

Original: Hence, (29), (35), (37), and (38) ensure that, for all K ≥ 1 ,
Revised: Hence, (20), (26), (28), and (29) ensure that, for all K ≥ 1 ,

Original: [Equation]
Revised: [Equation]

Original: Therefore, from (26), for all K ≥ 1 ,
Revised: Therefore, from (17), for all K ≥ 1 ,

Original: [Equation]
Revised: [Equation]

Original: From (28) and (39), for all K ≥ 1 ,K K X
Revised: From (19) and (30), for all K ≥ 1 ,

Original: [Equation]
Revised: [Equation]

Original: √

Original: M

Original: [Equation]

Original: q

Original: [Equation] z [Equation] (ii) A discussion similar to the one showing (ii) in Theorem 3.1 for SGD would show (ii) in Theorem 3.1 for Adam.
Revised: (ii) A discussion similar to the one showing (ii) in Theorem 3.1 for SGD would show (ii) in Theorem 3.1 for Adam.

Original: (iii)
Revised: (iii)

Original: An argument similar to that which obtained (iii) in Theorem 3.1 for SGD would prove (iii) in Theorem 3.1 for Adam.
Revised: An argument similar to that which obtained (iii) in Theorem 3.1 for SGD would prove (iii) in Theorem 3.1 for Adam.

Original: A. A DDITIONAL NUMERICAL RESULTS
Revised: A. A DDITIONAL NUMERICAL RESULTS

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]