Original: A NEW PERSPECTIVE IN UNDERSTANDING OF A DAMT YPE ALGORITHMS AND BEYOND
Revised: A NEW PERSPECTIVE IN UNDERSTANDING OF A DAMT YPE ALGORITHMS AND BEYOND

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: First-order adaptive optimization algorithms such as Adam play an important role in modern deep learning due to their super fast convergence speed in solving large scale optimization problems.
Revised: First-order adaptive optimization algorithms such as Adam play an important role in modern deep learning due to their super fast convergence speed in solving large scale optimization problems.

Original: However, Adam’s non-convergence behavior and regrettable generalization ability make it fall into a love-hate relationship to  deep learning community.
Revised: However, Adam’s non-convergence behavior and disturbing generalization ability make it fall into a love-hate relationship to the deep learning community.

Original: Previous studies on Adam and its variants (refer as AdamType algorithms) mainly rely on theoretical regret bound analysis, which overlook the natural characteristic reside in such algorithms and limit our thinking.
Revised: Previous studies on Adam and its variants (refer as Adam-Type algorithms) mainly rely on theoretical regret bound analysis, which overlooks the natural characteristics residing in Adam-Type algorithms and limits our understanding.

Original: In this paper, we aim at seeking a different interpretation of Adam-Type algorithms so that we can intuitively comprehend and improve them.
Revised: In this paper, we aim at seeking a different interpretation of Adam-Type algorithms so that we can intuitively comprehend and improve them.

Original: The way we chose is based on a traditional online convex optimization algorithm scheme known as mirror descent method.
Revised: The way we chose is based on a traditional online convex optimization algorithm scheme known as mirror descent method.

Original: By bridging Adam and mirror descent, we receive a clear map of the functionality of each part in Adam.
Revised: By bridging Adam and mirror descent algorithm, we receive a clear map of the functionality of each term in Adam.

Original: In addition, this new angle brings us a new insight on identifying the non-convergence issue of Adam.
Revised: In addition, this new angle brings us a new insight on identifying the non-convergence issue of Adam and explaining the superior convergence speed of Adam than other ﬁrst-order methods.

Original: Moreover, we provide  new variant of Adam-Type algorithm, namely AdamAL which can naturally mitigate the non-convergence issue of Adam and improve its performance.
Revised: Moreover, we provide a new variant of Adam-Type algorithm, namely AdamAL which can naturally mitigate the non-convergence issue of Adam and improve its performance.

Original: We further conduct experiments on various popular deep learning tasks and models, and the results are quite promising.
Revised: We further conduct experiments on various popular deep learning tasks and models, and the results are quite promising.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In recent years, ﬁrst-order optimization algorithms with adaptive learning rate have become the dominant method to train deep neuron networks because these methods show extraordinary power on solving large-scale machine learning optimization problems.
Revised: In recent years, ﬁrst-order optimization algorithms with adaptive learning rate have become the dominant method to train deep neural networks because these methods show extraordinary power in solving large-scale machine learning optimization problems.

Original: By cooperating with ﬁrst-order information, adaptive methods iteratively update parameters by moving them to the direction of the negative gradient of the cost function with non-ﬁxed learning rate.
Revised: By cooperating with ﬁrst-order information, adaptive methods iteratively update parameters by moving them to the direction of the negative gradient of the cost function with non-ﬁxed learning rate.

Original: The ﬁrst algorithm in this line of research can be dated back to (McMahan & Streeter, 2010), where they  demonstrate that the convergence rates can often be dramatically improved through the use of preconditioning.
Revised: The ﬁrst algorithm in this line of research can be dated back to McMahan & Streeter (2010), where they adaptively choose regularization functions for bounding objective function parameters based on the loss functions observed at each iteration. Then, in Streeter & McMahan (2010), they demonstrate that the convergence rates can often be dramatically improved through the use of preconditioning.

Original: Then
Revised: The insight of these two methods is parallel but effective, that is, they try to modify the gradients’ magnitude with adaptive pre-coordinate learning rates.

Original:  AdaGrad (Duchi et al., 2011)  provides  ﬁrst practical adaptive algorithm with  theoretical guarantee based on (Zinkevich, 2003) regret analysis.
Revised: Later, AdaGrad (Duchi et al., 2011) carefully chooses the preconditioner and provides the ﬁrst practical adaptive algorithm with a tight theoretical guarantee based on Zinkevich (2003) regret analysis.

Original: Although AdaGrad achieves signiﬁcant improvement on sparse settings, the rapid decay of the  learning rate limits it usage.
Revised: Although AdaGrad achieves a great success in the sparse settings, the rapid decay of the adaptive learning rate limits its usage.

Original: This is because AdaGrad use the past gradient accumulation as adaptive learning rate.
Revised: This is due to AdaGrad accumulating all the past gradients as its learning rate.

Revised: And this unbounded learning rate becomes extremely small when one has a large number of training iterations.

Original: To address this, several variants of AdaGrad, such as  RMSProp (Hinton et al., 2012), AdaDelta (Zeiler, 2012), Adma Kingma & Ba (2014) have been proposed to mitigate the rapid decay of the learning rate. 
Revised: To address this, several variants of AdaGrad, such as AdaDelta (Zeiler, 2012) and RMSProp (Hinton et al., 2012) have been proposed to mitigate the rapid decay of the learning rate. Based on these, Adma (Kingma & Ba, 2014) incorporating with ﬁrst momentum correction accelerates the convergence speed of ﬁrst-order optimization algorithms to a new height.

Original: In particular, Adam use  exponential moving average (EMA)  to obtain smooth rate.
Revised: The use of exponential moving average (EMA) in Adam becomes a key to Adam’s success.

Original:  Denote g t ∈ R d as the gradient of generic optimization problem f with respect to its parameters x ∈ R d at iteration t , then the generic updaing rule of adaptive methods can be express asfollows (Reddiet al., 2019):
Revised: Up to this point, we can summarize the adaptive or non-adaptive ﬁrst-order optimization algorithms as follows. Denote g t ∈ R d as the gradient of generic optimization problem f with respect to its parameters x ∈ R d at iteration t , then the generic updating rule of adaptive methods can be expressed as (Reddi

Original: [Equation]
Revised: [Equation]

Original: where (cid:12) denotes the entry-wise or Hadamard product.
Revised: where (cid:12) denotes the entry-wise or Hadamard product and the α t is the base learning rate.

Original: In  equation above, m t
Revised: In the equation above, m t

Original: = ϑ ( g 1 , · · · , g t ) is a function relates to the historical gradients; v t =
Revised: = ϑ ( g 1 , · · · , g t ) is a function related to the historical gradients up to t ; and v t =

Original: υ ( g 1 , · · · , g t ) is a n-dimension vector with nonnegative entry; α t is the base learning rate; For Adam, in particular, the m t and v t are computed by
Revised: υ ( g 1 , · · · , g t ) is a function that produces a d-dimensional vector with non-negative entry; The design of v t is simple of non-adaptive methods, such as vanilla stochastic gradient descent (Vanilla SGD), where we have v t = I , however, it is crucial for Adam-Type algorithms.

Original:  EMA of gradient, with coefﬁcient β 1 and β 2 where
Revised: For Adam, in particular, the m t and v t are computed by EMA of gradient, with coefﬁcient β 1 and β 2 where

Original: [Equation]
Revised: [Equation]

Original: Adam, the most popular adaptive method, has been widely adopted by  deep learning community.
Revised: Adam, the most popular adaptive method, has been widely adopted by the deep learning community.

Original: The root cause of the fast convergence of Adam and its variants in convex or non-convex optimization problems remains an open question (Chen et al., 2018).
Revised: The root cause of the fast convergence of Adam and its variants in convex or non-convex optimization problems remains an open question (Chen et al., 2018).

Original: In addition, the generalization ability and out-of-sample behavior of Adam and all other adaptive methods are even worse than traditional non-adaptive counterparts such as vanilla stochastic gradient descent (Vanilla SGD) (Wilson et al., 2017).
Revised: In addition, the generalization ability and out-of-sample behavior of Adam-Type methods are even worse than traditional non-adaptive counterparts such as Vanilla SGD (Wilson et al., 2017).

Original: In order to understand the insight behind Adam algorithm and close the generalization gap, several Adam-Type algorithms have been proposed including (Reddi et al., 2019; Luo et al., 2019;
Revised: In order to understand the insight behind those adaptive algorithms and close the generalization gap, several Adam-Type algorithms have been proposed including (Reddi et al., 2019; Luo et al., 2019;

Original: Zhou et al., 2018; Balles & Hennig, 2017; Liu et al., 2019).
Revised: Zhou et al., 2018b; Balles & Hennig, 2017; Liu et al., 2019).

Original: Although they propose many different kinds of viewpoints in understanding the performance of Adam and demonstrate a series of correction methods to improve Adam, we think the behavior of Adam-Type algorithms is still unclear.
Revised: Although they propose many different kinds of viewpoints in understanding the performance of Adam and demonstrate a series of correction methods to improve Adam, we think the mechanism of Adam-Type algorithms is still unclear.

Original: For example, one common thinking about about m t and v t in Adam is ﬁrst and second moments of unbiased estimator g t , however, why this second moments can be used as adaptive learning rate? 
Revised: For example, one common thinking about about m t and v t in Adam is ﬁrst and second moments of unbiased estimator g t , however, why this second moment can be used as adaptive learning rate? Does the v t really behave as the second moment of the gradient g t ?

Original: Also, another commonly asked question is where the Adam adopts such a fast convergence speed?
Revised: Also, another commonly asked question is where the Adam adopts such a faster convergence speed than other ﬁrst-order methods?

Original: In this paper, in order to answer the questionsmentioned above and have a deep comprehension on  Adam-Type algorithms , we provide a new insight into adaptive learning rate methods, which brings a new perspective  on identifying the non-convergence issue of Adam.
Revised: These mysterious questions on adaptive methods ﬁnally leads us to rethink the Adam-Type algorithms in a neutral way. In this paper, we provide a new insight into Adam-Type algorithms, which brings a new perspective of comprehension on Adam-Type algorithms and it allows us to easily identify the misbehavior of Adam such as non-convergence issues.

Original: In the previous work, thebehavior analysis of Adam is based on Kingma & Ba (2014) framework, which limits our understanding.
Revised: In the aforementioned works, the analysis of Adam-Type methods is based on Kingma & Ba (2014) framework, which limits our understanding.

Original: In fact,  the  adaptiveﬁrst-order methods has a long history. Adaptive learning rate was ﬁrst mentioned in Streeter & McMahan (2010); McMahan & Streeter (2010) but it is highly related to adaptive regularization of follow-the-proximally-regularized-leader (FTPRL). We notice that the intrinsic design of Adam can  be related to the traditional mirror descent method (Xiao, 2010).
Revised: In fact, when we look back to the origin of adaptive learning rate methods mentioned in Streeter & McMahan (2010), we notice that the design of Adam is highly related to adaptive regularization of follow-the-proximally-regularized-leader (FTPRL) method and it can also be regarded as a variant of traditional mirror descent method (Xiao, 2010).

Original: The more detail and our motivation can be found in  next section.
Revised: The more detail and our motivation can be found in the next section.

Revised: For simplicity, we use standard Adam as our touchstone through the paper to convey our main idea, and their variants follow the same thoughts.

Original: We summarize our contribution in two folds:
Revised: Now, we summarize our contribution in three folds:

Original: We provide a new perspective in understanding the non-convergence behavior of AdamType algorithms based on mirror descent approach.
Revised: We provide a new perspective in understanding the non-convergence behavior of AdamType algorithms based on mirror descent approach.

Revised: Our analysis agrees well with the previous works but much more intuitive and effective.

Original: Based on our observation, we identify potential fault in Adam-Type algorithms and we provide a new Adam variant algorithm,  AdamAL.
Revised: Based on our observation, we identify potential faults in Adam-Type algorithms and we provide a new Adam variant algorithm, named AdamAL.

Revised: We conduct a series of experiments on different machine learning tasks and models by using our AdamAL algorithm, and the results are promising and the performance of AdamAL is never worse than Adam.

Original:  P RELIMINARIES AND MOTIVATIONS
Revised:  P RELIMINARIES AND MOTIVATIONS

Original: Notations Given a vector x ∈ R d we denote its i-th entry by x i ; We use || x || to denote its l 2 norm; for a vector x t in the t-th iteration, the i-th coordinate of x t is denoted as x t,i . 
Revised: Notations Given a vector x ∈ R d we denote its i-th entry by x i ; We use || x || to denote its l 2 norm; for a vector x t in the t-th iteration, the i-th coordinate of x t is denoted as x t,i . We also deﬁne x 1: t = (cid:80) ti =1 x i .

Original: Given two vectors x, y ∈ R , we use (cid:104) x, y (cid:105) to denote their inner product, x (cid:12) y to denote element-wise product, xy to denote entry-wise division, the max( x, y ) to denote entry-wise maximum and min( x, y ) to denote entry-wise minimum.
Revised: Given two vectors x, y ∈ R , we use (cid:104) x, y (cid:105) to denote their inner product, x (cid:12) y to denote element-wise product, xy to denote entry-wise division, the max( x, y ) to denote entry-wise maximum and min( x, y ) to denote entry-wise minimum.

Original: We use S + to denote the set of all positive deﬁnite matrices M .
Revised: We use S + to denote the set of all positive deﬁnite matrices M .

Original: We use Mto denote √ M .
Revised: We use M 1 2 to denote √ M .

Original: 2. P RELIMINARIES
Revised: 2. P RELIMINARIES

Original: Nonlinear projected subgradient methods and mirror descent algorithm Iterative gradient descent (GD) scheme, which can be traced back to (Cauchy, 1847), is the simplest strategy to minimize convex optimization problems.
Revised: Nonlinear projected subgradient methods and mirror descent algorithm Iterative gradient descent (GD) scheme, which can be traced back to (Cauchy, 1847), is the simplest strategy to minimize convex optimization problems.

Original: It was further developed as Zinkevich Online Greedy Subgradient
Revised: It was further developed as Zinkevich Online Greedy Subgradient

Original: Project (OGSP) (Zinkevich, 2003), which can be considered as a variation of project gradient descent (PGD) algorithm with following updating rules:
Revised: Project (OGSP) (Zinkevich, 2003), which can be considered as a variation of project gradient descent (PGD) algorithm with following updating rules:

Original: [Equation]
Revised: [Equation]

Original: where P A X is a distance-based projector denoting the projection of a point y onto X by P A X ( y )
Revised: where P A X is a distance-based projector denoting the projection of a point y onto X by P A X ( y )

Original: =
Revised: =

Original: arg min x ∈X || x − y
Revised: arg min x ∈X || x − y

Original: || A ,  || ·||
Revised: || A , and || ·

Original:  A = (cid:104) x, Ax (cid:105) , and g t ∈ ∂f t ( x t ) =
Revised: || A = (cid:104) x, Ax (cid:105) , and g t ∈ ∂f t ( x t ) =

Original: f
Revised: f

Original: (cid:48) ( x t ) is the subgradient of the objective function f t ( · ) .
Revised: (cid:48) ( x t ) is the subgradient of the objective function f t ( · ) .

Original: The major issue with applying PGD is that PGD only works inHilbert space
Revised: The well-known bottleneck of using PGD is that it can only work in

Original:  H and it cannot be extended to more general situation (in modern machine learning) of optimization  in some Banach space B where the Euclidean norms cannot be computed.
Revised: Hilbert space H and it cannot be extended to more general cases (in modern machine learning) of optimization plays in some Banach space B where the Euclidean norms cannot be computed.

Original: To this end, the mirror descent algorithm (MDA) introduced by (Nemirovsky & Yudin, 1983) overcomes such infeasibility by using the linearity on dual vector space B ∗ and a carefully designed mirror map (Bubeck, 2014).
Revised: To this end, the mirror descent algorithm (MDA) introduced by (Nemirovsky & Yudin, 1983) overcomes such infeasibility by using the linearity on dual vector space B ∗ and a carefully designed mirror map (Bubeck, 2014).

Original:  MDA has following updating schemes:
Revised: Hence, MDA has following updating schemes:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: the nonlinear projection therefore encoded in the term D ψ .

Original: By replacing the Euclidean quadratic norms in Equation.
Revised: By replacing the Euclidean quadratic norms in Equation.

Original: 3 with more general distance-liked settings such as Bregman distance function D ψ ( · ) deﬁne on ψ ( · ) , the equivalence of PGD algorithm and 
Revised: 3 with more general distance-liked settings such as Bregman distance function D ψ ( · ) deﬁne on ψ ( · ) , the equivalence of PGD algorithm and MDA has been proved by Beck &

Original: MDA has been proved by Beck & Teboulle (2003).
Revised: Teboulle (2003).

Original: For example, the simplest version of MDA is that: taking ψ ( x ) =
Revised: For example, the simplest version of MDA is that: taking ψ ( x ) =

Original: 12 || x || 22 , and D ψ ( x, x t ) =
Revised: 12 || x || 22 , and D ψ ( x, x t ) =

Original: 12 || x − x t || 22 then plugging into Equation.
Revised: 12 || x − x t || 22 then plugging into Equation.

Original: 4, we have:
Revised: 4, we have:

Original: [Equation]
Revised: [Equation]

Original: Update scheme in the above equation.
Revised: Update scheme in the above equation.

Original: 5 is also known as proximal point algorithm
Revised: 5 is also known as proximal point algorithm

Original: (Rockafellar, 1976).
Revised: (Rockafellar, 1976).

Original: The equivalence of Equation.
Revised: The equivalence of Equation.

Original: 5 and PGD can be achieved by taking the derivative of our target function on x , and rearranging the formula:
Revised: 5 and PGD can be achieved by taking the derivative of our target function on x , and rearranging the formula:

Original: [Equation]
Revised: [Equation]

Original: Now, we see the last two expressions in Equation.
Revised: Now, we see the last two expressions in Equation.

Original: 6 are well-known subgradient descent update.
Revised: 6 are well-known subgradient descent update.

Original: We restate the proposition 3.2 in Beck & Teboulle (2003) as:
Revised: We restate the proposition 3.2 in Beck & Teboulle (2003) as:

Original: Proposition 1.
Revised: Proposition 1.

Original: Assume X is a closed convex subset in R with non-empty interior, and objective function f : X → R is a convex and Lipschitz function.
Revised: Assume X is a closed convex subset in R with non-empty interior, and objective function f : X → R is a convex and Lipschitz function.

Original: Suppose the optimal set of x denoted by X ∗ is non-empty, we can compute the subgradient of f at x as g ∈ ∂f ( x ) .
Revised: Suppose the optimal set of x denoted by X ∗ is non-empty, we can compute the subgradient of f at x as g ∈ ∂f ( x ) .

Original: For a convex mirror mapping function ψ : X → R with conjugate function ψ ∗ deﬁned by ψ ∗ ( y ) = max x ∈X {(cid:104) x, y (cid:105) − ψ ( x ) } .
Revised: For a convex mirror mapping function ψ : X → R with conjugate function ψ ∗ deﬁned by ψ ∗ ( y ) = max x ∈X {(cid:104) x, y (cid:105) − ψ ( x ) } .

Original: Then the sequence { x t } ⊆
Revised: Then the sequence { x t } ⊆

Original: X generated by MDA is equivalent to the sequence generated by PGD.
Revised: X generated by MDA is equivalent to the sequence generated by PGD.

Original: We state the equivalence of PGD algorithm and MDA, particularly, the general gradient descent (DG or SGD) can be directly derived from the Equation.
Revised: We state the equivalence of PGD algorithm and MDA, particularly, the general gradient descent (or SGD) can be directly derived from the Equation.

Original: Follow the proximally-reqularized leader (FTPRL) FTPRL is introduced by McMahan & Streeter (2010) belongs to the family of follow-the-regularized-leader (FTRL) algorithm such as Regularized Dual Averaging (Xiao, 2010).
Revised: Follow the proximally-reqularized leader (FTPRL) FTPRL is introduced by McMahan & Streeter (2010) belongs to the family of follow-the-regularized-leader (FTRL) algorithms such as Regularized Dual Averaging (Xiao, 2010).

Original: In general, FTRL-Type has following update rule:
Revised: In general, FTRL-Type has following update rule:

Original: [Equation]
Revised: [Equation]

Original: where the subgradient of objective function f (cid:48) τ ( x τ ) is approximated by the gradient at x τ and
Revised: where the subgradient of objective function f (cid:48) τ ( x τ ) is approximated by the gradient at x τ and

Original: R 1: t ( x ) is deﬁned as regularization.
Revised: R 1: t ( x ) is deﬁned as regularization.

Original: Particularly, the formal FTPRL is:
Revised: Particularly, the formal FTPRL is:

Original: [Equation]
Revised: [Equation]

Original: with φ 1: t · x + Ψ( x ) can be considered as non-smooth composite term which is orthogonal to our  paper, more detail can be found in (McMahan, 2010b).
Revised: with φ 1: t · x + Ψ( x ) can be considered as non-smooth composite term which is orthogonal to our [Table] paper, more details can be found in McMahan (2010b).

Original: The last term in  above equation is stabilizing regularization that ensure low regret.
Revised: The last term in the above equation is stabilizing regularization to ensure low regret.

Original: It is also worth mentioning that the Q τ can be  regarded as  generalized learning rate which plays crucial role in this paper.
Revised: We mention that the Q τ can be either regarded as scale of regularization or generalized learning rate which plays crucial role in our paper.

Original: As we can see, FTPRL appears quite different from MDA stated in Equation.
Revised: As we can see, FTPRL appears quite different from MDA stated in Equation.

Original: 5, however, in McMahan (2010a;b) they show that in the case of selecting quadratic stabilizing regularization, the FTPRL and generalized MDA only has differences in parameter centering.
Revised: 5, however, in McMahan (2010a;b) they show that in the case of selecting quadratic stabilizing regularization, the FTPRL and generalized MDA only has differences in parameter centering.

Original: In fact, MDA illustrates in  Equation.
Revised: In fact, MDA is illustrated in the Equation.

Original: 5 regularizing the parameter to be close to the origin, on contrast, FTPRL is regularizing the parameter at current feasible point.
Revised: 4 regularizing the parameter to be close to the origin, on contrast, FTPRL is regularizing the parameter at current feasible point.

Original: No surprising, McMahan (2010a) propose the equivalence proof of FTPRL and a variation algorithm of the MD as follow.
Revised: No surprising, McMahan (2010a) propose the equivalence proof of FTPRL and a variation algorithm of the MDA as follow.

Original: Proposition 2.
Revised: Proposition 2.

Original:  Let R t be a sequence of differentiable convex functions ( ∇ R t (0) = 0 ), and let Ψ be an arbitrary convex function.
Revised: (McMahan, 2010a) Let R t be a sequence of differentiable convex functions ( ∇ R t (0) = 0 ) , and let Ψ be an arbitrary convex function.

Original: Deﬁne the proximal-MDA with updating rule:
Revised: Deﬁne the proximal-MDA with updating rule:

Original: [Equation]
Revised: [Equation]

Original:  where the Bregman distance function D R t with respect to R t where R t ( x ) =
Revised: (9) where the Bregman distance function D R t with respect to R t where R t ( x ) =

Original: R t ( x − x t ) .
Revised: R t ( x − x t ) .

Original: And applying FTPRL to the same objective function, with:
Revised: And applying FTPRL to the same objective function, with:

Original: [Equation]
Revised: [Equation]

Original: when φ t ∈ ∂ Ψ , such that g 1: t + φ 1: t + ∇ R 1: t ( x t ) = 0 .
Revised: when φ t ∈ ∂ Ψ , such that g 1: t + φ 1: t + ∇ R 1: t ( x t ) = 0 .

Original: Then the two above update scheme are equivalent.
Revised: Then the two above update scheme are equivalent.

Original: At this moment, we state a series of equivalence proposition from FTPRL to proximal-MDA (variation of MDA) and MDA to PGD.
Revised: At this moment, we state a series of equivalence proposition from FTPRL to proximal-MDA (variation of MDA) and MDA to PGD.

Original: Now, we can construct the equivalence proof of FTPRL and
Revised: Now, we can construct the equivalence proof of FTPRL and

Original: PGD properly.
Revised: PGD properly.

Original: Although the direct proof of equivalence between FTPRL and PGD is provided unofﬁcially in McMahan (2010a), we would like to elaborate the intuition behind each algorithm and deduce our perspective to understanding the-sate-of-art algorithm such as adaptive method.
Revised: Although the direct proof of equivalence between FTPRL and PGD is provided unofﬁcially in McMahan (2010a), we would like to elaborate the intuition behind each algorithm and deduce our perspective in understanding the Adam-Type algorithms.

Original: 2. M OTIVATION
Revised: 2. M OTIVATION

Original: Non-Adam-Type and Adam-Type algorithms to mirror descent (FTPRL) Non-Adam-Type online gradient descent algorithm including SGD, SGD with momentum, Polyak’s HB and Nesterov’s accelerated gradient method can be easily understood as ﬁrst order optimization with different momentum function.
Revised: Non-Adam-type and Adam-type algorithms to mirror descent (FTPRL) Non-Adam-type online gradient descent algorithms including SGD, SGD with momentum , Polyak’s HB and Nesterov’s accelerated gradient method can be easily understood as ﬁrst order optimization with different momentum function.

Original: And, interestingly, most of them have physical interpretations.
Revised: And, interestingly, most of them have physical interpretations.

Original: However, as shown in Table, Adam-Type algorithms do not rely on the non-increasing learning rate when iteratively updating their parameters, instead, they perform adaptively learning rate element-wise on parameters according to the ﬁrst order information.
Revised: However, as shown in Table 1, Adam-Type algorithms do not rely on the non-increasing learning rate when iteratively updating their parameters. Instead, they perform adaptively learning rate element-wise on parameters according to the ﬁrst order information.

Original: This leads to the most mystery part in the
Revised: However, this leads to the most mysterious part in the

Original: Adam-Type algorithm where the adaptively update is represented as − η 0 √ v t (cid:12) m t .
Revised: Adam-Type algorithm where the adaptive update is represented as − η 0 √ v t (cid:12) m t .

Original: One commonly interpretation of this updating representation is regarding − η 0 √ v t as adaptive learning rate, and regarding m t as general ﬁrst order gradient.
Revised: One common interpretation of this updating representation is regarding − η 0 √ v t as adaptive learning rate, and regarding m t as general ﬁrst order gradient.

Original: However, we can not treat Adam-Type algorithm in this way because the ﬁrst order information g t resides in both m t and v t and we are unable to simply decorrelate them as learning rate scheme.
Revised: However, we can not treat Adam-Type algorithm in this way because the ﬁrst order information g t resides in both m t and v t and we are unable to simply decorrelate them as common gradient with step-size scheme.

Original: Another possible interpretation of such updating is related to the Newton’s second order method, but there is no free lunch for expressing in this way.
Revised: Another possible interpretation of such updating is related to the Newton’s second order method, but there is no free lunch for expressing in this way.

Original: We will discuss it later.
Revised: We will discuss it later.

Original: In order to dissect Adam-Type algorithm, we recall the FTPRL algorithm mentioned in the previous section.
Revised: In order to dissect Adam-Type algorithm, we recall the FTPRL algorithm mentioned in the previous section.

Original: A natural question raise: can we interpret Adam-Type algorithm as a variant of Mirror Descent method?
Revised: A natural question arises: can we interpret Adam-Type algorithm as a variant of mirror descent method?

Original: Before answer the question, let us explain why we present Adam-Type algorithm as MD is beneﬁcial?
Revised: Before answering the question, let us explain why we present Adam-Type algorithm as MDA is beneﬁcial.

Original: We summary in the following:
Revised: We summary below: [Table]

Original: Implicit updates: This concept was ﬁrst derived by Kivinen & Warmuth (1997) and later pointed by Kulis & Bartlett (2010).
Revised: Implicit updates: This concept was ﬁrst derived by Kivinen & Warmuth (1997) and later pointed by Kulis & Bartlett (2010).

Original: It refers that the without using explicit ﬁrst-order update rules to efﬁciently compute the parameters.
Revised: It refers to without using explicit ﬁrst-order update rules to efﬁciently compute the parameters.

Original: The mirror descent algorithms, in fact, adopt implicit update rules very well, therefore, the explicit update will roll into some regularizedliked terms which can elaborate more insights.
Revised: The mirror descent algorithms, in fact, adopt implicit update rules very well, therefore, the explicit update will roll into some regularizedliked terms which can elaborate more insights.

Original: In contrast, like Adam, the learning rate is explicitly deﬁned as η 0 √ v t which is hard to identify from intuition unless relying on the theoretically proofs.
Revised: In contrast, like Adam, the learning rate is explicitly deﬁned as η 0 √ v t which is hard to identify from intuition unless relying on the theoretical proofs.

Original: First-order information dissection The entire adaptive update involves in Hadamard product and division.
Revised: First-order information dissection The entire adaptive update involves in Hadamard product and division.

Original: Both numerator and denominator are highly related to the construction function  respect to the ﬁrst-order gradient g .
Revised: Both numerator and denominator are highly related to the construction function with respect to the ﬁrst-order gradient g .

Original: The underlying relation between two functions m t and v t can not be decorrelated.
Revised: The underlying relation between two functions m t and v t can not be decorrelated.

Original: However, as show in Table, we represent Adam-Type algorithms in MD way so that we can separate the numerator and denominator as simple additive scheme where the hard-understanding division disappear and meanwhile, the alignment of m t and v t gone.
Revised: However, as shown in Table 2, we represent Adam-Type algorithms in MDA way so that we can separate the numerator and denominator as simple additive scheme where the hard-understanding division disappears and meanwhile, the alignment of m t and v t gone.

Original: Equivalence guarantee The original adaptive method AdaGrad isbuild upon non-
Revised: Equivalence guarantee The original adaptive method AdaGrad is

Original: linear subgradient projection, andthe Adam actually inherit such design indicate by their regret proof.
Revised: build upon non-linear subgradient projection, and Adam actually inherit such design indicate by their regret proof.

Original: To re-represent Adam-Type algorithm as FTPRL style, we do require the theoretically equivalence guarantee so that we can transform safely.
Revised: To re-represent Adam-Type algorithm as FTPRL style, we do require the theoretical equivalence guarantee so that we can transform safely.

Original: As we discuss in previous (Proposition 1& 2), we successfully build such  bridge in between two types  algorithm.
Revised: As we discuss in previous (Proposition 1& 2), we successfully build such a bridge in between two types of algorithm.

Original: In general, these ﬁrst-order algorithms can be written in FTPRL style:
Revised: In general, these ﬁrst-order algorithms can be written in FTPRL style (also see Table 2):

Original: [Equation]
Revised: [Equation]

Original: The understanding of above representation is highly related to our analysis on Adam-Type algorithms.
Revised: The understanding of above representation is highly related to our analysis on Adam-Type algorithms.

Original: Term A has two parts, the ﬁrst part g 1: t · x is an approximation to f 1: t based on the gradient; the second part C 1: t · x refers as ﬁrst-order momentum correction or fault tolerant in this literature.
Revised: Term A has two parts, the ﬁrst part g 1: t · x is an approximation to f 1: t based on the gradient; the second part C 1: t · x refers to ﬁrst-order momentum correction or fault tolerant in this literature.

Original: Term B is similar to the setting of FTPRL, but we usually consider it as l 2 regularization.
Revised: Term B is similar to the setting of FTPRL, but we usually consider it as l 2 -regularization.

Original: In addition, term C stabilizing regularization plays very crucial role in this transformation because (1) the implicit updates from x t to x t +1 happens in this term; (2) the Q τ reside in norms can be regarded as generalized learning rate or even more complicated format; (3) the rule of parameter centering to  current feasible solution is ﬁgured.
Revised: In addition, term C stabilizing regularization plays a crucial role in this transformation because (1) the implicit updates from x t to x t +1 happen in this term; (2) the Q τ residing in norms can be regarded as generalized learning rate or even more complicated format; (3) the rule of parameter centering to the current feasible solution is ﬁgured.

Original: Finally, the leading ρ is a balancing coefﬁcient , aims at controlling the tendency of minimization between term A and C. Smaller ρ value will guide the minimization process  relies more on term C, otherwise term A will dominate the minimization.
Revised: Finally, the leading ρ is a balancing coefﬁcient , aims at controlling the tendency of minimization between term A and C. Smaller ρ value will guide the minimization process and relies more on term C, otherwise term A will dominate the minimization.

Original: The simplest format transformation from Vanilla SGD to MD is illustrate in Equation.
Revised: The simplest format transformation from Vanilla SGD to MD is illustrated in Equation.

Original: Now, we rewrite it as FTPRL style according to Eqution.
Revised: Now, we rewrite it as FTPRL style according to Equation.

Original: 8 (for the sake of simplicity, we do not consider the Ψ temporarily, we will discuss it later):
Revised: 8 (for the sake of simplicity, we do not consider the Ψ ):

Original: [Equation]
Revised: [Equation]

Original: where the ρ = 1 , C 1: t is zero, and Q τ sets to be Q τ = σ 2 τ I .
Revised: where the ρ = 1 , C 1: t is zero, and Q τ sets to be Q τ = σ 2 τ I .

Original: If Vanilla SGD with learning rate η t = 1 t , the (cid:80) tτ =1 σ τ = t .
Revised: If Vanilla SGD with learning rate η t = 1 t , the (cid:80) tτ =1 σ τ = t .

Original: If it with ﬁxed learning rate η t = 1 k , then (cid:80) tτ
Revised: If it with ﬁxed learning rate η t = 1 k , then (cid:80) tτ

Original: =
Revised: =

Original: 1 σ τ = k .
Revised: 1 σ τ = k .

Original: To be mentioned here, ﬁxed learning rate SGD is kind of special, it can be easily explained as Equation.
Revised: To be mentioned here, ﬁxed learning rate SGD is kind of special, it can be easily explained as Equation.

Original: 5, however, in FTPRL format (Equation. 12), we need to be more careful.
Revised: 5, however, in FTPRL format (Equation. 12), we need to be more careful.

Original: Mirror descent liked Vanilla SGD with non-increasing learning rate show the great insight of understanding current algorithms.
Revised: Mirror descent like Vanilla SGD with non-increasing learning rates shows the great insight of understanding current algorithms.

Original: First, we are always looking for the next step x t +1 in the opposite direction of the current gradient because cos( π ) = − 1 minimize Equation.
Revised: First, we are always looking for the next step x t +1 in the opposite direction of the current gradient because cos( π ) = − 1 minimize Equation.

Original: Second, x t +1 is being bounded inthe a region centered at previous step x t .
Revised: Second, x t +1 is being bound in a region centered at previous step x t .

Original: This is due to the fact that we do not want the new solution  be far away from the current feasible solution, otherwise, may cause slow convergence,  McMahan (2010b) conﬁrm this view.
Revised: This is due to the fact that we do not want the new solution to be far away from the current feasible solution, otherwise, may cause slow convergence, and McMahan (2010b) conﬁrm this view.

Original: Third, (cid:80) tτ =
Revised: Third, (cid:80) tτ =

Original: 1 σ τ should be non-decreasing alone the time, similar, when σ τ get smaller, the bounding region expand reducing the convergence speed.
Revised: 1 σ τ should be non-decreasing alone time. Similar, when σ τ gets smaller, the bounding region expands, reducing the convergence speed.

Original:  N EW P ERSPECTIVE ON A DAM
Revised:  N EW P ERSPECTIVE ON A DAM

Original: In this section, we deliver a new perspective on Adam-Type algorithm from the Mirror Descent point of view.
Revised: In this section, we deliver a new perspective on Adam-Type algorithm from the Mirror Descent point of view.

Original: In this lecture, we mainly focus on Adam to demonstrate our analysis, for the other variants, they have similar results.
Revised: In this lecture, we mainly focus on Adam to demonstrate our analysis, for the other variants, they have similar results.

Original: 3. A DAM ON M IRROR D ESCENT
Revised: 3. A DAM ON M IRROR D ESCENT

Original: According to the Table, the subgradient projected Adam can be written as follows (entry-wise).
Revised: According to the Table 2, the subgradient projected Adam can be written as follows (entry-wise).

Original: In order to make a clear comparison, we also show SGDM alongside.
Revised: In order to make a clear comparison, we also show SGDM alongside.

Original: [Equation]
Revised: [Equation]

Original:  where both β 1 ≤ 1 (commonly chose β 1 = 0 . 9 ), the (cid:80) tτ=1 σ τ,i =(cid:113) (1 − β
Revised: t (cid:88) τ = σ ∗ τ || x ,i − x τ,i || 22 } (13) where both β 1 ≤ 1 (commonly chose β 1 = 0 . 9 ), the (cid:80) tτ =

Original: [Equation] (cid:80) τ[Equation] g
Revised: 1 σ τ,i = (cid:113) (1 − β 2 ) (cid:80) τ

Original: [Equation] i in Adam settings with β 2 = 0 .
Revised: = t τ =1 β t − τg 2 τ,i in Adam settings with β 2 = 0 .

Original: In SGDM, (cid:80) t τ =1 σ ∗ τ performs differently, (1) SGDM is nonadaptive method, the τ ∗ applies to all entry on the x ; (2) (cid:80) tτ
Revised: In SGDM, (cid:80) tτ = 1 σ ∗ τ performs differently, (1) SGDM is nonadaptive method, the τ ∗ applies to all entry on the x ; (2) (cid:80) tτ

Original: =1 σ ∗ σ = t if we have learning rate η t = 1 t , it also means σ ∗ = 1 for all τ ∈ T .
Revised: =1 σ ∗ σ = t if we have learning rate η t = 1 t , it also means σ ∗ = 1 for all τ ∈ T .

Original: First, let us recall Vanilla SGD and compare it with
Revised: First, let us recall Vanilla SGD and compare it with

Original: SGDM:
Revised: SGDM:

Original: Corollary 1.
Revised: Corollary 1.

Original: Compare to Vanilla SGD, SGDM employs ﬁrst-order momentum correction deﬁned in Section.
Revised: Compare to Vanilla SGD, SGDM employs ﬁrst-order momentum correction deﬁned in Section.

Original: 2.2 to correct the possible wrong direction prediction, making a smooth optimization trajectory.
Revised: 2.2 to correct the possible wrong direction prediction, making a smooth optimization trajectory.

Original: (This is a well known truth.
Revised: (This is a well known truth.

Original: We verify it by experiments show in Appendix.)
Revised: We verify it by experiments show in Appendix.)

Original: Besides, we also notice that Adam has ρ = 1 while SGDM has ρ = 1 1 − β 1 = 10  . 
Revised: Besides, we also notice that Adam has ρ = 1 while SGDM has ρ = 1 1 − β 1 = 10 (a common setting of β = 0 . 9 ).

Original: Recall the previous section, by deﬁnition of ρ , we have following corollary:
Revised: Recall the previous section, by deﬁnition of ρ , we have following corollary:

Original: Corollary 2.
Revised: Corollary 2.

Original: Compare to Vanilla SGD or Adam, SGDM has larger ρ value indicates that the SGDM is more sensitive on the value change of loss function .
Revised: Compare to Vanilla SGD or Adam, SGDM has larger ρ value indicates that the SGDM is more sensitive on the value change of loss function .

Original: To explain this, we know one of the drawbacks of SGDM is that SGDM is prone to oscillation around the optimal point, because SGDM has relative weak bound in proximal term, and is very sensitive on small change of loss, that is, ρ factor ampliﬁes this loss change up to tenfold. 
Revised: To explain this, we know one of the drawbacks of SGDM is that SGDM is prone to oscillation around the optimal point, because SGDM has relatively weak bound in proximal terms, and is very sensitive to small change of loss, that is, ρ factor ampliﬁes this loss change up to tenfold. We notice that Adam avoids such imbalance between objective function and its parameter regularization by using EMA wisely.

Original: Now, back tothe Adam, before we move forward, we deﬁne some terms in  Equation.
Revised: Now, back to Adam, before we move forward, we deﬁne some terms in the Equation.

Original: 13.Adam,
Revised: 13.Adam,

Original: Proximal Searching Region refer as D = || x − x τ || 22 , this Euclidean quadratic norm reﬂects the geometry of given constraints feasible set X .
Revised: Proximal Searching Region refers to D = || x − x τ || 22 , this Euclidean quadratic norm reﬂects the geometry of given constraints feasible set X .

Original: We can also treat it as a regularization process.
Revised: We can also treat it as a regularization process.

Original: This region is inversely proportional ( D ∝ 1 B ) to the Regularization
Revised: This region is inversely proportional ( D ∝ 1 B ) to the Regularization

Original: Budget deﬁned below.
Revised: Budget deﬁned below.

Original: Regularization Budget refer as B = t (cid:88) τ
Revised: Regularization Budget refer to B = t (cid:88) τ

Original: = σ τ,i , it indicates total ”weight” we can distribute to the Proximal Searching Region.
Revised: = σ τ,i , it indicates total ”weight” we can distribute to the Proximal Searching Region.

Original: More weight it has been given will lead to a stronger regularization in bounding, and a small searchable space centering at x τ .
Revised: More weight it has been given will lead to a stronger regularization in bounding, and a small search space centering at x τ .

Original: In Hoffer et al.
Revised: In Hoffer et al.

Original: (2017), they deﬁne an ultra slow diffusion phenomenon when they evaluate the distance from current weight to initialization weight point with || x t −[-x 
-]
Revised: (2017), they deﬁne an ultra slow diffusion phenomenon when they evaluate the distance from current weight to initial weight point with || x t −[-x 
-]

Original:  || 22 ∼ log t .
Revised: x 0 || 22 ∼ log t .

Original: Interestingly, this result is entirely consistent with our Proximal Searching Region analysis, because for any t ∈ T , we have || x t +1 − x t || ∼ ( logt ) (cid:48)
Revised: Interestingly, this result is entirely consistent with our Proximal Searching Region analysis, because for any t ∈ T , we have || x t +1 − x t || ∼ (log t ) (cid:48)

Original: = 1 t .
Revised: = 1 t .

Original: Now, we summarise our result as follow:
Revised: Now, we summarise our results as follows:

Original: Hyper-parameter β 1 : β 1 exponential smooth eliminates the presence of imbalance in between the goal of minimizing loss  function and the constraints of searching in proximal region.
Revised: Hyper-parameter β 1 : β 1 exponential smooth eliminates the presence of imbalance in between the goal of minimizing loss of function and the constraints of searching in proximal region.

Original: In other words, Adam treats both conditions fairly.
Revised: In other words, Adam treats both conditions fairly.

Original: First-order momentum: the usage of m t leads to a smooth optimization trajectory which avoid the sharp twist such as SGD.
Revised: First-order momentum: the usage of m t leads to a smooth optimization trajectory which avoids the sharp twist such as SGD.

Original: It can beneﬁt Adam if searching in  wrong direction, the momentum correction C τ can compensate the party of penalty directly from the loss function.
Revised: It can beneﬁt Adam if searching in the wrong direction, the momentum correction C τ can compensate the party of penalty directly from the loss function.

Original: Adam-Type algorithm:
Revised: Adam-Type algorithm:

Original: Adam-Type algorithms such as AdamG (this paper), AMSGrad AdaBound, AdaShift, NosAda, etc can be regarded as making a correction on one of regularization term (most on || Q 

Revised: Adam-Type algorithms such as AdamAL (this paper), AMSGrad AdaBound, AdaShift, NosAda, etc. can be regarded as making a correction on one of regularization term (most on || Q

Original: τ || x − x τ || 22 ).
Revised: τ || x − x τ || 22 ).

Original: Learning rate: mirror descent corporate with implicit update makes learning rate (step size) act as a scalar factor of regularization.
Revised: Learning rate: mirror descent corporate with implicit update makes learning rate (step size) act as a scalar factor of regularization.

Original: By transferring the Adam to MD-liked method, we successfully disassemble the Adam updating − m t √ v t into two additive terms and identify their functionality separately.
Revised: By transferring the Adam to MDA-liked method, we successfully disassemble the Adam updating − m t √ v t into two additive terms and identify their functionality separately.

Original: We mainly focus on the m t part in this section, we will move our eye on 1 √ v t in  next section.
Revised: We mainly focus on the m t part in this section, we will move our eye on 1 √ v t in the next section.

Original: 3. A DAM V t AND THE NON - CONVERGENCE OF A DAM
Revised: 3. A DAM V t AND THE NON - CONVERGENCE OF A DAM

Original: A commonly thinking of Adam’s adaptive learning rate 1 √ v t will treat v t as second moment approximation.
Revised: A common thinking of Adam’s adaptive learning rate 1 √ v t will treat v t as second moment approximation.

Original: However, in our perspective, we regard it as adaptive regularize scalar and  performs implicit update by replacing the explicit learning rate.
Revised: However, in our perspective, we regard it as adaptive regularize scalar and it performs implicit updates by replacing the explicit learning rate.

Original: Two adaptive regularize from AdaGrad and
Revised: Two adaptive regularize from AdaGrad and

Original: Adam show in below: 
Revised: Adam show in below: Adam:

Original: [Equation] 
Revised: [Equation] σ

Revised: τ,

Revised: [Equation]

Revised: = (cid:118)(cid:117)(cid:117)(cid:116) (

Revised: [Equation]

Revised: ) τ = t (cid:88) τ =

Revised: [Equation]

Revised: β t

Revised: [Equation]

Revised: g

Revised: [Equation]

Revised: ,i

Revised: AdaGrad: t (cid:88) τ

Revised: = σ τ,i = (cid:118)(cid:117)(cid:117)(cid:116)

Original:  By guiding with this intuition and our regularization budget deﬁnition, we know can conclude that
Revised: τ = t (cid:88) τ = g 2 τ,i (14) By guiding with this intuition and our regularization budget deﬁnition we now can conclude that

Original: Strictly speaking, non-Adam-Type algorithm such as SGD(M) has only proximal searching region center at x t , however, Adam-Type algorithms achieve globally stabilizing regulations via proximal searching region through { x 1 , x 2 , · · · , x t − 1 , x t } .
Revised: Strictly speaking, non-Adam-Type algorithm such as SGD(M) has only proximal searching region center at x t , however, Adam-Type algorithms achieve globally stabilizing regulations via proximal searching region through { x 1 , x 2 , · · · , x t − 1 , x t } .

Original: Therefore, in our settings, SGD has σ function with a constant value where { σ τ = k | τ = t } .
Revised: Therefore, in our settings, SGD has σ function with a constant value where { σ τ = k | τ = t } .

Original: A special case is when η t = 1 t where the SDG has a descending step size 1 t , we notice that (cid:80) tτ = 1 σ τ = t and σ t = 1 .
Revised: A special case is when η t = 1 t where the SDG has a descending step size 1 t , we notice that (cid:80) t τ =1 σ τ = t and σ t = 1 .

Original: In fact, in our theory, we can regard SGD with decreasing learning rate as a adaptive regularization withthe respect to training iterations.
Revised: In fact, in our theory, we can regard SGD with decreasing learning rate as an adaptive regularization with respect to training iterations.

Original: Adam-Type algorithms, in contrast, have stronger bounding constraints with each proximal term || x − x τ || 22 .
Revised: Adam-Type algorithms, in contrast, have stronger bonding constraints with each proximal term || x − x τ || 22 .

Original: With the training iterations increasing, in order to retain thethe similar regularization strength, the natural way is increasing the regularization budget (cid:80) tτ
Revised: With the training iterations increasing, in order to retain the similar regularization strength, the natural way is increasing the regularization budget (cid:80) tτ

Original: =
Revised: =

Original: 1 σ τ such that B t ≥ B t − 1 .
Revised: 1 σ τ such that B t ≥ B t − 1 .

Original: We have the following comments for Adam: 2.1 Pro: Non-decreasing regularization budget B t can beneﬁt for Adam, however, unbounded budget cause the inﬁnitely small proximal searching region, on the surface, training will stop without parameters update.
Revised: We have the following comments for Adam: 2.1 Pro: Non-decreasing regularization budget B t can beneﬁt for Adam, however, unbounded budget causes the inﬁnitely small proximal searching region. On the surface, training will stop without parameters updates.

Original: Adam over come this problem wisely, exponential moving average (EMA) preforms as a sliding window, in other words,  regularization budget is bounded.
Revised: Adam overcomes this problem wisely, exponential moving average (EMA) performs as a sliding window. In other words, the regularization budget is bounded.

Original: 2.2 Con: EMA, on the one hand controls the regularization budget in a range, but it fails to satisfy the primary requirement that regularization budget B t should be in nondecreasing manner.
Revised: 2.2 Con: EMA, on the one hand, controls the regularization budget in a range, but it fails to satisfy the primary requirement that regularization budget B t should be a non-decreasing manner.

Original: Lot of previous works point out this issue such as (Reddi et al., 2019; Luo et al., 2019; Chen et al., 2018).
Revised: Lots of previous works point out this issue such as (Reddi et al., 2019; Luo et al., 2019; Chen et al., 2018).

Original: However, the way the identify such problem is not natural, for example, the objective function with periodicity gradient rarely seen in real scenario.
Revised: However, the way they identify such problems is not natural, for example, the objective function with periodicity gradient rarely seen in real scenarios.

Original: Our way seems more easy to access
Revised: Our way seems more easy to access.

Original: The non-convergence of Adam This issue was ﬁrst identiﬁed by Reddi et al.
Revised: The non-convergence of Adam This issue was ﬁrst identiﬁed by Reddi et al.

Original: (2019), which points out that the key issue in the convergence of Adam lies in the quantity
Revised: (2019), which points out that the key issue in the convergence of Adam lies in the quantity

Original: [Equation]
Revised: [Equation]

Original: which assumes to be a non-negative value, but in training, this assumption dose not always hold in Adam.
Revised: which assumes to be a non-negative value, but in training, this assumption does not always hold in Adam.

Original: Reddi et al.
Revised: Reddi et al.

Original: (2019) construct an objective function with periodicity gradient to illustrate the non-convergence of Adam which is hard to follow.
Revised: (2019) construct an objective function with periodicity gradient to illustrate the non-convergence of Adam which is hard to follow.

Original: And (Luo et al., 2019) using the similar way but conduct a heuristic experiment shows thatthe Adam will generate extreme learning rates (also extreme v t value)  can effect the convergence.
Revised: And Luo et al. (2019) using a similar way but conduct a heuristic experiment shows that Adam will generate extreme learning rates (also extreme v t value) that can affect the convergence.

Original: Fundamentally, they are identifying the same problem in different expressions.
Revised: Fundamentally, they are identifying the same problem in different expressions.

Original: In Reddi et al.
Revised: In Reddi et al.

Original: (2019) non-convergence Theorem.1 the repeated occurrence gradient − C is, in fact, the extreme learning rates.
Revised: (2019) non-convergence Theorem.1 the repeated occurrence gradient − C is, in fact, the extreme learning rate.

Original: In our analysis, the emerge of extreme learning rates is due to decreasing regularization budget B t − B t − 1 < 0 which  equivalent to Γ < 0 , the negative regularization coefﬁcient σ τ < 0 makes the corresponding proximal searching region || x − x τ || 22 to be inﬁnite large when minimizing the Equation.
Revised: In our analysis, the emergence of extreme learning rates is due to decreasing regularization budget B t − B t − 1 < 0 which is equivalent to Γ < 0 , the negative regularization coefﬁcient σ τ < 0 makes the corresponding proximal searching region || x − x τ || 22 to be arbitrarily large when minimizing the Equation.

Original: 13.Adam.
Revised: 13.Adam.

Original: Again, the parameters behavior in out of the control manner for example || x − x τ || 22 → + ∞ .
Revised: Again, the parameters behavior in out of the control manner for example || x − x τ || 22 → + ∞ .

Original: To this end, Reddi et al.
Revised: To this end, Reddi et al.

Original: (2019) propose a very intuitive modiﬁcation on Adam where
Revised: (2019) proposes a very intuitive modiﬁcation on Adam where

Original: [Equation]
Revised: [Equation]

Original: Although this setting solve the decreasing regularization budget issue, it still remain the problem so called nonalignment projection, we state this problem in  next section.
Revised: Although this setting solves the decreasing regularization budget issue, it still remains the problem so called nonalignment projection, we state this problem in the next section.

Original: 3. NON - ALIGNMENT PROJECTION AND A DAM AL
Revised: 3. NON - ALIGNMENT PROJECTION AND A DAM AL

Original: AMSGrad (Reddi et al., 2019) using Equation.
Revised: AMSGrad (Reddi et al., 2019) uses Equation.

Original: 16 to ensure the Γ t ≥ 0 for all t ∈ T .
Revised: 16 to ensure the Γ t ≥ 0 for all t ∈ T .

Original: They derive it mainly from an unrealistic objective function.
Revised: They derive it mainly from an unrealistic objective function which has extreme gradient appearance in periodical manner.

Original: Does it really solve the problem or dose this design violate the intuition of Adam-Type algorithm?
Revised: Does AMSGrad really outperform Adam or does AMSGrad’s design follow the general sketch of Adam-Type algorithm?

Original: We conduct the experiment to illustrate the fundamental problem of AMSGrad (or its variant AdaShift).
Revised: We conduct an experiment to illustrate the fundamental problem of AMSGrad (or its variant AdaShift).

Original: We call this problem as nonalignment projection .
Revised: We call this problem as non-alignment projection .

Original: To illustrate the non-alignment projection problem, we trace a series of entries generated by AMSGrad and sample them from  v t,i : j , our goal is counting the total  times of that entry being swap with v t − 1 ,i : j and meanwhile, we record the interval between two swaps.
Revised: To illustrate the non-alignment projection problem, we trace a series of entries generated by AMSGrad and sample them from vector v t,i : j , our goal is counting the total number of times of that entry being swap with v t,i : j and meanwhile, we record the interval between two swaps.

Original: We employ a heatmap to visualize this result.
Revised: We employ a heat map to visualize this result.

Original: As show in ﬁgure.1, we can ﬁnd that (1)  different entry has very different swapping counts; (2) the swapping intervals are nonuniform.
Revised: As shown in Figure 1, we can ﬁnd that (1) the different entry has very different swapping counts; (2) the swapping intervals are nonuniform.

Original: Note here sampling v t,i from different neuron network layers and different batch size have different results, but we demonstrate the presence of such problem.
Revised: Note here sampling v t,i from different neuron network layers and different batch size have different results, but we demonstrate the presence of such problems.

Original: To be more speciﬁcally, we present a simple one-step AMSGrad swapping at iteration t and ﬁgure out the ill-condition problem.
Revised: To be more speciﬁc, we present a simple one-step AMSGrad swapping at iteration t and ﬁgure out the ill-condition problem.

Original: Suppose at iteration t  we have corresponding v t , then next step.
Revised: Suppose in iteration t , we receive a corresponding v t,i at i th element.

Original:  we have v t [-+
-] = 
Revised: Then in the next step, we would like to perform the update of v t,i to v t +1 ,i via v t [-+
-]+1 ,i = β 2 v t,i + (1 − β 2 )

Original: β 2v t
Revised: g 2 t

Original: +(1 − β 2 ) g 2 t +1 , at this moment, according to AMSGrad, we  evaluate v t +1 = max { v t +1 , v t } ,if the swap happens, we will receive v t +1 = v t . We likely to use this v t +1 for next step updating namely v t +2 = 
Revised: +1 ,i based on the original design of the Adam. However, if we choose AMSGrad to evaluate v t +2 ,i , we ﬁrst make the comparison between new v t +1 ,i and old v t,i to ensure Γ t +1 to be non-negative and choose ˆ v t +1 ,i = max { v

Original: β 2 v t[-+
-]
Revised: t[-+
-]

Original: + (1 − β 2 ) g 2 t +2 .
Revised: +1 ,i , v t,i } .

Original: However, the real value of v t +2 should be
Revised: If this swap happens, we will have ˆ v t +1 ,i = v t,i and AMSGrad uses it for t + 2 update, and results in

Original: [Equation]
Revised: [Equation]

Original: and AMSGrad will gives us
Revised: However, the true step for updating v t +2 ,i should be

Original: [Equation]
Revised: [Equation]

Original: The difference is quite obvious show in here, but another question is Why AMSGrad still working?
Revised: The difference is quite obvious show in Equation 17 and Equation 18. But another question is Why AMSGrad still working?

Original: AMSGrad can still work is because we choose very big β 2 value as 0.999 which makes the v [-amst +
-] ≈ v t +2  .
Revised: AMSGrad can still work is because we choose very big β 2 value as 0. which makes the v [-amst +
-]ams t +2 ,i ≈ v t +2 ,i .

Original: In fact, iterative method will accumulate this small error into each step and as a consequence, the non-alignment of v t will lead the model to ﬁnd a suspicious local optimal.
Revised: However, iterative method will accumulate this small error into each step and as a consequence, the non-alignment of v t will lead the model to ﬁnd a suspicious local optimal.

Original: Another non-alignment refers to m t and v k , AMSGrad updates its i th parameter by
Revised: Another non-alignment refers to m t and v k , AMSGrad updates its i th parameter by

Original: [Equation]
Revised: [Equation]

Original: where k (cid:54) =
Revised: where k (cid:54) =

Original: t
Revised: t

Original: Recall in the Zinkevich’s greedy projection, we minimize the objective loss function relies on current gradient approximation g t then we perform the projection by projector P −√ v k X to resolve the x t +1 .
Revised: Recall in the Zinkevich’s greedy projection, we minimize the objective loss function relies on current gradient approximation g t then we perform the projection by projector P −√ v k X to resolve the x t +1 .

Original: We see the v k is not the actual projection matrix for x t approximate by m t .
Revised: We see the v k is not the actual projection matrix for x t approximate by m t .

Original: To address above issue, recall the deﬁne of non-decreasing regularization budget, in Adam setting,
Revised: To address above issue, recall the deﬁne of non-decreasing regularization budget, in Adam setting,

Original: B t ≥ B t − 1 equivalent to v t ≥ v t − 1 that is
Revised: B t ≥ B t − 1 equivalent to v t ≥ v t − 1 that is

Original: [Equation]
Revised: [Equation]

Original: Now, the solution for resolving Adam’s non-convergence and non-alignment of AMSGrad is clear.
Revised: Now, the solution for resolving Adam’s non-convergence and non-alignment of AMSGrad is clear.

Original: That is before we evaluate v t , we modify g t to guarantee the non-decreasing condition of v t − 1 to v t .
Revised: That is before we evaluate v t , we modify g t to guarantee the non-decreasing condition of v t − 1 to v t .

Original: We illustrate the update detail of AdamAL in Algorithm.1.
Revised: We illustrate the update detail of AdamAL in Algorithm.1.

Original: Using a similar one-step example to
Revised: Using a similar one-step example to

Original: [Equation] [Equation] 
Revised: [Equation] [Equation] [Equation]

Original: 9: endfor illustrate how AdamAL can mitigate non-alignment issue when update v t +2 :
Revised: [Equation] [Equation] [Equation] end illustrate how AdamAL can mitigate non-alignment issue when update v t +2 :

Original: [Equation]
Revised: [Equation]

Original: which reconstruct the v t +2 in correct expression.
Revised: which reconstruct the v t +2 in correct expression.

Original: The major difference of AdamAL and AMSGrad is we do not skip the gradient update for v t so that we guarantee the alignment of g t and v t so as m t .
Revised: The major difference of AdamAL and AMSGrad is we do not skip the gradient update for v t so that we guarantee the alignment of g t and v t so as m t .

Original: We have the following bound for Adamal.
Revised: We have the following bound for AdamAL.

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: Suppose that the Assumption in Section.
Revised: Let { x t } and { v t } be the sequences obtained from AdamAl

Original: II are satisﬁed, β 1 is chosen such that β 1 ≥ β 1 ,t with β 1 ,t  ∈ (0 , 1) is non-increasing, and for some constant G, we have || α t √ v t m t  || ≤ G , for all t, then the Algorithm. 1 satisfy:
Revised: (Algorithm 1). η t = η/t 1 / 2 , β 1 ,t = β 1 , β 1 ,t non-decreasing for all t ∈ [ T ] . Assume that feasible convex set F such for all x t ∈ F with bounded diameter D 2 and ||∇ f t ( x t ) || ≤ G .

Original: [Equation] where A 0 , A 1 , A 2 , A 3 are constants related to β 1 G.
Revised: For x t generated using the AdamAL, we have the following bound on the regret:

Original: [Figure]
Revised: [Equation]

Original:  EXPERIMENTS
Revised:  EXPERIMENTS

Original: In this section, we turn to an empirical study of different models to compare new variants with popular optimization methods including SGD(M), Adam, and AMSGrad.
Revised: In this section, we turn to an empirical study of different models to compare new variants with popular optimization methods including SGD(M), Adam, and AMSGrad.

Original: We focus on image classiﬁcation task on CIFAR10.
Revised: We focus on image classiﬁcation tasks on CIFAR10 and CIFAR100 with different state-of-the-art deep models including

Revised: ResNet18, ResNet50, VGG16 and VGG19.

Original: From the experiment results show in Figure.
Revised: From the experiment results shown in Figure.

Original: 2, we notice that AdamAL outperforms Adam and AMSGrad!
Revised: 2, we notice that AdamAL outperforms Adam and AMSGrad!

Original: This result is desirable because we ﬁx the  non-alignment projection issue reside in AMSGrad.
Revised: This result is desirable because we ﬁx the [Figure] non-alignment projection issue residing in AMSGrad.

Original: In general, from the experiments, AdamAL constanly achieve 1%  accuracy gain than Adam.
Revised: In general, from the experiments, AdamAL constantly achieve 1% more accuracy gain than Adam.

Original: Notice that we only conduct our experiments with 80 epochs, this is due to that fact that we observe there is no further accuracy improvement without performing any hyperparameter tuning.
Revised: Notice that we only conduct our experiments with 80 epochs, this is due to that fact that we observe there is no further accuracy improvement without performing any hyperparameter tuning.

Original: If we perform hyperparameter tuning, the result show in Table.1.
Revised: If we perform hyperparameter tuning, the results show in Table.1.

Original: AdamAL can ﬁnally reach to 95% accuracy in average on test data, however, the best run of Adam  still lower than AdamAL.
Revised: AdamAL can ﬁnally reach to 95% accuracy on average on test data, however, the best run of Adam is still lower than AdamAL.

Original: It is also worth mentioning that AMSGrad have even worse performance than Adam due to non-alignment.
Revised: It is also worth mentioning that AMSGrad have even worse performance than Adam due to non-alignment.

Original: We also compare the result of AdamAL using different min-batch settings, the result shows that AdamAL is also not sensitive to min-batch size.
Revised: We also compare the result of AdamAL using different mini-batch settings, the result shows that AdamAL is also not sensitive to min-batch size.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original:   CONCLUSION
Revised:  DISCUSS AND CONCLUSION

Original: In this paper, we present a new angle to look at the ﬁrst-order method with adaptive learning rate.
Revised: The Newton second-order method and v t The mystery of v t in Adam is fascinating. As we discussed in the section 2.2, using second moment to describe v t runs counter to its primary purpose. Because AdaMax (Kingma & Ba, 2014), p-NosAdam (Huang et al., 2018) and Padam (Zhou et al., 2018a), they apply the p-norm to v t as v 1 /pt and Adam still performs well. Another evidence from

Revised: AdaShift (Zhou et al., 2018b), in their settings, v t is no longer acting as second moment, instead they treat it as a learning rate scalar.

Revised: In our point of view, recall the α -exp-concavity (Hazan et al., 2007), it structures the relation from ﬁrst-order to second-order Hessian with α ≤ H/G 2 where gradient upper bound is G and H > 0 is the lower bound of Hessian.

Revised: This relaxed condition results of g 2 t to be an possible approximation of Hessian with scaling factor α at iteration t .

Revised: Beyond the Adam-Type method Assume the above assumption on g 2 t is true, the fast convergence speed of Adam-Type algorithms seems easy to be explained.

Revised: We also notice that most of the previous works improve the Adam by modifying (or correcting) the term C in Equation 11.

Revised: Therefore, such corrections cannot speed up the Adam, instead, they deﬁne different regularization budget.

Revised: To make one step furthers to Adam may rely on close approximations of Hessian.

Revised: In this paper, we present a new angle to look at the ﬁrst-order method with adaptive learning rate.

Original: We decouple the Adam updating rule as an addition of two regularized terms.
Revised: We decouple the Adam updating rule as an addition of two regularized terms.

Original: In this way, we can identify the intuition behind Adam-Type algorithm.
Revised: In this way, we can identify the intuition behind Adam-Type algorithm.

Original: Additionally, we naturally ﬁgure out the nonconvergence issue resides in AMSGrad and Adam, we propose another variant of Adam algorithm to mitigate such problem. 
Revised: Additionally, we naturally ﬁgure out the nonconvergence issue resides in AMSGrad and Adam, we propose another variant of Adam algorithm to mitigate such problem. A A UXILIARY L EMMAS Lemma A.1. (McMahan & Streeter, 2010) Let Q ∈ S n ++ with A = Q 12 .

Revised: Let F be a convex set, and let u 1 , u 2 ∈ R n with x 1 = P A F ( u 1 ) and x 2 =

Revised: P A F ( u 2 ) ,then

Revised: [Equation]

Revised: Proof.

Revised: Deﬁne:

Revised: [Equation]

Revised: we can write as:

Revised: [Equation]

Revised: Then, note that ∇ x B ( x, u 1 ) =

Revised: Qx − Qu 1 , and so we must have ( Qx 1 − Qu 1 ) (cid:62) ( x 2 − x 1 ) ≥ 0 ; otherwise for δ sufﬁciently small the point x 1 + δ ( x 2 − x 1 ) would belong to F (by convexity) and would be closer to u 1 than x 1 .

Revised: Similarly, we must have ( Qx 2 − Qu 2 ) (cid:62) ( x 1 − x 2 ) ≥ 0 .

Revised: Thus

Revised: [Equation]

Revised: Letting ˆ u = u 2 − u 1 , and ˆ x = x 2 − x 1 , we have ˆ x (cid:62) Q ˆ x ≤ ˆ u (cid:62) Q ˆ x .

Revised: Since Q is semi-deﬁnite, we have (ˆ u − ˆ x ) (cid:62) Q (ˆ u − ˆ x ) ≥ 0 , or equivalently ˆ u (cid:62) Q ˆ u + ˆ x (cid:62) Q ˆ x − 2ˆ x (cid:62) Q ˆ u ≥ 0 .

Revised: Thus

Revised: [Equation]

Revised: and so

Revised: [Equation]

Revised: B P ROOF OF T HEOREM 3.

Revised: Proof.

Revised: Let x ∗ =

Revised: arg min x ∈F (cid:80) Tt =1 f t ( x ) , exist and F is feasible convex set.

Revised: We have

Revised: [Equation]

Revised: |

Revised: [Equation]

Revised: [Equation]

Revised: Applying Lemma A.1 we have:

Revised: [Equation]

Revised: That is

Revised: [Equation]

Revised: We rearrange the Equation 27 to have ﬁrst inequality.

Revised: And the second inequality follows the CauchySchwarz and Yong’s inequality.

Revised: Next, we have

Revised: [Equation]

Revised: The ﬁrst inequality is due to convexity of function f .

Revised: To bound the above regret, we need the following intermediate result.

Revised: Lemma B

Revised: [Equation]

Revised: For the parameter settings and conditions assumed in Theorem

Revised: [Equation]

Revised: [Equation]

Revised: we have:

Revised: [Equation]

Revised: Proof.

Revised: To prove, we need

Revised: [Equation]

Revised: t || 2 + η T d (cid:88) i

Revised: = m 2 T,i v 1 / 2 T,i =

Revised: T − 1 (cid:88) t =

Revised: η t

Revised: || v − 1 / 4 t (cid:12) m t

Revised: || 2 + η d (cid:88) i

Revised: =

Revised: ((1 − β 1 ) (cid:80) Tj =1 β T − jg j,i )(cid:112) T ( β 2 v t − 1 ,i + (1 − β 2 )

Revised: ˆ g t,i ) =

Revised: T − 1 (cid:88) t =

Revised: η t

Revised: || v − 1 / 4 t (cid:12) m t

Original: {+|| 
+}
Revised: {+|| 
+}

Revised: + η (1 − β 1 )d (cid:88)

Revised: i =

Revised: ( (cid:80) Tj =1 β T − jg j,i )(cid:112) T ( β 2 v t − 1 ,i + (1 − β 2 )

Revised: ˆ g t,i ) ≤

Revised: T − 1 (cid:88) t =

Revised: η t

Revised: || v − 1 / 4 t (cid:12) m t

Original: {+|| 
+}
Revised: {+|| 
+}

Revised: + η (1 − β 1 )d (cid:88)

Revised: i = (cid:80)

Revised: Tj =1 β 2( T − j )(cid:80)

Revised: Tj =1 g 2 j,i (cid:112) T ( β 2 v t − 1 ,i + (1 − β 2 )ˆ g t,i ) =

Revised: T − 1 (cid:88) t =

Revised: η t

Revised: || v − 1 / 4 t (cid:12) m t

Original: {+|| 
+}
Revised: {+|| 
+}

Revised: + η (1 − β 1 )(1 − β 2 T 1 ) 1 + βd (cid:88)

Revised: i

Revised: =

Revised: (cid:80) Tj =1 g 2 j,i (cid:112) T ( β 2 v t − 1 ,i + (1 − β 2 )

Revised: ˆ g t,i ) =

Revised: T − 1 (cid:88) t =

Revised: η t

Revised: || v − 1 / 4 t (cid:12) m t

Original: {+|| 
+}
Revised: {+|| 
+}

Revised: + η (1 − β 1 )(1 − β 2 T 1 ) 1 + βd (cid:88)

Revised: i

Revised: =

Revised: (cid:80) Tj =1 g 2 j,i (cid:113) T (1 − β 2 ) (cid:80)

Revised: T j =1 β T − jˆ g j,i =

Revised: T − 1 (cid:88) t =

Revised: η t

Revised: || v − 1 / 4 t (cid:12) m t

Original: {+|| 
+}
Revised: {+|| 
+}

Revised: + η (1 − β 1 )(1 − β 2 T 1 ) (1 + β 1 ) (cid:112) T (1 − β 2 ) d (cid:88) i

Revised: = (cid:80)

Revised: Tj =1 g 2 j,i (cid:113)(cid:80)

Revised: Tj =1 β T − jˆ g j,i ≤

Revised: T − 1 (cid:88) t =

Revised: η t

Revised: || v − 1 / 4 t (cid:12) m t

Original: {+|| 
+}
Revised: {+|| 
+}

Revised: + η (1 − β 1 )(1 − β 2 T 1 ) (1 + β 1 ) (cid:112) T (1 − β 2 ) d (cid:88) i =

Revised: T (cid:88)

Revised: j = g 2 j,i (cid:113)

Revised: β T − jˆ g j,i =

Revised: T − 1 (cid:88) t =

Revised: η t

Revised: || v − 1 / 4 t (cid:12) m t

Original: {+|| 
+}
Revised: {+|| 
+}

Revised: + η (1 − β 1 )(1 − β 2 T 1 ) (1 + β 1 ) (cid:112) T (1 − β 2 ) d (cid:88) i =

Revised: T (cid:88)

Revised: j = γ T − j g 2 j,i (cid:112)

Revised: ˆ g j,i ≤

Revised: T − 1 (cid:88) t =

Revised: η t

Revised: || v − 1 / 4 t (cid:12) m t

Original: {+|| 
+}
Revised: {+|| 
+}

Revised: + η (1 − β 1 )(1 − β 2 T 1 ) (1 + β 1 ) (cid:112) T

Revised: [Equation]

Revised: The ﬁrst inequality follows from Cauchy-Schwarz inequality.

Revised: We let γ = 1 / √ β 2 .

Revised: The last inequality due to the Algorithm 1 that ˆ g j,i = max { g j,i , v j − 1 ,i } .

Revised: By using similar upper bounds for all time steps, the quantity in above can further be bounded as follows:

Revised: [Equation]

Revised: i ||

Revised: [Equation]

Revised: The ﬁrst inequality due to the fact that 1 − β 2 j 1 < 1 and (cid:80) Tj =1 γ j − t ≤ 1 / (1 − γ ) .

Revised: The second inequality is due to simple application of Cauchy-Schwarz inequality.

Revised: The ﬁnal inequality is due to the following harmonic sum bound with (cid:80)

Revised: Tt =1 1 /t ≤ (1 + log T ) .

Revised: Now we return to the proof of Theorem 3.1, by using above lemma, we have:

Revised: T

Revised: [Equation]

Revised: 1 / 4 t (cid:12) ( x t − x ∗ ) ||+ 2(1 − β 1 )

Revised: T (cid:88) t

Revised: = {|| v 1 / 4 t (cid:12) ( x t − x ∗ )

Original: {+||η t − || v 1 / 4 t − 1 (cid:12) ( x t − x ∗ ) ||η t −} + η √ 
+}
Revised: {+||η t − || v 1 / 4 t − 1 (cid:12) ( x t − x ∗ ) ||η t −} + η √ 
+}

Revised: + log T (1 + β 1 )(1 − γ ) (cid:112) (1 − β 2 ) d (cid:88)

Revised: i =

Revised: ||

Revised: g 1: T,i ||= 2 η 1 (1 − β 1 ) d (cid:88)

Original: {+i = v 1 / 2 1 ,i ( x 1 ,i − x ∗ ,i ) 
+}
Revised: {+i = v 1 / 2 1 ,i ( x 1 ,i − x ∗ ,i ) 
+}

Revised: + 2(1 − β 1 )

Revised: T (cid:88) t = d (cid:88) i = v 1 / 2 t,i ( x t,i − x ∗ ,i )η t + 2(1 − β 1 )

Revised: T (cid:88) t = d (cid:88) i =

Revised: ( x t,i − x ∗ ,i ) 2 { v 1 / 2 t,i η t − || v 1 / 2 t − 1 ,i η t − 1 } + η √ 1 + log T (1 + β 1 )(1 − γ ) (cid:112) (1 − β 2 ) d (cid:88)

Revised: i =

Revised: ||

Revised: g 1: T,i ||(30)

Revised: For the sake of clarity, we further suppose that the feasible region of parameter x with bound D 2 , the Equation 30 becomes to:

Revised: [Equation]

Revised: Therefore, we ﬁnish the proof of Theorem 3.1 as desired.